Explaining how Copy-on-Write optimizes performance
Introduction
The primary post explained how the Copy-on-Write mechanism works. It highlights some areas where copies are introduced into the workflow. This post will give attention to optimizations that be sure that this won’t slow the common workflow down.
We utilize a way that pandas internals use to avoid copying the entire DataFrame when it’s not crucial and thus, increase performance.
I’m a part of the pandas core team and was heavily involved in implementing and improving CoW to date. I’m an open source engineer for Coiled where I work on Dask, including improving the pandas integration and ensuring that Dask is compliant with CoW.
Removal of defensive copies
Let’s start with probably the most impactful improvement. Many pandas methods performed defensive copies to avoid negative effects to guard against inplace modifications afterward.
df = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
df2 = df.reset_index()
df2.iloc[0, 0] = 100
There isn’t any must copy the information in reset_index, but returning a view would introduce negative effects when modifying the result, e.g. df can be updated as well. Hence, a defensive copy is performed in reset_index.
All these defensive copies aren’t any longer there when Copy-on-Write is enabled. This affects many methods. A full list will be found here.
Moreover, choosing a columnar subset of a DataFrame will now all the time return a view as a substitute of a replica as before.
Let’s take a look at what this implies performance-wise after we mix a few of these methods:
import pandas as pd
import numpy as npN = 2_000_000
int_df = pd.DataFrame(
np.random.randint(1, 100, (N, 10)),
columns=[f"col_{i}" for i in range(10)],
)
float_df = pd.DataFrame(
np.random.random((N, 10)),
columns=[f"col_{i}" for i in range(10, 20)],
)
str_df = pd.DataFrame(
"a",
index=range(N),
columns=[f"col_{i}" for i in range(20, 30)],
)
df = pd.concat([int_df, float_df, str_df], axis=1)
This creates a DataFrame with 30 columns, 3 different dtypes and a pair of million rows. Let’s execute the next method chain on this DataFrame:
%%timeit
(
df.rename(columns={"col_1": "new_index"})
.assign(sum_val=df["col_1"] + df["col_2"])
.drop(columns=["col_10", "col_20"])
.astype({"col_5": "int32"})
.reset_index()
.set_index("new_index")
)
All of those methods perform a defensive copy without CoW enabled.
Performance without CoW:
2.45 s ± 293 ms per loop (mean ± std. dev. of seven runs, 1 loop each)
Performance with CoW enabled:
13.7 ms ± 286 µs per loop (mean ± std. dev. of seven runs, 100 loops each)
An improvement by roughly an element of 200. I selected this instance explicitly for example the potential advantages of CoW. Not every method will get that much faster.
Optimizing copies triggered by inplace modifications
The previous section illustrated many methods where a defensive copy is not any longer crucial. CoW guarantees you could’t modify two objects directly. Which means we’ve got to introduce a replica when the identical data is referenced by two DataFrames. Let’s take a look at techniques to make these copies as efficient as possible.
The previous post showed that the next might trigger a replica:
df.iloc[0, 0] = 100
The copy is triggered if the information that’s backing df is referenced by one other DataFrame. We assume that our DataFrame has n integer columns, e.g. is backed by a single Block.
Our Reference tracking object can also be referencing one other Block, so we are able to not modify the DataFrame inplace without modifying one other object. A naive approach can be to repeat the entire block and be done with it.
This may arrange a brand new reference tracking object and create a brand new Block that’s backed by a fresh NumPy array. This Block doesn’t have any more references, so one other operation would have the ability to change it inplace again. This approach copies n-1 columns that we do not necessarily must copy. We utilize a way we call Block splitting to avoid this.
Internally, only the primary column is copied. All other columns are taken as views on the previous array. The brand new Block doesn’t share any references with other columns. The old Block still shares references with other objects because it is simply a view on the previous values.
There’s one drawback to this method. The initial array has n columns. We created a view on columns 2 till n, but this keeps the entire array alive. We also added a brand new array with one column for the primary column. This may keep a bit more memory alive than crucial.
This method directly translates to DataFrames with different dtypes. All Blocks that usually are not modified in any respect are returned as is and only Blocks which can be modified inplace are split.
We now set a brand new value into column n+1 the float Block to create a view on columns n+2 to m. The brand new Block will only back column n+1.
df.iloc[0, n+1] = 100.5
Methods that may operate inplace
The indexing operations we checked out don’t generally create a brand new object; they modify the present object inplace, including the information of said object. One other group of pandas methods doesn’t touch the information of the DataFrame in any respect. One distinguished example is rename. Rename only changes the labels. These methods can utilize the lazy-copy mechanism mentioned above.
There’s one other third group of methods that may actually be done inplace, like replace or fillna. These will all the time trigger a replica.
df2 = df.replace(...)
Modifying the information inplace without triggering a replica would modify df and df2, which violates CoW rules. That is one in all the the explanation why we consider keeping the inplace keyword for these methods.
df.replace(..., inplace=True)
This may eliminate this problem. It’s still an open proposal and might go into a unique direction. That said, this only pertains to columns which can be actually modified; all other columns are returned as views anyway. Which means just one column is copied in case your value is simply present in one column.
Conclusion
We investigate how CoW changes pandas internal behavior and the way this may translate to improvements in your code. Many methods will get faster with CoW, while we are going to see a slowdown in a few indexing related operations. Previously, these operations all the time operated inplace, which might need produced negative effects. These negative effects are gone with CoW and a modification on one DataFrame object won’t ever impact one other.
The following post on this series will explain how you possibly can update your code to be compliant with CoW. Also, we are going to explain which patterns to avoid in the long run.
Thanks for reading. Be happy to succeed in out to share your thoughts and feedback about Copy-on-Write.