PYTHON PROGRAMMING

Easy methods to inspect Pandas data frames in chained operations without breaking the chain into separate statements

Debugging lies in the center of programming. I wrote about this in the next article:

This statement is sort of general and language- and framework-independent. Once you use Python for data evaluation, you must debug code no matter whether you’re conducting complex data evaluation, writing an ML software product, or making a Streamlit or Django app.

This text discusses debugging Pandas code, or relatively a particular scenario of debugging Pandas code during which operations are chained right into a pipe. Such debugging poses a difficult issue. Once you don’t know easy methods to do it, chained Pandas operations appear to be far harder to debug than regular Pandas code, that’s, individual Pandas operations using typical project with square brackets.

To debug regular Pandas code using typical project with square brackets, it’s enough so as to add a Python breakpoint — and use the pdb interactive debugger. This might be something like this:

>>> d = pd.DataFrame(dict(
...     x=[1, 2, 2, 3, 4],
...     y=[.2, .34, 2.3, .11, .101],
...     group=["a", "a", "b", "b", "b"]
.. ))
>>> d["xy"] = d.x + d.y
>>> breakpoint()
>>> d = d[d.group == "a"]

Unfortunately, you possibly can’t do this when the code consists of chained operations, like here:

>>> d = d.assign(xy=lambda df: df.x + df.y).query("group == 'a'")

or, depending in your preference, here:

>>> d = d.assign(xy=d.x + d.y).query("group == 'a'")

On this case, there isn’t any place to stop and have a look at the code — you possibly can only accomplish that before or after the chain. Thus, certainly one of the solutions is to interrupt the foremost chain into two sub-chains (two pipes) in a…