Use Python Logging like a Pro

I can bet that just about every Python developer sometimes uses “print” for debugging. There’s nothing incorrect with that for prototyping, but for production, there are far more effective ways to handle the logs. In this text, I’ll show five practical the explanation why Python “logging” is far more flexible and powerful and why you absolutely should use it if you’ve not began before.

Let’s get into it.

Code

To make things more practical, let’s consider a toy example. I created a small application that calculates a linear regression for 2 Python lists:

import numpy as np
from sklearn.linear_model import LinearRegression
from typing import List, Optional
def do_regression(arr_x: List, arr_y: List) -> Optional[List]:
""" LinearRegression for X and Y lists """
try:
x_in = np.array(arr_x).reshape(-1, 1)
y_in = np.array(arr_y).reshape(-1, 1)
print(f"X: {x_in}")
print(f"y: {y_in}")
reg = LinearRegression().fit(x_in, y_in)
out = reg.predict(x_in)
print(f"Out: {out}")
print(f"Rating: {reg.rating(x_in, arr_y)}")
print(f"Coef: {reg.coef_}")
return out.reshape(-1).tolist()
except ValueError as err:
print(f"ValueError: {err}")
return None
if __name__ == "__main__":
print("App began")
ret = do_regression([1,2,3,4], [5,6,7,8])
print(f"LinearRegression result: {ret}")

This code works, but can we do it higher? We obviously can. Let’s see five benefits of using “logging” as an alternative of “print” on this code.

1. Logging levels

Let’s change our code a bit:

import logging
def do_regression(arr_x: List, arr_y: List) -> Optional[List]:
"""LinearRegression for X and Y Python lists"""
try:
x_in = np.array(arr_x).reshape(-1, 1)
y_in = np.array(arr_y).reshape(-1, 1)
logging.debug(f"X: {x_in}")
...
except ValueError as err:
logging.error(f"ValueError: {err}")
return None
if __name__ == "__main__":     
logging.basicConfig(level=logging.DEBUG, format='%(message)s')
logging.info("App began")
ret = do_regression([1,2,3,4], [5,6,7,8])
logging.info(f"LinearRegression result: {ret}")

Here I replaced “print” calls with “logging” calls. We made a small change, however it makes the output far more flexible. Using the “level” parameter, we will now set different logging levels. For instance, if we use “level=logging.DEBUG”, then all output will likely be visible. After we are sure that our code is prepared for production, we will change the extent to “logging.INFO”, and debugging messages won’t be displayed anymore:

And what is vital is that no code change is required except the initialization of the logging itself!

By the best way, all available constants might be present in the logging/__init__.py file:

ERROR = 40
WARNING = 30
INFO = 20
DEBUG = 10
NOTSET = 0

As we will see, the “ERROR” level is the best; by enabling the “ERROR” log level, we will suppress all other messages, and only errors will likely be displayed.

2. Formatting

As we will see from the last screenshot, it is straightforward to manage the logging output. But we will do far more to enhance that. We can even adjust the output by providing the “format” string. For instance, I can specify formatting like this:

logging.basicConfig(level=logging.DEBUG,
format='[%(asctime)s] %(filename)s:%(lineno)d: %(message)s')

Without every other code changes, I’ll have the ability to see timestamps, file names, and even the road numbers within the output:

There are about 20 different parameters available, which might be present in the “LogRecord attributes” paragraph of the manual.

3. Saving logs to a file

Python logging is a really flexible module, and its functionality might be easily expanded. Let’s say we wish to save lots of all our logs right into a file for future evaluation. To do that, we’d like so as to add only two lines of code:

logging.basicConfig(level=logging.DEBUG, 
format='[%(asctime)s] %(message)s',
handlers=[logging.FileHandler("debug.log"), 
logging.StreamHandler()])

As we will see, I added a brand new parameter “handlers”. A StreamHandler is displaying the log on the console, and the FileHandler, as we will guess from its name, saves the identical output to the file.

This method is actually flexible. Plenty of various “handler” objects can be found in Python, and I encourage readers to envision the manual on their very own. And as we already know, logging works almost routinely; no further code changes are required.

4. Rotating log files

Saving logs right into a file is a great option, but alas, the disk space shouldn’t be unlimited. We will easily solve this problem by utilizing the rotating log file:

from logging.handlers import TimedRotatingFileHandler
...
if __name__ == "__main__":
file_handler = TimedRotatingFileHandler(
filename="debug.log",
when="midnight",
interval=1,
backupCount=3,
)
logging.basicConfig(level=logging.DEBUG, 
format='[%(asctime)s] %(message)s',
handlers=[file_handler, logging.StreamHandler()])

All parameters are self-explanatory. A TimedRotatingFileHandler object will create a log file, which will likely be modified every midnight, and only the last three log files will likely be stored. The previous files will likely be routinely renamed to something like “debug.log.2023.03.03”, and after a 3-day interval, they will likely be deleted.

5. Sending logs via socket

Python’s logging is surprisingly flexible. If we don’t want to save lots of logs into a neighborhood file, we will just add a socket handler, which is able to send logs to a different service using a particular IP and port:

from logging.handlers import SocketHandler
logging.basicConfig(level=logging.DEBUG, format='[%(asctime)s] %(message)s',
handlers=[SocketHandler(host="127.0.0.1", port=15001), 
logging.StreamHandler()])

That’s it; no more code changes are required!

We can even create one other application that can hearken to the identical port:

import socket
import logging
import pickle
import struct
from logging import LogRecord
port = 15001
stream_handler = logging.StreamHandler()
def create_socket() -> socket.socket:
"""Create the socket"""
sock = socket.socket(socket.AF_INET)
sock.settimeout(30.0)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
return sock
def read_socket_data(conn_in: socket.socket):
"""Read data from socket"""
while True:
data = conn_in.recv(4)  # Data: 4 bytes length + body
if len(data) > 0:
body_len = struct.unpack(">L", data)[0]
data = conn_in.recv(body_len)
record: LogRecord = logging.makeLogRecord(pickle.loads(data))
stream_handler.emit(record)
else:
logging.debug("Socket connection lost")
return
if __name__ == "__main__":
logging.basicConfig(level=logging.DEBUG, format='[%(asctime)s] %(message)s',
handlers=[stream_handler])
sock = create_socket()
sock.bind(("127.0.0.1", port))  # Local connections only
sock.listen(1)  # One client might be connected
logging.debug("Logs listening thread began")
while True:
try:
conn, _ = sock.accept()
logging.debug("Socket connection established")
read_socket_data(conn)
except socket.timeout:
logging.debug("Socket listening: no data")

The tricky part here is to make use of the emit method, which adds all distant data received by a socket to an lively StreamHandler.

6. Bonus: Log filters

Finally, a small bonus for readers who were attentive enough to read until this part. Additionally it is easy so as to add custom filters to logs. Let’s say we wish to log only X and Y values into the file for future evaluation. It is straightforward to create a brand new Filter class, which is able to save to log only strings containing “x:” or “y:” records:

from logging import LogRecord, Filter
class DataFilter(Filter):
"""Filter for logging messages"""
def filter(self, record: LogRecord) -> bool:
"""Save only filtered data"""
return "x:" in record.msg.lower() or "y:" in record.msg.lower()

Then we will easily add this filter to the file log. Our console output will stay intact, however the file could have only “x:” and “y:” values.

file_handler = logging.FileHandler("debug.log")
file_handler.addFilter(DataFilter())
logging.basicConfig(level=logging.DEBUG, 
format='[%(asctime)s] %(message)s',
handlers=[file_handler, logging.StreamHandler()])

Conclusion

On this short article, we learned several easy ways to include logs into the Python application. Logging in Python is a really flexible framework, and it is unquestionably price spending a while investigating how it really works.

Thanks for reading, and good luck with future experiments.

In the event you enjoyed this story, be at liberty to subscribe to Medium, and you’ll get notifications when my recent articles will likely be published, in addition to full access to 1000’s of stories from other authors.