Mastering the Art of Python Project Setup: A Step-by-Step Guide
Oct 21, 2022
Whether you’re a seasoned developer or simply getting began with 🐍 Python, it’s vital to know tips on how to construct robust and maintainable projects. This tutorial will guide you thru the technique of organising a Python project using a number of the hottest and effective tools within the industry. You’ll learn tips on how to use GitHub and GitHub Actions for version control and continuous integration, in addition to other tools for testing, documentation, packaging and distribution. The tutorial is inspired by resources akin to Hypermodern Python and Best Practices for a brand new Python project. Nonetheless, this isn’t the one strategy to do things and you may have different preferences or opinions. The tutorial is meant to be beginner-friendly but additionally cover some advanced topics. In each section, you’ll automate some tasks and add badges to your project to point out your progress and achievements.
The repository for this series may be found at github.com/johschmidt42/python-project-johannes
- OS: Linux, Unix, macOS, Windows (WSL2 with e.g. Ubuntu 20.04 LTS)
- Tools: python3.10, bash, git, tree
- Version Control System (VCS) Host: GitHub
- Continuous Integration (CI) Tool: GitHub Actions
It is anticipated that you simply are acquainted with the versioning control system (VCS) git. If not, here’s a refresher for you: Introduction to Git
Commits can be based on best practices for git commits & Conventional commits. There’s the traditional commit plugin for PyCharm or a VSCode Extension that provide help to to jot down commits on this format.
Overview
Structure
- Documentation framework (mkdocs, diataxis)
- Configuration (mkdocs.yml)
- Construct documentation locally (index.html)
- GitHub Pages (gh-pages)
- CI (pages.yml)
- Docstrings (mkdocstrings)
- Badge (Documentation)
- Bonus (Plugins: Swagger)
As developers, we love writing code. But code alone may be difficult to understand sometimes. That’s why we’d like to make our code readable, usable and comprehensible for others who might encounter it. Whether we have now customers or colleagues who require documentation, or whether we just need to help our future selves in just a few months, we should always document our code! It’ll make our lives easier and our code higher, trust me!
There are tools that allow us to generate really nice-looking and modern documentation from Markdown files and docstrings mechanically. These tools reduce the trouble as we link the already existing information within the code and the pages that we manually create. On this series, we introduced fastAPI, a REST API framework that uses the mkdocs library as its documentation framework. Their documentation pages/static website looks like this:
In case you think that this documentation looks really good and are eager about organising you own documentation with mkdocs and the material theme, follow along! You don’t need any frontend development skills to construct a surprising documentation. You may see the here.
sphinx is one other popular documentation library, however it uses reStructuredText as a substitute of markdown because the default plaintext markup language. I personally prefer mkdocs for that reason.
So let’s start by making a recent branch: feat/docs
Create a brand new dependency group called docs and add the mkdocs library and the fabric theme to it. We use a separate group because we only need to use the libraries which are needed to create the documentation in our CI pipeline.
> poetry add --group docs mkdocs mkdocs-material
For our landing page, we must create a Markdown file index.md that provides a brief description of the project and allows us to navigate to other pages (markdown files). I’ll follow the perfect practices for project documentation as described by Daniele Procida within the Diataxis documentation framework Due to this fact, besides the index.md I’ll create 4 additional markdown files within the docs directory:
To create our landing page, we’d like a Markdown file called index.md that provides a temporary overview of the project and links to other pages (markdown files). I’ll use the perfect practices for project documentation from Daniele Procida’s Diataxis documentation framework. So, besides the index.md, I’ll make 4 more markdown files within the docs directory:
> mkdir docs
> cd docs/ && tree
.
├── explanation.md
├── how-to-guides.md
├── index.md
├── reference.md
└── tutorials.md
Each file can be full of some text in markdown. So the content of the index.md could appear to be this:
The markdown pages are referenced on this file.
To construct documentation based on these files, we’d like so as to add yet one more configuration file where we set just a few options: mkdocs.yml
This file lets us set the navigation tab, the positioning name, the theme, the choice to make use of directory urls and more. We will even add plugins (mkdocstrings etc.) to this yml file later to get more cool features in our documentation page. Don’t forget to ascertain out the Bonus part at the underside!
As a substitute of constructing the navigation ourselves, we could simply point to the folder where our documentation is stored and let it’s generated mechanically:
Constructing the positioning locally is so simple as running:
> mkdocs construct
INFO - Cleansing site directory
INFO - Constructing documentation to directory: /Users/johannes/workspace/python-project-johannes/site
INFO - Documentation in-built 0.40 seconds
This makes a directory called site that has an index.html file. We will open it in our browser and see our static documentation site:
We also can navigate through the pages that we created:
Now we have now a basic but nice-looking documentation that we are able to see locally. Let’s share it with everyone by deploying the positioning content with GitHub Pages (to the gh-pages branch). mkdocs makes this very easy for us, so we just must run
> mkdocs gh-deploy -m "docs: update documentation" -v --force
which returns information in regards to the steps being performed:
...
INFO - Documentation in-built 0.55 seconds
WARNING - Version check skipped: No version laid out in previous deployment.
INFO - Copying '/Users/johannes/workspace/python-project-johannes/site' to 'gh-pages' branch and pushing to GitHub.
Enumerating objects: 55, done.
Counting objects: 100% (55/55), done.
Delta compression using as much as 8 threads
Compressing objects: 100% (51/51), done.
Writing objects: 100% (55/55), 473.92 KiB | 3.18 MiB/s, done.
Total 55 (delta 8), reused 0 (delta 0), pack-reused 0
distant: Resolving deltas: 100% (8/8), done.
distant:
distant: Create a pull request for 'gh-pages' on GitHub by visiting:
distant: https://github.com/johschmidt42/python-project-johannes/pull/recent/gh-pages
distant:
distant: GitHub found 1 vulnerability on johschmidt42/python-project-johannes's default branch (1 moderate). To search out out more, visit:
distant: https://github.com/johschmidt42/python-project-johannes/security/dependabot/1
distant:
To github.com:johschmidt42/python-project-johannes.git
* [new branch] gh-pages -> gh-pages
INFO - Your documentation should shortly be available at: https://johschmidt42.github.io/python-project-johannes/
It tells us that the construct artifacts (html, css files etc.) were generated and pushed to the distant branch gh-pages. That’s the default branch, but we are able to name it whatever we would like, e.g. my-super-cool-branch-for-docs . Behind the scenes, mkdocs will use the ghp-import tool to commit them to the gh-pages branch and push the gh-pages branch to GitHub. After a short while, our site needs to be available at
https://johschmidt42.github.io/python-project-johannes/
If the positioning doesn’t show up once you open the URL in a browser, you could do these steps in your Settings section of the Github repository:
That is all thoroughly explained in GitHub Pages & MkDocs deployment.
We will check that the branch is created and the content has some website files (html files etc.).
Let’s put these recent commands in our Makefile before we move on to CI/CD:
# Makefile...
##@ Documentation
docs-build: ## construct documentation locally
@mkdocs construct
docs-deploy: ## construct & deploy documentation to "gh-pages" branch
@mkdocs gh-deploy -m "docs: update documentation" -v --force
...
clean-docs: ## remove output files from mkdocs
@rm -rf site
To maintain our documentation updated, we’d like a GitHub actions workflow that runs these commands each time we commit to our default branch. We will create a workflow by making a file called .github/workflow/pages.yml and adding some content just like what we did in lint.yml or test.yml.
The documentation is updated by a construct & deploy job after we merge a PR to the most important branch. Awesome!
We also can use docstrings in our code to generate documentation with the mkdocs plugin mkdocstrings. Let’s make a brand new branch: feat/docs-docstrings and add the library to our docs group:
> poetry add --group docs "mkdocstrings[python]"
To make use of docstrings within the source code to seem within the documentation, we’d like to first create docstrings! We are going to follow the Google python docstrings style, because it is my favourite docstrings style and in addition supported with the plugin. Please note that it doesn’t make much sense so as to add docstrings to fastAPI endpoints, since the documentation for these endpoints needs to be provided within the decorator function parameters. Nonetheless, we’ll create docstrings for app.py anyway and moreover create one other file service.py for demonstration purposes.
Our app.py looks like this now:
and we created on the identical level because the app.py the src/example_app/service.py with this very generic content, that has some docstring tests:
In our pyproject.toml we now have the mkdocstrings in our dependency group docs. Please note that we have now added docstring-tests within the src code (service.py ) but pytest would only search for tests within the tests directory. That’s since the attribute testpaths only points to the tests directory. So we have now to update this part by adding the src directory as input for pytest to search for tests as well. Our pyproject.toml is updated like this:
# pyproject.toml[tool.poetry.group.docs.dependencies]
mkdocs = "^1.3.1"
mkdocs-material = "^8.4.3"
mkdocstrings = {extras = ["python"], version = "^0.19.0"}
...
[tool.pytest.ini_options]
testpaths = ["src", "tests"]
addopts = "-p no:cacheprovider" # deactivating pytest caching.
But that’s not the one change, we’d like to do. By default, pytest doesn’t search for docstring tests, we also must add the --docstest-modules flag when running pytest.
So our Makefile can be updated to this:
# Makefileunit-tests:
@pytest --doctest-modules
unit-tests-cov:
@pytest --doctest-modules --cache-clear --cov=src --junitxml=pytest.xml --cov-report=html --cov-report term-missing | tee pytest-coverage.txt
...
Our test.yml doesn’t need to vary as we’re using these commands that we just updated.
Okay, so we have now added docstrings & included them in our testing pipeline. Now we are able to semi-automatically generate documentation based on the source code. I say semi-automatic because we have now to do the next step. We’d like so as to add some blocks with a special ::: notation in one in all the markdown files for the documentation. This tells mkdocstrings which files to make use of for autodocs. I’ll use the references.md file, which is for the technical documentation, and add these blocks for example_app.app.py and example_app.service.py:
# docs/reference.md...
::: example_app.app
options:
show_root_heading: true
::: example_app.service
options:
show_root_heading: true
......
Because our handler (python) needs to search out the module example_app, we are able to conveniently add this information within the mkdocs.yml:
# mkdocs.yml...
plugins:
- mkdocstrings:
handlers:
python:
paths: [src]
...
The trail points to the src directory that comprises the package example_app. There’s a greater explanation on tips on how to find the modules within the “finding modules” documentation.
We’re able to create documentation from our source code. We just need to construct the documentation and deploy it to the gh-pages branch with the CI (commit to most important)
Before we jump to the last section, we shouldn’t forget so as to add a search bar to our site and add the Github URL next to it:
# mkdocs.yml...
plugins:
- search:
- mkdocstrings:
handlers:
python:
paths: [src]
repo_url: https://github.com/johschmidt42/python-project
And now we see each features on our documentation site:
To explore the complete potential of the fabric theme, let me show a number of the features that we are able to leverage:
The features section of the material theme within the mkdocs.yml file is used to enable or disable specific features of the theme.
On this case, the next features are enabled:
navigation.tabs: This feature enables tabs within the navigation bar.navigation.indexes: This feature enables indexes within the navigation bar.navigation.fast: This feature enables fast search within the navigation bar.
The result looks like this now:
There is just the obligatory badge adding process for us to be left for the core of this section.
To get the badge, we are able to click on a workflow run
and choose the most important branch. The badge markdown may be copied and added to the README.md:
Our landing page of the GitHub now looks like this ❤:
In case you are interested in how this badge reflects the most recent status of the pipeline run within the most important branch, you may take a look at the statuses API on GitHub.