The paradox of choice in Python packaging tools makes dependency management more complex than it needs to be. In this Quick Hit, we will explore Python dependency management and demonstrate an easy-to-implement workflow to generate reproducible environments.

Note: this article assumes familiarity with virtual environments.

Background

We use dependency management tools to create a list of modules that our application requires. In Python the standard convention for tracking dependencies is to list them in a requirements.txt file stored in the root project directory.

Why is Dependency Management Important

Including all application dependencies in requirements.txt ensures correct packages will be downloaded and installed when we distribute code.

Python will not notice missing dependencies until it tries to import them into the program. This can lead to lead to runtime exceptions around missing modules.

Problems with Python Dependency Management

When we add packages to requirements.txt, we have to ensure we pin the correct version and include all sub-dependencies.

Correct Version

Pin dependency to exact version in requirements.txt

All listings in requirements.txt must be pinned to an exact version, i.e. requests==2.22.0 versus requests

This ensures that we install the correct package when distributing code. Unpinned dependencies can silently update resulting in unexpected behavior.

Include Sub-dependencies

Include all dependencies in requirements.txt

Our dependencies have dependencies. Their dependencies have dependencies. And those dependencies have even more dependencies. Not pinning dependencies and sub-dependencies adds risk to our deployment. It results in an environment that is not completely reproducible.

Pinning all dependencies does have one disadvantage: we have to be careful when upgrade dependencies. We need a way to separate direct dependencies and sub-dependencies.

pip-tools

pip-tools is a package that allows us to separate direct dependencies from their sub-dependencies. pip-tools generates a dependency graph and uses this information to create a bespoke requirements.txt file for our project.

Instructions

1. pip install pip-tools
2. List all of direct dependencies in requirements.in
3. Generate requirements file: pip-compile --output-file=requirements.txt requirements.in

We can now use the requirements.txt file in all of our build processes.

Adding new dependency

1. Add pinned requirement to requirements.in
2. Generate requirements file: pip-compile --output-file=requirements.txt requirements.in

Upgrading dependency

When we upgrade packages, we only want to update our direct dependencies:

1. Update requirement in requirements.in
2. Generate requirements file: pip-compile --output-file=requirements.txt requirements.in
3. Test to ensure functionality is as expected

If tests pass, we can safely update the package.

Advantages

• Leverage existing workflows that use requirements.txt
• Does not require us to learn new tools that add unnecessary complexity

Disadvantages

• Manual process to add and update dependencies

Comparison to Other Tools

pip

• Keep track of direct dependencies and sub-dependencies in requirements.txt
• Great for simple project

Pipenv

• Use a Piplock file to keep track of direct dependencies and sub-dependencies
• Manage virtual environment
• Can generate requirement.txt

Poetry

• Use a poetry.lock file to keep track of direct dependencies and sub-dependencies
• Manage virtual environment
• Can generate requirement.txt
• Build packages and publishes artifacts to PyPI

Conclusion

There are many dependency management options in the Python ecosystem. Before adding unnecessary complexity to your workflow, take a look at pip-tools and see if its fits what you are trying to do.