Siv Scripts

Solving Problems Using Code

Mon 27 January 2020

Managing Python dependencies with pip-tools and requirements.txt

Posted by Aly Sivji in Quick Hits   

The paradox of choice in Python packaging tools makes dependency management more complex than it needs to be. In this Quick Hit, we will explore Python dependency management and demonstrate an easy-to-implement workflow to generate reproducible environments.

Note: this article assumes familiarity with virtual environments.


We use dependency management tools to create a list of modules that our application requires. In Python the standard convention for tracking dependencies is to list them in a requirements.txt file stored in the root project directory.

Why is Dependency Management Important

Including all application dependencies in requirements.txt ensures correct packages will be downloaded and installed when we distribute code.

Python will not notice missing dependencies until it tries to import them into the program. This can lead to lead to runtime exceptions around missing modules.

Problems with Python Dependency Management

When we add packages to requirements.txt, we have to ensure we pin the correct version and include all sub-dependencies.

Correct Version

Pin dependency to exact version in requirements.txt

All listings in requirements.txt must be pinned to an exact version, i.e. requests==2.22.0 versus requests

This ensures that we install the correct package when distributing code. Unpinned dependencies can silently update resulting in unexpected behavior.

Include Sub-dependencies

Include all dependencies in requirements.txt

Our dependencies have dependencies. Their dependencies have dependencies. And those dependencies have even more dependencies. Not pinning dependencies and sub-dependencies adds risk to our deployment. It results in an environment that is not completely reproducible.

Pinning all dependencies does have one disadvantage: we have to be careful when upgrade dependencies. We need a way to separate direct dependencies and sub-dependencies.


pip-tools is a package that allows us to separate direct dependencies from their sub-dependencies. pip-tools generates a dependency graph and uses this information to create a bespoke requirements.txt file for our project.


  1. pip install pip-tools
  2. List all of direct dependencies in
  3. Generate requirements file: pip-compile --output-file=requirements.txt

We can now use the requirements.txt file in all of our build processes.

Adding new dependency

  1. Add pinned requirement to
  2. Generate requirements file: pip-compile --output-file=requirements.txt

Upgrading dependency

When we upgrade packages, we only want to update our direct dependencies:

  1. Update requirement in
  2. Generate requirements file: pip-compile --output-file=requirements.txt
  3. Test to ensure functionality is as expected

If tests pass, we can safely update the package.


  • Leverage existing workflows that use requirements.txt
  • Does not require us to learn new tools that add unnecessary complexity


  • Manual process to add and update dependencies

Comparison to Other Tools


  • Keep track of direct dependencies and sub-dependencies in requirements.txt
  • Great for simple project


  • Use a Piplock file to keep track of direct dependencies and sub-dependencies
  • Manage virtual environment
  • Can generate requirement.txt


  • Use a poetry.lock file to keep track of direct dependencies and sub-dependencies
  • Manage virtual environment
  • Can generate requirement.txt
  • Build packages and publishes artifacts to PyPI


There are many dependency management options in the Python ecosystem. Before adding unnecessary complexity to your workflow, take a look at pip-tools and see if its fits what you are trying to do.