Environments and Package Management

This section will include notes for Python and will be structured based on areas related to Python such as the base language associated libraries and other related tooling

Virtual Environments

A virtual environment is a self-contained directory that isolates a Python project and its dependencies, avoiding conflicts with other projects or system-wide Python packages.

Working with an environment

• Create a virtual environment via python -m venv <env_name>
• Activate the virtual environment interpreter via source <env_name>/bin/activate

Installing dependencies

• Install a package pip install <package_name>
• Collect the currently installed packages pip freeze > requirements.txt
• Install all dependencies from a package file pip install -r requirements.txt

Note

The requirements.txt is fairly rudimentary and will not prevent many dependency consistency issues. Only use this for very simple projects and otherwise use Poetry for anything bigger.

PyEnv

PyEnv is a popular tool for managing multiple Python versions on a single system, offering flexibility for development environments. It allows you to easily install and switch between different Python versions without interfering with the system Python. PyEnv is especially useful for ensuring compatibility with projects requiring specific Python versions. It integrates well with virtual environments and avoids the complexities of modifying PATH manually.

• Install PyEnv: Follow this guide.
• Install Python Versions: pyenv install <version>
• List Available Python Versions: pyenv install --list
• Set Global Python Version: pyenv global <version>
• Set Local Python Version: pyenv local <version> (per directory/project)
• Show Current Python Version: pyenv version
• Uninstall a Python Version: pyenv uninstall <version>

Caution

Using PyEnv is very valuable and should be used whenever you need multiple python versions and/or are working on projects which have differing Pythons.

Poetry

Poetry is a powerful tool for managing Python dependencies, virtual environments, and project packaging. It simplifies the process of creating and managing project dependencies by resolving and locking them, ensuring consistent environments across deployments. Poetry eliminates the need for requirements.txt by using a declarative pyproject.toml file, and it also integrates seamlessly with publishing workflows for Python packages.

poetry.lock File

This file ensures consistency across environments by locking the exact versions of all resolved dependencies, including their transitive dependencies. It is automatically updated when dependencies are added, removed, or updated using Poetry. This file ensures repeatable builds by freezing dependency resolutions.

Note

The lock file should always be part of the versioning as it ensures that the builds are frozen and match exactly.

Common Commands

Activity	Command	Explanation
Create a new project	poetry new <project_name>	Creates an entirely new project as a new folder
Initialize a project	poetry init	Interactively creates a pyproject.toml file.
Install dependencies	poetry install	Installs all dependencies listed in the pyproject.toml file taking into account the lock file or making one.
Add a dependency	poetry add <package>	Adds a new dependency, installs it and updates the lock file.
Remove a dependency	poetry remove <package>	Removes a dependency and updates the lock file.
Update dependencies	poetry update	This will update all the dependencies to the latest compatible versions
Refresh the lock file	poetry lock --no-update	If you manually added some dependency, run this before install to get just that and update the lockfile

pyproject.toml File

This file is the core configuration file for a Poetry-managed project. It contains project metadata, dependencies, and build system requirements. It is human-readable and serves as the main point for declaring what your project needs.

Very useful!

Package Mode

In most cases we are not publishing a library or planning to push to a package index. There we do not then need the project itself to be packaged but just the dependencies. This is similar to running the command poetry install --no-root

[tool.poetry]
package-mode = false

Working with Groups

Very often we have testing dependencies, which we do not want installed in the final product. Those can be added with the command poetry add <package_name> --group <group_name>

Later on we can specify a group to be optional in the toml file. Which prevents the basic install from installing these. When we do want them installed we can run poetry install --with test. Alternatively, we can also prevent installation of a specific group via poetry install --without test

[tool.poetry.group.test]
optional = true

[tool.poetry.group.test.dependencies]
pytest = "*"

Setting up secondary sources

You can add a different source if you have a private Package Registry via this command, for example a GitLab Registry

poetry source add --priority=supplemental gitlab https://gitlab.com/api/v4/projects/<project-id>/packages/pypi

This looks like this in the toml file:

[[tool.poetry.source]]
name = "gitlab"
url = "https://gitlab.com/api/v4/projects/<project-id>/packages/pypi"
priority = "supplemental"

Note

In most cases you will need to authenticate, here is an example for gitlab

poetry config http-basic.gitlab <username> <personal-access-token>