Python Package Management

Overview

IVPM provides first-class support for Python packages, managing both source packages (editable installs) and binary packages (from PyPI) within a project-local virtual environment.

Virtual Environment Creation

IVPM automatically creates a project-local Python virtual environment in packages/python/ when you run ivpm update.

Automatic Creation

$ ivpm update

Creates:

packages/
└── python/
    ├── bin/
    │   ├── python
    │   ├── pip
    │   └── activate
    ├── lib/
    │   └── python3.x/
    │       └── site-packages/
    └── ...

The virtual environment is created using Python’s built-in venv module.

Package Manager Selection

IVPM supports two Python package managers:

uv - Fast, modern Python package installer (recommended)
pip - Traditional Python package installer

Auto-detection (default):

IVPM checks if uv is available. If found, uses uv; otherwise, uses pip.

Explicit selection:

# Use uv
$ ivpm update --py-uv
$ ivpm clone https://github.com/org/project --py-uv

# Use pip
$ ivpm update --py-pip
$ ivpm clone https://github.com/org/project --py-pip

Managing Installation

Skip Python package installation:

$ ivpm update --skip-py-install

Use this when packages are already installed and you only want to fetch non-Python dependencies.

Force reinstallation:

$ ivpm update --force-py-install

Use this to re-install all Python packages, useful after:

Updating dependencies in ivpm.yaml
Corrupted virtual environment
Switching between dev/release dependency sets

Editable vs Binary Packages

IVPM installs Python packages in two modes:

Editable Packages

Source: Git repositories, local directories

Installation: pip install -e (editable mode)

Characteristics:

Source code directly accessible
Changes to source immediately available
Can modify and commit
Full development setup

Example:

deps:
  - name: my-library
    url: https://github.com/org/my-library.git

Result:

packages/
├── my-library/          # Source code
│   ├── setup.py
│   ├── src/
│   └── ...
└── python/
    └── lib/python3.x/site-packages/
        └── my-library.egg-link  # Points to ../../../my-library

Import behavior:

import my_library  # Uses code from packages/my-library/

Binary Packages

Source: PyPI

Installation: pip install (normal mode)

Characteristics:

Pre-built wheels or source distribution
Installed into site-packages
Cannot modify (read-only)
Faster installation

Example:

deps:
  - name: requests
    src: pypi
    version: ">=2.28.0"

Result:

packages/
└── python/
    └── lib/python3.x/site-packages/
        ├── requests/
        ├── requests-2.31.0.dist-info/
        └── ...

Import behavior:

import requests  # Uses installed package

PyPI Package Configuration

Basic PyPI Packages

deps:
  # Latest version
  - name: requests
    src: pypi

  # Specific version
  - name: numpy
    src: pypi
    version: "==1.24.0"

  # Version range
  - name: pandas
    src: pypi
    version: ">=1.5.0,<2.0"

  # Minimum version
  - name: pytest
    src: pypi
    version: ">=7.0"

Version Specifiers

IVPM supports PEP 440 version specifiers:

Specifier	Meaning
`==1.2.3`	Exactly version 1.2.3
`>=1.2.3`	Version 1.2.3 or higher
`<2.0`	Any version below 2.0
`>=1.2,<2.0`	1.2 or higher, but below 2.0
`~=1.2.3`	Compatible release (>=1.2.3, <1.3.0)
`*` or unspecified	Latest version

Pre-release Packages

To include pre-release versions (alpha, beta, rc):

$ ivpm update --py-prerls-packages

Or in ivpm.yaml, specify the pre-release version explicitly:

deps:
  - name: package
    src: pypi
    version: ">=1.0.0a1"  # Includes alpha releases

Source Packages

Git-based Python Packages

Any Git repository with setup.py, setup.cfg, or pyproject.toml is automatically detected as a Python package.

Example:

deps:
  - name: my-package
    url: https://github.com/org/my-package.git
    branch: develop

Auto-detection: IVPM scans for Python build files after fetching.

Manual specification:

deps:
  - name: my-package
    url: https://github.com/org/my-package.git
    type: python  # Explicitly mark as Python

Installation: Installed as editable: pip install -e packages/my-package

Local Development Packages

Use src: dir for co-developed packages:

deps:
  - name: co-dev-lib
    url: file:///home/user/projects/library
    src: dir
    link: true  # Symlink instead of copy

Result:

packages/co-dev-lib → symlink to /home/user/projects/library
Installed as editable
Changes in original location immediately available

Setup Dependencies

Some packages must be installed before others. Use setup-deps to specify installation order:

package:
  name: my-project

  setup-deps:
    - wheel
    - setuptools
    - cython

  dep-sets:
    - name: default-dev
      deps:
        - name: wheel
          src: pypi
        - name: setuptools
          src: pypi
        - name: cython
          src: pypi
        - name: my-cython-package
          url: https://github.com/org/cython-pkg.git

Behavior:

Install wheel, setuptools, cython first (in order)
Then install all other packages (topologically sorted)

This ensures build dependencies are available before packages that need them.

Building Native Extensions

The `build` Command

For Python packages with native extensions (C, C++, Cython):

$ ivpm build

This runs the build process for all Python packages in packages/.

Options:

# Build with debug symbols
$ ivpm build --debug

# Build specific dependency set
$ ivpm build -d default-dev

Debug Builds

To build native extensions with debug symbols:

$ ivpm build --debug

Or set environment variable:

$ DEBUG=1 ivpm build

This passes -g flag to the C/C++ compiler and disables optimization.

Example: Building a Package with Native Extensions

ivpm.yaml:

setup-deps:
  - cython
  - setuptools

dep-sets:
  - name: default-dev
    deps:
      - name: cython
        src: pypi
      - name: setuptools
        src: pypi
      - name: my-fast-lib
        url: https://github.com/org/fast-lib.git

Build process:

$ ivpm update    # Fetch sources, install packages
$ ivpm build     # Build native extensions

Using the Virtual Environment

Activate Command

Run commands within the virtual environment:

# One-off command
$ ivpm activate -c "python script.py"
$ ivpm activate -c "pytest"

# Interactive shell
$ ivpm activate
(venv) $ python
(venv) $ pip list
(venv) $ exit

Behind the scenes: ivpm activate sources packages/python/bin/activate

Direct Access

You can also use the virtual environment directly:

# Use Python directly
$ packages/python/bin/python script.py

# Install additional packages
$ packages/python/bin/pip install extra-package

# Run installed scripts
$ packages/python/bin/pytest

Environment Variables

When activated, these variables are set:

VIRTUAL_ENV → packages/python
PATH → packages/python/bin:$PATH
PYTHONPATH → (can be customized via env-sets)

Complete Examples

Example 1: Pure Python Project

ivpm.yaml:

package:
  name: data-analyzer
  default-dep-set: default-dev

  dep-sets:
    - name: default
      deps:
        - name: pandas
          src: pypi
          version: ">=2.0"
        - name: numpy
          src: pypi
          version: ">=1.24"

    - name: default-dev
      deps:
        - name: pandas
          src: pypi
          version: ">=2.0"
        - name: numpy
          src: pypi
          version: ">=1.24"
        - name: pytest
          src: pypi
        - name: black
          src: pypi
        - name: mypy
          src: pypi

Usage:

$ ivpm update
$ ivpm activate -c "pytest"
$ ivpm activate -c "black src/"
$ ivpm activate -c "mypy src/"

Example 2: Mixed Source and Binary

ivpm.yaml:

package:
  name: ml-pipeline
  default-dep-set: default-dev

  dep-sets:
    - name: default-dev
      deps:
        # Binary packages from PyPI
        - name: torch
          src: pypi
          version: ">=2.0"
        - name: transformers
          src: pypi

        # Source package (co-developed)
        - name: custom-models
          url: https://github.com/org/models.git

        # Local development
        - name: data-utils
          url: file:///home/user/projects/utils
          src: dir

Usage:

$ ivpm update
$ ivpm activate -c "python train.py"

# Edit custom-models or data-utils, changes immediate
$ ivpm activate -c "python train.py"  # Uses modified code

Example 3: Native Extensions with Build

ivpm.yaml:

package:
  name: signal-processor

  setup-deps:
    - cython
    - numpy

  dep-sets:
    - name: default-dev
      deps:
        - name: cython
          src: pypi
        - name: numpy
          src: pypi
        - name: scipy
          src: pypi
        - name: fast-dsp
          url: https://github.com/org/fast-dsp.git

fast-dsp has C extensions that need compilation

Usage:

$ ivpm update       # Fetch and install
$ ivpm build        # Build native extensions
$ ivpm activate -c "python process.py"

# Debug build
$ ivpm build --debug
$ ivpm activate -c "gdb python"

Troubleshooting

Package Not Importable

Problem: ModuleNotFoundError when importing

Check:

Is the package in the active dependency set?
```
$ cat ivpm.yaml  # Check dep-sets
```
Was ivpm update run?
```
$ ivpm update
```

Is the virtual environment activated?

$ ivpm activate -c "python -c 'import package'"

List installed packages:
```
$ ivpm activate -c "pip list"
```

Editable Install Not Working

Problem: Changes to source package not reflected

Check:

Is it installed in editable mode?

$ ivpm activate -c "pip list"
# Look for -e /path/to/package

Check the egg-link:

$ cat packages/python/lib/python3.*/site-packages/package.egg-link

Solution: Reinstall:

$ ivpm update --force-py-install

Build Failures

Problem: Native extension build fails

Common causes:

Missing build dependencies (compiler, headers)
Missing setup-deps
Wrong order of installation

Solutions:

# Install system dependencies (Ubuntu/Debian)
$ sudo apt-get install build-essential python3-dev

# Add setup-deps to ivpm.yaml
$ vim ivpm.yaml  # Add setuptools, wheel, cython

# Rebuild
$ ivpm update --force-py-install
$ ivpm build

Version Conflicts

Problem: Package version conflict errors

Check versions:

$ ivpm activate -c "pip list"

Solution: Adjust version specs in ivpm.yaml:

deps:
  # Too restrictive
  - name: package-a
    src: pypi
    version: "==1.0.0"

  # Better
  - name: package-a
    src: pypi
    version: ">=1.0.0,<2.0"

Best Practices

Use version ranges instead of exact versions for flexibility
Specify setup-deps for packages with build dependencies
Use editable installs for packages under active development
Keep dev tools separate in default-dev dependency set
Pin versions for reproducible builds (release deps)
Use –force-py-install after changing dependency sets
Leverage uv for faster package installation
Test with both pip and uv if distributing to diverse users

Python Package Management

Overview

Virtual Environment Creation

Automatic Creation

Package Manager Selection

Managing Installation

Editable vs Binary Packages

Editable Packages

Binary Packages

PyPI Package Configuration

Basic PyPI Packages

Version Specifiers

Pre-release Packages

Source Packages

Git-based Python Packages

Local Development Packages

Setup Dependencies

Building Native Extensions

The build Command

Debug Builds

Example: Building a Package with Native Extensions

Using the Virtual Environment

Activate Command

Direct Access

Environment Variables

Complete Examples

Example 1: Pure Python Project

Example 2: Mixed Source and Binary

Example 3: Native Extensions with Build

Troubleshooting

Package Not Importable

Editable Install Not Working

Build Failures

Version Conflicts

Best Practices

See Also

The `build` Command