Custom packages¶

Up until now we were only consuming packages in flox environments. But in your project you can also define new custom packages.

While defining custom packages can be a tedious task, it is an important step in the journey to be able to let others use your package in their environment.

If you haven't already, please install flox before continuing.

Advanced Nix knowledge required

At flox we are aware of the steep learning curve of Nix. It is part of our mission to to flatten this learning curve.

We don't want to discurage you from reading this tutorial, but we feel it is our duty to inform you that advanced Nix knowledge is required to write build recipes with Nix.

The good news is that we are aware of this and are already building a solution that will make writing build recipes as easy as writing a Dockerfile.

We welcome you to share your experiences and suggestions with us on our discourse.

1. Prepare example project¶

In this tutorial we package a simple hello-world app written in python from the ground up. The source to the example can be found here: flox-examples/hello-python.

$ git clone https://github.com/flox-examples/hello-python.git --branch "pre-flox" # (1)!
$ cd hello-python

The tutorial starts with the project in the tutorial/pre-flox branch state.

The repo is structured like this:

  .
  ├── hello
  │  ├── __init__.py
  │  └── __main__.py
  ├── README.md
  └── setup.py

What does the code in this hello-python look like?

If we examine the content we can see that this is a very simple hello-world application written in python.

hello/__init__.py

def hello(): # (1)!
    print("Hello world!")

hello/__main__.py

from . import hello
if __name__ == '__main__':
    hello()

setup.py

from setuptools import setup, find_packages
setup(
    name="hello-python",
    version="0.1",
    packages=find_packages(),
    entry_points={
        'console_scripts': [
            'hello = hello:hello',
        ],
    }
)

2. `flox init` - Create an initial project structure¶

We start by invoking flox init to generate an initial build recipe for your project. The process begins by prompting you to choose an appropriate template, and in this case we will choose package-python.

$ flox init
Found git repo: /home/USER/dev/hello-python
wrote: /home/USER/dev/hello-python/flake.nix
> Enter package name (hello-python) <enter>
? Select a template for flox init
  package-bazel: Bazel package template
  package-go: Go package tempalate
  package-java: Java package template
  package-js-yarn: Javascript(yarn) package template
  package-mix: Mix package template
  package-perl: Perl package template
> package-python: Python package template
  package-ruby: Ruby package template
  package-rust: Rust package template
  package-simple: Simple package template
> Enter package name hello-python
> Select a template for flox init package-python: Python package template
HINT: avoid selecting a template next time with:
      $ flox init --template package-python
wrote: /home/USER/dev/hello-python/pkgs/__PACKAGE_NAME__/default.nix
wrote: /home/USER/dev/hello-python/pkgs/__PACKAGE_NAME__/flox.nix
wrote: /home/USER/dev/hello-python/pkgs/__PACKAGE_NAME__
wrote: /home/USER/dev/hello-python/pkgs
[INFO] [flox_rust_sdk::models::project] moved: /home/USER/dev/hello-python/pkgs/__PACKAGE_NAME__ -> /home/USER/dev/hello-python/pkgs/hello-python

After flox initializes the package, we find that the repository has changed:

$ git status --short
 A flake.nix
 A pkgs/hello-python/default.nix
 A pkgs/hello-python/flox.nix

Let's inspect the content of the files just generated:

flake.nix - the project glue code defining external resources and project configuration

{
  description = "A flox project"; # (1)!

  inputs.flox-floxpkgs.url = "github:flox/floxpkgs"; # (2)!

  outputs = args @ {flox-floxpkgs, ...}: flox-floxpkgs.project args (_: {}); # (3)!
}

A description of the project.
A definition of an external resource.

flox/floxpkgs is a collection of utilities and packages that provide an easy way to start working with flox.
Glue code that scans your project for package and environment configurations.

pkgs/hello-python/default.nix - a build description

{
  self,
  python3Packages,
}: # (1)!

python3Packages.buildPythonPackage { # (2)!
  pname = "hello-python"; # (3)!
  version = "0.0.0"; # (4)!
  src = self; # (5)!
  propagatedBuildInputs = with python3Packages; [ # (6)!
    requests
  ];
  meta.description = "An example hello-python flox package."; # (7)!
}

Section of the file where we import things that we will later need in the build recipe.

In our case we import:

self: An object that holds reference to our project's content.

python3Packages: A Nixpkgs collection of python 3 packages and utilities.
A buildPythonPackage utility function from Nixpkgs which is used to describe a python package in Nix.
A package name.
A package version.
Source code of your project.
Runtime dependencies.

propagatedBuildInputs in Nix this concept refers loosely to runtime dependencies.

buildPythonPackage function does not know how to extract dependencies from setup.py. You need to provide them manually.
Description of a package. It will appear in flox search results.

We will cover the purpose of the flox environment file (pkgs/hello-python/flox.nix) in detail a bit later in this tutorial when talking about developing a custom package.

3. `flox build` - Build the custom package¶

With both a build recipe (pkgs/hello-python/default.nix) and the glue code (flake.nix) in place we can now build a package.

$ flox build # (1)!
warning: not writing modified lock file of flake 'git+file:///home/USER/dev/hello-python':
...
• Added input 'flox-floxpkgs/tracelinks/flox-floxpkgs':
    follows 'flox-floxpkgs'

$ ./result/bin/hello # (2)!
Hello python

This command tells flox to build your project and then creates a symbolic link to the package at ./result.
Let's run just built hello binary.
Running hello binary.

4. `flox run` - Build and run the custom package¶

These 2 steps can be simplified into one command.

Set the name of the main binary program in pkgs/hello-python/default.nix:

Index: pkgs/hello-python/default.nix
===================================================================
--- pkgs/hello-python/default.nix (revision 100644)
+++ pkgs/hello-python/default.nix (working copy)
@@ -12,4 +12,5 @@ python3Packages.buildPythonPackage {
     requests
   ];
   meta.description = "an example flox package";
+  meta.mainProgram = "hello";
 }

Build and run:

$ flox run # (1)!
warning: not writing modified lock file of flake 'git+file:///home/USER/dev/hello-python':
...
• Added input 'flox-floxpkgs/tracelinks/flox-floxpkgs':
    follows 'flox-floxpkgs'
Hello python

In the background, this command runs flox build and `./result/bin/hello' for you.

4. `flox develop` - Developing a package¶

With the build recipe (pkgs/hello-python/default.nix) having all the required information, we can provide a development environment for our package. Not only that, we can extend the development environment with extra development tools.

Adding extra development tools is done via the flox environment file (pkgs/hello-python/default.nix).

Lets add the black package to our flox environment file (pkgs/hello-python/flox.nix) to be able to format our python code.

pkgs/hello-python/flox.nix

{
  # flox environment configuration
  #
  # flox.nix options: https://flox.dev/docs/reference/flox-nix-config
  # Getting started with flox: https://flox.dev/docs
  # Get help: https://discourse.flox.dev
  #
  # Happy hacking!

  packages.nixpkgs-flox.black = {}; # (1)!

  environmentVariables.PIP_DISABLE_PIP_VERSION_CHECK = "1"; # (2)!
}

black is a commonly used python formatter.
The PIP_DISABLE_PIP_VERSION_CHECK environment variable makes sure that pip will not try to update itself.

Now lets enter the development environment for our hello-python package.

$ flox develop
warning: creating lock file '/home/USER/dev/hello-python/flake.lock'
warning: not writing modified lock file of flake 'git+file:///home/USER/dev/hello-python':
• Updated input 'flox-floxpkgs/nixpkgs/nixpkgs':
    follows 'flox-floxpkgs/nixpkgs/nixpkgs-stable'
  → 'github:flox/nixpkgs/1b1f50645af2a70dc93eae18bfd88d330bfbcf7f' (2023-01-23)
warning: not writing modified lock file of flake 'git+file:///home/USER/dev/hello-python':
• Updated input 'flox-floxpkgs/nixpkgs/nixpkgs':
    follows 'flox-floxpkgs/nixpkgs/nixpkgs-stable'
  → 'github:flox/nixpkgs/1b1f50645af2a70dc93eae18bfd88d330bfbcf7f' (2023-01-23)
Executing setuptoolsShellHook
Obtaining file:///home/USER/dev/hello-python
  Preparing metadata (setup.py) ... done
Installing collected packages: hello-python
  Running setup.py develop for hello-python
Successfully installed hello-python-0.1
Finished executing setuptoolsShellHook

$ black hello # (1)!
reformatted hello/__main__.py

All done! ✨ 🍰 ✨
1 file reformatted, 1 file left unchanged.

$ python -c "import requests; print(requests.__name__)" # (2)!
requests

$ hello
Hello world!

$ sed -i -e 's/world/everyone/' hello/__init__.py

$ hello # (3)!
Hello everyone!

$ exit

Command black is now available in the environment.
Python is configured with the required modules.
With Nix $PATH is configured to invoke hello directly from source code, which allows for rapid iterative development

Note that changes to the source code are immediately reflected in subsequent invocations.

Where to next?¶

Convert environments into container images.
Publish custom packages for others to use.

Custom packages¶

1. Prepare example project¶

2. flox init - Create an initial project structure¶

3. flox build - Build the custom package¶

4. flox run - Build and run the custom package¶

4. flox develop - Developing a package¶

Where to next?¶

2. `flox init` - Create an initial project structure¶

3. `flox build` - Build the custom package¶

4. `flox run` - Build and run the custom package¶

4. `flox develop` - Developing a package¶