AI Stack as Code: Define It Once, Run It Everywhere
“Do you have some AI stack-as-code examples?” It’s one of the most common questions we get from customers.
What they want are reference patterns for implementing declarative stacks for things like model-training, retrieval-augmented generation (RAG), embeddings, and other AI use cases. So we went ahead and built some.
One pattern composes four modular AI environments to create a reproducible, cross-platform, completely declarative AI RAG stack. It can be used either for development or as the foundation for a production RAG pattern.
A second pattern, a llamacpp-powered model serving environment, uses Flox’s documentation and articles from our blog to provide a RAG-powered knowledge assistant for asking questions about Flox. It demonstrates a production pattern for deploying AI stacks-as-code.
Let’s dig into what a stack-as-code is… and why it’s the best way to create, share, and manage AI stacks.
AI Stacks That Run on Every Machine
If you’ve ever experienced the mounting frustration, the mix of fear and loathing, of attempting to run someone’s AI stack on your local system, the genius of AI-stacks-as-code should resonate with you.
Say you’ve built something on your MacBook using Apple’s MPS APIs and you go to share it with the rest of your team… but not everybody can use it because they're on Windows or Linux machines with NVIDIA CUDA GPUs. Or imagine the reverse, where ML and AI Stacks someone built for CUDA won't run accelerated on your MacBook using MPS or Apple’s MLX framework. If It runs at all, it’s in a container, with no access to a GPU.
The ML- or AI-stack-as-code makes these problems go away. Instead of packaging and materializing an AI or ML stack’s runtime dependencies for a specific machine platform, you declare them, like so:
[install]
python.pkg-path = "python3"
python.version = "3.13.13"
uv.pkg-path = "uv"
git.pkg-path = "git"
torch.pkg-path = "python313Packages.torch"
numpy.pkg-path = "python313Packages.numpy"
jupyterlab.pkg-path = "python313Packages.jupyterlab"
sentence-transformers.pkg-path = "python313Packages.sentence-transformers"
chromadb.pkg-path = "python313Packages.chromadb"The TOML above is an example of one kind of stack-as-code implementation: a Flox environment. It declares the basic dependencies for a reproducible, cross-platform AI RAG stack. Paste this into a Flox manifest and you get a runtime environment with exactly these declared dependencies. (Note: Flox isn’t the only declarative, stack-as-code technology out there. Others include Nix and Guix.) If you need GPU-accelerated packages for MPS/MLX Macs and NVIDIA CUDA to coexist in the same environment, no worries: just declare platform-specific versions of PyTorch and other performance-sensitive packages:
## cuda-accelerated dependencies
torch-cuda.pkg-path = "flox-cuda/python3Packages.torch"
torch-cuda.version = "2.10.0"
torch-cuda.systems = ["x86_64-linux", "aarch64-linux"]
numpy-cuda.pkg-path = "flox-cuda/python3Packages.numpy"
numpy-cuda.systems = ["x86_64-linux", "aarch64-linux"]
## mps-accelerated dependencies
mps-torch.pkg-path = "python313Packages.torch"
mps-torch.version = "2.10.0”
mps-torch.systems = ["aarch64-darwin"]
mps-numpy.pkg-path = "python313Packages.numpy"
mps-numpy.systems = ["aarch64-darwin"]You’re not building (or re_building_) anything, least of all a multi-gigabyte container image. You’re delegating to a graph-backed resolver the work of pulling in just the right platform- and hardware-optimized dependencies. The resolver uses the stack’s declarative spec to identify compatible versions of runtime dependencies. These get materialized into an immutable, cryptographically hashed store that lives on each machine.
In other words: You declare what the stack needs; the tool resolves the dependency graph and figures out how to materialize what you’ve declared locally. Nothing gets installed globally, so your machine won’t break. You can even install conflicting versions of packages side-by-side… on the same system.
AI Stacks That Run at Any Time
This pattern also obviates one of the most frustrating parts of working with AI stacks: you’ve got the right dependencies in the right places, but the stack still won’t run because… because what now? Because the runtime environment itself isn’t standardized: the Jupyter notebook starts with the wrong Python; or the dynamic linker finds the wrong versions of native libraries; or the model cache lives in the cloud and needs an Oauth token; or the GPU runtime only works when certain shell variables get exported.
An AI stack-as-code runs and behaves the same way everywhere because it doesn’t just declare that stack’s runtime software packages… it declares the environment in which the stack itself is to run.
[vars]
PYTHONNOUSERSITE = "1"
HF_HOME = "$FLOX_ENV_CACHE/huggingface"
SENTENCE_TRANSFORMERS_HOME = "$FLOX_ENV_CACHE/sentence-transformers"
CHROMA_DATA_DIR = "$FLOX_ENV_PROJECT/.chroma"
[hook]
on-activate = '''
export JUPYTER_HOST="${JUPYTER_HOST:-0.0.0.0}"
export JUPYTER_PORT="${JUPYTER_PORT:-8888}"
export JUPYTER_SERVER_TOKEN="${JUPYTER_SERVER_TOKEN:-}"
export NOTEBOOK_DIR="${NOTEBOOK_DIR:-$FLOX_ENV_PROJECT}"
export INSTALL_REQUIREMENTS="${INSTALL_REQUIREMENTS:-false}"
# Generate random token if not provided
if [ -z "$JUPYTER_SERVER_TOKEN" ]; then
export JUPYTER_SERVER_TOKEN="$(openssl rand -hex 32)"
echo "ℹ️ Generated random token: $JUPYTER_SERVER_TOKEN"
fi
# Setup venv
export VENV_DIR="$FLOX_ENV_CACHE/venv"
if [ ! -d "$VENV_DIR" ]; then
echo "Creating Python virtual environment..."
python3.13 -m venv "$VENV_DIR"
fi
# Activate venv with guard
if [ -f "$VENV_DIR/bin/activate" ]; then
source "$VENV_DIR/bin/activate"
# Install requirements if requested
if [ -f "$FLOX_ENV_PROJECT/requirements.txt" ] && [ "$INSTALL_REQUIREMENTS" = "true" ]; then
echo "Installing packages from requirements.txt..."
pip install -r "$FLOX_ENV_PROJECT/requirements.txt" --quiet
fi
fi
'''
[services]
jupyter.command = '''
if [ -f "$VENV_DIR/bin/activate" ]; then
source "$VENV_DIR/bin/activate"
fi
exec jupyter-lab \
--no-browser \
--ip="$JUPYTER_HOST" \
--port="$JUPYTER_PORT" \
--IdentityProvider.token="$JUPYTER_SERVER_TOKEN" \
--notebook-dir="$NOTEBOOK_DIR"
'''The Bash shell logic you see above is all that’s required to exactly reproduce the same environment across macOS, Linux, and Windows with WSL2, on x86 or ARM. If a machine has a CUDA or MPS-/MLX-compatible GPU, the runtime detects and uses it. Under the hood, Flox, which is built on Nix, wires up the environment, setting declared packages on PATH so they take priority over local versions. This means commands like python, uv, and jupyter always resolve to their declared versions.
There’s one more thing. An AI stack-as-code travels across space and time. Five years from now, it will resolve to exactly the same declared dependencies and reproduce exactly the same runtime environment on each supported target platform. (Assuming its platform, hardware, and accelerator APIs remain supported by upstream vendors!) The upshot is that an AI stack-as-code created two or more years ago (and pinned at that time to now-historical dependencies) still runs and behaves exactly the same today.
The Power of Declarative Environments
The way this works is simple. Let’s use Flox as an example: You declaratively define software packages, environment variables, setup logic, services, even build recipes as part of the Flox manifest. This makes recreating an AI stack’s runtime environment as simple as copying the .flox directory that’s co-located alongside the AI project’s Git repo or (even more convenient) activating it on-demand from FloxHub. Each Flox environment generates a lockfile that feezes software in place until you run flox upgrade to update one or more packages. Note: you can also pin packages to specific versions to keep them from being upgraded.
A Flox AI stack-as-code standardizes the complex moving parts of an AI stack, declaring Python versions and dependencies, native C/C++ libraries, document parsers, embedding models, vector databases, notebooks, and other software … even runtime services. It runs and behaves reproducibly, pulling in the same versions of the same packages. It lets you replace READMEs or runbooks with one command:
flox activate -sDefining AI stacks as code gives teams declarative artifacts they can version, diff, review, and promote (or roll back) atomically. The same Flox AI stack-as-code runs everywhere: on macOS, Linux, and Windows (with WSL2); on x86 and ARM; on Nvidia CUDA or Apple Metal/MLX. It’ll even run (slowly) on bare CPUs, if GPU acceleration isn’t available. An AI stack-as-code can also travel from local dev (for model R&D/training, RAG prototyping, document-ingestion workflows, etc.) to eval, CI, and production.
Finally, the declarative stack-as-code simplifies CVE remediation: not only can you catalog exactly which environments run which dependencies, but identifying vulnerabilities is as simple as a database query, while promotion or rollback takes the form of an atomic edit to a text file:
[install]
jupyterlab.pkg-path = "python313Packages.jupyterlab"
torch-cuda.pkg-path = "flox-cuda/python313Packages.torch"
sentence-transformers.pkg-path = "python313Packages.sentence-transformers"
openssl.pkg-path = "openssl"
-openssl.version = "3.6.1"
+openssl.version = "3.6.2"You might be thinking: “I can do basically the same thing with containers.” But, respectfully, you basically cannot.
A Dockerfile is neither purely declarative nor perfectly reproducible. Teams must rebuild OCI images to patch CVEs. And with AI workloads, especially, multi- or double-digit-gigabyte OCI images are costly to build, store, move, and run. Declarative stacks-as-code eliminate image rebuild churn; tools like Flox and Nix can even be used to generate distroless containers from declarative stack-as-code specs.
Flox AI Stack-as-Code Patterns
Flox’s RAG pattern defines:
- A Python 3.13 base;
- A document-parsing layer;
- An embeddings layer consisting of
torch,sentence-transformers; ChromaDB vector store); - A notebook layer consisting of JupyterLab, configured to run as a service; and
- A working, extractive-RAG pipeline with an evaluation harness.
This stack runs locally, without phoning home to external services. You can pull and run it just by doing flox activate -s flox-labs/ask-flox-llamacpp; alternatively, you need only clone this GitHub repo to get the same stack locally. Helper functions (ai-doctor, ingest, index, brief, etc.) run the RAG-building workflow.
Flox’s llama.cpp production model server environment can either layer on top of this RAG stack (for building and testing) or run on its own in production. It ships a built-in llama.cpp model server for Linux and macOS, along with Anthropic’s Claude Code, OpenAI’s Codex, OpenCode, and a slew of other agentic harnesses. The RAG pattern augments users’ prompts with the context required to answer them. It’s a neat demonstration of how you can use Flox to build, package, and publish anything: including (in this case) the index and contents of a ChromaDB vector database!
Run it by doing flox activate -s -r flox-labs/ask-flox-llamacpp. A pre-defined helper function, ask-flox <model_name>, runs Claude Code with the optional local model of your choice. If you don’t specify a local model, it runs against the latest version of Claude Opus. However you run it, Claude Code uses the search_flox_docs MCP tool to query ChromaDB and enrich the prompts it sends to the agent. To learn more about this environment, including how we built, packaged, and published it so it runs on both Linux and macOS, check out this GitHub repo.
▐▛███▜▌ Claude Code v2.1.198
▝▜█████▛▘ unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_M with medium effort · API Usage Billing
▘▘ ▝▝ ~/dev/ask-flox-llamacpp
❯ how does flox auto-activation work?
I'll explain how Flox auto-activation works. Let me search the Flox documentation for this information.
Called ask-flox
Based on the Flox documentation, here's how auto-activation works:
Overview
Auto-activation is an experimental feature that automatically activates Flox environments when you enter their directory and deactivates them when you leave. This eliminates the need to manually run flox activate or flox deactivate commands.
How it works:
1. Feature Flag: Auto-activation requires enabling the feature flag with FLOX_FEATURES_AUTO_ACTIVATE=true or by running flox config --set features.auto_activate true.
2. Prompt Hook: The Flox prompt hook is installed in your shell and is responsible for monitoring directory changes. This hook is installed through any in-place activation.
3. Directory Discovery: When you change directories, the hook walks up the filesystem from your current directory to the root, collecting all directories containing a .flox directory.
4. Eligibility Check: Each discovered environment is only auto-activated if you've explicitly allowed it using flox activate allow.
5. Activation Process: Eligible environments are activated outermost-first, setting environment variables and running hooks. Services aren't started unless services.auto-start = true is set in the manifest.
6. Deactivation: When leaving a directory, the environment is automatically deactivated, reverting shell changes.
Setup Requirements:
- Enable the feature flag (FLOX_FEATURES_AUTO_ACTIVATE=true or via config)
- Install the prompt hook (done automatically when using in-place activation)
- Allow specific environments to be auto-activated using flox activate allow
The feature is experimental and subject to change, but provides a convenient way to automatically manage environment activation based on directory context.
✻ Cooked for 3sExplore the Reference Patterns
The examples in this article are just a starting point. FloxHub hosts other AI-stack-as-code patterns that address different stages of the AI lifecycle, including model serving, generative image serving, inference model training, and other use cases.
You can clone these environments into your own projects, compose them with other Flox environments, or activate them directly from FloxHub without copying anything into your repos.