Blog

Nix in the Wild: Ockam

Flox Team | 23 August 2024

Mrinal Wadhwa likes to keep things simple, as befits the CTO of a technology startup called Ockam.

Ockam describes itself as “a toolkit for developers to build applications that can trust data in motion.” So when Ockam needed a scalable, sustainable solution for standardizing its build environment and improving reproducibility, Mrinal and his team looked for a solution that would be just as simple to implement, scale, and maintain.

They found it in Nix, a cross-platform build and packaging platform that enables reproducible, atomic package management. Ockam settled on Nix only after an extended period of trial and error—experimenting with Bazel and Gradle, among other tools.

But because Ockam uses a single code repository, or monorepo, Nix ultimately emerged as the only credible, viable choice, Mrinal maintains.

“Across all of these options of monorepo-related tools, Nix has worked best for us. And I feel like a very similar set of people will go down this track: people who are building complex systems that require lots of different languages, and who expect their team to grow rapidly,” he says.

The Need for Nix

For Mrinal and his team, using a monorepo was a proven, pragmatic way to support a polyglot programming model, allowing contributors to leverage the strengths of languages like Rust, Elixir, Swift, and Typescript to build software optimized for use case-specific requirements.

On the other hand, a monorepo had several drawbacks, not just because it made it more difficult to manage dependencies across dissimilar development environments, but also because it increased the complexity of Ockam’s continuous integration (CI) workflows. Ockam’s testing workflows are more complicated than most, not just because of the monorepo, but because it has to recreate common customer scenarios during testing—sometimes, using older software versions.

A third challenge was growth: in a sense, Ockam was a victim of its own success. Its regular release cadence and rapid evolution had fueled uptake and attracted more than 350 community contributors. This not only accelerated development, but expanded the size and complexity of its codebase.

This was good and bad, a testament both to the strength of Ockam’s community and its rapid maturation. But it also exacerbated the project’s existing dependency management and CI scaling issues. This mostly virtuous circle drove Mrinal to look for solutions to all three of these problems.

Dev Containers Are Not the Answer

For many, the obvious solution would have been containers.

But Mrinal and Ockam had already been down that path. In their experience, many would-be contributors balked at the complexity involved in managing and running containers, as well as the run-time resources they required. Others didn’t like that they weren’t as fast as native code. Still others couldn’t easily integrate them into their preferred workflows. After all, Mrinal points out, not everybody likes using a command-line terminal.

The biggest issue, though, had to do with their size; with all required dependencies, Ockam’s dev container images topped out at 6GB. Having to use images of this size not only discouraged potential contributors, but also caused headaches for maintainers.

“Our dev container image, which we used to call our Builder, very quickly grew to be 5 or 6 GB in size. All the dev tools, the Rust dev tools, the Elixir dev tools—everything is baked into this Docker image,” he explains. “So as that thing was growing bigger and bigger, it didn't serve its purpose, because a new person is just not going to download a 5GB thing before they touch us.”

Coming to Nix

First, Mrinal and his team experimented with Bazel. Then they tried Gradle, which they liked because it was language-agnostic, but which “has its own set of challenges. It’s one of those [tools] that only a few people know how to use,” Mrinal says.

Mrinal got interested in Nix when a new engineer, already proficient with the tool, built a prototype of Ockam’s build system based on it. This encouraged team members to experiment with Nix themselves. Like many technologists new to Nix, they were intimidated—at first.

There was its unique expression language, which—even though it allows users to define and manage environments in a declarative manner—has a steep learning curve. There was its (at that point) ad-hoc installation process. And prior to the development of Flakes, there were also questions about reproducibility.

“Everybody had a positive reaction to Nix as an idea, but nobody knew how to change it,” he says. So the prototype moved to the backburner while the team experimented with other tools.

During this time of trial and error, a lot was happening in the Nix community. Determinate Systems introduced its Nix installer, which helped standardize the Nix installation experience. And Flox and Determinate both emerged as open-core supporters of Nix, employing several prominent Nix contributors. Both companies focused on supporting Nix development while also building tools and services to make Nix more suitable for enterprise users.

Collectively, these changes helped address almost all of the Ockam team’s issues with Nix. The forgotten prototype moved back to the frontburner, and Ockam’s Nix journey got underway.

Early Results

One year into this journey, Mrinal says Nix has delivered on his expectations, radically simplifying dependency management, improving reproducibility, and—not least—helping Ockam continue to grow.

“It’s easing onboarding quite a bit. We can say to new people on the team: ‘Install Nix, here’s the one-liner for doing that, and then just type make nix_[your specific target], and then everything will build,” Mrinal explains. “That’s just a very, very different experience from ‘Install this tool, then these tools, then these tools and that tool, and also these tools, and—oh yeah—hope it all works.”

With Nix, Mrinal says, new engineers tend to get up to speed very quickly. The install process is flawless—as it should be—and builds just work. “They do the install and the build succeeds the first time, and [with other tools and frameworks] people often struggle with these steps for days.”

Getting Ahead of the Nix Learning Curve

Yes, he acknowledges, new team members usually aren’t familiar with Nix, but in the first place, newly onboarding contributors aren’t expected to configure Nix on their own, and, in the second place, there’s help from an unexpected source—GPT4, Claude, and other large language models (LLM), which have been extensively trained on Nix’s documentation, along with other Nix-related resources.

Believe it or not, he says, learning-Nix-by-LLM kinda/sorta works: “They actually give pretty good debugging advice. It’s been a pretty good way to help people ramp up to understand what’s going on with Nix.”

Mrinal’s team took other common-sense steps to help make the transition to Nix as painless as possible—both for veteran contributors and newcomers. “We added nix_ targets for all of our make [targets]. You can use make directly if you have the tools on your machine, or you can just type make nix_[target] and it will drop into a shell and do it there,” he says. “It helps when someone is new and they’re really only interested in building the Rust-specific codebase for Ockam. They can understand what’s going on in the Rust-specific targets and just use the Nix version of [the make binary]."

Not least, the team continues to collect notes, comments, lessons learned, mistakes made, and other feedback from users, documenting this information in an internal guide. Call it the Nix-o-nomicon.

…And Getting Comfortable with Nix

Once team members get comfortable with Nix, he says it’s easy enough for them to figure out how to tweak it—for example, by adding new tools.

“When we need to add new tools to the build process, that’s pretty fast as well. There are like five .nix files in the codebase—and once they get familiar with what’s going on, people feel comfortable tweaking them to add new tools,” he indicates. “Most of our CI targets are now using the Nix path so everything gets built inside nix.”

And speaking of CI, thanks to Nix, Ockam has achieved near parity between local development and CI: “The environment I’m testing in locally and the environment I’m testing in in CI are approximately the same,” he indicates. “They can feel confident [their code is] not going to do unexpected things in CI.”

Mrinal offers this surprising takeaway: one year on, Nix has in certain ways become invisible. At first, yes, Nix’s declarative language was intimidating for new users. There was no getting around that, but Ockam has been able to work through it using the interventions described above.

The upshot is that Ockam’s internal users just aren’t intimidated by Nix. In most cases, Mrinal says, they don’t even notice it. The larger proof is in Ockam’s success in onboarding newbies: “It just works and works well. It’s become invisible. It does what it’s supposed to do, which is ensure that people during [local] development and CI have approximately the same environments.”

Futures

Mrinal and his team haven’t yet had a chance to push the limits, if any, of reproducibility in Nix.

For their first year, they focused on easing the transition to Nix, especially for core Ockam contributors, most of them internal team members. One goal for the future is to try to make the Nix onboarding experience easier for external, open-source community contributors, too.

“This whole journey is now a year long. In that year we will have made it faster to onboard onto our codebase both for internal developers joining the team, which we plan to grow as rapidly as we can, and faster onboarding from the open-source community,” he says. “But I think the internal data points are proof that can be true externally as well. And if that's the case, that allows us to grow the open-source community faster, which has its own kind of network effect-like benefits.”

And then there’s reproducibility. While Ockam hasn’t rigorously tested and validated the reproducibility of its Nix-powered build processes, Mrinal emphasizes that this is on the roadmap.

“We still want to get to that reproducibility goal. I think we're somewhere between 90 to 95% of the way there, we just haven’t invested the time in conducting the tests that will prove to us that, hey, we can in fact reproduce the builds from a month ago, or two months ago, right? Is that true or not?” he explains, adding that in unofficial testing, Nix has consistently enabled reproducible build outcomes.

Even if Ockam can’t get above 90% to 95%, there’s tremendous value there, Mrinal argues.

“If I have to hunt down the problem, my [problem space] is that remainder, that 5% to 10% remainder,” he says. “And actually, dependency trees are getting horrible, right? So even 90% of reproducibility means you don’t have to tackle the depths of hundreds of dependency trees, right?”

Final Thoughts

The come-to-Nix experience of Mrinal and his team resembles that of so many adopters: there was initial interest, tempered by intimidation in the face of Nix’s ostensible complexity, followed by a gradual circling back. For Ockam, the decision to adopt Nix came down to the following factors:

• The availability of a Nix installer;
• The development of Flakes;
• The fact that Nix ran in a conventional shell, using a shell-like syntax;
• The realization that contributors didn’t have to be Nix Ninjas to effectively use it;
• Active support from vendors, demonstrating commitment to and backing for the Nix project.

Drilling down into these: the Nix installer obviated concerns about the complexity involved in installing Nix. Flakes brought more rigor (and determinism!) to package and environment management in Nix, giving Ockam the confidence it could achieve reproducibility at scale. The ability to run Nix in a shell, using familiar commands, aligned perfectly with the workflows of most Ockam contributors; and, for contributors who preferred using IDEs, Ockam was also able to implement effective workarounds.

Finally, effectively using Nix, especially to bootstrap an Ockam build environment, didn’t require mastering its language. In fact, Mrinal’s team has discovered that for most use cases, neither they nor contributors need to directly modify Nix expressions, or navigate the nixpkgs repository configurations. “Only a few people like to touch Nix code because of the intimidation factor. But that’s fine,” he says. “Slowly but steadily, we’ve now reached a point where everything we build in CI is built in Nix. Most people are using Nix and make to build. Some people using IDEs bypass Nix.”

Besides, Mrinal argues, for organizations or projects opting to use a monorepo, there are very few viable, scalable options. In this context, the “complexity” of Nix pales in comparison to that of other tools, like Gradle or Bazel.

He sees Nix as the most pragmatic solution to a wicked problem: delivering a flexible, consistent, and reliable cross-platform development experience—one that supports multiple programming languages, and crucially spans both local and remote development and testing. “I went down the Gradle track, I for a little bit went down the Bazel track. And yes, Nix is complicated, but those other things are also incredibly complicated and very frustrating in their own right,” he concludes.