Workflow improvements with Nix

Development workflows are always interesting to examine and are oftentimes beneficial to revise for your own sake once in a while. The more efficient/correct your workflow, the less of a grind it is for you to get started and iterate, that much is clear. At times, investments made in this direction also facilitate adoption by fellow developers (think Vagrant). In all honesty, the amount of time I’ve invested in adopting a flow that works well for me, however, is probably disproportionate to the time I’ve spent actually hacking away at something meaningful.

Just by using Nix/NixOS you already opt-in to some of many awesome features. Conveniently, the buck doesn’t stop there. In this series, we have yet to examine nix-shell, another powerful tool in the Nix toolset, and how this can be coupled with direnv to make development truly seamless and buttery smooth.

Smooth sailing

There are a whole number of factors that contribute to the time it takes to set-up a new project or begin hacking on an existing project. My primary concerns were:

  • to be able to easily fetch dependencies for multi-language projects; and
  • to isolate these development environments to that specific project, keeping it local instead of polluting the global system state.

Both of these goals are met using nix-shell.

nix-shell

nix-shell is perhaps one of the most valuable tools in the Nix toolset. In a sentence, it allows users to enter a sub-shell with specific Nix packages set-up in a sort-of virtual environment. It is similar to nix-build in that it receives a file defining a package as input, except it does not execute the build, stopping beforehand and entering the environment to be used for building the package. Through this, it’s possible to define a single default.nix for a piece of software you are developing, and use this to package the project for Nix, whilst also using the same file to build an environment suitable for the development of said project.

Here is an example shell.nix file I use for my Rust projects.

{ pkgs ? import <nixpkgs> {} }:

pkgs.mkShell {
  buildInputs = with pkgs; [
    latest.rustChannels.stable.rust
  ];

  RUST_BACKTRACE = 0;
}

It’s really just that simple. There are a couple of things to keep in mind.

  • In the example, I’m using the Rust overlay in order to have specific versions of tools like cargo available in my environment.
  • Key/value pairs in the set passed to pkgs.mkShell are exported by Nix as environment variables (in fact, this piggybacks straight off of stdenv.mkDerivation, see NixOS/nixpkgs#30975). This is convenient for us to export variables to the nix-shell.

Here is another example of a quick C++ hack I was working on last year.

{ pkgs ? import <nixpkgs> {} }:

pkgs.mkShell {
  buildInputs = with pkgs; [
    gbenchmark
    gcc49 # Can specify a specific compiler version.
    ispc
    tbb
  ];
}

This is already 10-steps forward from other available solutions just in terms of memory footprint and simplicity. No virtualization layers, etc. just plain-old Nix package management. To make things even better however, we can save ourselves some sub-shell madness using direnv.

In tandem with direnv

direnv is an environment management tool developed by zimbatm. It essentially sets and removes environment variables in the shell depending on the current directory and the presence of a .envrc file.

Before each prompt, direnv checks for the existence of a “.envrc” file in the current and parent directories. If the file exists (and is authorized), it is loaded into a bash sub-shell and all exported variables are then captured by direnv and then made available to the current shell.

It should be clear at this point that direnv is the perfect partner to nix-shell. It even has built-in support for capturing the nix-shell environment. direnv can be used with nix-shell through a one-liner.

use_nix

To see it all in effect, we just need to enter the directory with both files defined.

$ cd project
$ direnv allow # One-time command.
direnv: loading .envrc
/home/rycwo/dev/repo/bigint-base10/.direnv/nix/shell.drv
direnv: export +AR +AS +CC +CONFIG_SHELL +CXX +HOST_PATH +IN_NIX_SHELL +LD +NIX_BINTOOLS +NIX_BINTOOLS_WRAPPER_x86_64_unknown_linux_gnu_TARGET_HOST +NIX_BUILD_CORES +NIX_BUILD_TOP +NIX_CC +NIX_CC_WRAPPER_x86_64_unknown_linux_gnu_TARGET_HOST +NIX_ENFORCE_NO_NATIVE +NIX_HARDENING_ENABLE +NIX_INDENT_MAKE +NIX_LDFLAGS +NIX_STORE +NM +OBJCOPY +OBJDUMP +RANLIB +READELF +RUST_BACKTRACE +SIZE +SOURCE_DATE_EPOCH +STRINGS +STRIP +TEMP +TEMPDIR +TMP +TMPDIR +WINDRES +_PATH +buildInputs +builder +configureFlags +depsBuildBuild +depsBuildBuildPropagated +depsBuildTarget +depsBuildTargetPropagated +depsHostHost +depsHostHostPropagated +depsTargetTarget +depsTargetTargetPropagated +doCheck +doInstallCheck +name +nativeBuildInputs +nobuildPhase +out +outputs +phases +propagatedBuildInputs +propagatedNativeBuildInputs +shell +stdenv +strictDeps +system ~PATH

Exiting the directory cleans everything up.

$ cd ..
direnv: unloading

Needless to say, I have yet to come across any simpler workflow!

Drawbacks of this approach

One of the difficulties I found in transitioning to using Nix was drawing the line between using Nix to manage project dependencies/packages vs. their respective solutions (cargo, pip, vim.plug). Using Nix works well for development projects that expect you to manage your own dependencies such as C/C++, whereas I’ve found that with projects written in say Rust or Python, you end up having to work against an existing solution, generating additional files just to be able to work in a “Nix-like” manner.

In one sense it could be described as an anti-pattern. The amount of effort needed from the community to collect every single package/module and their dependencies etc. seems to be more effort than it’s worth even despite the automate-able nature of the problem. On the developer’s side, everything works fine if you follow the Nix workflow until you have to package dependencies yourself, whether they are niche packages not available in nixpkgs or your own packages that act as local dependents.

In almost all of the provided tooling, language packages/modules would typically be downloaded to somewhere within the user’s home directory, which seems like a sane idea to me and avoids pollution of the system itself. Perhaps ultimately I have yet to run into a problem with this setup that required me to re-consider going down the Nix path.

Additional tips

Aside from the magical nix-shell and direnv combination, there are a few other little bits that I wanted to share that didn’t seem long enough to have their own post.

Patching st

As a fan of the suckless philosophy, I try to keep my set-ups lightweight and minimal. Mileage down either of these lines varies on a case-by-case basis, but my terminal-emulator-of-choice is an example of where I am quite happy with my progress. st is one of the most simple implementations of fully-featured terminal emulators I’ve encountered. Aside from being incredibly small and light, it shines in its approach to configuration.

All the configuration for st is defined in config.h. The configuration is baked into the compiled executable, which works great because let’s be honest - how often do you modify your terminal settings anyway? It saves the software from being bloated with runtime customization options that are really only used one-percent of the total time spent on the terminal.

There are a few strategies we could use to configure the st Nix package, but by far the most straightforward method is to take advantage of the patch phase of stdenv.mkDerivation. Generating a patch with your configuration options can be done using diff -Naur a b > override-st-config.patch, where a and b are the original and modified versions of the config respectively. The final step is to override the patches attribute when listing packages in configuration.nix.

environment.systemPackages = with pkgs; [
  # ...

  (pkgs.st.overrideAttrs (oldAttrs: {
    patches = [ /path/to/override-st-config.patch ];
  }))
];

Wrapping (Neo)Vim

As a Vim user, I have invested quite a bit time in tuning my Vim configuration and plugins (fellow Vim users will understand)! Some of you may have come across plugins such as Syntastic, ALE, or the more recent LanguageClient (to be used in conjunction with language servers). Each of these plugins augments Vim with IDE-like functionality, but they all depend on having external tools (Clang, Pyflakes, to name a couple) available in the system PATH. The knee-jerk reaction would be to install these tools directly with your package manager, but then we would be introducing project-specific tools into the global environment. Naturally, as this series is about using Nix, I would like to share a Nix-y solution that has been working well for me.

One of the neat things about the way packages are built in Nix is their “composability”. Digging through nixpkgs, it appears to be quite a common pattern to wrap existing packages to provide additional functionality. We can apply this mentality to wrapping Vim so that all of our plugin dependencies are available during runtime, and it works well because the dependencies are closed within the wrapper environment, keeping our system clean. This can be done quite simply with wrapProgram. To see an example, you can refer to my own NeoVim wrapper.


Taking over a year to wrap-up a series is a testament to my millennial attention span, though I’m happy to say this is the last post in the series. I hope it was helpful in some way to you and may have in some way convinced you to give Nix/NixOS a try. If it is too daunting to dive straight in, consider playing around with it in a VM. I did so myself and discovered a few advantages along the way:

  • you can hack away as you like without having to re-install the OS if you brick it, just nixos-rebuild switch --rollback;
  • and when you’re happy with the configuration, you can copy your configuration.nix when you install the OS on your host machine and nixos-rebuild switch will give you an identical system configuration!

Have fun, and happy hacking! :v:

Useful links