NervesLocalSetup
How to get going with Nerves on Linux using a VM. Assumes you know how to use Elixir and qemu, and that you are operating on a Debian Linux system or something tolerably close to it.
Up to date as of April 2024.
Setup and install
The Nerves intro documentation is a little… scattered. It wants you to go through the livebook or something, which is fine but if you don’t want to do that and/or don’t have hardware to run it on right there, it takes a bit of diggging to set up. You have to go to Getting Started, which isn’t actually enough to get you started because you actually have to dig down further to Installation first. And none of it tells you anything about how to get going on a virtual machine, which seems like the obvious place to start when you don’t necessarily want to buy hardware for this, or don’t know what hardware you want to use yet.
So, let’s go through the setup process in order, and get all the pieces sorted out. First, install basic stuff:
sudo apt install build-essential automake autoconf git squashfs-tools ssh-askpass pkg-config curl libmnl-dev elixir libssl-dev libncurses5-dev bc m4 unzip cmake python3 libwxgtk3.2-devNext install fwup, which appears to be a firmware image
builder/bundler tool. It isn’t in apt, so we will install
it from Github. Note that fwup is NOT the same as
fwupd, helpfully. I have no idea whether they’re
related.
wget https://github.com/fwup-home/fwup/releases/download/v1.10.2/fwup_1.10.2_amd64.deb
sudo apt install ./fwup_1.10.2_amd64.debOk, this is where we hit our first hiccup. We SHOULD just have to do this:
# Update hex and rebar to latest (hex-2.0.6 and rebar3 1-14), and install Nerves base stuff.
# Requires Elixir >= 1.11.2
mix local.hex
mix local.rebar
mix archive.install hex nerves_bootstrap
# Create project template
mix nerves.new hello_nerves
cd hello_nerves
# Set MIX_TARGET, which specifies your target board. With fish shell:
set -x MIX_TARGET x86_64
# bash:
# export MIX_TARGET=x86_64
# Install deps and build firmware image
mix deps.get
mix firmwareBut if you’re like me and running on Debian 12 Bookworm, when you do
mix deps.get this will conveniently print an error message
starting with something like this:
==> hello_nerves
Nerves environment
MIX_TARGET: x86_64
MIX_ENV: dev
** (Mix) Major version mismatch between host and target Erlang/OTP versions
Host version: 25
Target version: 26
...Given that OTP 25 was released in late 2022 and OTP 26 still isn’t in
the Debian Unstable repo, and people are already talking about
what the OTP 27 release will bring, it seems a safe bet to bypass
the the Debian repo and install our own version of Erlang/Elixir. At
least until I have the time and energy to become a Debian maintainer and
give a hand to poor Sergei Golovan who has been packaging Erlang
mostly-singlehandedly since 2006. So I guess I’ll try this
asdf tool that the Nerves docs talk about. No, not that asdf. No, not
that one either. This one..
Ok, to install:
# The Nerves docs at https://hexdocs.pm/nerves/installation.html#linux say to use branch v0.11.0
# but I had some Issues with it and 0.14.0 seems to work more smoothly
git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.14.0
# There's also shell files for sh/bash/zsh, elvsh, nushell, and powershell; I'm frankly impressed.
source ~/.asdf/asdf.fish
# Tell asdf which things you want it to use
asdf plugin add erlang
asdf plugin add elixir
# Tell asdf to download and build the versions of things you want.
# It gives some scary-looking warnings along the lines of "asdf_26.0.2 is
# not a kerl-managed Erlang/OTP installation" but seems to work fine in practice.
asdf install erlang 26.0.2
asdf install elixir 1.15.4-otp-26
# Tell asdf which versions of things to use by default
asdf global erlang 26.0.2
asdf global elixir 1.15.4-otp-26Now running erl or iex should print out
“OTP 26” among the version text. If you want to use your system-wide
installs then just open a new shell and don’t source
~/.asdf/asdf.fish or whatever. Despite my bitching
asdf seems pretty sensible, it puts everything into its
install dir and you just run the shell script to access it, and if it
doesn’t know what version of something to use it just asks you. Not
perfect, but it’s something. Don’t want it anymore? Just delete
~/.asdf. I’m sure you can also set tools and versions and
stuff in a file in the project dir, but I’m not interested enough to
find out yet.
Right, on with the show. You have your MIX_TARGET still
set correctly after all that rigmarole? You still in your Elixir project
dir? Well it’s probably garbage now if you tried to build it with the
wrong version of Elixir, doing mix firmware I get the
following error:
|nerves| Building OTP Release... [216/1805]
* [Nerves] validating vm.args
** (Mix) Incompatible vm.args.eex
The procedure for starting IEx changed in Elixir 1.15. The rel/vm.args.eex for
this project starts IEx in an incompatible way for the version of Elixir you're
using and won't work.
To fix this, either change the version of Elixir that you're using or make the
following changes to vm.args.eex:
Please remove the following lines:
* rel/vm.args.eex:47:
-user Elixir.IEx.CLI
Please ensure the following lines are in rel/vm.args.eex:
-user elixir
-run elixir start_iex You can probably fix it, but easier to just nuke your project and start over.
cd ..
rm -rf hello_nerves
mix local.hex
mix local.rebar
mix archive.install hex nerves_bootstrap
mix nerves.new hello_nerves
cd hello_nerves
# Get deps for our target and build the actual firmware.
set -x MIX_TARGET x86_64
mix deps.get
mix firmwareYou should now have a hello_nerves.fw file in
_build/x86_64_dev/nerves/images/hello_nerves.fw, or
wherever mix firmware has told you it lives. On my system
it is 26 MB, which is pretty ok for a full Linux system with busybox,
Erlang and Elixir. The .fw file is just a zip archive with
whatever layout fwup has decided it needs. Peeking into it,
it’s some metadata, a few grub files, and a squashfs filesystem
containing about as little as humanly possible to run a Linux
system.
Whew! Ok. This is a lot, but it’s actually notably better than most embedded workspace setups I’ve had to deal with.
Actually running stuff
Now that we have a disk image, let’s make a VM that can actually run
this. The Nerves
x86_64 target seems… fairly underused at the moment, but we don’t
need to do anything fancy. No information on how to turn this
.fw file into something qemu can actually boot, but
investigating the fwup docs gives us the answer:
-a is “apply the firmware update”, so copying an image
into something instead of just building it. -d is the
“device” file, -i is the image, -t complete is
the “task” to perform, whatever
that means. So now we have myimage.img sitting in the
current dir, which inspection proves to be a raw disk image; you can
look at the partition table with cfdisk or your tool of
choice. Oh well then, let’s fire it up in qemu:
It boots, decompresses Linux, and then appears to hang forever. Well that’s inconvenient. The last lines are:
ALSA device list:
No soundcards found.
input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
Waiting for root device PARTUUID=04030201-02...
tsc: Refined TSC clocksource calibration: 3593.117 MHz
clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x33caed1ef19, max_idle_ns: 440795230794 ns
clocksource: Switched to clocksource tscIncorrect detours
So it looks like Linux does early boot stuff and then decides to send
its output somewhere else? Somewhere qemu doesn’t show by default with
-display curses. That’s rather odd, the Nerves
x86_64 platform documentation says
IEx terminal Display - tty0, so it should be
printing to the normal qemu screen. Looking at the man page for qemu, it
appears we might want -nographic, which says it redirects
everything to the terminal, including serial ports and stuff. But that
appears even worse, just hanging forever on
Booting partition A..., though at least this time qemu lets
us do C-a x to exit the emulator. So it’s either trying to
output its info to somewhere we can’t see it (I’m SSH’ed into a server,
so I have no graphics on the dev machine at the moment), or it’s not
even managing to boot linux properly. Probably the first option, since
nothing else in the config changed, but playing “where the hell is my
output going” is one of those joys of embedded programming in general.
I’m a bit rusty on qemu though, and it appears this system has version
7.2.9 rather than the version 6.2.0 all my osdev scripts are written
for. C-a c gets us to qemu’s very useful and critically
underdocumented system console, which shows… that things appear to be
working properly and the instruction pointer is just sitting in a tight
loop somewhere?
Huh. FINE I’ll fire up VNC over a heckin’ cell phone internet
connection and look at it for real! And I see… the same thing.
clocksource: Switched to clocksource tsc, and then nothing.
…WAIT after a few minutes it printed out
random: crng init done. Is the linux kernel for some reason
getting Really Confused about being in a VM? qemu’s hardware defaults
are usually pretty good about that, if only because it pretends to be a
Gateway shitbox from like 2003. Maybe that’s too old for Nerves’
chopped-down kernel to appreciate.
Let’s try something where Nerves probably works well, such as a RPi. Problem there is I have no idea how well qemu works emulating it, but here goes…
set -X MIX_TARGET=rpi0
mix deps.get
mix firmware
#fwup -a -d myimage.img -i _build/rpi0_dev/nerves/images/hello_nerves.fw -t complete
# Wait we can use nerves to create the image for us, just give it a normal filename to write to.
# Would that work with an x86_64 target? Huh, good question; I didn't even try it 'cause I assumed it wouldn't.
# But some experimenting shows that it DOES in fact work.
mix firmware.burn --device rpi0.img
# qemu complains if the SD card image is not a power of 2 size
qemu-img resize -f raw rpi0.img 512M
qemu-system-aarch64 -machine raspi0 -drive file=rpi0.img,format=rawHAH, there we– wait, no, nope, it just has a blank screen. Goddammit.
…nope, I’m just an idiot, the nerves_system_rpi0 docs say
“IEx terminal: UART ttyAMA0”. Well I don’t really want to
figure out the incantation to make qemu output the serial terminal to
the console right now. The Nerves docs for the rpi0 platform do note
that output can be switched to the HDMI output, but doing so “won’t be a
good development experience”.
:-|
No no, this is fine, qemu is always this bitchy. The RPi3 Nerves docs say that the RPi3 terminal is “HDMI and USB keyboard”. Let’s try that…
set -X MIX_TARGET=rpi3
mix deps.get
mix firmware
mix firmware.burn --device rpi3.img
qemu-img resize -f raw rpi3.img 512M
qemu-system-aarch64 -machine raspi3b -drive file=rpi3.img,format=raw…same result. Maybe VNC over cell phone is a mistake?
Ok after a lot of messing around and general mysteries, it
appears that the clocksource: Switched to clocksource tsc
is a red herring, and the random: crng init done message is
just it saying that it has enough hardware entropy (from me desperately
mashing stuff into the keyboard) to seed the kernel RNG. The
real error is a few lines above that, where it says
waiting for root device PARTUUID=04030201-02, but running
blkid on the image file shows that it does indeed have
PTUUID=“04030201”….
AHA! It works if I tell qemu to use the virtio drive type instead of
qemu’s default IDE interface! It does seem like a safe assumption that
Nerves would not bother compiling IDE support into its kernel. So it
boots if I run it with
qemu-system-x86_64 -drive file=amd64.img,format=raw,if=virtio -m 512.
This drops me into an iex terminal full of convenient
Unix-style shortcuts like ls/0 and pwd/0, and
I can indeed run the HelloNerves.hello() function and get
the expected output.
whew
The Whole Shebang
All the false turns and stuff are left in here on purpose, but that makes this harder to use as a reference. So here’s the whole working process I came up with:
# Install basic deps
sudo apt install build-essential automake autoconf git squashfs-tools ssh-askpass pkg-config curl libmnl-dev elixir libssl-dev libncurses5-dev bc m4 unzip cmake python3 libwxgtk3.2-dev
# Install fwup
wget https://github.com/fwup-home/fwup/releases/download/v1.10.2/fwup_1.10.2_amd64.deb
sudo apt install ./fwup_1.10.2_amd64.deb
# Install asdf
git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.14.0
source ~/.asdf/asdf.fish
# Tell asdf which things you want it to use
asdf plugin add erlang
asdf plugin add elixir
# Tell asdf to download and build the versions of things you want.
# It gives some scary-looking warnings along the lines of "asdf_26.0.2 is not a kerl-managed Erlang/OTP installation"
# but seems to work fine in practice.
# This will take a few minutes 'cause it has to build the Erlang system from source.
asdf install erlang 26.0.2
asdf install elixir 1.15.4-otp-26
# Tell asdf which versions of things to use by default
asdf global erlang 26.0.2
asdf global elixir 1.15.4-otp-26
# Install nerves and create template project
mix local.hex
mix local.rebar
mix archive.install hex nerves_bootstrap
mix nerves.new hello_nerves
cd hello_nerves
# Set target machine and get target deps
set -x MIX_TARGET x86_64
mix deps.get
# build stuff and make fwup package
mix firmware
# Turn that into a disk image file called hello.img
mix firmware.burn --device hello.img
# Run the sucker in qemu
qemu-system-x86_64 -drive file=hello.img,format=raw,if=virtio -m 512…and now that have figured all that out I see that there is in fact a
build target called mix nerves.gen.qemu_script. RIP.
Conclusions
As mentioned, in my experience setting up embedded toolchains always
sucks ass. This is the real reason that Arduino and Raspberry Pi
succeeded; not ’cause the hardware was amazing, not even only because it
was cheap, but because made it easy to get started and use the tools
with minimum hassle so that newbies could start experimenting quickly,
and people who kinda knew what they were doing but were new to the
domain (like me) spent a minimal amount of time screaming at header
paths and linker errors. Toolchains like clang and Rust’s
cargo show that cross-compiling doesn’t have to be
a nightmare, but it’s still an imperfect process.
So all in all, Nerves seems to do a pretty nice job of getting you
going with embedded programming. It uses other tools like
fwup and asdf but is a thin enough layer over
them that it doesn’t hide much of what’s going on, just provides nice
shortcuts. Not mentioned here but apparently it also uses Buildroot to construct kernels and
platform packages. So these are generally tools you want to be using
anyway. It definitely gives you a sane toolkit for building embedded
linux systems, so you don’t have to figure out how to do it from
scratch, which is extremely useful. If I had to build an embedded Linux
system without Nerves, I now know far, far more about how to do it well
than I did before I went through this. That said, there’s still some
hiccups:
- As mentioned, the documentation is somewhat scattered and bass-ackwards from what I expected/wanted. But that’s a Hard problem because different people learn in different ways.
- IMO VM’s deserve more love. As a prototyping and exploration tool they should not be underrated.
- More/better troubleshooting docs would be nice, but they tend to come from the system being used, and Nerves is still a very small community.
Lots of things seem to be just due to the small community. Erlang/Elixir programmers are already a niche, and embedded programmers are a very different niche, and the Venn diagram of the overlap is rather small. Still, it’s better than it looks. For example, the hardware support appears a little anemic, being basically Raspberry Pi, Beaglebone, and a few miscellaneous things… but there’s more unofficial platform packages out there and it doesn’t seem too difficult to create a custom one. It’d be be useful if they advertised that a little more, and more docs on how to create your own platform would be nice. I already have a bunch of devices in mind where Nerves would be useful, or at least fun… Anbernic handheld consoles and other crazy handheld consoles, Gateworks industrial boards for when I want a RPi that can survive being banged around a bit and someone else is paying for it, a large pile of Pine64 single-board computers and touch devices…
So yeah. A bit rough, but very very promising. I am excited.
