LanguageCompilationSpeed

Written between April and May 2021. The world may have changed since then.

Introduction

One of the annoying things about programming in Rust is waiting for things to compile. “Why is Rust slow to compile” is a complicated question that has lots of moving parts that I won’t go into right now, but either way, rustc takes a while to do its work. But I’ve always felt like it was partially a problem of expectations. People used to fast-turnaround languages like JS and Python seem to complain about it loudest (even though I’ve had Typescript projects that take minutes to compile anyway). Meanwhile I don’t seem to see as many people used to C++ bitching about Rust’s compile times. Regardless of expectations though, everyone can agree on the fundamental issue: Faster compilation times are better. Nobody’s ever said “boy I wish I had to wait an extra ten minutes to see if this thing I made even works”. What people argue about is how MUCH better fast compiles are, and what the cost/benefit tradeoffs are.

So for a long time I have wanted to actually benchmark the speed of compiling Rust code vs. other programming languages, but it’s been stymied by one major problem: Lack of suitable benchmark code. A good benchmark for this sort of thing would be as direct a translation of a program from one language to another as possible by someone who is an expert in both, but in many ways Rust is unique enough to make that pretty hard. It has a type system much more like Haskell than C++, and the borrow checker encourages some patterns that are a little odd compared to both manual memory management and garbage collected laguages. A good benchmark would thus have to be a nontrivial program written in Rust and then translated to C++ as accurately as possible, or vice versa, and then one could still get tangled up in what is idiomatic code or whether the compilers are actually doing the same thing. A better benchmark would be several of these programs with different problem domains, with proper statistics and stuff, but frankly, I am not going to put that much work into trying to figure this out when I could be doing something actually useful instead.

But then I had a thought! A thought that is silly enough to be fun but also might be realistic enough to be useful. Each language I’m interested in looking at has at least one large, real-world codebase that solves similar problems in fairly similar ways, and is written by experts in a style idiomatic to that language: its compiler. So, I’m going to look at the compilation speed of different languages by benchmarking how fast various compilers compile themselves.

Of course this is not actually an apples-to-apples comparison. Different compilers can be very different beasts that potentially use very different technology under the hood. There’s also a vast range in the size of compilers: clang+LLVM is millions of lines of code, while chibicc is barely more than 10,000, and both are functional C compilers. rustc is written in Rust… except for the giant chunk of C++ that is LLVM. We could look at the Roslyn C# compiler as well… except C# is generally executed by a JIT runtime, where a one-and-done compiler execution is kinda the worst case for the JIT’s optimization capabilities. So this isn’t going to be a perfect comparison. But hopefully digging into what the differences are and what impact they have is usually also pretty interesting. Fortunately, most of these languages also have multiple implementations, so we can get at least a couple data points for each language and see what the spread looks like.

Methods

Hardware doesn’t matter. It’s a Ryzen 2400G in case someone cares. What I care about is relative speeds, not absolute, and all the tests are done on the same hardware.

We will measure speed in terms of lines compiled per second, using a single process/thread. “Lines” will be measured by the tokei program, and will include comments. Most of these compilers do whole-module or whole-program optimizations, which makes “lines per second” rather meaningless since the cost to optimize it varies based on the total complexity of the program, not a constant multiplier that scales with lines of input. Many algorithms used in compilers are apparently also super-linear in complexity, so doubling the size of the program may increase the amount of time the compiler spends on it by more than two. So this is already a hopeless idea, but I’m going to try to scrape a few fragments of meaning from it, but I don’t bother too hard to try to analyze and correct for any of these confounding factors.

So on the scale of “lies, damned lies, and statistics”, the results of this experiment will hover somewhere around the region of “damned lies”. Like cosmology and theoretical physics, if we get something within 10x of the correct answer, I’m going to be happy with it. If I tried to get entirely rigorous statistics for everything I would probably be at this for months; I would rather skim lightly over a lot of things and accept the results will be vague guidelines at best.

C/C++

I know these are separate languages, but it appears the main C compilers, gcc and clang, both compile both C and C++, and gcc at least is written mostly in C with a fair amount of C++ mixed in. So I’m going to pretend they overlap.

gcc

gcc 10.2.0 compiled with the gcc 10.2.0 provided on my Debian system. Requires libgmp, libmpfr, and libmpc. And libgcc for multilib??? To build I did ./configure --enable-languages=c --disable-multilib --disable-bootstrap, followed by time make -j1

GNU docs kinda suck ass, as always. And the build system spends a significant amount of time running configure scripts, even when I try to make it do all that up front. And apparently gcc is written largely in C++ these days so this isn’t really an apples-to-apples test; I could have sworn it used to be mostly C. There’s still a lot of C in the auxiliary libraries though. The problem here is figuring out what the flying fuck is and isn’t actually getting compiled. Per these docs (findable in gcc/doc/install.texi) by default it does a full 3-stage bootstrap build: builds itself with the system C compiler, builds itself with the version of itself just built, then builds itself again and compares tests against the stage two compiler. Configuring with --disable-bootstrap turns this off, apparently, so we only compile the compiler once. Parts of the build process appear to be interested in the location of numpy, clang, and Eris only knows what else, and it also builds some amount of runtime library junk. It includes or fetches a copy of zlib, it generates about 1.5 million lines of code for various specifications and state machines like an instruction parser, builds the obtusely-named libiberty at least twice, and all sorts of hairy things like that.

So in general it does everything possible to make it impossible to count what I want to count. I’m going to try anyway, but don’t expect these numbers to be remotely correct.

Executed in   18.49 mins   fish           external 
   usr time  1012.76 secs  467.00 micros  1012.76 secs 
   sys time   98.19 secs   92.00 micros   98.19 secs 

Okay, now can I tell it to bootstrap itself once by hand? It should give identical results but I want to see what actually happens. Instead I… can’t even find where it put the gcc output executable. RIP. Heck, let’s build gcc with clang and see what happens! Is that even possible? Apparently! Works flawlessly, just by setting the CC environment variable. Building with clang we get:

Executed in   19.88 mins   fish           external 
   usr time  1072.10 secs  550.00 micros  1072.10 secs 
   sys time  121.03 secs  108.00 micros  121.03 secs 

tokei gcc/ libgcc/ libatomic/ libiberty/ turns up:

-------------------------------------------------------------------------------
 Language            Files        Lines         Code     Comments       Blanks
-------------------------------------------------------------------------------
 Ada                  6233      1473225       808366       379355       285504
 Alex                  215         6623         6623            0            0
 Assembly              505        80595        68559         2639         9397
 Autoconf               46        30229        20932         4202         5095
 BASH                    1          138          110            8           20
 C                   56442      4781658      3276207       846600       658851
 C Header             2047       567076       387633        90201        89242
 C++                   285       182221       131712        24270        26239
 D                    2285       173438       120269        23466        29703
 FORTRAN Legacy        504        19447        12668         5509         1270
 FORTRAN Modern       6321       229706       167504        34321        27881
 Go                   1088        82178        64885         7864         9429
 Haskell                59          217          191            0           26
 Makefile               14          622          439           49          134
 Markdown              422       463941       463941            0            0
 Module-Definition     154        42887        37848            1         5038
 OCaml                   1          358          285           29           44
 Objective C           528        24726        16837         3004         4885
 Objective C++         368        17491        11818         2218         3455
 Perl                    4         1580         1107          265          208
 Python                  5         1493          953          259          281
 ReStructuredText       65        61884        61884            0            0
 Shell                  25         3234         2028          788          418
 Standard ML (SML)       1          277          215           28           34
 TeX                     3        10601         6477         3307          817
 Plain Text             28         5344         5344            0            0
-------------------------------------------------------------------------------
 Total               77649      8261189      5674835      1428383      1157971
-------------------------------------------------------------------------------

So if we only take the 5,530,955 lines of C, C++ and header files, our results are something like:

  • gcc: 5,000 loc/second
  • clang: 4,600 loc/second

Again, these numbers are absolute BS because there’s tons of weird stuff going on here besides “just” building a compiler. But it’s a number.

g++

Same deal as above, except configured with ./configure --enable-languages=c++ --disable-multilib --disable-bootstrap. It took almost exactly the same amount of time, so I’m going to assume they’re doing pretty much the same thing and the results are redundant.

clang

Ok, we are using clang version 9.0.1, and building the same version. We are going to build the LLVM lib and clang compiler frontend, since that seems the best apples-to-apples comparison that involves the whole compiler. This… is going to take a while.

…and, it got killed with an out of memory error while linking clang, after 196 minutes. Silly me, trying to build LLVM on a system with only 13 GB of memory free. How dare I? I tried another time or two with the same results. I could try to link it with lld, but that doesn’t sound very rewarding. And now I know why the LLVM people are building their own linker at all, it’s because their code is so gigantic they actually need it. After some wizardly help from someone on Discord, it turns out that building clang without debug info enabled works, and doesn’t OOM my entire system. Have to include -DCMAKE_BUILD_TYPE=Release in the cmake command line. I guess including debug symbols for a 157 megabyte binary is a lot of work for the linker to sort out.

So the working build string is: mkdir build; cd build; cmake -DLLVM_ENABLE_PROJECTS=clang -DCMAKE_BUILD_TYPE=Release -G "Unix Makefiles" ../llvm; time make -j1

Time results are:

Executed in  201.62 mins   fish           external
   usr time  192.27 mins  1118.00 micros  192.27 mins
   sys time    9.09 mins  190.00 micros    9.09 mins

So that was building it with the default compiler on my system, which is gcc. Let’s build it with clang:

cmake -DCMAKE_CXX_COMPILER=/usr/bin/clang++ -DCMAKE_C_COMPILER=/usr/bin/clang -DLLVM_ENABLE_PROJECTS=clang -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release ../llvm; time make -j1

Executed in  191.03 mins   fish           external
   usr time  184.09 mins  551.00 micros  184.09 mins
   sys time    6.10 mins  100.00 micros    6.10 mins

(Sidetrack: Does using Ninja instead of makefiles result in any speedup? Turns out it makes it 1 minute faster for single-process builds. Less than 1%, so, not enough to be interesting to me here.)

Ok, measuring with tokei, the llvm and clang directories contain 4,635,178 lines of C/C++/header code. …and 1,360,633 of assembly that I’m going to ignore. So our results are:

  • gcc: 4,635,178 / (201 * 60) = 384 lines/second
  • clang: 4,635,178 / (191 / 60) = 404 lines/second

If we include all the C-family code in the entire project directory that’s 7,385,315 lines, which brings the numbers up to:

  • gcc: 612 lines/second
  • clang: 644 lines/second

So the “real” number is probably somewhere between those extremes. Of course we’re ignoring assembly language, we’re ignoring the demonstrably not-ignorable fact that the linker is a non-trivial part of this process, and so on. But still, 600 lines of code per second is probably an underestimate, it is impressive. And not in a great way. I blame C++.

chibicc

Chibicc is an educational but functional C compiler designed to be small and readable. I needed something small and satisfying to try to compile. Building it is easy, basically no configuration is required. Times measured with time make -j1.

Building with GCC:

Executed in  781.51 millis    fish           external 
   usr time  661.18 millis    0.00 micros  661.18 millis 
   sys time  120.50 millis  713.00 micros  119.79 millis 

With chibicc built by GCC:

Executed in  900.55 millis    fish           external 
   usr time  753.50 millis  746.00 micros  752.76 millis 
   sys time  147.35 millis  131.00 micros  147.22 millis 

With chibicc built by itself:

Executed in    1.66 secs   fish           external 
   usr time  1502.81 millis  609.00 micros  1502.20 millis 
   sys time  154.26 millis  107.00 micros  154.15 millis 

SLOC measured by tokei:

-------------------------------------------------------------------------------
 Language            Files        Lines         Code     Comments       Blanks
-------------------------------------------------------------------------------
 C                      50        11270         8694          771         1805
 C Header               13          718          537           69          112
 Makefile                1           50           31            3           16
 Markdown                1          209          209            0            0
 Shell                   6          363          255           43           65
-------------------------------------------------------------------------------
 Total                  71        12610         9726          886         1998
-------------------------------------------------------------------------------

Our lines of C+header code is 11,918. So the results are:

  • gcc: 11918 loc / 781 ms (has to parse comments and stuff as well, so!) = 15,300 loc/second
  • chibicc built with gcc: 13,200 loc/second
  • chibicc built with itself: 7,200 loc/second

So we see that chibicc compiles code a little bit slower than gcc, but its optimizations suck in comparison.

lcc

Sure why not try some other C compilers; I have fond memories of using lcc back in the days as the one free C compiler that actually worked on Windows. Building it is a little bit of a pain though, you have to set Magic Environment Variables correctly or else it will just do something random. It’s a real throwback to the Good Ol’ Days of of the late 90’s/early oughts, when men were men, errors were silent, and computers expected you to tell them information they could figure out themselves. Read the docs. All of them.

For me the build command ended up being:

mkdir out
HOSTFILE=etc/linux.c BUILDDIR=out make all -j1

Building with gcc:

Executed in    4.06 secs   fish           external 
   usr time    3.43 secs  787.00 micros    3.43 secs 
   sys time    0.63 secs  129.00 micros    0.63 secs 

Compiling it with itself didn’t work ’cause it looks for all the various bits of itself in the wrong places. Not sure how to fix that yet.

tokei output for the following directories: src/ include/ lburg/ cpp/ etc/:

-------------------------------------------------------------------------------
 Language            Files        Lines         Code     Comments       Blanks
-------------------------------------------------------------------------------
 C                      50        16356        14821          566          969
 C Header               72         3637         3191           49          397
 Happy                   1          202          202            0            0
 Markdown                6         5796         5796            0            0
 Shell                   1           52           42            4            6
-------------------------------------------------------------------------------
 Total                 130        26043        24052          619         1372
-------------------------------------------------------------------------------

Total lines of C+header: 19,993

  • gcc: 19,993 loc / 4.06 sec = 4,900 loc/second

tcc

I wasn’t going to do this but it turned out to be really easy, so here it is. We are using version 0.9.27. Just do configure --cc=gcc or whatever. I did have issues bootstrapping it; for some reason it had trouble finding header files if it wasn’t installed in the correct place. So I compiled it with the Debian packaged version of tcc, which has the same version number, and then installed that copy to /usr/local with its make install command. Then it could build itself fine.

  • building with gcc: 6.02 seconds
  • building with system tcc, presumably built with gcc: 0.48 seconds
  • building with tcc built with itself: 0.495 seconds

Tokei output:

-------------------------------------------------------------------------------
 Language            Files        Lines         Code     Comments       Blanks
-------------------------------------------------------------------------------
 Assembly                9         1368         1183            0          185
 Batch                   1          189          171            0           18
 C                     160        50572        41058         4489         5025
 C Header               93        40384        34289         1553         4542
 Makefile                5          926          665          105          156
 Module-Definition       5         3398         3346            0           52
 Perl                    1          427          306           63           58
 Shell                   2          560          488           28           44
 Plain Text              1          168          168            0            0
-------------------------------------------------------------------------------
 Total                 277        97992        81674         6238        10080
-------------------------------------------------------------------------------

This includes a couple example programs but they’re like 150 lines total. More importantly, the win32/ dir has 400 lines of C and 32750 lines of header files. That also includes the code for the ARM targets and such, which is another 8000ish lines. So, omitting those, our total is 60,244 lines, which may still be an overestimate. That is indeed quite tiny; not exactly a weekend project, but well within the reach of a reasonably motivated person doing this for fun.

So our results are:

  • gcc: 10,000 lines/sec
  • tcc built w/ gcc: 125,500 lines/sec
  • tcc bootstrapped with itself: 121,700 lines/sec

Dang, that’s fast. These are also one of the few sets of numbers where I’m sure enough of what is and isn’t actually being compiled to trust that it’s pretty accurate.

Rust

rustc

Finally, something with a toolchain that wasn’t built by rabid baboons. …well, maybe it was, but they’re my rabid baboons, and I know how they think, and where they write things down. So, following the README.md file, we just cp config.toml.example config.toml, and do time ./x.py build. …okay, that starts by cloning git repos for a big pile of documentation, a small pile of auxiliary crates, and LLVM, and building a build tool. However, if you do ./x.py help then it does all the downloading and configuring and such, getting everything ready for you to do the actual run without needing to download anything more.

Let’s use the rustc 1.52 version tag, since that’s the version of rustc I have on my computer already. First off, we need to make it build LLVM or use a pre-built version of LLVM, so we only look at the time spent compiling the Rust part of the compiler. The full commands are:

cp config.toml.example config.toml
# Get everything downloaded
./x.py help
# Build only LLVM first, since we only want to measure rust
./x.py build llvm
# Build the rest of the compiler.
time ./x.py build -j 1 --stage 1

This doesn’t seem to build clang with debug symbols, so it all works out. We are doing a “stage 1” build. This still re-does a bit of work: it builds the Rust stdlib with your existing Rust compiler, builds a new Rust compiler using that stdlib, and then re-builds the stdlib with the new compiler. For now that’s fine, I suppose. Results

Executed in   43.05 mins   fish           external
   usr time   42.49 mins  552.00 micros   42.49 mins
   sys time    0.54 mins  115.00 micros    0.54 mins

By now the rust repo directory contains everything, including all of LLVM, various tools, unit tests, etc. I THINK that what actually gets built with stage1 is only the code in compiler/ and library/. There is a src/ directory as well but it appears full of unit tests, CI stuff, checked out source for tools such as cargo and rustdoc, etc. The outputs in build/x86_64-unknown-linux-gnu/stage1 are only for rustc, rustdoc and the standard lib, I think, so I am going to omit src/. This might not be correct, but most of the bulk of code in src/ is unit tests which I know aren’t getting built, so hopefully this is close-ish. So the tokei output for compiler/ and library/ summed together is:

-------------------------------------------------------------------------------
 Language            Files        Lines         Code     Comments       Blanks
-------------------------------------------------------------------------------
 Assembly                1          372          347            0           25
 Autoconf                4         2683         2167          180          336
 C                      32        15620        11217         1984         2419
 C Header                6         1512          906          277          329
 C++                     6         3968         3248          261          459
 Dockerfile             35          508          428           20           60
 HTML                    1        93399        84614           32         8753
 JSON                    1           34           34            0            0
 Markdown              508        18309        18309            0            0
 Python                  1          236          196            5           35
 Rust                 2325       921697       642237       195016        84444
 Shell                  28        10437         7694         1581         1162
 Plain Text              1         8465         8465            0            0
 TOML                   89         2033         1733           64          236
 XML                     1       148137       147871            0          266
 YAML                    1            2            2            0            0
-------------------------------------------------------------------------------
 Total                3040      1227412       929468       199420        98524
-------------------------------------------------------------------------------

Or, 921,697 lines of Rust code. So, our results are:

  • rustc: 357 lines/sec

I’m preeeeeetty sure this is an underestimate: the stage0 build that happens before stage1 builds the stdlib again, and seems to build a few other minor tools as well. I could try to omit that from the calculations, but frankly I didn’t do that with any of the other compilers, so in the interests of fairness I’m not going to try. A bit of noodling around suggests it might shave a couple minutes off the time at most, anyway.

mrustc

The Other Rust Compiler, written in C++ to be able to bootstrap rustc without needing a copy of rustc to bootstrap itself. It only implements a subset of Rust, but it’s the subset used by rustc itself so it’s a pretty broad and slightly weird one; the main shortcut is it assumes the code it’s given is valid, so does very little actual checking. It’s designed basically to be a clean-room way to bootstrap rustc without needing rustc to already exist on a system, so this is actually a perfect test for it.

So let’s build the sucker. make -j1 takes 448 seconds with g++. It’s about 108,000 lines of C++. So, that’s about 240 lines/second, with gcc. And since it’s not written in the language it compiles, for once I know for sure it’s not sneakily doing any bootstrapping stuff to make my life hard. mrustc, written in C++, compiles more slowly in terms of lines/second than rustc written in Rust does, which is just hilarious.

So how fast does mrustc build rustc, I wonder? Once mrustc is built we do make RUSTCSRC to download the rustc source (it builds rustc 1.29, so this isn’t quite apples-to-apples with what we did in the previous section). Then we do time make -f minicargo.mk -j1 to build the build tool, the Rust standard lib, and rustc. …However, this also needs to build LLVM as well, so we have to sneak into the source dir and build it by hand, THEN go back and tell mrustc to try to build everything. It correctly detects that LLVM is already built, and so just keeps on trucking with everything else.

And we have:

Executed in   44.28 mins   fish           external
   usr time   42.07 mins  490.00 micros   42.07 mins
   sys time    0.83 mins  127.00 micros    0.83 mins

And tokei reports…. 2.3 million lines of Rust in the rustc-1.29.0-src directory, which seems a little concerning given that rustc 1.52 was about 900k lines. It seems that most of those lines are in the vendor and tests directory, so if I remove those, then tokei measures rustc as weighing in at… 965k lines. Seems about right. This calls into question how accurate the previous result for rustc actually is; it presumably still uses most of these vendored libs, but doesn’t actually include their source in the main repository anymore but instead downloads them from crates.io. But if the previous results are underestimating the amount of work done by about 2.5x, well, that’s still “correct” to within a factor of 10 so I’m going to stick with it for now. So that would give us:

  • mrustc: 965k lines / 44 minutes = 363 lines/second

Free Pascal

Let’s try some other languages! How bad could it be? …holy hell, I thought lcc was obtuse. Latest source seems to be Free Pascal version 3.2.0, same as what’s on my Debian system. It claims it does some kind of bootstrapping and testing automatically, we’re just gonna have to keep an eye out for that, but I’m not going to try too hard to bootstrap the thing with itself. In the end though, while the docs and build system are somewhat obtuse to describe it seems pretty straightforward to actually use. Just make rtl for the runtime lib, make compiler for the actual compiler.

make rtl: 3.70 seconds. Hard to tell how much code it’s actually building, the runtime lib is full of platform-specific code. It’s 99264 lines of Pascal total though. …And 118,643 lines of Makefile, which is absolutely hilarious, even if they’re generated by the fpcmake program. It looks like the x86_64+linux backend plus common code is something along the lines of 10,000 lines, and it also takes a significant amount of time just for the make parts of it to run. So I think I’m going to ignore this bit.

make compiler: 7.84 seconds. The compiler dir has 430,960 lines of Pascal in it, but a lot of that is for non-x86 backends. I don’t know much about FPC so I don’t really have a great grip on how many of those lines are actually used, since it doesn’t appear to build all possible cross-compilers. Some rough digging to look at the different backends suggests that the actual compiler + x86_64 backend is more like 350,000 lines of code.

So the final result is:

  • fpc: 44,600 loc/second

Tragically, I don’t seem to be able to find another Pascal compiler to test against, at least not easily. GNU Pascal seems dead and Delphi is Windows only? Either way, I’m done for now.

Go

Go claims it compiles quickly. Does it? Let’s find out.

Go’s building docs are less complete than Rust’s, and it doesn’t seem to have a way to tell it to not do a full three-stage bootstrap. So I just stopped the build when it printed out “Building Go toolchain3”. Took 41 seconds.

tokei output for go/src:

-------------------------------------------------------------------------------
 Language            Files        Lines         Code     Comments       Blanks
-------------------------------------------------------------------------------
 Alex                    2          118          118            0            0
 Assembly              502       140814       127778           70        12966
 Autoconf                9          283          274            0            9
 BASH                   12          768          448          220          100
 Batch                   5          314          186           71           57
 C                      70         6122         4757          600          765
 C Header               10          373          148          168           57
 C++                     1           34           17            9            8
 CSS                     1            1            1            0            0
 Go                   4386      1822009      1439338       238118       144553
 HTML                    1            1            1            0            0
 JSON                   13         1712         1712            0            0
 Makefile                4           17            7            7            3
 Markdown                6          945          945            0            0
 Objective C             2           21           15            3            3
 Perl                    9         1343         1019          169          155
 Python                  1          606          404           70          132
 Shell                   7         1842         1057          653          132
 Plain Text            704        44107        44107            0            0
-------------------------------------------------------------------------------
 Total                5745      2021430      1622332       240158       158940
-------------------------------------------------------------------------------

So the results are indeed pretty quick:

  • 1.82 million lines / 41 seconds = 44k lines/second

Others

  • Swift? A rumor mentions it’s bulky and slow to work on. It seems to be about 50% C++ though, so I’m not going to try to untangle it the way I did for rustc.
  • ocamlc – Ok this is actually impossible to measure; it builds a C program that is its bytecode interpreter, uses that to run a pre-compiled compiler bytecode file, and then uses that to compile the actual compiler to build itself again as native code. I could measure something, but Eris only knows what it would be.
  • Haskell? F#? Zig isn’t self-hosted yet. Nim?
  • Modula-2? Apparently the ADW compiler is nice, though Windows-only?

Conclusions

What conclusions can we make?

This was a terrible idea!

And most of these numbers are absolute frothing nonsense. Compilers are complicated! They use nontrivial libraries! They involve lots of weird code generation and bootstrapping stuff! Each one has its own boutique toolchain to make the bootstrapping cycle easy, and isn’t designed for someone to take them apart and poke around in the middle of the stages! And most of these toolchains are among the least user-friendly imaginable! Even if you manage to do it, most of the time you have tons of confounding factors like the overhead of configure scripts or make itself, the exact libraries and settings you use, and what languages you’re building for! Good heckin’ luck actually untangling that all.

For better benchmarks, you might want to check out this project which takes a different approach: it tries to generate programs in each language and compile them all. It has the eternal problem of “synthetic benchmarks vs. real benchmarks”, but at least you can measure synthetic benchmarks.

This was kinda cool to do though. tcc pleased me the most, because I expected it to be difficult and old-fashioned as heck like lcc was, and it was actually really nice. It is apparently unmaintained these days; someone should pick it up and dust it off!

This data is too terrible to be honored with a graph, but let’s make a table for at least some kind of summary:

Language Compiler compiled with Speed it was compiled at Error bounds
C and C++ gcc gcc 5k loc/sec big
C and C++ gcc clang 4.6k loc/sec big
C++ clang gcc 380 loc/sec biggish
C++ clang clang 400 loc/sec biggish
C chibicc gcc 15k loc/sec chibi-sized
C chibicc gcc-built chibicc 13k loc/sec chibi-sized
C chibicc bootstrapped chibicc 7k loc/sec chibi-sized
C lcc gcc 5k loc/sec no idea
C tcc gcc 10k loc/sec smallish
C tcc gcc-built tcc 125k loc/sec smallish
C tcc bootstrapped tcc 121k loc/sec smallish
C++ mrustc gcc 240 loc/sec smallish
Rust rustc rustc 350 loc/sec big
Rust rustc mrustc 360 loc/sec big
Pascal fpc fpc 45k loc/sec biggish
Go go go 44k loc/sec mediumish

So what have we learned from this? Nothing terribly useful, but:

  • Can you compile C fast? – Yes.
  • Can you compile C++ fast? – No information
  • Can you compile Rust fast? – No information
  • Can you compile Pascal fast? – Yes.
  • Can you compile Go fast? – Yes.

On the flip side:

  • Can you compile C slowly? – Yes but the worst case I found was still 10x faster than Rust/C++’s worst case
  • Can you compile C++ slowly? – Hell yeah
  • Can you compile Rust slowly? – I would be disappointed if you couldn’t
  • Can you compile Pascal slowly? – No information
  • Can you compile Go slowly? – No information