Testing command line applications in Rust is something that I found was a bit of a grey area. The main documentation for Rust has a section Testing CLI applications by running them which explains how they suggest you test command lines applications:

use std::process::Command;  // Run programs
use assert_cmd::prelude::*; // Add methods on commands
use predicates::prelude::*; // Used for writing assertions

#[test]
fn file_doesnt_exist() -> Result<(), Box<std::error::Error>> {
    let mut cmd = Command::cargo_bin("grrs")?;
    cmd.arg("foobar")
        .arg("test/file/doesnt/exist");
    cmd.assert()
        .failure()
        .stderr(predicate::str::contains("No such file or directory"));

    Ok(())
}

Basically you use a #[test] inside an integration test (in the tests folder) which uses a std::process to spawn the command line tool you’ve developed. This works, but it has two failings in my opinion:

  1. If you want to test the same command line tool with multiple inputs (maybe you want to test that it fails in a specific way?) you’re probably going to add a bunch of tests and include all the boiler plate code to call them.
  2. There isn’t a really clean way to check that complex output from your application appears as expected.

I’m coming from the background of writing compilers, where the state of the art for testing is LLVM’s Lit - a tester that combined with the LLVM tool FileCheck lets you do awesome things like:

; RUN: opt < %s -constprop -S | FileCheck %s

; CHECK-LABEL: shift_undef_64
define void @shift_undef_64(i64* %p) {
  %r1 = lshr i64 -1, 4294967296 ; 2^32
  ; CHECK: store i64 undef
  store i64 %r1, i64* %p

  %r2 = ashr i64 -1, 4294967297 ; 2^32 + 1
  ; CHECK: store i64 undef
  store i64 %r2, i64* %p

  %r3 = shl i64 -1, 4294967298 ; 2^32 + 2
  ; CHECK: store i64 undef
  store i64 %r3, i64* %p

  ret void
}

Basically it is:

  • Running a program of your choosing with the options that you want.
  • The current file is fed as input to the program.
  • Then the output is parsed through FileCheck which also takes the same input.
  • And it uses special CHECK: statements hidden in comments to assert that the output matches some of the stuff that we expect.

This lets us perform some pretty impressive checks that the output matches what we want, and crucially lets us not check some things, which is pretty useful for the things we don’t care about.

I’m currently working on a compiler framework written in Rust for the craic. I’m not yet at the point where I want to link y’all to the bodgey code and it is in no way usable in any way, shape, or form yet. But one thing I wanted was to be able to unit test the compiler framework by using the command line tools similar to how LLVM did it above. This will let me generate a huge number of tests in a really easy to extend format - I know this because I’ve used LLVM’s tooling for years and can attest it is very powerful.

Attempt 1 - Use LLVM’s Lit?⌗

So my first though was just to use LLVM’s lit tool directly. This quickly fell apart as an idea for a number of reasons:

  • The actual lit tooling is a bunch of python scripts, and I don’t want to require anything else beyond Rust.
  • The tool that actually does the heavy lifting in lit is the FileCheck tool, which is an executable shipped with LLVM.
  • The FileCheck executable is only included in an LLVM you build yourself - you can’t just get it with an LLVM install.

So all in all this didn’t seem like a good way to integrate this sort of testing with Rust. So I was about to try and roll my own when I noticed a really awesome crate lit, so I decided to investigate using it.

The lit crate⌗

So first of all, we need to use the crate. And since we only require the crate for testing we do:

[dev-dependencies]
lit = "0.3.4"

Next up we want to add an integration test that uses this. An integration test can depend on the entire main crate you are developing. In my case I’ve got a library (yair-io) and two executables (yair-as and yair-dis) that use it, and I want to drive these from the integration test. The way you do it is like so:

extern crate lit;

#[cfg(test)]
mod tests {
    use std::env;
    use std::env::consts;
    use std::path::PathBuf;

    fn bin_dir() -> PathBuf {
        env::current_exe()
            .ok()
            .map(|mut path| {
                path.pop();
                path.pop();
                path
            })
            .unwrap()
    }

    fn yair_as_exe() -> String {
        bin_dir()
            .join(format!("yair-as{}", env::consts::EXE_SUFFIX))
            .to_str()
            .unwrap()
            .to_string()
    }

    fn yair_dis_exe() -> String {
        bin_dir()
            .join(format!("yair-dis{}", env::consts::EXE_SUFFIX))
            .to_str()
            .unwrap()
            .to_string()
    }

    #[test]
    fn lit() {
        lit::run::tests(|config| {
            config.add_search_path("tests/lit");
            config.add_extension("ya");

            config.constants.insert("yair_as".to_owned(), yair_as_exe());
            config.constants.insert("yair_dis".to_owned(), yair_dis_exe());
        })
        .expect("Lit tests failed");
    }
}

So lets break this down:

  • We use the same #[cfg(test)] that we’d expect from a Rust test.
  • And we have a single #[test] tagged function that’ll run our lit tests.
  • The lit() function calls into the lit crate to run the tests (lit::run::tests).
    • We need to tell it where our test cases live by using add_search_path to point it at the folder containing your text files to run.
    • And also what file extensions we want to consider ok for testing by using add_extension. In my case my human-readable IR files are like foo.ya so the extension is just ya.
    • And then we need to create some variables that map to where the executables built by rust are located. I add two constants for yair_as and yair_dis to map to where the actual executables exist on disk. To get these locations I use a helper function bin_dir to work out where these executables will be placed on the file-system based on where the integration test is on the file-system.

Note you can add any other constants you want here too for use in the lit scripts. So if you wanted to get the OS being used in the scripts you could add config.constants.insert("os".to_owned(), consts::OS.to_owned()); for instance.

Now we’ve got this you can run cargo test and it’ll have a new integration test that’ll run your lit tests! But since you don’t have a single actual test yet, nothing will actually run yet.

So let’s look at an example of how to write a test case for lit that’ll be picked up by the above integration test.

// RUN: @yair_as @file | @yair_dis -

// CHECK: mod "😀" {
mod "😀" {
  // CHECK: fn foo(a : u8, b : <u8, 2>) : <u8, 2> {
  fn foo(a: u8, b: <u8, 2>) : <u8, 2> {
    // CHECK: b0(v2 : u8, v3 : <u8, 2>):
    bar(a: u8, b: <u8, 2>):
      // CHECK: [[name:\w+]] = insert v3, v2, 1
      c = insert b, a, 1
      // CHECK: ret $$name
      ret c
  }
}

So let’s break down what I’ve done piece by piece:

// RUN: @yair_as @file | @yair_dis -

The first line tells the lit crate how to run this test case by using the RUN: statement. To access constants in lit you use the @foo syntax, so in our case to access the yair-as tool that I setup previously to map to the constant yair_as I use @yair_as to run it. There is a special constant that represents the path to the current file - @file. My yair-as assembler tool takes a positional argument for the input file, so what we are doing here is assembling the current file using my tool, and spitting out the output to the default which is stdout. This produces the binary representation of my intermediate representation, which is kinda hard to test! So what I’m doing is roundtripping it back out via the dissassembler yair-dis (mapped to the @yair_dis constant) and then we can check that the output of the disassembler, which is human readable, matches what the input was.

You’ll note that there is a pipe operator between the two tools, passing the stdout from one into the stdin of the other. Also, the stdout from the total run is what is fed into the lit checks to follow. Also - we do not have a separate FileCheck tool like LLVM uses - which is awesome! It just works, which I really appreciate.

// CHECK: mod "😀" {
mod "😀" {

Next up I have a module 😀 being declared, and I make sure that this is output correctly. You’ll notice that the rest of the file follows a similar pattern - some statements in the original IR matched with some CHECK:’s that ensure that the output was correct. You don’t have to check every line, you could check no lines if you wanted to - but this sparse matching of the output to ensure that some parts of it match what you want is so powerful.

// CHECK: [[name:\w+]] = insert v3, v2, 1
c = insert b, a, 1
// CHECK: ret $$name
ret c

You can also use some more powerful features of checks - variables. Basically you can check that something you’ve matched on earlier appears later in the file. So in the above example I am storing the name of the result from the insert instruction into the variable name, and then getting that name in the subsequent check using $$name. This means that if I changed the naming convention at a later date everything would keep working with the test case - nice!

Lastly - the output from cargo test of lit is as follows:

running 1 test
PASS :: tests/lit/var/basic.ya
PASS :: tests/lit/var/array.ya
PASS :: tests/lit/var/struct.ya
PASS :: tests/lit/var/vector.ya
PASS :: tests/lit/var/export.ya
PASS :: tests/lit/fn/one_arg.ya
PASS :: tests/lit/fn/bad_export.ya
PASS :: tests/lit/fn/one_arg_body.ya
PASS :: tests/lit/fn/export.ya
PASS :: tests/lit/fn/no_args.ya
PASS :: tests/lit/mod/multiple_modules.ya
PASS :: tests/lit/mod/bad_fn_or_var.ya
PASS :: tests/lit/mod/simple.ya
PASS :: tests/lit/mod/missing_closing_brace.ya
PASS :: tests/lit/mod/bad_quoted.ya
PASS :: tests/lit/mod/unicode.ya
PASS :: tests/lit/mod/empty.ya
PASS :: tests/lit/mod/missing_opening_brace.ya
PASS :: tests/lit/mod/missing_module_name.ya
PASS :: tests/lit/mod/bad_simple.ya
PASS :: tests/lit/insts/insert.ya
test tests::lit ... ok

You can see that it appears like a single running test (because like I said previously we are adding a single integration test that powers the lit runs), but the output from that test is a bunch of runs telling you what passes and what fails. If I manually make a test case fail, the output is like so:

FAIL :: tests/lit/insts/insert.ya
tests/lit/insts/insert.ya:15: could not find match: 'something not there'
next line: '}'

Telling me the line with the CHECK that it could not find.

Conclusion⌗

I really love this crate - it is super useful for testing these kind of applications. Full kudos to Dylan McKay for writing such an awesome crate, and I hope I’ve convinced a few more people to try it out.