With the upcoming version of Clang 6, support for collecting code coverage information on Windows is now mature enough to be used in production. As a proof, we can tell you that we have been using Clang to collect code coverage information on Windows for Firefox.

In this post, I will show you a simple example to go from a C++ source file to a coverage report (in a readable format or in a JSON format which can be parsed to generate custom nice reports or upload results to Coveralls/Codecov).

Build

Let’s say we have a simple file, main.cpp:

#include <iostream>

int main() {
  int reply = 42;

  if (reply == 42) {
    std::cout << "42" << std::endl;
  } else {
    std::cout << "impossible" << std::endl;
  }

  return 0;
}

In order to make Clang generate an instrumented binary, pass the ‘--coverage’ option to clang:

clang-cl --coverage main.cpp

In the directory where main.cpp is, both the executable file of your program and a file with extension ‘gcno’ will be present. The gcno file contains information about the structure of the source file (functions, branches, basic blocks, and so on).

09/01/2018  16:21    <DIR>          .
09/01/2018  16:21    <DIR>          ..
09/01/2018  16:20               173 main.cpp
09/01/2018  16:21           309.248 main.exe
09/01/2018  16:21            88.372 main.gcno

N.B.: You might have to manually link to the Clang clang_rt.profile-YOUR_ARCH.lib (https://bugs.llvm.org/show_bug.cgi?id=40877).

Run

Now, the instrumented executable can be executed. A new file with the extension ‘gcda’ will be generated. It contains the coverage counters associated with the ‘gcno’ file (how many times a line was executed, how many times a branch was taken, and so on).

09/01/2018  16:22    <DIR>          .
09/01/2018  16:22    <DIR>          ..
09/01/2018  16:20               173 main.cpp
09/01/2018  16:21           309.248 main.exe
09/01/2018  16:22            21.788 main.gcda
09/01/2018  16:21            88.372 main.gcno

At this point, we need a tool to parse the gcno/gcda file that was generated by Clang. There are two options, llvm-cov and grcov. llvm-cov is part of LLVM and can generate bare bones reports, grcov is a separate tool and can generate LCOV, Coveralls and Codecov reports.

We had to develop grcov (in Rust!) to have a tool that could scale to the size of Firefox.

Parse with llvm-cov

You can simply run:

llvm-cov gcov main.gcno

Several files with extension gcov will be generated (one for each source file of your project, including system header files). For example, here’s main.cpp.gcov:

        -:    0:Source:main.cpp
        -:    0:Graph:main.gcno
        -:    0:Data:main.gcda
        -:    0:Runs:1
        -:    0:Programs:1
        -:    1:#include <iostream>
        -:    2:
        -:    3:int main() {
        1:    4:  int reply = 42;
        -:    5:
        1:    6:  if (reply == 42) {
        1:    7:    std::cout << "42" << std::endl;
        1:    8:  } else {
    #####:    9:    std::cout << "impossible" << std::endl;
        -:   10:  }
        -:   11:
        1:   12:  return 0;
        -:   13:}

Parse with grcov

grcov, can be downloaded from GitHub (in the Releases page).

Simply execute grcov with the ‘--llvm’ option, pointing it to the directory containing your gcda/gcno files. The “-t” option allows you to specify the output format:

  • “lcov” for the LCOV format, which you can then translate to a HTML report using genhtml;
  • “coveralls” for a JSON format compatible with Coveralls/Codecov;
  • “coveralls+” for an extension of the former, with addition of function information.

Example:

grcov --llvm PATH_TO_YOUR_DIRECTORY -t coveralls+ --token unused --commit-sha unused > report.json

grcov has other options too, simply run it with no parameters to list them. The most important one probably being “--branch”, which adds the branch information to the output.

In production

We are using Clang to collect code coverage on Windows for Firefox. The grcov tool is used to parse the gcno/gcda files generated by LLVM and to emit a report that can be converted to HTML or uploaded to services such as Coveralls or Codecov.

We have fixed several bugs in LLVM, Clang and compiler-rt (rL314201, rL315677, rL316048, rL317705, rL317709, rL321702, rL321703) that were preventing GCOV coverage to be usable in Clang 5 (and previous versions), but with Clang 6 the situation should be pretty good (if it works with the millions of lines of code of Firefox, with its highly parallel architecture and with several diverse kinds of test suites, you can rest assured that it will work for most projects!).