Testing C/C++ for Security and Reliability

Building robust C/C++ applications is a highly challenging endeavor that requires thorough testing. While C/C++ enables us to write high-performance code, the memory-unsafety nature of the language brings a broad spectrum of security risks. Memory corruption issues constitute the vast majority of bugs and security vulnerabilities found in C/C++ projects, and their impact is best demonstrated by the Heartbleed bug on OpenSSL. Regular unit and integration tests are essential to test that our code functions correctly, they are not enough to uncover memory-corruption bugs. On the other hand, fuzz testing has established itself as the best practical method to find these issues in large code bases such as Google Chrome.

In this example, we demonstrate how you can integrate fuzz testing into your CMake projects. This is enabled by the CMake integration provided by CI Fuzz.

C/C++ demo

The demo is a C++ CMake project that demonstrates two types of memory-corruption issues: a buffer overflow and a use-after-free issue. These bugs can only be triggered with specific inputs. This means that a blackbox approach is extremely unlikely to trigger them. Here, we demonstrate how you can create fuzz tess and integrate them into your CMake project and how you can use CI Fuzz to run them and find the bugs.

Develop Fuzz Tests

To add a fuzz test, you should define a function that has the following signature

FUZZ_TEST(const uint8_t *data, size_t size) {
  // Do something interesting with inputs generated by CI Fuzz. 
}

CI Fuzz will then execute with method in a loop and in each iteration provide new inputs that maximize code coverage and trigger interesting behavior in your application.

CI Fuzz provides an add_fuzz_test CMake command that you can use to add your fuzz tests into your CMake project. This command behaves exactly as the usual add_executable command and you can configure dependencies and include paths as you would do with an executable target.

Run Fuzz Tests

1. (Once) Install the command line interface named cifuzz. You can get the latest release from GitHub or by running our install script:

   sh -c "$(curl -fsSL https://raw.githubusercontent.com/CodeIntelligenceTesting/cifuzz/main/install.sh)"

   If you are using Windows you can download the latest release and execute it.

2. Login to our CI App

   cifuzz login

   This will create an API access token that cifuzz uses to communicate with the CI App. When logged in, the cifuzz can provide more details about the findings including severity. You will also be able to run your tests at scale in our SaaS.

3. Run the fuzz test with CI Fuzz. For that you just need to provide the test class containing the fuzz test.

   > cifuzz run heap_buffer_overflow_test
   ▄  Build in progress... Done.                                                                                                                                                                                                                 
   Running heap_buffer_overflow_test                                                                                                                                                                                                             
   Storing generated corpus in .cifuzz-corpus/heap_buffer_overflow_test                                                                                                                                                                          
   Starting from an empty corpus                                                                                                                                                                                                                 
                                                                                                                                                                                                                                              
   Use 'cifuzz finding <finding name>' for details on a finding.
   
   💥 [prickly_hare] heap buffer overflow in heap_buffer_overflow (src/heap_buffer_overflow.cpp:17:5)                                                                                                                                            
                                                                                                                                                                                                                                              
   Note: The reproducing inputs have been copied to the seed corpus at:
   
       heap_buffer_overflow_test_inputs/prickly_hare
   
   They will now be used as a seed input for all runs of the fuzz test,
   including remote runs with artifacts created via 'cifuzz bundle' and
   regression tests. For more information on regression tests, see:
   
       https://github.com/CodeIntelligenceTesting/cifuzz/blob/main/docs/Regression-Testing.md
   
   Execution time: 2s                                                                                                                                                                                                                            
   Average exec/s: 1060290                                                                                                                                                                                                                       
   Findings:       1                                                                                                                                                                                                                             
   Corpus entries: 9 (+9)    

4. You can check the finding details as follows

   cifuzz finding prickly_hare 

5. You can also check the code covered by CI Fuzz

   cifuzz coverage heap_buffer_overflow_test
   Building heap_buffer_overflow_test                                                                                                                                                                                                            
   ▀ Build in progress... Done.                                                                                                                                                                                                                 
   Running heap_buffer_overflow_test on corpus
   
   ✅ Coverage Report:
                                File | Functions Hit/Found |  Lines Hit/Found | Branches Hit/Found
        src/heap_buffer_overflow.cpp |      1 / 1 (100.0%) | 11 / 11 (100.0%) |   17 / 18  (94.4%)
   src/heap_buffer_overflow_test.cpp |      2 / 2 (100.0%) |   4 / 4 (100.0%) |     0 / 0 (100.0%)
                                     |                     |                  |                   
                                     | Functions Hit/Found |  Lines Hit/Found | Branches Hit/Found
                               Total |               3 / 3 |          15 / 15 |            17 / 18

   In addition, you also get a lcov coverage report that you can observe in your browser. Having a look at coverage report helps understand the testing progress and observe the code areas that CI Fuzz has not yet covered. This is valuable so that you can improve and optimize your tests.

Conclusion

In this short tutorial, we have shown how to use CI Fuzz to test your C/C++ CMake project. cifuzz offers many more features, and if you are interested simply cifuzz help.