Chapter:C testing

From Juneday education
Jump to: navigation, search

Contents

Meta information about this chapter

Expand using link to the right to see the full content.

Introduction

Purpose

Goal

The student shall:


Instructions to the teacher

Common problems

Chapter videos

All videos in this chapter:

Compiling with sufficient flags

Look at the following code (download bad.c) for a while:

#include <stdio.h>

int
main(void)
{
  int arr[] = { 1, 1, 2, 5, 8, 13};
  print_array();
}


void
print_array(int *arr, int size)
{
  int i;
  printf ("[");
  for (i=0;i<size;i++);
    {
      printf("%d ", arr[i]);
    }
  printf ("]\n");
}

Can you see any problematic with the code above? Let's see what the compiler says when compiling this file (bad.c).

$ gcc  bad.c
bad.c: In function ‘main’:
bad.c:7:3: warning: implicit declaration of function ‘print_array’ [-Wimplicit-function-declaration]
   print_array();
   ^~~~~~~~~~~
bad.c: At top level:
bad.c:12:1: warning: conflicting types for ‘print_array’
 print_array(int *arr, int size)
 ^~~~~~~~~~~
bad.c:7:3: note: previous implicit declaration of ‘print_array’ was here
   print_array();

Ok, the compiler warns a bit, but who cares? Let's execute the program.

$ ./a.out
Segmentation fault (core dumped)

Let's look at what the compiler says:

bad.c:7:3: warning: implicit declaration of function ‘print_array’ [-Wimplicit-function-declaration]
   print_array();
   ^~~~~~~~~~~

Before declaring the function print_array we use it. Doing this makes C assume that the function returns an integer and has no parameters.

bad.c:12:1: warning: conflicting types for ‘print_array’
 print_array(int *arr, int size)
 ^~~~~~~~~~~

So when later on declaring the function C complains about the function having conflicting types, since it made an implicit declaration above.

First of all, warnings like this should be treated with proper respect. Fact is, we suggest you start treating them as errors. Let's make gcc treat all warnings as errors by using the following option:

gcc -Werror  bad.c
bad.c: In function ‘main’:
bad.c:7:3: error: implicit declaration of function ‘print_array’ [-Werror=implicit-function-declaration]
   print_array();
   ^~~~~~~~~~~
bad.c: At top level:
bad.c:12:1: error: conflicting types for ‘print_array’ [-Werror]
 print_array(int *arr, int size)
 ^~~~~~~~~~~
bad.c:7:3: note: previous implicit declaration of ‘print_array’ was here
   print_array();
   ^~~~~~~~~~~
cc1: all warnings being treated as errors

Now gcc is not creating a program for us. This is way better. We rather have the compiler than the customer find bugs for us. Let's add a couple of more flags:

$ gcc  -pedantic -Wconversion -Wall -Werror  -Wextra -Wstrict-prototypes bad.c
bad.c: In function ‘main’:
bad.c:7:3: error: implicit declaration of function ‘print_array’ [-Werror=implicit-function-declaration]
   print_array();
   ^~~~~~~~~~~
bad.c:6:7: error: unused variable ‘arr’ [-Werror=unused-variable]
   int arr[] = { 1, 1, 2, 5, 8, 13};
       ^~~
bad.c: At top level:
bad.c:12:1: error: conflicting types for ‘print_array’ [-Werror]
 print_array(int *arr, int size)
 ^~~~~~~~~~~
bad.c:7:3: note: previous implicit declaration of ‘print_array’ was here
   print_array();
   ^~~~~~~~~~~
bad.c: In function ‘print_array’:
bad.c:16:3: error: this ‘for’ clause does not guard... [-Werror=misleading-indentation]
   for (i=0;i<size;i++);
   ^~~
bad.c:17:5: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the ‘for’
     {
     ^
cc1: all warnings being treated as errors

Uh oh, here are some nasty warnings:

bad.c:16:3: error: this ‘for’ clause does not guard... [-Werror=misleading-indentation]
   for (i=0;i<size;i++);
   ^~~
bad.c:17:5: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the ‘for’
     {
     ^

We have a ; after the for statement. This means that everything between for (i=0;i<size;i++) and ; will be executed in each loop. That is, nothing. After the loop is done we're going to execute (once!):

  {
      printf("%d ", arr[i]);
    }

BTW, the function print_array is for internal use only so let's make it static.


Let's alter the code a bit:

#include <stdio.h>

static void print_array(int *arr, int size); /* added prototype. static since internal use */

int
main(void)
{
  int arr[] = { 1, 1, 2, 5, 8, 13};
  print_array(arr, 6); /* pass the array and size */
}


static void
print_array(int *arr, int size)
{
  int i;
  printf ("[");
  for (i=0;i<size;i++) /* removed the ";" */
    {
      printf("%d ", arr[i]);
    }
  printf ("]\n");
}

The lesson to learn here is not what we did to fix the code but rather to see that the compiler is our friend and with proper flags we can get more help from the compiler. Download the improved file: better.c.

Black box test

We will not spend any time on this. Well, you're reading this so some time will be spent. With black box test we mean that we're testing our software as a black box - we don't know anything about what units/modules exist and how they work. Typically we write test cases, sometimes based on a so called requirement specification.

So if we look at our calculator (hand in assignment 01-03) we could do some test of this. Here's an example:

$ ./calc 1 + 2 
3

We're testing the calculator as one black box. We don't care how the program reads and interprets data from the user or how to program does the caluculations. We focus in making sure that this balck box functions. Before we end this scetion we would like to talk a bit on what can be interesting to test for us:

  1. normal use
  2. bad input (non-number, no operator, missing parameters)
  3. big (positive and negative) numbers
  4. big (positive and negative) result

normal use

$ ./calc 1 + 2
3
$ ./calc 1 - 2
-1
$ ./calc 1 x 2
2
$ ./calc 1 / 2
0
$ ./calc 1 % 2
1

We could use a for loop in bash to do the above a bit shorter:

for op in + - x / % ; do ./calc 1 $op 2; done

and if we want to make sure the program calulates correctly:

for op in + - x / % ; do  RES=$(./calc 1 $op 2); BASH_OP=$(echo $op| sed 's,x,\*,g'); BASH_RES=$(( 1 $BASH_OP 2 )) ; if [ $BASH_RES -ne $RES ] ; then echo "1 $op 2 failed"; break; fi; done

and if we want to test a lot (1000) of different operands:

CONT=true; CNT=1000; while [ $CNT -ge 0 ] && [ $CONT="true" ] ; do OP1=$RANDOM; OP2=$RANDOM; echo "Testing $OP1 $OP2 ($CNT tests left)";  for op in + - x / % ; do RES=$(./calc $OP1 $op $OP2); BASH_OP=$(echo $op| sed 's,x,\*,g'); BASH_RES=$(( $OP1 $BASH_OP $OP2 )) ; if [ $BASH_RES -ne $RES ] ; then echo "$OP1 $op $OP2 failed"; CNT=false; break ; fi; done; CNT=$(( $CNT - 1)); done

Of course we will not examine the above or even ask you to understand it - even though it is kind of easy. But bash is dealt with in another course. We would like to show show you that it is possible to do tons of tests and leave that work to the computer.


bad input (non-number, no operator, missing parameters)

./calc 1 + a

this should fail since a is not a number.

./calc 1 a 2

this should fail since a is not an operator.

There will be a lot of tests in the end. We suggest you automate these kind of tests. More on that in the bash course.

big (positive and negative) numbers

./calc 1111111111111111111111111111111111111111 + 22222222222222222222222222222222222222222222222

Should fail since the values are bigger than the maximum value of an integer. The same goes for negavtive numbers.

big (positive and negative) result

./calc 1000000000 x 1000000000

Should fail since the result is bigger than the maximum value of an integer. The same goes for negavtive numbers. Note that the operands (1000000000) are valid integers.

Unit test

Let's think about building ships for a short while. It would be rather stupid to test the ship (typically see if it floats) by putting it in water. Imagine if we had tested the engines, the bolts, the weld work etc before putting the ship in water. It makes sense doesn't it? If you don't agree that this is a good idea we would like to hear from, so get in contact with us. We will from now on assume you see the benefits from testing a ship and will now go from the ship to software. If it was possible to test smaller units (components/modules) of a program wouldn't it make sense to do this? Of course it would. So let's do that.

Testing a unit of a program is called unit test (surprise!).

We have, in a hand in assignment, developed a calculator. We could, and should, of course test the entire calculator like a block box (see below). But if, or rather when, something goes wrong, how do we know in which unit the bug is located? Having unit tests we can in an easier find the units that work and the ones who don't.


Now, let's assume you fixed a bug in your calculator and all of a sudden the calculator does not work anymore. Where's the error located now? Good thing we have the unit tests. Using the existing unit tests we can, again find the bugs or limit the amount of code we need to look for bugs in.

- Should I test my program using unit tests are black box tests?

- Both!

But let's start with unit tests. Let's assume we have the following piece of code which should be used in a big program. The code can be found here: simple-math.c simple-math.h simple-math-test.c

int
max(int a, int b)
{
  if (a>b)
    {
      return a;
    }
  return b;
}

Can something really go wrong here? Probably not. What if we pass values bigger than the maximum value of an int (INT_MAX) or smaller then the minimum value (INT_MIN)? Well, we'd say that it is misuse of the function and that by using proper flags to your compiler you should not be able to do this.

Let's continue with the function, mean, below. The function:

  • calculates the mean value of
  • integers passed in an array together with
  • the size (int) of the array
  • result is stored in the memory pointed to by int *result)
  • returns an integer with the status (0 on success).
int
mean(int *array, int size, int *result)
{
  int i ;
  int sum;

  // initialise sum to 0                                                                                                                        
  sum = 0 ;
  for (i=0; i<size; i++)
    {
      // add the value to sum                                                                                                                   
      sum = sum + array[i];
    }

  *result = sum/size;
  return 0;
}

Before you go on about some obvious flaws we would like you to write tests for these flaws instead of rightfully complaining our code of stinking. What do we need to test? Here are some thoughts:

  1. someone passes NULL as array - this really should return not 0 (e g 1)
  2. someone passes a negative size - this should return not 0 (e g 2)
  3. someone passes NULL as result - this really should return not 0 (e g 1)
  4. the sum is bigger than the maximum (or smaller the the minimum) value of an int - this should return not 0 (e g 3)
  5. the result is rounded incorrectly - this shall be fixed. We shall round "up"

Uh oh, the list is big. Before we continue developing the complete program it is, really - it is!, a good idea to write tests for these cases. Let's do them one by one:

someone passes NULL as array

void
test_null_array(void)
{
  int res;

  assert( (mean(NULL, 1, &res)==1));
}

In the above statement we assert that the return value shall be 2. If this is not true the program "explodes to pieces" - the manual says assert - abort the program if assertion is false.

If we execute this simple test we get a Segmentation fault. So we need to fix the actual code:

int
mean(int *array, int size, int *result)
{
  int i ;
  int sum;

  if (array==NULL)
    {
      return 1;
    }
   .......

Now the function does not end in a seg fault when passed a NULL pointer. Let's move on.

someone passes a negative size

void
test_negative_size(void)
{
  int arr[] = {2, 2, 1};
  int res;

  assert( (mean(arr, -1, &res)==2));
}

In the above statement we assert that the return value shall be 2. If we execute this simple test the program aborts (due to failed assert). So we need to fix the actual code:

int
mean(int *array, int size, int *result)
{
  int i ;
  int sum;

  if (array==NULL)
    {
      return 1;
    }

  if (size<0)
    {
      return 2;
    }

  ....

Now the function returns 2 if passwed a negative size.

Note: is it a good idea at all to be able to pass a negative size? How about using an unsigned type instead?

someone passes NULL as result

void
test_null_result(void)
{
  int arr[] = {2, 2, 1};

  assert( (mean(arr, 1, NULL)==1));
}

In the above statement we assert that the return value shall be 1. If we execute this simple test we get a Segmentation fault. So we need to fix the actual code:

int
mean(int *array, int size, int *result)
{
  int i ;
  int sum;

  if ( (array==NULL) || (result==NULL) )
    {
      return 1;
    }

  if (size<0)
    {
      return 2;
    }

  ......

Now the function does not end in a seg fault when passed a NULL pointer. Let's move on again.

the sum is bigger than the maximum (or smaller the the minimum) value of an int

void
test_too_big_sum(void)
{
  int res;
  int arr[] = {INT_MAX, INT_MAX};

  assert( (mean(arr, 2, &res)==3));
}

In the above statement we assert that the return value shall be 2. If we execute this simple test we get Assertion `(mean(arr, 2, NULL)==3)' failed.. So, guess what, we need to fix the actual code:

int
mean(int *array, int size, int *result)
{
  int i ;
  // Store the sum in a long long instead                                                                                                       
  long long sum;

  if ( (array==NULL) || (result==NULL) )
    {
      return 1;
    }

  if (size<0)
    {
      return 2;
    }

  // initialise sum to 0                                                                                                                        
  sum = 0 ;
  for (i=0; i<size; i++)
    {
      // add the value to sum                                                                                                                   
      sum = sum + array[i];

      // Check if the sum fits in an int                                                                                                        
      if ((sum>INT_MAX) || (sum<INT_MIN) )
        {
          return 3;
        }
    }

  DEBUG("%lld / %d => %lldd\n", sum, size, sum/size);
  *result = (int)sum/size;
  return 0;
}

Now the function detects if the max values are reached. Adding a test to check if the min value is handled is easy. If you're in doubt, look at the code in our git repository (see link above). Let's move on again.

the result is rounded incorrectly

void
test_mean()
{
  int arr[] = {2, 2, 1};
  int size = 3;

  int ret;
  int res;

  ret = mean(arr, size, &res);
  assert(ret==0);
  assert(res==2);
}

In the above statement we assert that the result shall be 2. If we execute this simple test we get Assertion `res==2' failed.. So, guess what, we need to fix the actual code:

  *result = (int)round(((double)sum/(double)size));


In short we typecast the ints to doubles and do a floating point division which we "round". This values is typecasted to an int. It's kind of a cheat to do it this way, but we're not here to learn how to write math code.

To compile this code you need to compile and tell the compiler, actually the linker, to look for undefined references (the round function) in an optional so called library call m as in math. To compile and link with m you simply type:

gcc simple-math.c simple-math-test.c -lm



BTW, writing tests before the actual code is part of a software development process called Test-driven_development

.... and as some of you may ask, how do we test the unit tests?

Code coverage

So we're saying it is good to write test code. What lines in our unit have we tested? Sometimes it is hard to know. But if we use a code coverage tool it is easy. If we compile our code with some extra options, run the program and let a small program analyse some files that have been created when we executed our program we will get a nice view of the code tested using our tests.

Let's say we want to see wht lines of code in simple-math.c have been tested. Then we need to compile like this:

$ gcc -pedantic -Wconversion -Wall -Werror  -Wextra -Wstrict-prototypes -g  --coverage -fprofile-arcs -ftest-coverage simple-math.c simple-math-test.c -o simple-math-test -lm

It is the flags -fprofile-arcs -ftest-coverage that are important here. After the compilation we run the program (simple-math-test):

$ ./simple-math-test

Now it's time to let the program gcov (comes with [GCC])

$ gcov -b simple-math.c

The command above produces a text file called simple-math.c.gcov which contains information about what lines have been executed (covered). The file is long so we will only show a small part of it here:

function mean called 9 returned 100% blocks executed 100%
        9:   28:mean(int *array, int size, int *result)
        -:   29:{
        -:   30:  int i ;
        -:   31:  // Store the sum in a long long instead
        -:   32:  long long sum;
        -:   33:  
        9:   34:  if ( (array==NULL) || (result==NULL) )
branch  0 taken 89% (fallthrough)
branch  1 taken 11%
branch  2 taken 13% (fallthrough)
branch  3 taken 88%
        -:   35:    {
        2:   36:      return 1;
        -:   37:    }
        -:   38:
        7:   39:  if (size<0)
branch  0 taken 14% (fallthrough)
branch  1 taken 86%
        -:   40:    {
        1:   41:      return 2;
        -:   42:    }
        -:   43:  
        -:   44:  // initialise sum to 0
        6:   45:  sum = 0 ;
       26:   46:  for (i=0; i<size; i++)
branch  0 taken 85%
branch  1 taken 15% (fallthrough)
        -:   47:    {
        -:   48:      // add the value to sum
       22:   49:      sum = sum + array[i];
        -:   50:
        -:   51:      // Check if the sum fits in an int
       22:   52:      if ((sum>INT_MAX) || (sum<INT_MIN) )
branch  0 taken 91% (fallthrough)
branch  1 taken 9%

Ok, if we read this carefully we will get a picture of what lines have been covered. But it is way easier if we use a tool called lcov (works on MacOS and GNU/Linux. On Windows you need to build lcov yourself). gcovr is a similar tool that can be used on all platforms. We will focus on lcov here, it is not important for the course to learn HOW to do code coverage bur rather that it is possible to do code coverage:

lcov --directory . --capture --output-file simple-math.info

And finally use genhtml to generate the html page.

genhtml -o coverage simple-math.info

Once this is executed we will a nice html report which is easy to read. We provide two screenshots to give you a feeling of what it looks like.


Overview of the tested (covered) files

parts of the report of the file simple-math.c

100% may not say anything at all

It is most project managers' dream to have 100% code coverage. We will show you a short example on why 100% code coverage does not autoatically means that the code is perfect.


Here is some code we would like to test print-fun.c:

#include <stdio.h>


int
errprint(char *str)
{
  return fprintf(stderr, "%s\n", str);
}

and its coresponding header file print-fun.h:

int
errprint(char *str);

We test the code above using this simple test print-fun-test.c:

#include <assert.h>
#include <stdio.h>
#include "print-fun.h"

int
main(void)
{
  printf ("Start test\n");
  assert (errprint("test")==5);
  printf ("Test finished\n");
}

So, let's test the code and see what code coverage percent we'll get on our function.

$ gcc -pedantic -Wconversion -Wall -Werror  -Wextra -Wstrict-prototypes -g  --coverage -fprofile-arcs -ftest-coverage print-fun.c print-fun-test.c -o print-fun-test
$ ./print-fun-test 
Start test
test
Test finished
$ gcov -b print-fun.c
File 'print-fun.c'
Lines executed:100.00% of 2
No branches
Calls executed:100.00% of 1
Creating 'print-fun.c.gcov'
$lcov --directory . --capture --output-file print-fun.info && genhtml -o coverage print-fun.info
Capturing coverage data from .
Found gcov version: 6.2.1
Scanning . for .gcda files ...
Found 4 data files in .
Processing print-fun.gcda
Processing simple-math-test.gcda
Processing simple-math.gcda
Processing print-fun-test.gcda
Finished .info-file creation
Reading data file print-fun.info
Found 4 entries.
Found common filename prefix "/home/hesa/opt/prog-unbook/programming-with-c/testing"
Writing .css and .png files.
Generating output.
Processing file wiki-material/print-fun-test.c
Processing file wiki-material/simple-math-test.c
Processing file wiki-material/print-fun.c
Processing file wiki-material/simple-math.c
Writing directory view page.
Overall coverage rate:
  lines......: 88.1% (59 of 67 lines)
  functions..: 83.3% (10 of 12 functions)
$ firefox coverage/index.html

It is easy to see that we've tested 100% of the function. Should satisfy anyone shouldn't it? Well, the code smells and the test smells accordingly. What if we use this code in an important software and, by chance, pass a NULL pointer to the function? It will crash on some platforms and not on some. So even if we have a100% code coverage our code is not good. ... hope this proves that 100% does not automatically mean all is well.

Checking for memory leaks and other errors

If we write academic programs, that is programs with roughly 20 lines of code doing nothing useful, it is easy to keep track of our memory. But if we write real programs we need to make sure, at least be "as sure as we can", that we don't leak memeory. If a program has one memory leak of say 20 bytes it is not a big deal but if a program looses track of say 20 bytes of memory (loosing track of memory is a memory leak) in a loop which is executed tons of times we have a big problem. Furthermore it is really awkward to have a memory leak that someone detects without your knowledge. So it is, for several reasons, good to keep track of memory. So how do we check if we've leaked memory?

Let's wait with the answer on that one. Instead we're going to ask one more question. Look at the following code outside-array.c which uses print_array.c and print_array.h:

void
print_int_array(int *array, int size)
{
  int i ;
  for (i=0;i<size;i++)
    {
      printf("array[%d]: %d\n", i, array[i]);
    }
}


int
main(void)
{
  int crap_ints[] = { 9, 8, 7, 6, 5, 4, 3, 2, 1 };

  print_int_array(crap_ints, 10);

  return 0;
}

In the code above we tell the function to print 10 integers while passing an array with 9 integers. The code will compile fine and most likely will execute without problems (apart from printing some random value) the last round in the loop. Can we check if we read outside array?

Yes, we can. There are several tools out there to check memory usage in our programs. We will show you the program Valgrind. Valgrind is available on GNU/Linux and Mac OS X 10.10 (with limited support for 10.11 and 10.12). We provide a Virtualbox image which you can use to try out valgrind (and Ubuntu).

Testing your error handling code

We've been talking a lot about making your code robust. One important aspect of this is to check return values when performing a critical operation, such as converting a string to an integer or when allocating memory. It is almost always possible to check this and in the following sections we provide a couple of solution to this problem.

Before we start by discussing initiated by a student. The student asked how he could generate an error in fclose. He wanted to test if his error handling code for a fclose failure worked. The student had tried several different ways to generate an flcose failure, e g removing the file before closing it. If no data to flush to the stream fclose really doesn't do anything so this did not work. One could have written tons of data to the file ..... but are we testing the right thing? We need to discuss the question a bit further. What exactly is it you want to test?

  • that fclose returns non 0 (actually EBADF) if something goes wrong
  • that our code can handle a failure

For us, the authors of this book, we are quite sure about fclose. We don't feel we need to test this function. Fact is, we are really sure that this works fine. And of course it is our code handling the failure we need to test. So how do we "fake" a fclose failure. Well, it is easy, but let's rephrase the question a bit. How can we fake that we got "not 0" from fclose? Let's ask the question this way, hoping to make it even more obvious, how do you set a variable to "not 0" (e g EBADF)? It's kind of obvius isn't it?

We could simply do something like this:

int ret = fclose (fd);
ret = EBADF;  /* THIS IS ADDED TO FAKE A FAILURE */
if (ret == EBADF )
  {
   /* error handling code */ 
  }

If we add this code we can test our error handling code. Is it this easy? Yes, it is. But let's discuss this strategy and some other below.

Note: this strategy is a bit dangerous ..... what will happen if we forget to remove EBADF?

Using macros

We could define a macro like this:

#define fclose(f) EBADF

This macro will replace all calls to fclose with EBADF so code like this:

int ret = fclose (fd);

will be looking like this (after the pre proccessor is done)

int ret = 9;

This is all good isn't it? Well, no. What if we forget to remove the macro. Let's make the macro optional, that is we have to "force" the compiler to use the macro under one condition (that we define ENABLE_FCLOSE_FAILURE).

#ifdef ENABLE_FCLOSE_FAILURE
#define fclose(f) EBADF
#endif  /* FCLOSE_FAILURE */


If we want to compile and stop after pre procssing, we type like this (assuming our file is called file.c:

gcc -E file.c

Now the important part of our code looks like this:

  int ret = fclose (fd);

If we instead compile like this (note that we define ENABLE_FCLOSE_FAILURE):

gcc -DENABLE_FCLOSE_FAILURE -E file.c

... the important part of our code looks like this:

  int ret = 9;

Neat trick, isn't it?

Using a function wrapper

Simple program

Here's a short program that reads and prints a textfile.

#include <stdio.h>
#include <errno.h>

/*
 * Example program that reads and prints a text file
 *
 *
 */

#define READ_BUFF_SIZE 100

int
main(int argc, char **argv) 
{
  char buff[READ_BUFF_SIZE];
  char *default_file_name = "fclose-wrapper.c";
  char *file_name = default_file_name;
  FILE *fd;
  int ret;
  
  if (argc>1)
    {
      file_name = argv[1];
    }
  
  fd=fopen(file_name, "r");
  if (fd==NULL)
    {
      fprintf(stderr, "Failed opening file: '%s'\n", file_name);
      return 1;
    }
  
  while (fgets(buff, READ_BUFF_SIZE, fd)!=NULL)
    {
      printf ("%s",buff);
    }
  
  ret = fclose(fd);
  if (ret!=0)
    {
      fprintf(stderr, "Failed closing file: '%s'\n", file_name);
      return 1;
    }

  return 0;
}

Introducing the wrapper code

Assume we want to check if our fclose failure handling code works. Let's write a wrapper around fclose.

int closefile(FILE *fd)
{
  int ret;
  ret = fclose(fd);
#ifdef ENABLE_FCLOSE_FAILURE
  ret = EBADF;
#endif
  return ret;
}

and replace the code calling fclose, with this:

  ret = closefile(fd);
  if (ret!=0)
    {
      fprintf(stderr, "Failed closing file: '%s'\n", file_name);
      return 1;
    }

Download the code: fclose-wrapper.c

Enabling the faked failure

To enable the test we simply compile like this (note that we're defining ENABLE_FCLOSE_FAILURE):

gcc  -pedantic -Wconversion -Wall -Werror  -Wextra -Wstrict-prototypes -DENABLE_FCLOSE_FAILURE fclose-wrapper.c -o fclose-wrapper

and to execute the program (with the faked failure enabled) we type:

./fclose-wrapper fclose-wrapper.c

This prints out the content of the c file itself (fclose-wrapper.c) and then prints

Failed closing file: 'fclose-wrapper.c'

Complete source code: fclose-wrapper.c.

Replacing the function with your own

If we use the same problem as in the previous example we can, instead of writing a wrapper, write our own fclose.

Introducing the fclose replacement

int fclose(FILE *fd)
{
  return EBADF;
}

This code always returns a failure code. This might be a "bit" problematic in production code so we don't want to use the code - if not explicitly enable it. So let's use the same trick as earlier:


#ifdef ENABLE_FCLOSE_FAILURE
int fclose(FILE *fd)
{
  return EBADF;
}
#endif


Enabling the faked failure

To enable the test we simply compile like this (note that we're defining ENABLE_FCLOSE_FAILURE):

pgcc -DENABLE_FCLOSE_FAILURE fclose-replacement.c -o fclose-replacement

Note: we can't use the -Wextra flag to gcc in this example. Gcc will complain about an unused parameter in fclose.


and to execute the program (with the faked failure enabled) we type:

./fclose-replacement fclose-replacement.c

This prints out the content of the c file itself (fclose-replacement.c) and then prints

Failed closing file: 'fclose-replacement.c'

Complete source code: fclose-replacement.c.

Simply replacing the return value

We could simply do something like this:

int ret = fclose (fd);
ret = EBADF;  /* THIS IS ADDED TO FAKE A FAILURE */
if (ret == EBADF )
  {
   /* error handling code */ 
  }

The obvious drawback of this strategy is the risk of forgetting the change it back so we don't advise you to go for this strategy. You could of course use the #ifdef trick as we've used before but then the real code will be cluttered with test stuff which will make it hard to read.

Example testing alloc code

So let's look at some code to allocate, reallocate and free memory. Here's a bit of code. The code really doesn't do anything useful - but it will do as an example:

Example program

int
main(void)
{
  /* Create memory for 4 characters plus ending \0 */
  char * name = calloc(5, sizeof(char));

  /* Check if it all went well */
  if (name == NULL)
    {
      fprintf(stderr, "Shit....we failed (c)allocating memory\n");
      /* Uh oh, we did not get any memory */
      exit (1);
    }

  /* Use the memory - in this case copy a name to it and print it */
  strcpy (name, "Adam");
  fprintf(stdout, "name: %s\n", name);

  /* increase the memory */
  char * tmp = realloc(name, sizeof(char)*17);

  /* Check if it all went well */
  if (tmp == NULL)
    {
      fprintf(stderr, "Shit....we failed reallocating memory\n");
      /* Uh oh, we did not get any memory. Free the old mem and exit */
      free(name);
      exit (1);
    }

  /* Ok, so we did get some memory. Make name point to the new memory                               
     location (could be the same, could be a new address */
  name = tmp;

  /* Use the memory - in this case copy a name to it and print it*/
  strcpy (name, "Kenny McCormick");
  fprintf(stdout, "name: %s\n", name);

  /* Free the memory */
  free(name);

  return 0;
}


Error handling code to check

How do we check the following?

  /* Create memory for 4 characters plus ending \0 */
  char * name = calloc(5, sizeof(char));

  /* Check if it all went well */
  if (name == NULL)
    {
      fprintf(stderr, "Shit....we failed (c)allocating memory\n");
      /* Uh oh, we did not get any memory */
      exit (1);
    }

and this

  /* increase the memory */
  char * tmp = realloc(name, sizeof(char)*17);

  /* Check if it all went well */
  if (tmp == NULL)
    {
      fprintf(stderr, "Shit....we failed reallocating memory\n");
      /* Uh oh, we did not get any memory. Free the old mem and exit */
      free(name);
      exit (1);
    }

Adding macros to fake alloc failures

If we add the following (typically after the include statements):

/*                                                                                                  
 * Macro name: REALLOC_FAILURE                                                                      
 *                                                                                                  
 * Used to fake a realloc failure. Use it by compiling like this:                                   
 *   gcc -DREALLOC_FAILURE simple-realloc-chk.c                                                     
 */
#ifdef REALLOC_FAILURE
#define realloc(mem,newsize) NULL
#endif

/*                                                                                                  
 * Macro name: CALLOC_FAILURE                                                                       
 *                                                                                                  
 * Used to fake a calloc failure. Use it by compiling like this:                                    
 *   gcc -DCALLOC_FAILURE simple-realloc-chk.c                                                      
 */
#ifdef CALLOC_FAILURE
#define calloc(nr,typesize) NULL
#endif


You can find the source code, including comments, with the macros here: simple-realloc-chk.c

Using the macros when testing

We can now compile and run the program like this:

$ gcc  -pedantic -Wconversion -Wall -Werror  -Wextra -Wstrict-prototypes simple-realloc-chk.c -o simple-realloc-chk
$ ./simple-realloc-chk 
name: Adam
name: Kenny McCormick

If we want to test our code with a calloc failure we can do like this:

$ gcc -DCALLOC_FAILURE -pedantic -Wconversion -Wall -Werror  -Wextra -Wstrict-prototypes simple-realloc-chk.c -o simple-realloc-chk
$ ./simple-realloc-chk 
Shit....we failed (c)allocating memory


If we want to test our code with a realloc failure we can do like this:

$ gcc -DREALLOC_FAILURE -pedantic -Wconversion -Wall -Werror  -Wextra -Wstrict-prototypes simple-realloc-chk.c -o simple-realloc-chk
$ ./simple-realloc-chk name: Adam
Shit....we failed reallocating memory


and it would be neat to also check for memory leaks, which under GNU/Linux, can be done using Valgrind like this:

valgrind --leak-check=full ./simple-realloc-chk 
==8422== Memcheck, a memory error detector
==8422== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==8422== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==8422== Command: ./simple-realloc-chk
==8422== 
name: Adam
Shit....we failed reallocating memory
==8422== 
==8422== HEAP SUMMARY:
==8422==     in use at exit: 0 bytes in 0 blocks
==8422==   total heap usage: 2 allocs, 2 frees, 1,029 bytes allocated
==8422== 
==8422== All heap blocks were freed -- no leaks are possible
==8422== 
==8422== For counts of detected and suppressed errors, rerun with: -v
==8422== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

In the valgrind example above we tested with the realloc failure. We could, no let's make that should, check for memory leaks in all cases.

Using your own alloc code instead of libc's

Sometimes you can't replace a call to calloc or any of its friends so the call is implicit (e g you call readline or strdup) The source code for strdup in GNU's libc looks like this (alloc code is highlighted):

char *
__strdup (const char *s)
{
  size_t len = strlen (s) + 1;
  void *new = malloc (len);

  if (new == NULL)
    return NULL;

  return (char *) memcpy (new, s, len);
}

Note:the source code for the entire file containing strdup can be found here: strdup.c.

On Unix like operating systems this is easy. Let's do it in three steps:

Implement the alloc functions

Typicall create a file, alloc-null.c, with the functions implemented as follows:

#include <stdlib.h>

void *malloc(size_t size) { return NULL; } 
void free(void *ptr) { return ;} 
void *calloc(size_t nmemb, size_t size) { return NULL; } 
void *realloc(void *ptr, size_t size)  { return NULL; }

All functions allocating memory are implemented returning NULL and free simply returns. You can find the source code file here: https://github.com/progund/programming-with-c/blob/master/testing/wiki-material/alloc-null.c.

Create a shared object file

Next step is to create a shared object file (which will be loaded later on).

$ gcc -shared  alloc-null.c  -o liballoc-null.so
$ file liballoc-null.so 
liballoc-null.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=e035ec09c7d118f9e08909bb0ea6349853e01178, not stripped
Load the objects

The remaining part is to make sure to make your program use your functions instead of the ones in libc. So we're going to use LD_PRELOAD. But let's start with using the file simple-realloc.c without any faked memory problems.

$ gcc -g simple-realloc.c -o simple-realloc 
$ valgrind --leak-check=full ./simple-realloc 
==6696== Memcheck, a memory error detector
==6696== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==6696== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==6696== Command: ./simple-realloc
==6696== 
name: Adam
name: Kenny McCormick
==6696== 
==6696== HEAP SUMMARY:
==6696==     in use at exit: 0 bytes in 0 blocks
==6696==   total heap usage: 3 allocs, 3 frees, 1,046 bytes allocated
==6696== 
==6696== All heap blocks were freed -- no leaks are possible
==6696== 
==6696== For counts of detected and suppressed errors, rerun with: -v
==6696== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Ok, the program (still) works. Let's replace the libc versions osf the alloc functions with the ones in the shared object file we've just created. Set the variable LD_PRELOAD to point to the shared object file and this file's objects will be loaded before libc.

$ LD_PRELOAD=./liballoc-null.so ./simple-realloc

Nothing is printed so nothing got allocated. It seem to work. We can test if our code can deal with memory failures (such as allocating memory returning NULL).

We can add s small printout (in simple-realloc.c when failing to allocate memory:

  /* Check if it all went well */
  if (name == NULL)
    {
      /* Uh oh, we did not get any memory */
      fprintf(stderr, "Uh oh, failed allocating memory with calloc\n");
      exit (1);
    }

  /* Use the memory - in this case copy a name to it and print it */
  strcpy (name, "Adam");
  fprintf(stdout, "name: %s\n", name);

  /* increase the memory */
  char * tmp = realloc(name, sizeof(char)*17);
  
  /* Check if it all went well */
  if (tmp == NULL)
    {
      /* Uh oh, we did not get any memory. Free the old mem and exit */
      fprintf(stderr, "Uh oh, failed allocating memory with realloc\n");
      free(name);
      exit (1);
    }

Let's compile this and try again:

$ gcc -g simple-realloc.c -o simple-realloc
$ ./simple-realloc 
name: Adam
name: Kenny McCormick
$ LD_PRELOAD=./liballoc-null.so  ./simple-realloc 
Uh oh, failed allocating memory with calloc

Misc

gdb, where's my value?

Look briefly at the following code:

#include <stdio.h>
#include "const-enemy.h"

int nr = 12 ;

int main(void)
{
  int* nrp = &nr;
  
  PRINT_INT(nr);
  PRINT_INT(*nrp);

  return 0;
}

which uses this:

int print_var(char *name, int value, FILE* stream);

#define PRINT_INT(a) print_var(#a, a, stderr);

When compiling and executing the code (including the c file) above we should see the value 12 printed twice. So let's try it out:

$ gcc  -pedantic -Wconversion -Wall -Werror  -Wextra -Wstrict-prototypes -g  const.c const-enemy.c
$ ./a.out
nr=12
*nrp=-100

What???? Is this possible? Bring forth the debugger to find who's setting the value of nr to 100. Download the files XXXXX. To be honest, we have added some code to the file value-enemy.c. In this examplle it is quite easy to see that something strange is going on in the function print_var. In bigger files it is not quite as easy to find odd code. We made a simple example to make it easy to understand. Let's start compile and produce debugging information (use -g):

pgcc  -pedantic -Wconversion -Wall -Werror  -Wextra -Wstrict-prototypes -g  value.c value-enemy.c

Ok, we have a program to execute. Let's run the program in gdb:

$ gdb ./a.out 
GNU gdb (GDB) Fedora 7.11.1-75.fc24
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./a.out...done.
(gdb)

gdb has now loaded our program. Let's make the program break (pause) when entering the main function:

(gdb) b main
Breakpoint 1 at 0x40056e: file value.c, line 8.
(gdb)

start the program

(gdb) run
Starting program: /home/hesa/opt/prog-unbook/programming-with-c/testing/wiki-material/a.out 
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.23.1-10.fc24.x86_64

Breakpoint 1, main () at value.c:8
8	  int* nrp = &nr;
(gdb)

add a watchpoint on the nr variable

(gdb) watch nr
Hardware watchpoint 2: nr
(gdb)

ok, let's continue with executing the program. We tell the debugger (gdb) to continue either by typing continue or c

(gdb) c
Continuing.
nr=12

Hardware watchpoint 2: nr

Old value = 12
New value = -100
print_var (name=0x400690 "nr", value=12, stream=0x7ffff7dd4520 <_IO_2_1_stderr_>) at value-enemy.c:12
12	  return ret;

Uh oh.....we can see that someone is tampering our value (12). In the file value-enemy.c on line 12 something is happening. Let's leave the debugger:

(gdb) quit
A debugging session is active.

	Inferior 1 [process 19702] will be killed.

Quit anyway? (y or n) y

Let's look at the problematic code:

  unsigned long int iptr = (unsigned long int) &nr;
  ret = fprintf(stream, "%s=%d\n", name, value);
  *((int*)iptr) = -100;

Ok, this is kind of obvious. The idea with this short introduction is to show you how to use gdb to find who's tampering with your variables by setting a watchpoint. gdb is of course capable of more, much more. But we felt this related to the testing chapter .... well, well. Bye for now.

Exercises

Test exam_grade

In the section [Chapter:Functions#Writing_your_own Writing your own] in the chapter [Chapter:Functions Functions] you have and exercise (10) where you were asked to write the function: int exam_grade(int points). Write unit tests for this function.

Test course_grade

In the same section as above you have an exercise (11) where you were asked to write the function: int course_grade(int exam, int handin) . Write unit tests for these function.

Test max

Write unit test code (using assert, check or cunit) to verify the function max. Your tests shall cover the entire function.

Test min

Write unit test code (using assert, check or cunit) to verify the function min. Your tests shall cover the entire function.

Test mean

Write unit test code (using assert, check or cunit) to verify the function mean. Your tests shall cover the entire function.

Solutions

Expand using link to the right to see the full content.

Coming soon

Test exam_grade

Test course_grade

Test max

Test min

Test mean