Android:Test

From Juneday education
Jump to: navigation, search

Testing software

The procedures performed to get quality information about a system is called software testing. We can test programs by trying to find bugs or by verifying the correctness of a program. According to wikipedia, "In general, these properties indicate the extent to which the component or system under test:

  • meets the requirements that guided its design and development,
  • responds correctly to all kinds of inputs,
  • performs its functions within an acceptable time,
  • is sufficiently usable,
  • can be installed and run in its intended environments, and
  • achieves the general result its stakeholders desire."

- Software testing

We have no intention to make this page complete when it comes to all kinds of testing. Instead we will focus on two kinds of test and also compare how to write robust code in two different languages (C and Java). We will also give a short introduction to some useful tool and techniques.

Code example to get us going

The code on this page can be found at: github.com/progund/android-examples

C function to read an int from a user (stdin)

We have written a function hat reads input from a user (stdin).

File: user-io.c

#include <stdio.h>
#include "user-io.h"

int read_int(void)
{
  int i;
  scanf("%d", &i);
  return i;
}

File: user-io.h

int read_int(void);

To verify the function we have written a small test program.

File: test-user-io-read.c

#include <stdio.h>
#include "user-io.h"

int main(void)
{
  int ret;
  printf("Type an integer value: ");
  ret = read_int();
  printf("You typed: %d\n", ret);
  return 0;
}


To compile these files into a program (using gcc):

gcc -pedantic -Wconversion -Wall -Werror -Wextra -Wstrict-prototypes     test-user-io.c user-io.c   -o test-user-io

and to run it:

./test-user-io 
Type an integer value: 12
You typed: 12

All nice isn't it? All code is tested. We can check this if we do a code coverage analysis on the file user-io.c, using gcov and lcov:

gcc -pedantic -Wconversion -Wall -Werror -Wextra -Wstrict-prototypes --coverage -fprofile-arcs -ftest-coverage $(SOURCES) -o test-user-io -lm
for i in `seq -200 200` ; do echo "$i" | ./$(PROG) ; done
gcov -b user-io.c
lcov --directory .  --capture --output-file  user-info.info && genhtml -o coverage user-info.info

Html report from code coverage, showing that we have tested 100% of the code:

Gcov-lcov.png

This should keep all project managers happy? Fact is, it probably does. But have we really checked the program? How about checking with bad indata, such as "Henrik":

$ ./test-user-io 
Type an integer value: Henrik
You typed: 32689

Uh oh. Not so good, was it? How about not giving any input to the program. You can do this by either typing Ctrl-d (Control key followed by the key d) or using pipes as below:

$ echo -n | ./test-user-io 
Type an integer value: You typed: 32552

Uh oh, again. So we can see that writing tests and having all tests pass really don't tell if the code has no bugs. But people obsessed with software metrics for the sake of it will most likely be very pleased with this. However, the authors behind this book are not. The good thing with the tests we've done are that we now know that the code is full of bugs and therefore bad. Unfortunately this code was found in a book. And to make things even worse, we've seen code similar to this in several books and in educational material. Most likely code like this is copied between people like "Dawkins" memes.

A good way of thinking about your code once you're done, or rather once you think you're done: now, lets' break it. Try to break your code, fund bugs by passing bad indata, make sure file descriptors are faulty, make sure the memory has run out. .... Isn't it nicer to find the bugs yourself than someone else finding them?

Java

We've written a program in Java. The program outputs the content of a file, with a line number before the actual text.

Java file ReadFile.java:

import java.util.Scanner;
import java.io.File;
import java.io.FileNotFoundException;

public class ReadFile {

  public static void main(String[] args) {
    int lineCount=1;
    Scanner sc=null;
    try {
      sc = new Scanner(new File(args[0]));
    } catch (FileNotFoundException e) { ; }
    while (sc.hasNextLine()) {
      String line = sc.nextLine();
      System.out.println(lineCount + ":" + line);
      lineCount++;
    }
  }
  
}

Let's compile and test it (by passing the Java source code file as argument to the program).

$ javac ReadFile.java 
$ java ReadFile ReadFile.java 
1:import java.util.Scanner;
2:import java.io.File;
3:import java.io.FileNotFoundException;
4:
5:public class ReadFile {
6:
7:  public static void main(String[] args) {
8:    int lineCount=1;
9:    Scanner sc=null;
10:    try {
11:      sc = new Scanner(new File(args[0]));
12:    } catch (FileNotFoundException e) { ; }
13:    while (sc.hasNextLine()) {
14:      String line = sc.nextLine();
15:      System.out.println(lineCount + ":" + line);
16:      lineCount++;
17:    }
18:  }
19:  
20:}

Seems to work nice doesn't it. But as above perhaps we should test with some bad input. Let's start with not passing an argument:

$ java ReadFile 
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
	at ReadFile.main(ReadFile.java:11)

Hmm, didn't work well, did it? Let's pass a non existing file as argument to the program:

$ java ReadFile not-existing-file
Exception in thread "main" java.lang.NullPointerException
	at ReadFile.main(ReadFile.java:13)

Arg, crashed again. SO when testing with only "good" data the program worked. Testing with bad data and it crashed. Any lessons learned?

Conclusion

Sometimes a program or a unit does not do what we want it to do. Sometimes it works only with a limited input data, such as in the case of the program above. So how do we increase the quality of a program? Writing tests are not the silver bullet, but a often it is quite useful. Learning to write good quality code in the first place is another one. The latter is one thing we try hard to teach in our material. In this page we're going to look a bit at writing tests for Android and automating them.

Black Box test

"Black-box testing is a method of software testing that examines the functionality of an application without peering into its internal structures or workings. This method of test can be applied virtually to every level of software testing: unit, integration, system and acceptance. It is sometimes referred to as specification-based testing." - Black-box_testing (wikipedia)

If we want to test our Nobel Prize award winning class ReadFile as a block box, we simply run the program with different input:

Expected input:

$ java ReadFile ReadFile.java 
1:import java.util.Scanner;
2:import java.io.File;
3:import java.io.FileNotFoundException;
4:
5:public class ReadFile {
6:
7:  public static void main(String[] args) {
8:    int lineCount=1;
9:    Scanner sc=null;
10:    try {
11:      sc = new Scanner(new File(args[0]));
12:    } catch (FileNotFoundException e) { ; }
13:    while (sc.hasNextLine()) {
14:      String line = sc.nextLine();
15:      System.out.println(lineCount + ":" + line);
16:      lineCount++;
17:    }
18:  }
19:  
20:}

Run with missing argument

$ java ReadFile 
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
	at ReadFile.main(ReadFile.java:11)

Run with argument poitning out a file that does not exist:

$ java ReadFile does-not-exists
Exception in thread "main" java.lang.NullPointerException
	at ReadFile.main(ReadFile.java:13)

Run with argument poitning out a file that is not readable:

$ echo "Some text" > inputfile.txt
$ chmod a-rw inputfile.txt
$ java ReadFile inputfile.txt 
Exception in thread "main" java.lang.NullPointerException
	at ReadFile.main(ReadFile.java:13)

Unit test

"In computer programming, unit testing is a software testing method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures, are tested to determine whether they are fit for use." - Unit_testing

Instrumented test

We can't write unit tests since we have all our Java code in main - how about that for structured code.

Code review

"Code review is systematic examination (sometimes referred to as peer review) of computer source code. It is intended to find mistakes overlooked in software development, improving the overall quality of software. Reviews are done in various forms such as pair programming, informal walkthroughs, and formal inspections." - Code_review (Wikipedia)

Static code analysers

"Static program analysis is the analysis of computer software that is performed without actually executing programs (analysis performed on executing programs is known as dynamic analysis).[1] In most cases the analysis is performed on some version of the source code, and in the other cases, some form of the object code." - Static_program_analysis (wikipedia)

Checkstyle

"Checkstyle is a development tool to help programmers write Java code that adheres to a coding standard. It automates the process of checking Java code to spare humans of this boring (but important) task. This makes it ideal for projects that want to enforce a coding standard" - checkstyle.sourceforge.net

Let's say we want to use (a modified*) XML configuration file to check the file ReadFile.java we type:

$ checkstyle -c google_checks.xml ReadFile.java 
Starting audit...
[WARN] /home/hesa/opt/prog-unbook/android-examples/test/c/ReadFile.java:2: Wrong lexicographical order for 'java.io.File' import. Should be before 'java.util.Scanner'. [CustomImportOrder]
[WARN] /home/hesa/opt/prog-unbook/android-examples/test/c/ReadFile.java:3: Wrong lexicographical order for 'java.io.FileNotFoundException' import. Should be before 'java.util.Scanner'. [CustomImportOrder]
[WARN] /home/hesa/opt/prog-unbook/android-examples/test/c/ReadFile.java:8:18: WhitespaceAround: '=' is not preceded with whitespace. [WhitespaceAround]
[WARN] /home/hesa/opt/prog-unbook/android-examples/test/c/ReadFile.java:8:19: WhitespaceAround: '=' is not followed by whitespace. Empty blocks may only be represented as {} when not part of a multi-block statement (4.1.3) [WhitespaceAround]
[WARN] /home/hesa/opt/prog-unbook/android-examples/test/c/ReadFile.java:9:15: WhitespaceAround: '=' is not preceded with whitespace. [WhitespaceAround]
[WARN] /home/hesa/opt/prog-unbook/android-examples/test/c/ReadFile.java:9:16: WhitespaceAround: '=' is not followed by whitespace. Empty blocks may only be represented as {} when not part of a multi-block statement (4.1.3) [WhitespaceAround]
[WARN] /home/hesa/opt/prog-unbook/android-examples/test/c/ReadFile.java:12:39: '{' at column 39 should have line break after. [LeftCurly]
Audit done.

Uh oh. Plenty of fixes to do. We keep them as they are since we want to show the warnings produced by checkstyle.

  • ) We have removed some javadoc related tests

Debugger

"A debugger or debugging tool is a computer program that is used to test and debug other programs (the "target" program). The code to be examined might alternatively be running on an instruction set simulator (ISS), a technique that allows great power in its ability to halt when specific conditions are encountered, but which will typically be somewhat slower than executing the code directly on the appropriate (or the same) processor. Some debuggers offer two modes of operation, full or partial simulation, to limit this impact." - Debugger (wikipedia)

If you're interested in debugging C code, check out Chapter:C_testing.

jdb

"The Java Debugger, jdb, is a simple command-line debugger for Java classes. It is a demonstration of the Java Platform Debugger Architecture that provides inspection and debugging of a local or remote Java Virtual Machine" - jdb - The Java Debugger

jdb example session

In the example below we use highlighted text (yellow) to mark the text we typed. We're going to use jdb to inspect the Scanner sc after it has been constructed in main.

Compile for debugging:

$ javac -g ReadFile.java

Run debugger:

$ jdb ReadFile.java 
Initializing jdb ...
>

Set a breakpoint at main - halt execution when reaching main

> stop in ReadFile.main
Deferring breakpoint ReadFile.main.
It will be set after the class is loaded.

Run the class with the source code file as argument (this is an argument to the program, not to the debugger)

> run ReadFile ReadFile.java
run  ReadFile ReadFile.java
Set uncaught java.lang.Throwable
Set deferred uncaught java.lang.Throwable
> 
VM Started: Set deferred breakpoint ReadFile.main

Breakpoint hit: "thread=main", ReadFile.main(), line=8 bci=0
8        int lineCount=1;

main[1]

Continue execution - thre lines (using next three times)

main[1] next
> 
Step completed: "thread=main", ReadFile.main(), line=9 bci=2
9        Scanner sc=null;

main[1] next
> 
Step completed: "thread=main", ReadFile.main(), line=11 bci=4
11          sc = new Scanner(new File(args[0]));

main[1] next
> 
Step completed: "thread=main", ReadFile.main(), line=12 bci=22
12        } catch (FileNotFoundException e) { ; }

main[1]

Inspect the Scanner

main[1] print sc
 sc = "java.util.Scanner[delimiters=\p{javaWhitespace}+][position=0][match valid=false][need input=false][source closed=false][skipped=false][group separator=\,][decimal separator=\.][positive prefix=][negative prefix=\Q-\E][positive suffix=][negative suffix=][NaN string=\Q�\E][infinity string=\Q∞\E]"

Print surrounding lines:

main[1] list
8        int lineCount=1;
9        Scanner sc=null;
10        try {
11          sc = new Scanner(new File(args[0]));
12 =>     } catch (FileNotFoundException e) { ; }
13        while (sc.hasNextLine()) {
14          String line = sc.nextLine();
15          System.out.println(lineCount + ":" + line);
16          lineCount++;
17        }

Continue 'til the end...

main[1] cont
> 1:import java.util.Scanner;
2:import java.io.File;
3:import java.io.FileNotFoundException;
4:
5:public class ReadFile {
6:
7:  public static void main(String[] args) {
8:    int lineCount=1;
9:    Scanner sc=null;
10:    try {
11:      sc = new Scanner(new File(args[0]));
12:    } catch (FileNotFoundException e) { ; }
13:    while (sc.hasNextLine()) {
14:      String line = sc.nextLine();
15:      System.out.println(lineCount + ":" + line);
16:      lineCount++;
17:    }
18:  }
19:  
20:}
The application exited

Code coverage

"In computer science, code coverage is a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs. A program with high code coverage, measured as a percentage, has had more of its source code executed during testing which suggests it has a lower chance of containing undetected software bugs compared to a program with low code coverage." - Code coverage (wikipedia)

Profiler