Testing

How many bugs are in 1,000 lines of code?

• Typical code: 1-10
• Platform code: 0.1-1
• The best—NASA: 0.01-0.1

Never assume your software doesn’t have bugs.

Test-Driven Development

Test before you build!

• Specs are already written
• We know what the expected behavior is
• We can write tests for the expected behavior first
• All tests fail to start
• We know we are done writing code when all tests pass

“NYI” (not-yet implemented)

often, writing test exposes gaps in your specs

How NOT! not write tests

Random Sampling

• Pick one or two inputs and show your code works on it
• Why it doesn’t work: there maybe specific inputs that break your code

Exhaustive Testing

• Test for the domain of inputs
• Why it doesn’t work: tests run forever

How DO you write tsets

Black-Box Testing

• Pretend the code implementation is a black box
• All you know is what the specification; and what the input/output produces

White-Box Testing

• You can see the implementation
• You test for specific edge cases
• Off-by-one, running time, specific inputs, etc.

Malicious Testing

• What happens if a user is trying to break your system
• Sometimes, this is known as “pen-testing” or “white-hack hacking”
• Take CS340 Compsec

How BIG are your tests

Unit Testing

• Backbone of testing
• Typically, that means one test per function
• Tests choose representative inputs
• Idempotent: the state of the testing system should be a the beginning and end of the test (tests should revert) (setup + teardown tests)

Subsystem Testing

• Exercise multiple functions working together in a system
• Often takes longer
• OK to run these less frequently

End-to-End Integration

• Exercise the entire workflow
• May involve external libraries, hardware, etc.

Regression Testing

• Isolate the cause of the bug to the smallest possible test case
• Write a test assuming the bug is fixed
• Fix the bug
• Add the test to your test suite

How MUCH do we run tests

• Ideally, run tests every time code is committed
• Ideally—run tests that address the function
• Schedule long tests

What to test for

see also here

equivalence partitioning

Come up with one test case per equivalence class. For instance, for a function that uppercases letters, analyze the following:

1. Lowercase letters
2. Uppercase letters
3. Non-alpha letters
4. Non-printable letters
5. Combinations

Each group will therefore have nicely the requirements covered

boundary value analysis

In addition to just testing 1 element per class in equivalence partitioning, try to test boundary values (off-by-one, etc.) cases for each equivalence class if you can come up with them.

Arrange, Act, Assert

1. arrange for setup by setting up variables, etc., and define the expected result (yes we do it before to be more readable)
2. act do the thing
3. assert correctness by checking the expected result