Testing using Python

Henry Prêcheur <henry@precheur.org>

Vancouver Python user group February 2009

Plan

1. Introduction: What is testing?
2. Why test?
3. Unittest
4. Doctest
5. Other cool tools (coverage tools & nose)

What is testing?

Running your code to see if it breaks.

• Testing ≠ debugging
• Testing focuses on detecting errors, not correcting them.

Process

Different kinds of “developer test”

Unit test

function, class, and small program

Component test

class, package, small program

Integration test

combination of multiple units or components

Regression test

Make sure that what used to work Monday still works Friday

System test

Performance, resource usage, and/or security of the software in its final configuration

Why test?

You already test your code!

Running the code you have just written is testing.

Structure & reuse those tests to be more efficient

Why test?

1. Testing helps find bugs & errors
2. Writing tests and keeping them is a good way to document how the code works
3. Testing is a metric for Quality Assurance
4. Great for refactoring (contribution of the audience)

Why test?

Make sure new code is correct and defects are fixed

Why test?

Make sure defects are corrected

Automated testing is a great way to detect regressions.

Why test?

Important to migrate your project to Python 3.

You should have excellent unit tests with close to full coverage.
— PEP 3000

Why test?

2to3

Why test?

Contributing

Most big projects (like Python and Django) require tests with code contribution.

Why test?

Testing works.

It's not a trendy technology / language / framework pretending to solve all your problems.

Why test?

BUT ...

Why test?

... testing does not find all bugs

program testing can be a very effective way to show the presence of bugs, but is hopelessly inadequate for showing their absence.
— Edsger W. Dijkstra, The Humble Programmer

Why test?

... testing does not find all bugs

Individual testing steps (unit test, component test,, and integration test) typically find less than 50 percent of the errors present each. The combination of testing steps often finds less than 60 percent of the errors present.
— Steve McConnel, Code Complete 2ed

Questions so far?

How to test?

• Unittest also known as PyUnit, included in the Python standard library
• Doctest, included in the Python standard library
• py.test, PyPy's testing framework.
• pyUnitPerf, performance-based unit testing

Only focus on Unittest and Doctest.

Unittest

Similar to JUnit, the mother of all test frameworks.

Example 1/3

import unittest

class MyTestCase(unittest.TestCase):

    def test_add(self):
        # Could also be 'assertTrue' or 'failUnless'
        self.assert_(isinstance(2 + 4, int))

        self.assert_([] + [] == [])
        # More explicit error message if it fails
        self.assertEqual([] + [], [])

Example 2/3

def test_div(self):
    self.assertRaises(ZeroDivisionError,
                      lambda: 1 / 0)

    self.assertRaises(ZeroDivisionError,
                      int.__div__, 1, 0)

Example 3/3

if __name__ == '__main__':
    import unittest
    unittest.main()

Run the test

$ python test.py
..
----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

unittest.main() has handy features.
python test.py --help to learn more.

Fixtures 1/2

import unittest
import tempfile
import shutils

class MyTestCase(unittest.TestCase):
    def setUp(self):
        # create fixtures
        self.tempdir = tempfile.mkdtemp()

    def tearDown(self):
        # delete fixtures
        shutils.rmtree(self.tempdir)
    ...

Fixtures 2/2

def test_fixture_1(self):
    # self.tempdir was just created
    # Create a file
    open(os.path.join(self.tempdir, 'test'), 'w')
    # self.tempdir will be deleted after the function call

def test_fixture_2(self):
    # Check that the file 'test' doesn't exist
    self.assertRaise(IOError, open, os.path.join(self.tempdir, 'test'))

Grouping tests with TestSuite

suite1 = unittest.TestSuite(MyTestCase1(), MyTestCase2())
suite2 = unittest.TestSuite()
suite2.addTest(MyOtherTestCase())

big_suite = unittest.TestSuite(suite1)
big_suite.addTest(suite2)

You can also iterate on a TestSuite

TestRunner

• Using a GUI to run the tests
• Monitor performance
• Send an Email to developers when tests fail
• Run tests with different configurations (ex: different database backends)

And Now for Something
Completely Different

Doctest

Unittest is boring: exactly what you would expect from a test framework.

Doctest is cool and funky ...

Problem

How do I make sure that my documentation and my code are synchronized?

Wait ... Did we see this before?

Solution!

Documentation can be tests.

Doctest transforms documentation into real tests.

Example docstring

def factorial(n):
    '''Return the factorial of n, an exact integer >= 0.

        >>> factorial(1)
        1
        >>> [factorial(n) for n in range(6)]
        [1, 1, 2, 6, 24, 120]
    '''
    if n == 0:
        return 1
    else:
        n * factorial(n - 1)
    

Running doctests

if __name__ == '__main__':
    import doctest
    doctest.testmod()

Doctest in documentation

You can also put doctests in text files and execute those tests.

This is a documentation file.

The function hello_world() prints 'Hello World'.

    >>> hello_world()
    'Hello World'

More text ...

Doctest checks output and return value

def print_and_return(i):
    '''
    >>> print_and_return(42)
    42
    42
    '''
    print i
    return i

Potential gotcha!

Checking exception

def factorial(n):
    '''
    >>> factorial(-1)
    Traceback (most recent call last):
    ...
    ValueError: n must be >= 0
    '''
    if not n >= 0:
        raise ValueError("n must be >= 0")