Testing using Python

Henry Prêcheur <henry@precheur.org>

Vancouver Python user group February 2009


  1. Introduction: What is testing?
  2. Why test?
  3. Unittest
  4. Doctest
  5. Other cool tools (coverage tools & nose)

What is testing?

Running your code to see if it breaks.


Different kinds of “developer test”

Unit test
function, class, and small program
Component test
class, package, small program
Integration test
combination of multiple units or components
Regression test
Make sure that what used to work Monday still works Friday
System test
Performance, resource usage, and/or security of the software in its final configuration

Why test?

You already test your code!

Running the code you have just written is testing.

Structure & reuse those tests to be more efficient

Why test?

  1. Testing helps find bugs & errors
  2. Writing tests and keeping them is a good way to document how the code works
  3. Testing is a metric for Quality Assurance
  4. Great for refactoring (contribution of the audience)

Why test?

Make sure new code is correct and defects are fixed

Why test?

Make sure defects are corrected

Automated testing is a great way to detect regressions.

Why test?

Important to migrate your project to Python 3.

You should have excellent unit tests with close to full coverage.
— PEP 3000

Why test?


Why test?


Most big projects (like Python and Django) require tests with code contribution.

Why test?

Testing works.

It's not a trendy technology / language / framework pretending to solve all your problems.

Why test?

BUT ...

Why test?

... testing does not find all bugs

program testing can be a very effective way to show the presence of bugs, but is hopelessly inadequate for showing their absence.
— Edsger W. Dijkstra, The Humble Programmer

Why test?

... testing does not find all bugs

Individual testing steps (unit test, component test,, and integration test) typically find less than 50 percent of the errors present each. The combination of testing steps often finds less than 60 percent of the errors present.
— Steve McConnel, Code Complete 2ed

Questions so far?

How to test?

Only focus on Unittest and Doctest.


Similar to JUnit, the mother of all test frameworks.

Example 1/3

import unittest

class MyTestCase(unittest.TestCase):

    def test_add(self):
        # Could also be 'assertTrue' or 'failUnless'
        self.assert_(isinstance(2 + 4, int))

        self.assert_([] + [] == [])
        # More explicit error message if it fails
        self.assertEqual([] + [], [])

Example 2/3

def test_div(self):
                      lambda: 1 / 0)

                      int.__div__, 1, 0)

Example 3/3

if __name__ == '__main__':
    import unittest

Run the test

$ python test.py
Ran 2 tests in 0.000s


unittest.main() has handy features.
python test.py --help to learn more.

Fixtures 1/2

import unittest
import tempfile
import shutils

class MyTestCase(unittest.TestCase):
    def setUp(self):
        # create fixtures
        self.tempdir = tempfile.mkdtemp()

    def tearDown(self):
        # delete fixtures

Fixtures 2/2

def test_fixture_1(self):
    # self.tempdir was just created
    # Create a file
    open(os.path.join(self.tempdir, 'test'), 'w')
    # self.tempdir will be deleted after the function call

def test_fixture_2(self):
    # Check that the file 'test' doesn't exist
    self.assertRaise(IOError, open, os.path.join(self.tempdir, 'test'))

Grouping tests with TestSuite

suite1 = unittest.TestSuite(MyTestCase1(), MyTestCase2())
suite2 = unittest.TestSuite()

big_suite = unittest.TestSuite(suite1)

You can also iterate on a TestSuite


Run tests, used for reporting.

And Now for Something
Completely Different


Unittest is boring: exactly what you would expect from a test framework.

Doctest is cool and funky ...


How do I make sure that my documentation and my code are synchronized?

Wait ... Did we see this before?


Documentation can be tests.

Doctest transforms documentation into real tests.

Example docstring

def factorial(n):
    '''Return the factorial of n, an exact integer >= 0.

        >>> factorial(1)
        >>> [factorial(n) for n in range(6)]
        [1, 1, 2, 6, 24, 120]
    if n == 0:
        return 1
        n * factorial(n - 1)

Running doctests

if __name__ == '__main__':
    import doctest

Doctest in documentation

You can also put doctests in text files and execute those tests.

This is a documentation file.

The function hello_world() prints 'Hello World'.

    >>> hello_world()
    'Hello World'

More text ...

Doctest checks output and return value

def print_and_return(i):
    >>> print_and_return(42)
    print i
    return i

Potential gotcha!

Checking exception

def factorial(n):
    >>> factorial(-1)
    Traceback (most recent call last):
    ValueError: n must be >= 0
    if not n >= 0:
        raise ValueError("n must be >= 0")


>>> object() # Doesn't work well with doctest
<object object at 0x20e0c8120>

>>> object() # doctest:+ELLIPSIS
<object object at ...>

Integrating doctest and unittest

import unittest
import doctest
import my_module_with_doctests

# suite is a unittest.TestSuite()
suite = doctest.DocTestSuite(my_module_with_doctests)
runner = unittest.TextTestRunner()

Doctest good for simple tests, but noticeably bad when things become complex:

  • Lots of helpers are needed
  • A doctest of 10+ lines tend to be hard to follow

Example of ugly doctests (1/2)

    def __init__(self, f):
        >>> file_content = """title: test
        ... date: 2008-1-1
        ... author: test author
        ... encoding: utf-8
        ... test."""
        >>> from StringIO import StringIO
        >>> p = Post(StringIO(file_content))
        >>> p
        <Post('test', datetime.date(2008, 1, 1))>
        >>> p.title == 'test'
        >>> import datetime
        >>> p.date == datetime.date(2008, 1, 1)
        >>> p.content == 'test.'
        >>> p.encoding == 'utf-8'
        >>> p.author == 'test author'

Example of ugly doctests (2/2)

        >>> Post(StringIO('title: no payload\\ndate: 2008-1-1'))
        <Post('no payload', datetime.date(2008, 1, 1))>
        >>> Post(StringIO('title: no date'))
        Traceback (most recent call last):
        PostError: Error in post file <unknown filename>: No date defined
        >>> Post(StringIO("""title: bad encoding
        ... date: 2008-1-1
        ... encoding: bad-encoding"""))
        Traceback (most recent call last):
        PostError: Error in post file <unknown filename>: unknown encoding: \
        >>> Post(StringIO("""title: bad date
        ... date: 200008-101-10"""))
        Traceback (most recent call last):
        PostError: Error in post file <unknown filename>: Unable to parse \
date '200008-101-10'
        (Use YYYY-MM-DD [[HH:MM]:SS] format)

Unreadable mess which should be in a unittest.TestCase

Use Doctest
but don’t rely too much on it!

Doctest is for documenting first.

It is a good practice to duplicate relevant doctests in a unittest.TestCase.

More stuffs: code coverage

Code coverage describes the degree to which the source code of a program has been tested.

  • A good indicator of how good your tests are.
  • Developer usually overestimate their coverage.

More stuffs: code coverage

  • trace in the Python standard library
  • coverage.py
  • figleaf

More stuffs

nose: autodiscovery and execution of tests. Works with unittest and doctest.

$ nosetests path/to/project

Similar to py.test without resorting to too much magic.

nose also support coverage via coverage.py


Find the presentation’s slides at http://henry.precheur.org/vanpyz_test/

Contact me henry@precheur.org.