doctest

From Wikipedia, the free encyclopedia

doctest is a module included in the Python programming language's standard library that allows the easy generation of tests based on output from the standard Python interpreter shell, cut and pasted into docstrings.

Contents

[edit] Implementation specifics

doctest makes innovative use of the following Python capabilities:

  • docstrings
  • The Python interactive shell, (both command line and the included idle application).
  • Python introspection

When using the Python shell, the primary prompt: >>> , is followed by new commands. The secondary prompt: ... , is used when continuing commands on multiple lines; and the result of executing the command is expected on following lines. A blank line, or another line starting with the primary prompt is seen as the end of the output from the command.

The doctest module looks for such sequences of prompts in a docstring, re-executes the extracted command and checks the output against the output of the command given in the docstrings test example.

The default action when running doctests is for no output to be shown when tests pass. This can be modified by options to the doctest runner. In addition, doctest has been integrated with the Python unit test module allowing doctests to be run as standard unittest testcases. Unittest testcase runners allow more options when running tests such as the reporting of test statistics such as tests passed, and failed.

[edit] Literate Programming and Doctests

Although doctest does not allow a Python program to be embedded in narrative text, they do allow for verifiable examples to be embedded in docstrings, where the docstrings can contain other text. Docstrings can in turn be extracted from program files to generate documentation in other formats such as HTML or PDF. A program file can be made to contain the documentation, tests, as well as the code and the tests easily verified against the code. This allows code, tests, and documentation to evolve together.

[edit] Documenting libraries by example

Doctests are well suited to providing an introduction to a library by demonstrating how the API is used.

On the basis of the output of Python's interactive interpreter, text can be mixed with tests that exercise the library, showing expected results.

[edit] Examples

Example one shows how narrative text can be interspersed with testable examples in a docstring. In the second example, more features of doctest are shown, together with their explanation. Example three is set up to run all doctests in a file when the file is run, but when imported as a module, the tests will not be run.

[edit] Example 1: A doctest embedded in the docstring of a function

def list_to_0_index(lst):
    """A solution to the problem given in:
    http://rgrig.blogspot.com/2005/11/writing-readable-code.html
 
    'Given a list, lst, say for each element the 0-index where it appears for
    the first time. So the list x = [0, 1, 4, 2, 4, 1, 0, 2] is
    transformed into y = [0, 1, 2, 3, 2, 1, 0, 3]. Notice that for all
    i we have xyi = xi. Use any programming language and any data
    representation you want.'
 
    >>> x = [0, 1, 4, 2, 4, 1, 0, 2]
    >>> list_to_0_index(x)
    [0, 1, 2, 3, 2, 1, 0, 3]
    >>>
    """
 
    return [lst.index(i) for i in lst]

[edit] Example 2: doctests embedded in a README.txt file

======================
Demonstration doctests
======================

This is just an example of what a README text looks like that can be used with
the doctest.DocFileSuite() function from Python's doctest module.

Normally, the README file would explain the API of the module, like this:

   >>> a = 1
   >>> b = 2
   >>> a + b
   3

Notice, that we just demonstrated how to add two numbers in Python, and 
what the result will look like.

A special option allows you to be somewhat fuzzy about your examples:

   >>> o = object()
   >>> o                 # doctest: +ELLIPSIS
   <object object at 0x...>

Exceptions can be tested very nicely too:

   >>> x
   Traceback (most recent call last):
     ...
   NameError: name 'x' is not defined

[edit] Example 3: unique_words.py

This example also simulates input to the function from a file by using the python StringIO module

def unique_words(page):
  ''' Returns set of the unique words in list of lines of text
 
    Example:
 
    >>> from StringIO import StringIO
    >>> fileText = """the cat sat on the mat
    ... the mat was ondur the cat
    ... one fish two fish red fish
    ... blue fish
    ... This fish has a yellow car
    ... This fish has a yellow star"""
    >>> file = StringIO(fileText)
    >>> page = file.readlines()
    >>> words = unique_words(page)
    >>> print sorted(list(words))
    ['This', 'a', 'blue', 'car', 'cat', 'fish', 'has', 'mat', 
     'on', 'ondur', 'one', 'red', 'sat', 'star', 'the', 'two', 
     'was', 'yellow']
    >>>
  '''
 
  return set(word  for line in page  for word in line.split())
 
def _test():
    import doctest
    doctest.testmod()
 
if __name__ == "__main__":
    _test()

[edit] Critique of doctest

[edit] Advantages

  • Many programmers already use the Python shell to test their code. Cutting and pasting from the shell to a docstring is a very simple way of generating tests.
  • Tests and code are together. The tests are easily run making them much more likely to be kept up-to-date.
  • Some level of documentation, tests and code are kept together.
  • Testing of exceptions looks very intuitive
  • A simple concept.
  • Can test for returned exceptions
  • Nice output formatting/comparison integration (e.g. whitespace normalization, ndiff usage for failed examples, use of ellipsis)

[edit] Disadvantages

  • Large numbers of tests in a docstring can become unwieldy. docstrings should be pruned and excised tests put in external file(s).
  • Tests producing large amounts of output make for large docstrings.
  • Debugging integration is far from perfect
  • 'print' (or 'trace') debugging is not possible (because it intervenes with the test result)
  • Test setup has to be either copied or hidden away from the test, making the overall environment harder to understand.
  • A few of the complex assertions of existing unit tests frameworks do not exist, (e.g. assertAlmostEqual).
  • Failing assertions can be harder to debug if you compare a lot of output. This is especially the case for web applications if the expected result is a web page with a lot of HTML.

[edit] Doctest and Documentation generators

Both the EpyText format of Epydoc, and the reStructuredText format of Docutils, support the markup of doctest sections within docstrings.

[edit] References