4 Techniques for Testing Python Command-Line (CLI) Apps

“Lo-Fi” Debugging With Print

This is one of the simplest ways to test. All you have to do here is print a variable or object that you’re interested in–before a function call, after a function call, or within a function.

Respectively, these allow you to verify a function’s input, a function’s output, and a function’s logic.

If you save the code above as testapp.py and try to run it with python testapp.py you’ll see an error like so:

Traceback (most recent call last):
  File "testapp.py", line 60, in <module>
    print_person(final_transformed)
  File "testapp.py", line 23, in print_person
    parents = "and".join(person_data['parents'])
KeyError: 'parents'

There is a missing key in person_data that is passed into print_person. The first step would be to check the input to print_person and see why our expected output (a printed message) isn’t being generated. We will just add a print function call before the call to print_person:

final_transformed = final_transform(initial_transformed)
print(final_transformed)
print_person(final_transformed)

The print function does the job here, showing in its output that we don’t have the top-level parents key—nor the siblings key—but in the interest of our sanity, I’ll show you pprint, which prints multi-level objects in a more readable manner. To use it, add from pprint import pprint to the top of your script.

Instead of print(final_transformed), we call pprint(final_transformed) to inspect our object:

{'address': '123 Main St.\nFL, Anytown 99999',
 'city': 'Anytown',
 'name': 'John Q. Public',
 'relationships': {'parents': ['John Q. Public Sr.', 'Mary S. Public'],
                   'siblings': ['Michael R. Public', 'Suzy Q. Public']},
 'state': 'FL',
 'street': '123 Main St.',
 'zip': 99999}

Compare this with the expected final form above.

Because we know final_transform doesn’t touch the relationships dictionary, it’s time to see what is going on in initial_transform. Normally, I’d use a traditional debugger to step through this, but I want to show you another use of print debugging.

We can print the state of objects in code, but we aren’t limited to that. We can print whatever we want, so we can also print markers to see which logic branches are executed and when.

Because initial_transform is primarily a few loops, and because internal dictionaries are supposed to be handled by the inner for loop, we should check out what’s happening in there, if anything:

def initial_transform(data):
    """
    Flatten nested dicts
    """
    for item in list(data):
        if type(item) is dict:
            print "item is dict!"
            pprint(item)
            for key in item:
                data[key] = item[key]

    return data

If we come across a dictionary within our input data, then we will be alerted in the console and then we will see what the item looks like.

After running, our console output hasn’t changed. This is good evidence that our if statement isn’t working as expected. While we can continue printing to find the bug, this is a great way to demonstrate the strengths of using a debugger.

As an exercise, though, I recommend bug hunting this code using only print debugging. It’s good practice and will force you to think of all the ways to use the console to alert you about different things happening in the code.

Using a Debugger

Debuggers are great for when you want to step through the code one line at a time and inspect the entire application state. They help when you know roughly where errors are happening but can’t figure out why, and they give you a nice top-down view of everything happening inside your application at once.

There are many debuggers out there, and often they come with IDEs. Python also has a module called pdb that can be used in the REPL to debug code. Rather than get into implementation-specific details of all available debuggers, in this section I’ll show you how to use debuggers with common functions, such as setting breakpoints and watches.

Breakpoints are markers on your code that tell your debugger where to pause execution for you to inspect your application state. Watches are expressions that you can add during a debugging session to watch the value of a variable (and more) and are persisted through your app’s execution.

But let’s jump back to breakpoints. These will be added where you want to start or continue a debugging session. Since we are debugging the initial_transform method, we will want to put one there. I will denote the breakpoint with a (*):

def initial_transform(data):
    """
    Flatten nested dicts
    """
(*) for item in list(data):
        if type(item) is dict:
            for key in item:
                data[key] = item[key]

    return data

Now when we start debugging, execution will pause on that line and you’ll be able to see variables and their types at that particular point in the program’s execution. We have a few options to navigate our code: step over, step in, and step out are the most common.

Step over is the one you’ll use most often–this simply jumps to the next line of code.

Step in, attempts to go deeper into the code. You’ll use this when you come across a function call you want to investigate more deeply–you’ll be taken directly to that function’s code and be able to examine state there. You also use it often when confusing it for step over. Luckily step out can rescue us, this brings us back out to the caller.

We can also set a watch here, something like type(item) is dict, which you can do in most IDEs via an ‘add watch’ button during a debugging session. This will now show True or False no matter where you are in the code.

Set the watch, and now step over so that you are now paused on the if type(item) is dict: line. You should now be able to see the status of the watch, the new variable item, and the object data.

Even without the watch, we can see the issue: rather than type looking at what item points to, it’s looking at the type of item itself, which is a string. Computers do exactly what we tell them, after all. Thanks to the debugger, we see the error of our ways and fix our code like so:

def initial_transform(data):
    """
    Flatten nested dicts
    """
    for item in list(data):
        if type(data[item]) is dict:
            for key in data[item]:
                data[key] = item[key]

    return data

We should run it through the debugger again, and just make sure the code is going where we expect it to. And we are not, the structure now looks like this:

john_data = {
    'name': 'John Q. Public',
    'street': '123 Main St.',
    'city': 'Anytown',
    'state': 'FL',
    'zip': 99999,
    'relationships': {
        'siblings': ['Michael R. Public', 'Suzy Q. Public'],
        'parents': ['John Q. Public Sr.', 'Mary S. Public'],
    },
    'siblings',
    'parents',
}

Now that we’ve looked at how a visual debugger is used, let’s go deeper and put our new knowledge to the test by completing the exercise below.

john_data = {
    'name': 'John Q. Public',
    'street': '123 Main St.',
    'city': 'Anytown',
    'state': 'FL',
    'zip': 99999,
    'relationships': {
        'siblings': ['Michael R. Public', 'Suzy Q. Public'],
        'parents': ['John Q. Public Sr.', 'Mary S. Public'],
    },
    'parents': ['John Q. Public Sr.', 'Mary S. Public'],
    'siblings': ['Michael R. Public', 'Suzy Q. Public'],
}

def initial_transform(data):
    """
    Flatten nested dicts
    """
    for item in list(data):
        if type(data[item]) is dict:
            for key in data[item]:
                data[key] = data[item][key]

    return data

We’ve talked about the visual debugger. We’ve used the visual debugger. We love the visual debugger. There are still pros and cons to this technique, though, and you can review them in the section below.

Unit Testing with Pytest and Mocks

The previous techniques are tedious and can require code changes if you want to exhaustively test input-output combinations, ensuring you hit every branch of your code–especially as your app grows. In our example, the output of initial_transform still doesn’t look quite right.

While the logic in our code is fairly simple, it can easily grow in size and complexity, or become the responsibility of a whole team. How do we test an application in a more structured, detailed, and automated way?

Enter unit tests.

Unit testing is a testing technique that breaks down source code into recognizable units (usually methods or functions) and tests them individually.

You will essentially be writing a script or group of scripts that test each method with different inputs to ensure every logic branch within each method is tested–this is referred to as code coverage, and usually you want to aim for 100% code coverage. This isn’t always necessary or practical, but we can save that for another article (or a textbook).

Each test treats the method being tested in isolation: outside calls are overridden with a technique called mocking to give reliable return values and any object set up before the test is removed after the test. These techniques and others are done to assure the independence and isolation of the unit under test.

Repeatability and isolation are key to these kinds of tests, even though we are still continuing with our theme of comparing expected outputs to actual outputs. Now that you have an understanding of unit testing overall, you can take a quick detour and see how to unit test Flask applications with the minimum viable test suite.

import pytest
import testapp as app

@pytest.fixture(params=['nodict', 'dict'])
def generate_initial_transform_parameters(request):

@pytest.fixture(params=['nodict', 'dict'])
def generate_initial_transform_parameters(request):
    test_input = {
        'name': 'John Q. Public',
        'street': '123 Main St.',
        'city': 'Anytown',
        'state': 'FL',
        'zip': 99999,
    }
    expected_output = {
        'name': 'John Q. Public',
        'street': '123 Main St.',
        'city': 'Anytown',
        'state': 'FL',
        'zip': 99999,
    }

    if request.param == 'dict':
        test_input['relastionships'] = {
            'siblings': ['Michael R. Public', 'Suzy Q. Public'],
            'parents': ['John Q. Public Sr.', 'Mary S. Public'],
        }
        expected_output['siblings'] = ['Michael R. Public', 'Suzy Q. Public']
        expected_output['parents'] = ['John Q. Public Sr.', 'Mary S. Public']

    return test_input, expected_output

def test_initial_transform(generate_initial_transform_parameters):
    test_input = generate_initial_transform_parameters[0]
    expected_output = generate_initial_transform_parameters[1]
    assert app.initial_transform(test_input) == expected_output

def initial_transform(data):
    """
    Flatten nested dicts
    """
    for item in list(data):
        if type(data[item]) is dict:
            for key in data[item]:
                data[key] = data[item][key]
            data.pop(item)

    return data

def initial_transform(data):
    """
    Flatten nested dicts
    """
    for item in list(data):
        if type(data[item]) is dict:
            for key in data[item]:
                data[key] = data[item][key]
            data.pop(item)

    outside_module.do_something()
    return data

@pytest.fixture(params=['nodict', 'dict'])
def generate_initial_transform_parameters(request, mocker):
    [...]
    mocker.patch.object(outside_module, 'do_something')
    mocker.do_something.return_value(1)
    [...]

def initial_transform(data):
    """
    Flatten nested dicts
    """
    for item in list(data):
        if type(data[item]) is dict:
            for key in data[item]:
                data[key] = data[item][key]
            data.pop(item)

    outside_module.do_something()
    outside_module.do_something()
    return data

@pytest.fixture(params=['nodict', 'dict'])
def generate_initial_transform_parameters(request, mocker):
    [...]
    mocker.patch.object(outside_module, 'do_something')
    mocker.do_something.side_effect([1, 2])
    [...]

Integration Testing

Integration testing is one of the simpler testing methods here, but arguably one of the most important. This entails actually running your app end-to-end with real data in a production-like environment.

Whether this is your home machine, a test server that duplicates a production server, or just changing a connection to a test database from a production one, this lets you know that your changes will work when deployed.

Like in all the other methods, you’re checking that your application generates the expected outputs given some inputs–except this time you’re using actual external modules (unlike in unit testing, where they are mocked), perhaps writing to actual databases or files, and, in larger applications, ensuring that your code integrates well with the overall system.

How you do this is highly dependent on your application, for example, our test app can be run on its own with python testapp.py. However, let’s pretend our code is a segment of a large distributed application, like an ETL pipeline–in that case you would have to run the entire system on test servers with your code swapped in, run data through it, and make sure it made it through the whole system in the correct form. Outside of the command-line application world, tools like pyVows can be used for integration testing Django apps.

Putting It All Together

In conclusion—all CLI testing is a matter of comparing your expected outputs to your actual outputs, given some set of inputs. The methods I’ve discussed above are all ways of doing just that, and in many ways are complementary. These will be important tools for you to understand as you continue building command-line applications in Python, but this tutorial is just a starting point.

Python has a very rich ecosystem, and that extends to testing tools and methodologies, so branch out from this and investigate more–you may find a tool or technique that I didn’t mention here that you absolutely love. If so, I’d love to hear about it in the comments!

As a quick recap, here are the techniques we learned about today and how they’re employed:

• Print debugging - print out variables and other markers in code to see how execution flows
• Debuggers - controlling program execution to get a bird’s-eye view of application state and program flow
• Unit testing - breaking an application into individually testable units and testing all logic branches within that unit
• Integration testing - Testing your code changes in the context of the wider application

Now go forth and test! As you work with these techniques, be sure to let me know in the comments how you’ve employed them and which are your favorites.

To get a Python testing cheat sheet that summarizes the techniques demonstrated in this tutorial, click the link below: