Probably the most popular third-party open source testing framework in Python, pytest

1. Introduction

This article is the third in “Let’s talk about Python’s unit testing framework”. The first two articles introduced the standard library unittest and the third-party unit testing framework nose respectively. As the last article in this series, the finale is the most popular third-party unit testing framework in the Python world: pytest.

pytest project address:

https://github.com/pytest-dev/pytest

It has the following main features:

assert Prints detailed information when an assertion fails (no need to remember self.assert* names anymore)
Automatic discovery of test modules and functions
Modular fixtures to manage various testing resources
Fully compatible with unittest and basically compatible with nose
Very rich plug-in system, with more than 315 third-party plug-ins, and a prosperous community
Just like unittest and nose were introduced earlier, we will introduce the features of pytest from the following aspects.

2. Writing use cases

Like nose, pytest supports test cases in the form of functions and test classes. The biggest difference is that you can use the assert statement to make assertions without worrying about it missing in nose or unittest Questions with detailed contextual information.

For example, in the following test example, the assertion in test_upper is deliberately made to fail:

import pytest
 
def test_upper():
    assert 'foo'.upper() == 'FOO1'
 
class TestClass:
    def test_one(self):
        x = "this"
        assert "h" in x
 
    def test_two(self):
        x = "hello"
        with pytest.raises(TypeError):
            x + []

When using pytest to execute the use case, it will output detailed (and multi-color) context information:

================================== test session starts ========= ==========================
platform darwin -- Python 3.7.1, pytest-4.0.1, py-1.7.0, pluggy-0.8.0
rootdir: /Users/prodesire/projects/tests, inifile:
plugins: cov-2.6.0
collected 3 items
 
test.py F.. [100%]
 
======================================== FAILURES ========= ================================
________________________________________ test_upper ________________________________________
 
    def test_upper():
> assert 'foo'.upper() == 'FOO1'
E AssertionError: assert 'FOO' == 'FOO1'
E-FOO
E+FOO1
E ? +
 
test.py:4: AssertionError
=========================== 1 failed, 2 passed in 0.08 seconds =============== =============

It is difficult to see that pytest outputs both the test code context and the information about the measured variable value. Compared with nose and unittest, pytest allows users to write test cases in a simpler way and get a richer and more friendly test. result.

3. Use case discovery and execution

Pytest supports the use case discovery and execution capabilities supported by unittest and nose. pytest supports automatic (recursive) discovery of use cases:

By default, all test case files in the current directory that match test_*.py or *_test.py are found. Test functions starting with test or test methods starting with test in test classes starting with Test
Use the pytest command
Like nose2, the name pattern (fuzzy matching) of use case files, classes and functions can be configured by specifying specific parameters in the configuration file.
pytest also supports executing specified use cases:

Specify test file path
pytest /path/to/test/file.py
Specify test class
pytest /path/to/test/file.py:TestCase
Specify test method
pytest another.test::TestClass::test_method
Specify test function
pytest /path/to/test/file.py:test_function

4. Test fixtures (Fixtures)

The test fixture of pytest is very different from the styles of unittest, nose, and nose2. It can not only implement setUp and tearDown test pre-test and cleanup logic, as well as many other powerful functions.

4.1 Declaration and use

Test fixtures in pytest are more like test resources. You only need to define a fixture and then use it directly in your test cases. Thanks to the dependency injection mechanism of pytest, you do not need to display the import in the form of from xx import xx. You only need to specify the parameters with the same name in the parameters of the test function, such as :

import pytest
 
 
@pytest.fixture
def smtp_connection():
    import smtplib
 
    return smtplib.SMTP("smtp.gmail.com", 587, timeout=5)
 
 
def test_ehlo(smtp_connection):
    response, msg = smtp_connection.ehlo()
    assert response == 250

In the above example, a test fixture smtp_connection is defined, and a parameter with the same name is defined in the signature of the test function test_ehlo, then the pytest framework will automatically inject the variable .

4.2 Sharing

In pytest, the same test fixture can be shared by multiple test cases in multiple test files. Just define the conftest.py file in the package and write the definition of the test fixture in the file. Then all test cases of all modules in the package can use the tests defined in conftest.py Clamp.

For example, if a test fixture is defined in test_1/conftest.py in the following file structure, then test_a.py and test_b.py can use the test fixture; test_c.py cannot.

`-- test_1
| |-- conftest.py
| `-- test_a.py
| `-- test_b.py
`-- test_2
    `-- test_c.py
4.3 Effective Level

Both unittest and nose support the effective levels of test prepending and cleaning: test method, test class and test module.

The test fixture of pytest also supports various validation levels and is more abundant. Set by specifying the scope parameter in pytest.fixture:

function – function level, that is, the fixture will be regenerated before calling each test function
class – class level, fixtures will be regenerated before calling each test class
module – module level, before loading each test module, the fixture will be regenerated
package – package level, fixtures will be regenerated before loading each package
session – session level, fixture is only generated once before running all use cases
When we specify the effective level as module level, the example is as follows:

import pytest
import smtplib
 
 
@pytest.fixture(scope="module")
def smtp_connection():
    return smtplib.SMTP("smtp.gmail.com", 587, timeout=5)
4.4 Test Preparation and Cleanup

The test fixture of pytest can also implement test pre-production and cleanup. The two logics are split through the yield statement. The writing method becomes very simple, such as:

import smtplib
importpytest
 
 
@pytest.fixture(scope="module")
def smtp_connection():
    smtp_connection = smtplib.SMTP("smtp.gmail.com", 587, timeout=5)
    yield smtp_connection # provide the fixture value
    print("teardown smtp")
    smtp_connection.close()

In the above example, yield smtp_connection and the preceding statements are equivalent to the test preamble, and the prepared test resource smtp_connection is returned through yield; while the following statements will end at the end of the use case execution (to be precise, it is the declaration cycle of the effective level of the test fixture Executed after the end), which is equivalent to test cleanup.

If the process of generating test resources (such as smtp_connection in the example) supports the with statement, it can also be written in a simpler form:

@pytest.fixture(scope="module")
def smtp_connection():
    with smtplib.SMTP("smtp.gmail.com", 587, timeout=5) as smtp_connection:
        yield smtp_connection # provide the fixture value

In addition to the functions introduced in this article, pytest‘s test fixtures also have more advanced methods such as parameterized fixtures, factory fixtures, using fixtures in fixtures, etc. For details, please read “pytest fixtures: explicit, modular, scalable”.

5. Skip testing and expected failure

In addition to supporting the skip testing and expected failure methods of unittest and nosetest, pytest also provides corresponding methods in pytest.mark:

Skip tests directly through the skip decorator or pytest.skip function
Conditionally skip tests via skipif
Expect test failure via xfail
Examples are as follows:

@pytest.mark.skip(reason="no way of currently testing this")
def test_mark_skip():
    ...
 
def test_skip():
    if not valid_config():
        pytest.skip("unsupported configuration")
 
@pytest.mark.skipif(sys.version_info < (3, 6), reason="requires python3.6 or higher")
def test_mark_skip_if():
    ...
 
@pytest.mark.xfail
def test_mark_xfail():
    ...

For more tips on skipping tests and expecting failure, see “Skip and xfail: dealing with tests that cannot succeed”

6. Sub-test/parameterized test

In addition to supporting TestCase.subTest in unittest, pytest also supports a more flexible way of writing subtests, which is parameterized testing, implemented through the pytest.mark.parametrize decorator.

In the following example, defining a test_eval test function and specifying 3 sets of parameters through the pytest.mark.parametrize decorator will generate 3 subtests:

@pytest.mark.parametrize("test_input,expected", [("3 + 5", 8), ("2 + 4", 6), ("6*9", 42)])
def test_eval(test_input, expected):
    assert eval(test_input) == expected

In the example, the last set of parameters is deliberately caused to fail. You can see rich test result output by running the test case:

========================================= test session starts === ======================================
platform darwin -- Python 3.7.1, pytest-4.0.1, py-1.7.0, pluggy-0.8.0
rootdir: /Users/prodesire/projects/tests, inifile:
plugins: cov-2.6.0
collected 3 items
 
test.py ..F [100%]
 
============================================== FAILURES === ============================================
____________________________________________ test_eval[6*9-42] ____________________________________________
 
test_input = '6*9', expected = 42
 
    @pytest.mark.parametrize("test_input,expected", [("3 + 5", 8), ("2 + 4", 6), ("6*9", 42)])
    def test_eval(test_input, expected):
> assert eval(test_input) == expected
E AssertionError: assert 54 == 42
E + where 54 = eval('6*9')
 
test.py:6: AssertionError
================================= 1 failed, 2 passed in 0.09 seconds ========= =========================

If we change the parameters to pytest.param, we can also have more advanced gameplay. For example, we know that the last set of parameters failed, so we mark it as xfail:

@pytest.mark.parametrize(
    "test_input,expected",
    [("3 + 5", 8), ("2 + 4", 6), pytest.param("6*9", 42, marks=pytest.mark.xfail)],
)
def test_eval(test_input, expected):
    assert eval(test_input) == expected

If the values of multiple parameters of the test function want to be arranged and combined with each other, we can write like this:

@pytest.mark.parametrize("x", [0, 1])
@pytest.mark.parametrize("y", [2, 3])
def test_foo(x, y):
    pass

In the above example, x=0/y=2, x=1/y=2, and x=0/y=3 and x=1/y=3 are brought into the test function and executed as four test cases.

7. Test result output

The test result output of pytest is richer than that of unittest and nose. Its advantages are:

  • Highlight output, pass or fail will be distinguished by different colors
  • Richer context information, automatically output code context and variable information
  • Test progress display
  • The test result output layout is more friendly and easy to read

8. Plug-in system

pytest‘s plug-ins are very rich and plug-and-play, so users do not need to write additional code. For information on using plugins, see “Installing and Using plugins”.

In addition, thanks to pytest’s good architectural design and hook mechanism, its plug-in writing has become easy to get started. For information on writing plugins, see “Writing plugins”.

9. Summary

We list a horizontal comparison table to summarize the similarities and differences of these unit testing frameworks:

Python’s unit testing framework seems to have a wide variety, but in fact it has evolved from generation to generation, and there are traces to follow. By grasping its characteristics and combining the usage scenarios, you can easily make a choice.

If you don’t want to install or allow third-party libraries, then unittest is the best and only option. On the contrary, pytest is undoubtedly the best choice. Many Python open source projects (such as the famous requests) use pytest as the unit testing framework. Even nose2’s official documentation recommends that everyone use pytest. How admirable!

Finally, I would like to thank everyone who read my article carefully. Reciprocity is always necessary. Although it is not a very valuable thing, if you can use it, you can just take it away:

This information should be the most comprehensive and complete preparation warehouse for [software testing] friends. This warehouse has also accompanied tens of thousands of test engineers through the most difficult journey. I hope it can also help you!