Probably the most popular third-party open source testing framework in Python, pytest!

1. Introduction

This article is the third in “Let’s talk about Python’s unit testing framework”. The first two articles introduced the standard library unittest and the third-party unit testing framework nose respectively. As the last article in this series, the finale is the most popular third-party unit testing framework in the Python world: pytest.

It has the following main features:

  • assert Output detailed information when assertion fails (no need to remember self.assert* names anymore)
  • Automatic discovery of test modules and functions
  • Modular fixtures to manage various testing resources
  • Fully compatible with unittest and basically compatible with nose
  • Very rich plug-in system, with more than 315 third-party plug-ins, and a prosperous community

Like the previous introduction of unittest and nose, we will introduce the features of pytest from the following aspects.

2. Writing use cases

Like nose, pytest supports test cases in the form of functions and test classes. The biggest difference is that you can use the assert statement to make assertions without worrying about it missing in nose or unittest Questions with detailed contextual information.

For example, in the following test example, the assertion in test_upper is deliberately made to fail:

import pytest

def test_upper():
    assert 'foo'.upper() == 'FOO1'

class TestClass:
    def test_one(self):
        x = "this"
        assert "h" in x

    def test_two(self):
        x = "hello"
        with pytest.raises(TypeError):
            x + []

When using pytest to execute the use case, it will output detailed (and multi-color) context information:

================================== test session starts ========= ==========================
platform darwin -- Python 3.7.1, pytest-4.0.1, py-1.7.0, pluggy-0.8.0
rootdir: /Users/prodesire/projects/tests, inifile:
plugins: cov-2.6.0
collected 3 items

test.py F.. [100%]

======================================== FAILURES ========= ================================
________________________________________ test_upper ________________________________________

    def test_upper():
> assert 'foo'.upper() == 'FOO1'
E AssertionError: assert 'FOO' == 'FOO1'
E-FOO
E+FOO1
E ? +

test.py:4: AssertionError
=========================== 1 failed, 2 passed in 0.08 seconds =============== =============

It is not difficult to see that pytest not only outputs the test code context, but also outputs the information of the measured variable value. Compared with nose and unittest, pytest allows users to write test cases in a simpler way and get a richer and more friendly test. result.

Now I have also found a lot of test friends and created a communication group to share technology, sharing a lot of technical documents and video tutorials we collected.
If you don’t want to experience the feeling of not being able to find resources when studying on your own, having no one to answer your questions, and persisting for a few days before giving up.
You can join us to communicate. And there are many technical experts who have made certain achievements in automation, performance, security, test development, etc.
Share their experience, and also share many live lectures and technical salons
You can learn for free! Focus on it! Open source! ! !
QQ group number: 110685036

3. Use case discovery and execution

The use case discovery and execution capabilities supported by unittest and nose are also supported by pytest. pytest supports automatic (recursive) discovery of use cases:

  • By default, all test case files in the current directory that match test_*.py or *_test.py will be found, and the test functions starting with test or Test methods starting with test in test classes starting with Test
  • Use the pytest command
  • Like nose2, the name pattern (fuzzy matching) of use case files, classes and functions can be configured by specifying specific parameters in the configuration file

pytest also supports executing specified use cases:

  • Specify test file path
  • pytest /path/to/test/file.py
  • Specify test class
  • pytest /path/to/test/file.py:TestCase
  • Specify test method
  • pytest another.test::TestClass::test_method
  • Specify test function
  • pytest /path/to/test/file.py:test_function

4. Test fixtures (Fixtures)

The test fixture of pytest is very different from the styles of unittest, nose, and nose2. It can not only implement setUp and tearDown test pre-testing and cleanup logic, as well as many other powerful functions.

4.1 Declaration and use

Test fixtures in pytest are more like test resources. You only need to define a fixture and then use it directly in your test cases. Thanks to the dependency injection mechanism of pytest, you do not need to display the import in the form of from xx import xx. You only need to specify the parameters with the same name in the parameters of the test function, such as :

import pytest


@pytest.fixture
def smtp_connection():
    import smtplib

    return smtplib.SMTP("smtp.gmail.com", 587, timeout=5)


def test_ehlo(smtp_connection):
    response, msg = smtp_connection.ehlo()
    assert response == 250

In the above example, a test fixture smtp_connection is defined, and a parameter with the same name is defined in the signature of the test function test_ehlo, then the pytest framework will automatically inject the variable .

4.2 Sharing

In pytest, the same test fixture can be shared by multiple test cases in multiple test files. Just define the conftest.py file in the package and write the definition of the test fixture in the file, then all test cases of all modules in the package can be usedconftest.py.

For example, if a test fixture is defined in test_1/conftest.py with the following file structure, then test_a.py and test_b.py can use this Test fixture; test_c.py cannot be used.

`-- test_1
| |-- conftest.py
| `-- test_a.py
| `-- test_b.py
`-- test_2
    `-- test_c.py

4.3 Effective Level

Both unittest and nose support the effective levels of test prepending and cleaning: test method, test class and test module.

The test fixture of pytest also supports various validation levels and is more abundant. Set by specifying the scope parameter in pytest.fixture:

  • function – function level, that is, the fixture will be regenerated before calling each test function
  • class – class level, fixtures will be regenerated before calling each test class
  • module – module level, before loading each test module, the fixture will be regenerated
  • package – package level, fixtures will be regenerated before loading each package
  • session – session level, fixture is only generated once before running all use cases

When we specify the effective level as module level, the example is as follows:

import pytest
import smtplib


@pytest.fixture(scope="module")
def smtp_connection():
    return smtplib.SMTP("smtp.gmail.com", 587, timeout=5)

4.4 Test Preparation and Cleanup

The test fixture of pytest can also implement test pre-preparation and cleanup. The two logics are split through the yield statement. The writing method becomes very simple, such as:

import smtplib
importpytest


@pytest.fixture(scope="module")
def smtp_connection():
    smtp_connection = smtplib.SMTP("smtp.gmail.com", 587, timeout=5)
    yield smtp_connection # provide the fixture value
    print("teardown smtp")
    smtp_connection.close()

In the above example, yield smtp_connection and the preceding statements are equivalent to the test prefix, and the prepared test resource smtp_connection is returned through yield; and the following The statement will be executed after the use case execution ends (to be precise, the end of the declaration period of the effective level of the test fixture), which is equivalent to test cleanup.

If the process of generating test resources (such as smtp_connection in the example) supports the with statement, it can also be written in a simpler form:

@pytest.fixture(scope="module")
def smtp_connection():
    with smtplib.SMTP("smtp.gmail.com", 587, timeout=5) as smtp_connection:
        yield smtp_connection # provide the fixture value

In addition to the functions introduced in this article, pytest‘s test fixtures also have more advanced methods such as parameterized fixtures, factory fixtures, using fixtures in fixtures, etc. For details, please read “pytest fixtures: explicit, modular, scalable”.

5. Skip testing and expected failure

In addition to supporting the skip testing and expected failure methods of unittest and nosetest, pytest also supports pytest.mark Corresponding methods are provided in:

  • Skip tests directly through the skip decorator or pytest.skip function
  • Conditionally skip tests via skipif
  • Expect test failure via xfail

Examples are as follows:

@pytest.mark.skip(reason="no way of currently testing this")
def test_mark_skip():
    ...

def test_skip():
    if not valid_config():
        pytest.skip("unsupported configuration")

@pytest.mark.skipif(sys.version_info < (3, 6), reason="requires python3.6 or higher")
def test_mark_skip_if():
    ...

@pytest.mark.xfail
def test_mark_xfail():
    ...

For more tips on skipping tests and expecting failure, see “Skip and xfail: dealing with tests that cannot succeed”

6. Sub-test/parameterized test

In addition to supporting TestCase.subTest in unittest, pytest also supports a more flexible way of writing subtests, which is the parameter Testing is implemented through the pytest.mark.parametrize decorator.

In the following example, defining a test_eval test function and specifying 3 sets of parameters through the pytest.mark.parametrize decorator will generate 3 subtests:

@pytest.mark.parametrize("test_input,expected", [("3 + 5", 8), ("2 + 4", 6), ("6*9", 42)])
def test_eval(test_input, expected):
    assert eval(test_input) == expected

In the example, the last set of parameters is deliberately caused to fail. You can see rich test result output by running the test case:

========================================= test session starts === ======================================
platform darwin -- Python 3.7.1, pytest-4.0.1, py-1.7.0, pluggy-0.8.0
rootdir: /Users/prodesire/projects/tests, inifile:
plugins: cov-2.6.0
collected 3 items

test.py ..F [100%]

============================================== FAILURES === ============================================
____________________________________________ test_eval[6*9-42] ____________________________________________

test_input = '6*9', expected = 42

    @pytest.mark.parametrize("test_input,expected", [("3 + 5", 8), ("2 + 4", 6), ("6*9", 42)])
    def test_eval(test_input, expected):
> assert eval(test_input) == expected
E AssertionError: assert 54 == 42
E + where 54 = eval('6*9')

test.py:6: AssertionError
================================= 1 failed, 2 passed in 0.09 seconds ========= =========================

If we change the parameters to pytest.param, we can also have more advanced gameplay. For example, we know that the last set of parameters failed, so we mark it as xfail:

@pytest.mark.parametrize(
    "test_input,expected",
    [("3 + 5", 8), ("2 + 4", 6), pytest.param("6*9", 42, marks=pytest.mark.xfail)],
)
def test_eval(test_input, expected):
    assert eval(test_input) == expected

If the values of multiple parameters of the test function want to be arranged and combined with each other, we can write like this:

@pytest.mark.parametrize("x", [0, 1])
@pytest.mark.parametrize("y", [2, 3])
def test_foo(x, y):
    pass

In the above example, x=0/y=2, x=1/y=2, and x=0/y=3 and x=1/y=3 are brought into the test function and executed as four test cases.

7. Test result output

The test result output of pytest is richer than that of unittest and nose. Its advantages are:

  • Highlight output, pass or fail will be distinguished by different colors
  • Richer context information, automatically output code context and variable information
  • Test progress display
  • The test result output layout is more friendly and easy to read

8. Plug-in system

pytest‘s plug-ins are very rich and plug-and-play, so users do not need to write additional code. For information on using plugins, see “Installing and Using plugins”.

In addition, thanks to pytest‘s good architectural design and hook mechanism, its plug-in writing has become easy to get started. For information on writing plugins, see “Writing plugins”.

9. Summary

The three introductions to the Python testing framework are coming to an end here. Having written so much, readers may be tired from reading it. We might as well list a horizontal comparison table to summarize the similarities and differences of these unit testing frameworks:

Python’s unit testing framework seems to have a wide variety, but in fact it has evolved from generation to generation, and there are traces to follow. By grasping its characteristics and combining the usage scenarios, you can easily make a choice.

If you don’t want to install or allow third-party libraries, then unittest is the best and only option. On the contrary, pytest is undoubtedly the best choice. Many Python open source projects (such as the famous requests) use pytest as the unit testing framework. Even nose2 recommends everyone to use pytest in its official documentation. How admirable is this!

Finally, I would like to thank everyone who has read my article carefully. Looking at the increase in fans and attention, there is always some courtesy. Although it is not a very valuable thing, if you can use it, you can take it directly!

Software testing interview document

We must study to find a high-paying job. The following interview questions are the latest interview materials from first-tier Internet companies such as Alibaba, Tencent, Byte, etc., and some Byte bosses have given authoritative answers. After finishing this set I believe everyone can find a satisfactory job based on the interview information.

The knowledge points of the article match the official knowledge files, and you can further learn relevant knowledge. Python entry skill treeHomepageOverview 387029 people are learning the system