allure adds logs and failure screenshots (using appium for Android testing)

allure adds logs and failure screenshots

Use the python run_all_case.py command to run pytest and then generate the report through allure.

1. allure description

Allure is a lightweight and very flexible open source test reporting framework. It supports most testing frameworks, such as TestNG, Pytest, JUint, etc. It’s simple to use and easy to integrate.

1.1 How to use

# if __name__ == '__main__':: This is a common pattern in Python.
# If this file is run directly (i.e. you open it and run it), then the __name__ variable will be set to '__main__'.
# is the entrance to the python main function.
if __name__ == '__main__':
    # Used to run test cases and generate allure report content.
    pytest.main(["-vqs", "./xxx.py", "--alluredir=temp/html"])
    
    # Use the system function of the os module to execute a shell command. Generate an allure web report
    # "allure generate": Used to generate an Allure report from test run results.
    # "./temp/html": Specifies the parameter of the input directory, which means generating the Allure report from the result file in the "./temp/html" directory.
    # "-o ./report": Specifies the parameters of the output directory, indicating that the generated Allure report will be stored in the "./report" directory.
    # "--clean": This is an option parameter used to clear previously generated Allure report files.
    os.system("allure generate ./temp/html -o ./report --clean")
    
    # Open this allure web report
    os.system('allure open ./report')

1.2 Report Interpretation

[Picture]

  • Overview: Overview, showing use case execution status, severity distribution, environment information, etc.
  • Categories: Categories, classified by use case execution results, exception errors and failure errors.
  • Suites: Suites, classified by test case suites, directory -> test files -> test classes -> test methods.
  • Graphs: Charts showing use case execution distribution, status, severity, duration, duration trend, retry trend, category trend, and overall trend.
  • Timeline: Timeline, showing the time consumption of the use case, specific to the execution of the use case at each time point.
  • Behaviors: Behaviors, classified by use case behavior (displayed in the form of marked text, the use case needs to add allure related decorators).
  • Package: Package, classified by directory form, showing the execution status of different directory use cases.

1.3 allure related decorators

Usage Parameter value Parameter description
@allure.epic() epic description The concept in agile defines epic, followed by feature
@allure.feature() Module name Description of the function point, followed by story
@allure.story User story User story, below is title
@allure.title (title of the use case) Title of the test case Rename the name of the html report
@allure.testcase() Link to the test case Address Corresponds to the case in the functional test case system
@allure.issue() Defect Corresponding link in the defect management system
@allure.description Use case description Test case description
@allure.step() Operation steps Steps of test cases
@allure.severity() Use case level blocker, critical, normal, minor, trivial
@allure.link( ) Link Define a link to display in the report
@allure.attachment() Attachments Add attachments to report

2. pytest instructions

pytest is a full-featured Python testing tool that helps you write better programs. It is similar to the unittest testing framework that comes with Python, but pytest is more concise and efficient to use, and is compatible with the unittest framework. pytest supports simple unit testing and complex functional testing. It can be combined with requests to implement interface testing, and combined with selenium and appium to implement automated functional testing.

# "--clean-alluredir" clears the last record and runs "xxx.py"
# Output the running results to "temp/html"
# -v, --verbose Increase verbosity | Output detailed logs
# -q, --quiet Decrease verbosity | This parameter option is the opposite of -v and simplifies the output.
# -s Shortcut for --capture=no | Allow terminal output such as print while the test is running
pytest.main(["-vqs", "xxx.py", "--alluredir=temp/html", "--clean-alluredir"])

# The allure decorator mentioned above
@allure.feature("phone_status")
@allure.story("get cpu info")

# This is the defined test function.
# All pytest test functions should start with "test_".
# In this function you should write the actual test code.
def test_phone_status(self):

3. allure adding failure log

3.1 pytest.ini file explanation

pytest.ini is the configuration file that comes with pytest itself. Its main functions are: formulating use case discovery rules, defining labels, configuring log information, and other pytest runtime parameters can be configured in this configuration file to simplify the running complexity.

Introduction: Execute the program by reading the pytest.ini global configuration file
1) If pytest.ini has this parameter value, when executing, first read the parameters in the configuration file;
2) If not, take it from elsewhere (in the main function/command line).
The pytest.ini global configuration file is the core configuration file of the pytest unit testing framework.
Function: pytest.ini can change the default behavior of pytest
Location: Generally placed in the root directory of the project (i.e. under the top-level folder of the current project)
Name: pytest.ini, you cannot use any Chinese symbols, including Chinese characters, spaces, quotation marks, colons, etc.
Create a new configuration file and right-click (New->File->pytest.ini)
Encoding format: GBK or ANSI, you can use notepad + + to modify the encoding format
Running rules: Whether running in main function mode or command line mode, this global configuration file will be read.
This content cannot be displayed outside Feishu documents at the moment.

# Record output methods for different debugging levels
logging.debug('debug level, generally used to print some debugging information, the lowest level')
logging.info('info level, generally used to print some normal operation information')
logging.warning('waring level, generally used to print warning messages')
logging.error('error level, generally used to print some error information')
logging.critical('critical level, generally used to print some fatal error messages, the highest level')

# The two print function calls in this code output messages to standard output (stdout) and standard error (stderr) respectively.
print('Message outputted to stdout')
print('Message outputted to stderr', file=sys.stderr)

The format is generally fixed, and it is recommended to delete the Chinese characters. You can use the pytest –help command to view the setting options of pytest.ini. The processor decides whether to output to the terminal or to a file.
Two logging output locations are redirected here, so that both the console and allure reports have logging logs.
For print output, it can only be output to the console or the allure report. Use -s below to make the print output to the console instead of the report. Remove -s to generate the print log in the allure report.
The debug level log is the most detailed log in Python. The following settings will cause debug, info and other logs to be included in the console and reports.

[pytest]
; Turn on real-time log monitoring on the command line interface (CLI).
log_cli = true

; The logs displayed on the command line are debug and above.
log_cli_level = debug
; This is an assignment statement for command line options. -v -s indicates outputting detailed logs including print, and --alluredir ./temp/html is the output directory.
; Use -s here to print output to the console, otherwise output to allure,
addopts = -v -s --alluredir ./temp/html
; log_cli_format is the same format rule as log_date_format. If there is no such line, log_date_format will be used by default. There may be encoding abnormalities.
log_cli_format = %(asctime)s [%(levelname)s] %(message)s (%(filename)s:%(lineno)s)

;The log displayed in the file is above debug
log_file_level = debug
log_format = %(asctime) s [%(levelname) 5s] %(message)s [%(filename) s:%(lineno)-4s]
log_date_format = %Y-%m-%d %H:%M:%S

The effect of removing -s is as shown in the figure. If -s is added, there will be only the log file:

4. Screenshot of failure to add allure

4.1 Explanation of conftest.py file

This file is mainly used for global use case sharing, such as parameter sharing, test fixture definition, etc. Some initialization operations are usually placed in this file. The file name cannot be changed. There can be multiple in the entire project, but It needs to be in different directory levels, and its application scope is also different.

Use the pytest_runtest_makereport hook function to obtain the results of the case after the case execution is completed. If the case result fails, take a screenshot. Using driver screenshots to obtain the driver will cause relatively more changes (there are methods on the Internet that use the Feature tag to pass in the driver). Here we use adb to take system screenshots, then pull them to the project directory, and finally display them in the report.

# @pytest.hookimpl(tryfirst=True, hookwrapper=True) is a decorator,
# It tells pytest to register this function as a plugin hook implementation and ensure that this hook is executed before other test cases.
# tryfirst=True means that this hook is executed before the test function is executed, hookwrapper=True means that this hook can wrap other hooks.
@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    """
  This hook is used to take screenshots when the example is wrong.
    :param item: test case object
    :param call: test steps of test case
            The report report returned after executing the regular hook function has a property called report.when
          when=’call’ means returning the execution result of call
          when='setup' means returning the result of setup
    """
    outcome = yield # Creates a generator, and the yield keyword turns the function into a generator.
    rep = outcome.get_result() # Get the return result of the generator.
    # The when attribute may be "call", "setup" or "teardown", which are respectively for the execution link, environment initialization link and environment cleanup link of the application example.
    # The code focuses on the execution phase (i.e. when == "call") and the use case execution fails (i.e. rep.failed) or skips the part.
    if rep.when in ('setup', 'call') and rep.outcome in ('failed', 'skipped'):
-------------------------------------------------- -------------------------------------------------- --------
# Use driver mode to take screenshots

        if hasattr(_driver, "get_screenshot_as_png"):
            with allure.step('Add failure screenshot...'):
                allure.attach(_driver.get_screenshot_as_png(), "Failed screenshot", allure.attachment_type.PNG)

        _driver.quit()
-------------------------------------------------- -------------------------------------------------- --------
# Use adb to take screenshots
        formatted_date = datetime.now().strftime("%Y_%m_%d_%H_%M_%S")
        image_path = f"/sdcard/{<!-- -->formatted_date}.png"

        # Create a folder to store the pulled images
        output_dir = 'pulled_images'
        if not os.path.exists(output_dir):
            os.makedirs(output_dir)

        # screenshot
        os.popen(f"adb shell screencap -p {<!-- -->image_path}")
        time.sleep(1)

        # Extract the screenshot
        os.popen(f"adb pull {<!-- -->image_path} {<!-- -->output_dir}")
        time.sleep(1)

        # Delete screenshots on device
        os.popen(f"adb shell rm {<!-- -->image_path}")

        print(os.getcwd()) #Print the current working directory
        if os.path.exists(f"{<!-- -->output_dir}/{<!-- -->formatted_date}.png"):
            with allure.step('Add failure screenshot...'):
                allure.attach.file(f"{<!-- -->output_dir}/{<!-- -->formatted_date}.png", "Failed screenshot", allure.attachment_type.PNG)
        else:
            print("Screenshot failed")

The effect is as shown in the figure:

Reference

  • The allure tool of pytest (principle) is integrated with the pytest test framework
  • Run pytest – command line arguments
  • Automated key data recording (log logs, allure reports, exception screenshots)
  • Detailed explanation of pytest.ini configuration file [pytest series 12]
  • pytest implements screenshots of abnormal use cases and views them in the allure report
  • python-3.x – How to append Pytest logs to Allure Report