Beautiful Soup4 data analysis and extraction

Beautiful Soup4


Beautiful Soup is a Python library for parsing HTML and XML documents, providing convenient data extraction and manipulation functions. It helps extract required data from web pages such as tags, text content, attributes, etc.

Beautiful Soup will automatically convert input documents to Unicode encoding and output documents to UTF-8 encoding.

Beautiful Soup is relatively simple to use to parse HTML. The API is very user-friendly and supports multiple parsers.



Main features

Flexible parsing method:

Beautiful Soup supports multiple parsers, including the html.parser parser in the Python standard library, as well as third-party libraries lxml and html5lib. In this way, we can choose the appropriate parser for processing according to our needs.

Simple and intuitive API:

Beautiful Soup provides a concise and friendly API that makes parsing HTML documents very easy. We can use concise methods to select specific tags, get the text content in tags, extract attribute values, etc.

Powerful document traversal capabilities:

Through Beautiful Soup, we can traverse the entire HTML document tree, access, modify or delete each node, and even quickly locate the required node through nested selectors.

Tolerance for broken HTML:

Beautiful Soup can handle broken HTML documents, such as automatically correcting unclosed tags, automatically adding missing tags, etc., making data extraction more stable and reliable.


Beautiful relies on the parser when parsing. In addition to supporting the HTML parser in the Python standard library, it also supports some third-party libraries.

Supported parsers:

Parser Usage Advantages Disadvantages
Python standard library BeautifulSoup(markup, “html.parser”) Python’s built-in standard library has moderate execution speed and strong document fault tolerance The document fault tolerance in versions before Python 2.7.3 or 3.2.2) is poor
lxml HTML parser BeautifulSoup( markup, “lxml”) Fast speed and strong document fault tolerance Need to install C language library
lxml XML parser BeautifulSoup(markup, [“lxml-xml”]) BeautifulSoup(markup, “xml”) The only parser that supports XML with fast speed Requires installation C language library
html5lib BeautifulSoup(markup, “html5lib”) The best fault tolerance in the browser way Parse documents to generate documents in HTML5 format Slow and does not rely on external extensions

It can be seen that:

The lxml parser can parse HTML and XML documents and is fast and fault-tolerant, so it is recommended to use it. If you use lxml, then when initializing BeautifulSoup, set the second parameter to lxml.

Basic use of Beautiful Soup4

Installation library

pip install beautifulsoup4

pip install lxml

Create HTML file

By passing a document into the constructor of BeautifulSoup, you can get a document object. You can pass in a string or a file.

Create a test.html file here to create a document object.

<!DOCTYPE html>
<html lang="en">
    <meta charset="UTF-8">
         <li class="class01"><span index="1">H1</span></li>
         <li class="class02"><span index="2" class="span2">H2</span></li>
         <li class="class03"><span index="3">H3</span></li>

Basic usage

# Import module
from bs4 import BeautifulSoup

#Create beautifulsoup object, there are 2 ways to create it
# 1. Created by string, the second parameter is used to specify the parser
# soup = BeautifulSoup("html", 'lxml')
# 2. Create through file
soup = BeautifulSoup(open('test.html'), 'lxml')
#Print output
# print(soup.prettify())

# Get the element tag element, returning the first element by default

# Use .contents or .children to get child elements
# Back to list
# Return iterator

# Get element content

# Get the attribute value of the element. By default, the attribute value of the first element is taken.

The results of the operation are as follows:

<li class="class01"><span index="1">H1</span></li>

', <li class="class01"><span index="1">H1</span></li>, '\
', <li class="class02"><span class="span2"
index="2">H2</span></li>, '\
', <li class="class03"><span index="3">H3</span></li>, '\

<list_iterator object at 0x000001C18E475F10>



Object types of Beautiful Soup4

Beautiful Soup converts complex HTML documents into a complex tree structure. Each node is a Python object. All objects can be summarized into 4 types: Tag, NavigableString, BeautifulSoup, Comment

Tag object

In Beautiful Soup, Tag objects are objects used to represent tag elements in HTML or XML documents. Tag objects are the same as tags in native documents. The Tag object contains information such as the name, attributes, and content of the tag element, and provides a variety of methods to obtain, modify, and operate this information.

soup = BeautifulSoup('<b class="boldest">Extremely bold</b>')
tag = soup.b
# <class 'bs4.element.Tag'>

Common properties and methods of Tag objects


Attributes Description Example
name attribute Get the name of the tag tag_name =
string attribute Get the tag Text content within tag_text = tag.string
attrs attribute Get the attributes of the tag and return it in the form of a dictionary tag_attrs = tag.attrs
get() method Get the attribute value of the tag according to the attribute name attr_value = tag.get(‘attribute_name’)
find() method Find and return the first sub-tag element that meets the conditions child_tag = tag.find(‘tag_name’)
find_all() method Find and return all child tag elements that meet the conditions , returned in the form of a list child_tags = tag.find_all(‘tag_name’)
parent attribute Get the current tag’s Parent tag parent_tag = tag.parent
parents attribute Gets all ancestor tags of the current tag in the form of a generator Return for parent in tag.parents:
children attribute Get the direct child tag of the current tag and return it in the form of a generator for child in tag.children:
next_sibling attribute Get the next sibling tag of the current tag next_sibling_tag = tag.next_sibling
previous_sibling attribute Get the current tag’s Previous sibling tag previous_sibling_tag = tag.previous_sibling

NavigableString object

NavigableString object is a data type in the Beautiful Soup library, used to represent plain text content in HTML or XML documents. It inherits from Python’s basic string type, but has additional functions and features that make it suitable for processing textual content in documents.

Suppose we have the following HTML code snippet:

<p>This is a <b>beautiful</b> day.</p>

Use Beautiful Soup to parse it into a document object:

from bs4 import BeautifulSoup

html = '<p>This is a <b>beautiful</b> day.</p>'
soup = BeautifulSoup(html, 'html.parser')


The content of the label, which is actually a NavigableString object:

p_tag = soup.find('p')
content = p_tag.string
print(content) # Output: This is a
print(type(content)) # Output: <class 'bs4.element.NavigableString'>

You can also perform some operations on NavigableString objects, such as obtaining text content, replacing text, and removing whitespace characters:

# Get text content
text = content.strip()
print(text) # Output: This is a

# Replacement text
print(p_tag) # Output: <p>Hello<b>beautiful</b> day.</p>

# Remove whitespace characters
text_without_spaces = p_tag.string.strip()
print(text_without_spaces) #Output: Hello


Attributes Description Example
string attribute Used to obtain the text content of the NavigableString object text = navigable_string.string
replace_with() method Used to replace the current NavigableString object with another string or object navigable_string.replace_with(new_string)
strip() method Remove whitespace characters at both ends of the string stripped_text = navigable_string.strip()
parent attribute Get The parent node to which the NavigableString object belongs (usually a Tag object) parent_tag = navigable_string.parent
next_sibling attribute Get NavigableString The next sibling node of the object next_sibling = navigable_string.next_sibling
previous_sibling attribute Get the previous sibling node of the NavigableString object previous_sibling = navigable_string.previous_sibling

BeautifulSoup object

The BeautifulSoup object is the core object of the Beautiful Soup library and is used to parse and traverse HTML or XML documents.

Commonly used methods:

find(name, attrs, recursive, string, **kwargs): Find the first matching tag based on the specified tag name, attributes, text content, etc.

find_all(name, attrs, recursive, string, limit, **kwargs): Find all matching tags based on the specified tag name, attributes, text content, etc., and return a list

select(css_selector): Use CSS selector syntax to find matching tags and return a list

prettify(): Output the contents of the entire document in a beautiful way, including tags, text and indentation

has_attr(name): Check whether the current tag has the specified attribute name and return a Boolean value

get_text(): Get the text content of the current label and all sub-labels, and return a string

Common properties:

soup.title: Get the content of the first <title> tag in the document

soup.head: Get the <head> tag in the document

soup.body: Get the <body> tag in the document

soup.find_all('tag'): Get all matching <tag> tags in the document and return a list

soup.text: Get the plain text content in the entire document (remove tags, etc.)

Comment object

Comment object is a special type of object in the Beautiful Soup library, used to represent comment content in HTML or XML documents.

When parsing an HTML or XML document, Beautiful Soup represents the comment content as a Comment object. A comment is a special element in a document that is used to add notes, explanations, or temporarily delete part of the content. Comment objects can be automatically recognized and processed by the parser of the Beautiful Soup library.

Example of processing and accessing annotated content in HTML documents:

from bs4 import BeautifulSoup

# Create an HTML string containing comments
html = "<html><body><!-- This is a comment --> <p>Hello, World!</p></body></html>"

# Parse HTML document
soup = BeautifulSoup(html, 'html.parser')

# Use the type() function to get the type of the annotation object
comment = soup.body.next_sibling
print(type(comment)) # Output <class 'bs4.element.Comment'>

# Use the `.string` attribute to get the comment content
comment_content = comment.string
print(comment_content) # Output This is a comment

Search document tree

Beautiful Soup provides a variety of ways to find and locate elements in HTML documents.

Method selector

Use Beautiful Soup’s find() or find_all() method to select elements by tag name, combining attribute names and attribute values, combining text content, etc.

The difference between the two:

find returns the first element that meets the conditions

find_all returns a list of all elements that meet the criteria

For example: soup.find('div') will return the first div tag element, soup.find_all('a') will return all a tag elements.

1. Find elements by tag or tag list

# Find the element of the first div tag

# Find all a tag elements

# Find all li tags

# Find all a tags and b tags

2. Find elements through regular expressions

# Search for tags starting with sp
import re

3. Find elements by attributes

find = soup.find_all(
         "property name":"value"

# Find the first a tag element with href attribute
soup.find('a', href=True)

# Find all div tag elements with class attributes
soup.find_all('div', class_=True)

4. Find elements by text content

# Find the first element containing "Hello" text content

# Find all elements containing "World" text content

5. Find elements through keyword parameters


6. Mix it up

 'tag name',
   "property name":"value"

CSS selector

Use Beautiful Soup’s select() method to find elements through CSS selectors. CSS selectors are a powerful and flexible way to target elements based on tag names, class names, IDs, attributes, and combinations thereof.

1. Class Selector

To find elements by class name, use the . symbol plus the class name to find elements'.className')

2.ID selector

To find an element by ID, use the # symbol plus the ID to find an element.'#id')

3. Tag selector

Find elements by tag name, use tag names directly to find elements'p')

4.Attribute selector

Find elements by attributes: You can use the format of [attribute name = attribute value] to find elements with specific attributes and attribute values.'[attribute="value"]')'[href=""]')'a[href=""]')

5. Combination selector

Multiple selectors can be combined for more precise searches

# Return all a tag elements within the div element with a specific class name'div.class01 a')'div a')

Associated selection

In the process of element search, sometimes you cannot get the desired node element in one step. You need to select a certain node element, and then use this node as the basis to select its child nodes, parent nodes, sibling nodes, etc.

In Beautiful Soup, you can perform associated selection by using CSS selector syntax. You need to use the select() method of Beautiful Soup, which allows you to use CSS selectors to select elements.

1. Descendant selector (space): You can select all descendant elements under the specified element.

div a /* Select all a elements under the div element */

.container p /* Select all p elements under the element named container */

2. Direct descendant selector (>): You can select the direct descendant elements of the specified element.

div > a /* Select the a element that is a direct descendant of the div element */

.container > p /* Select p elements that are direct descendants of the element named container */

3. Adjacent sibling selector (+): You can select the next sibling element immediately adjacent to the specified element.

h1 + p /* Select the sibling p element immediately after the h1 element */

.container + p /* Select the sibling p element immediately following the element named container */

4. Universal sibling selector (~): can select all subsequent elements at the same level as the specified element.

h1 ~ p /* Select all p elements at the same level as the h1 element */

.container ~ p /* Select all p elements at the same level as the element named container */

5. Mixed selectors: You can combine multiple selectors of different types to select specific elements

div, p /* Select all div elements and all p elements */

.cls1.cls2 /* Select the class name is cls1 and the class name is cls2 */

Traverse the document tree

In Beautiful Soup, traversing the document tree is a common operation for accessing and processing individual nodes of an HTML or XML document

Child nodes and parent nodes of tags

contents: Get all child nodes of Tag and return a list

# Use list index to get an element

children: Get all child nodes of Tag and return a generator

for child in tag.contents:

The parent attribute gets the parent node of the label

parent_tag = tag.parent

Sibling nodes of the tag

You can use the .next_sibling and .previous_sibling properties to get the next or previous sibling node of a label.

next_sibling = tag.next_sibling
previous_sibling = tag.previous_sibling

Recursively traverse the document tree

You can use the .find() and .find_all() methods to recursively search for matching tags in the document tree.

You can use the .descendants generator iterator to traverse all descendant nodes of the document tree.

for tag in soup.find_all('a'):

for descendant in tag.descendants:

Traverse tag attributes

You can use the .attrs attribute to get all attributes of a tag and iterate over them.

for attr in tag.attrs:

———————————END——————- ——–


In this era of big data, how can you keep up with scripting without mastering a programming language? Python, the hottest programming language at the moment, has a bright future! If you also want to keep up with the times and improve yourself, please take a look.

Interested friends will receive a complete set of Python learning materials, including interview questions, resume information, etc. See below for details.

CSDN gift package:The most complete “Python learning materials” on the entire network are given away for free! (Safe link, click with confidence)

1. Python learning routes in all directions

The technical points in all directions of Python have been compiled to form a summary of knowledge points in various fields. Its usefulness is that you can find corresponding learning resources according to the following knowledge points to ensure that you learn more comprehensively.


2. Essential development tools for Python

The tools have been organized for you, and you can get started directly after installation! img

3. Latest Python study notes

When I learn a certain basic and have my own understanding ability, I will read some books or handwritten notes compiled by my seniors. These notes record their understanding of some technical points in detail. These understandings are relatively unique and can be learned. to a different way of thinking.


4. Python video collection

Watch a comprehensive zero-based learning video. Watching videos is the fastest and most effective way to learn. It is easy to get started by following the teacher’s ideas in the video, from basic to in-depth.


5. Practical cases

What you learn on paper is ultimately shallow. You must learn to type along with the video and practice it in order to apply what you have learned into practice. At this time, you can learn from some practical cases.


6. Interview Guide

Resume template

CSDN gift package:The most complete “Python learning materials” on the entire network are given away for free! (Safe link, click with confidence)

If there is any infringement, please contact us for deletion.