Use jsoup to crawl and parse web page data

If you think the content of this blog is helpful or inspiring to you, please follow my blog to get the latest technical articles and tutorials as soon as possible. At the same time, you are also welcome to leave a message in the comment area to share your thoughts and suggestions. Thank you for your support!

1. What is jsoup, its functions and advantages

Jsoup is a Java-based HTML parser that can easily grab and parse data from web pages. Its main role is to help developers process HTML documents and extract the required data or information.

The main advantages of Jsoup are as follows:

1. Ease of use: Jsoup provides an API similar to jQuery, making processing HTML documents very simple and easy to understand.
2. Good compatibility: Jsoup can parse various types of documents such as HTML and XML, and supports standard CSS selectors, making it easier to select elements.
3. Support modification: Jsoup can not only parse HTML documents, but also modify and manipulate them. For example, you can use Jsoup to modify elements, attributes, and text in web pages.
4. Efficient performance: Jsoup’s code is concise and uses efficient algorithms and data structures, so it has good performance and low memory consumption.
5. Open source and free: Jsoup is a completely open source software that can be used and modified for free, and the community supports it well, and you can get help if you encounter problems.

Jsoup is a powerful and easy-to-use HTML parser, suitable for various scenarios, such as data crawling, data mining, data analysis, etc.

2. How to use jsoup to grab HTML pages

1. Reference jsoup dependencies: First, you need to reference jsoup dependencies in the project. You can add the following code to the project’s pom.xml file:

<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.14.3</version>
</dependency>

Or download the jar package of jsoup and add it to the classpath of the project.

1. Create a connection: Use the Jsoup.connect(url) method to create a connection object.
2. Get the HTML document: Use the get() or post() method of the connection object to get the string form of the HTML document.
3. Parsing HTML documents: Use the Jsoup.parse(html) method to parse the HTML document string into a Document object, and then you can use the API provided by Jsoup to select, traverse, and modify elements.

The following is a simple Java code example that demonstrates how to use jsoup to grab a web page and print its title and links:

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;

public class JsoupDemo {
    public static void main(String[] args) {
        String url = "https://www.baidu.com/";
        try {
            Document doc = Jsoup.connect(url).get();
            String title = doc. title();
            System.out.println("The title of the page is: " + title);
            Elements links = doc. select("a[href]");
            for (Element link : links) {
                String linkHref = link.attr("href");
                String linkText = link. text();
                System.out.println("Link address: " + linkHref);
                System.out.println("Link text: " + linkText);
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

3. Introduction to common APIs for parsing HTML with jsoup

jsoup is a powerful Java library that can be used to parse HTML documents. It provides many common APIs for selecting, traversing, and modifying elements and attributes in HTML documents. Here are a few commonly used APIs:

1. Selector API: Used to select HTML elements based on CSS selector syntax.
2. Attribute (Attribute) API: Used to get, set, and remove attributes of HTML elements.
3. Traversal (Traversal) API: used to traverse the elements in the HTML document.
4. Operation (Manipulation) API: Used to modify elements and attributes in HTML documents.

Next, we will introduce these APIs one by one and give corresponding code examples.

Selector API

The selector API allows you to select HTML elements using CSS selector syntax. It provides some methods, such as ??Document.select()? and ??Elements.select()??, which can be used to select HTML elements.

// Select all p elements according to the CSS selector
Elements elements = doc. select("p");

// Select all elements with class "example" according to the CSS selector
Elements elements = doc. select(".example");

// Select the element with the id "example" according to the CSS selector
Element element = doc. select("#example"). first();

Attribute API

The Attributes API is used to get, set, and remove attributes of HTML elements. You can use the ??Element.attr()? method to get or set the value of a single attribute, or use the ??Element.attributes()?? method to get all attributes.

// Get the attribute value of the element
String href = element.attr("href");

// Set the attribute value of the element
element.attr("href", "http://example.com");

// remove attribute from element
element. removeAttr("href");

// Get all attributes of the element
Attributes attributes = element. attributes();

Traversal API

The Traversal API is used to traverse elements in an HTML document. You can use ??Element.parent()?, ??Element.children()?, ??Element.nextElementSibling()?? and other methods to traverse the elements in the HTML document.

//Get the parent element of the element
Element parentElement = element. parent();

// Get the child elements of the element
Elements childrenElements = element. children();

// Get the next sibling element of the element
Element nextSiblingElement = element.nextElementSibling();

Manipulation API

The Manipulation API is used to modify elements and attributes in HTML documents. You can use ??Element.html()?, ??Element.text()?, ??Element.append()?? and other methods to operate elements and attributes in HTML documents.

// Get the HTML content of the element
String html = element. html();

// Get the text content of the element
String text = element. text();

// Append HTML content to the element
element.append("<p>this is a new paragraph</p>");

// Append text content to the element
element.appendText("This is a new text");

4. Common selectors and usage examples of jsoup

Element selector

Element selector refers to selecting one or some elements. Commonly used element selectors include ??tagname?, ??tagname.class?, ??tagname#id??, etc. For example:

// Select all a elements
Elements links = doc. select("a");

// Select all div elements whose class attribute is news
Elements divs = doc. select("div. news");

// Select the div element whose id attribute is header
Element header = doc.select("div#header").first();

Attribute selector

An attribute selector refers to selecting an element with a certain attribute. Commonly used attribute selectors include ??[attr]?, ??[attr=value]?, ??[attr~=value]??, etc. For example:

// Select all a elements whose href attribute starts with "http"
Elements links = doc. select("a[href^=http]");

// Select all div elements whose class attribute contains news
Elements divs = doc. select("div[class~=news]");

Combination selector

Combined selectors refer to combining multiple selectors together. Commonly used combined selectors include space, greater than sign, plus sign, etc. For example:

// Select the p element in all div elements
Elements paragraphs = doc. select("div p");

// Select all p elements whose immediate children are divs
Elements directParagraphs = doc. select("div > p");

// Select the p element immediately after the div element
Elements adjacentParagraphs = doc. select("div + p");

Pseudo-class selector

A pseudo-class selector refers to selecting elements that do not conform to the syntax of conventional CSS selectors. Commonly used pseudo-class selectors include ??:contains(text)?, ??:empty?, ??:not(selector)??, etc. For example:

// Select all a elements containing "example" text
Elements links = doc. select("a:contains(example)");

// Select all div elements that do not contain child elements
Elements divs = doc. select("div:empty");

// Select all elements that are not div elements
Elements notDivs = doc. select(":not(div)");

5. jsoup handles special characters and encoding issues in HTML pages

Special characters and encoding issues are common pain points when working with HTML pages. Special characters refer to some characters with special meaning in HTML documents, such as <, >, & amp;, etc. These characters need to be escaped to be displayed correctly. The encoding problem is caused by different encoding formats. If the correct encoding process is not performed, problems such as garbled characters will appear. The following describes how to use jsoup to deal with special characters and encoding issues in HTML pages.

Escape special characters

In jsoup, you can use the ??text()?? method to get the text content of the element, this method will automatically escape the special characters in the HTML page. For example:

Element element = doc.select("p").first();
String text = element. text();

In this example, if ??

The ? tag contains special characters, such as ??, ??>?, ?? & amp;?, etc., the ??text()? method will automatically escape them into ?? >? etc.

String escaped = Entities. escape("<p>Hello, world!</p>");
System.out.println(escaped); // output: <p>Hello, world!</p>

Document doc = Jsoup. connect(url)
                .timeout(5000)
                .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64)")
                .get();
doc.outputSettings().charset("GBK");

Document doc = Jsoup. connect(url)
                .timeout(5000)
                .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64)")
                .get();
doc.outputSettings().charset(Charset.forName("ISO-8859-1"));

String url = "https://movie.douban.com/top250";
Document doc = Jsoup. connect(url)
                .timeout(5000)
                .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64)")
                .get();

Elements movieList = doc. select("ol. grid_view li");
for (Element movie : movieList) {
    String title = movie. select("div. hd a span. title"). text();
    String rate = movie. select("div. star span. rating_num"). text();
    String year = movie.select("div.bd p span.year").text();
    String directors = movie. select("div. bd p:first-child"). text();
    String actors = movie. select("div. bd p:nth-child(2)"). text();
    String imgUrl = movie.select("div.pic img").attr("src");
    
    System.out.println(title + " / " + rate + " / " + year + " / " + directors + " / " + actors + " / " + imgUrl);
}

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;

public class DoubanSpider {
    public static void main(String[] args) throws IOException {
        String url = "https://movie.douban.com/top250";
        Document doc = Jsoup. connect(url)
                .timeout(5000)
                .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64)")
                .get();
        Elements movieList = doc. select("ol. grid_view li");
        for (Element movie : movieList) {
            String title = movie. select("div. hd a span. title"). text();
            String rate = movie. select("div. star span. rating_num"). text();
            String year = movie.select("div.bd p span.year").text();
            String directors = movie. select("div. bd p:first-child"). text();
            String actors = movie. select("div. bd p:nth-child(2)"). text();
            String imgUrl = movie.select("div.pic img").attr("src");
            System.out.println(title + " / " + rate + " / " + year + " / " + directors + " / " + actors + " / " + imgUrl);
        }
    }
}

Document doc = Jsoup.connect("https://www.example.com/latest-articles").get();

Elements articles = doc. select("div. article-list ul li");

for (Element article : articles) {
    String title = article. select("h3. title"). text();
    String url = article. select("a. link"). attr("href");
    String date = article. select("span. date"). text();

    // Perform data processing, such as removing HTML tags, date formatting, etc.
    //...
}

// store the processed data in the database
db. save(title, url, date);

ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor();
scheduler.scheduleAtFixedRate(() -> {
    // Send request, parse HTML, process data, store data
    //...
}, 0, 1, TimeUnit.HOURS);

import org.jsoup.nodes.Element;
import org.jsoup.parser.Tag;

public class CustomElement extends Element {
    private static final String TAG_NAME = "custom";
    private static final String ATTRIBUTE_NAME = "attribute1";

    public CustomElement(String baseUri, String text) {
        super(Tag. valueOf(TAG_NAME), baseUri);
        this. text(text);
    }

    public String getAttribute1() {
        return this.attr(ATTRIBUTE_NAME);
    }

    public void setAttribute1(String value) {
        this.attr(ATTRIBUTE_NAME, value);
    }
}

CustomElement custom = new CustomElement("http://example.com", "This is custom text");
custom.setAttribute1("custom attribute value");

Document doc = Jsoup. parse("<html><body><div></div></body></html>");
Element div = doc. select("div"). first();
div.appendChild(custom);

System.out.println(doc.outerHtml());
// output result: <html>
 <head></head>
 <body>
  <div>
   <custom attribute1="custom attribute value">
    This is custom text
   </custom>
  </div>
 </body>
</html>

import org.jsoup.nodes.*;
import org.jsoup.select.NodeVisitor;

public class CustomNodeVisitor implements NodeVisitor {

    @Override
    public void head(Node node, int depth) {
        if (node instanceof Element) {
            Element element = (Element) node;
            System.out.println("Start element: " + element.tagName() + ", depth: " + depth);
        }
    }

    @Override
    public void tail(Node node, int depth) {
        if (node instanceof Element) {
            Element element = (Element) node;
            System.out.println("End element: " + element.tagName() + ", depth: " + depth);
        }
    }
}

Document doc = Jsoup. parse("<html><body><div><p>Paragraph 1</p><p>Paragraph 2</p></div></body></html>");
doc.traverse(new CustomNodeVisitor());