Full link tracking implementation based on Skywalking

In the previous article “Distributed application full-link tracking implementation”, several implementation methods of distributed application full-link tracking were introduced. This article will focus on the full-link implementation based on Skywalking, including the overall architecture and basic concepts and principles of Skywalking. Skywalking environment deployment, SpringBoot and Python integrated Skywalking monitoring implementation, etc.

1. Basic introduction to Skywalking
1.1 Skywalking overall architecture


Skywalking 8.x version architecture diagram

The overall architecture of SkyWalking is logically divided into four parts: probe Agent, platform back-end OAP, storage and UI interface

  • Probe Agent: Responsible for collecting link information from the application and sending it to SkyWalking’s OAP server. It collects Tracing and Metrics data and formats the data into a format suitable for SkyWalking. The probe is installed on the server where the service is located to facilitate data acquisition.
  • Platform back-end OAP: receives the data sent by the probe, uses the analysis engine to integrate the data in the memory, and then stores the data on the corresponding storage medium. OAP supports data aggregation, data analysis, and processes that drive data flow from probes to user interfaces. Analysis includes Skywalking native link tracking and performance metrics as well as third-party sources, including Istio and Envoy telemetry, Zipkin link tracking formatting, and more.
  • Storage: SkyWalking data is stored through an open plug-in interface. Currently supported memories include Elasticsearch, MySQL, ShardingSphere, TiDB, H2, etc.
  • User interface UI: Responsible for providing a console to view links, service indicators, etc. The UI is a highly customized web system based on the interface, allowing users to visually view and manage SkyWalking data.
1.2 Basic concepts of Skywalking
1.2.1 Core concepts in Skywalking

Unlike Prometheus, SkyWalking’s measurement mechanism is built around the following core concepts with a hierarchical structure:

  • Layer: represents an abstract framework in computer science, such as Operating System (OS_LINUX layer), Kubernetes (k8s layer). This layer will be the owner of different services detected from different technologies.
  • Service: Represents a group or set of workloads that provide the same behavior for incoming requests.
  • Service Instance: A single workload in a service group.
  • Endpoint: The service path for incoming requests.
  • Process: Operating system process. In some scenarios, service instance is not a process. For example, a Kubernetes Pod may contain multiple processes.
1.2.2 Indicator flow in Skywalking

Metric names and attributes (labels) are configured by the SkyWalking OAP server based on the data source and OAL and MAL. SkyWalking provides the ability to down-sampling time series indicators and generate data in different time periods (minutes, hours, days). The SkyWalking indicator flow is as follows:

2. Analysis of Skywalking Principle
2.1 Trace implementation in Skywalking

Skywalking implements core concepts such as Trace, Span, Tags, and Logs in OpenTracing. The difference is that a Segment concept is added between the Trace level and the Span level to represent the Span collection within a service instance.

2.1.1 Trace ID

In Skywalking, the global ID consists of three long type fields (part1, part2, part3), which record the ServiceInstanceId, Thread ID and Context generation sequence respectively. The format is as follows:

${ServiceInstanceId}.${Thread ID}.(${timestamp} * 10000 + thread increment sequence ([0, 9999]))
2.1.2 TraceSegment

In SkyWalking, TraceSegment is a concept between Trace and Span. It is a segment of a Trace and can contain multiple Spans. In the microservice architecture, a request will basically involve cross-process (and cross-thread) operations, such as RPC calls, asynchronous execution through MQ, HTTP requests for remote resources, etc. Processing a request requires multiple services. threads. TraceSegment records the execution flow of a request in a thread (i.e. Trace information). By concatenating the TraceSegments associated with the request, you can get the complete Trace corresponding to the request.

The core structure of TraceSegment is shown in the figure, including the following core fields:

  • traceSegmentId (ID type): Globally unique identifier of TraceSegment
  • refs (List type): It points to the parent TraceSegment
  • relatedGlobalTraces (DistributedTraceIds type): Record the Trace ID of the Trace to which the current TraceSegment belongs.
  • spans (List type): all spans contained in the current TraceSegment.
  • ignore (boolean type): The ignore field indicates whether the current TraceSegment is ignored.
2.1.3 Context

Each TraceSegment in SkyWalking is bound one-to-one to a Context object. The Context not only records the context information of the TraceSegment, but also provides functions related to managing the TraceSegment life cycle, creating Span, and cross-process (cross-thread) propagation.

2.2 Trace collection and sending
2.2.1 Context generation and sampling

When the application accesses, if no restrictions are imposed, a complete Trace will be generated for each request. When faced with a large number of business requests, a large amount of Trace data will be generated simultaneously, which will put huge pressure on the network and storage. Therefore, almost all Trace systems support the sampling function. In Skywalking Agent, it is implemented through the SamplingService service. The trySampling() method of SamplingService increments the samplingFactorHolder field. When it increases to the threshold (the default value is 3, which can be modified through the agent.sample_n_per_3_secs configuration), it will return false, indicating that the sampling is lost. This IgnoredTracerContext will be generated. IgnoredTracerContext is an empty Context implementation and will not record Trace information.

2.2.2 Trace collection

When TracingContext closes the last Span through the stopSpan() method, the finish() method will be called to close the corresponding TraceSegment. At the same time, all TracingContextListeners listening to the TracingContext closing event will also be notified. The main function of TraceSegmentServiceClient is to collect TraceSegment when it ends and send it to the back-end OAP cluster.

2.3 Skywalking OAP core architecture

Skywalking OAP adopts a microkernel architecture and uses ModuleManager (component manager) to manage multiple Modules (components). One Module can correspond to multiple ModuleProvider (component service providers). ModuleProvider is the real implementation of the bottom layer of Module.

When the OAP service is started, a Module can only choose to use one ModuleProvider to provide external services. A ModuleProvider may support a very complex large function. A ModuleProvider can contain multiple Services. A Service implements part of the functions of a ModuleProvider. By assembling and integrating multiple Services, the complete functions of the ModuleProvider can be obtained.

3. Skywalking environment deployment

The Skywalking test demo environment is as follows, testing the monitoring implementation of SpringBoot applications and Python programs respectively.

1) Unzip the installation package and use version 9.3.0

# tar -xzvf apache-skywalking-apm-9.3.0.tar.gz
# mv apache-skywalking-apm-bin/ /usr/local/skywalking

2) Modify the OAP configuration file and specify the storage type as MySQL

# vi config/application.yml
cluster:
  selector: ${SW_CLUSTER:standalone}
storage:
  selector: ${SW_STORAGE:mysql}
  mysql:
    properties:
      jdbcUrl: ${SW_JDBC_URL:"jdbc:mysql://192.168.112.121:3306/swtest?rewriteBatchedStatements=true & amp;allowMultiQueries=true"}
      dataSource.user: ${SW_DATA_SOURCE_USER:root}
      dataSource.password: ${SW_DATA_SOURCE_PASSWORD:123456}
      dataSource.cachePrepStmts: ${SW_DATA_SOURCE_CACHE_PREP_STMTS:true}
      dataSource.prepStmtCacheSize: ${SW_DATA_SOURCE_PREP_STMT_CACHE_SQL_SIZE:250}
      dataSource.prepStmtCacheSqlLimit: ${SW_DATA_SOURCE_PREP_STMT_CACHE_SQL_LIMIT:2048}
      dataSource.useServerPrepStmts: ${SW_DATA_SOURCE_USE_SERVER_PREP_STMTS:true}
    metadataQueryMaxSize: ${SW_STORAGE_MYSQL_QUERY_MAX_SIZE:5000}
    maxSizeOfBatchSql: ${SW_STORAGE_MAX_SIZE_OF_BATCH_SQL:2000}
    asyncBatchPersistentPoolSize: ${SW_STORAGE_ASYNC_BATCH_PERSISTENT_POOL_SIZE:4}

3) Configure the Webapp

# vi webapp/application.yml
serverPort: ${SW_SERVER_PORT:-18080}
# Comma seperated list of OAP addresses.
oapServices: ${SW_OAP_ADDRESS:-http://192.168.112.121:12800}

By default, port 8080 is used for access, and it is modified to 18080.

4) Download the MySQL connection jar and copy it to oap-libs

#Download link: mysql-connector-java-8.0.28.jar
https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.28/mysql-connector-java-8.0.28.jar
# cp mysql-connector-java-8.0.28.jar /usr/local/skywalking/oap-libs/

Just place the jar package in the oap-libs directory. If the jar package is not connected, the following exception will be thrown:

java.lang.RuntimeException: Failed to get driver instance for jdbcUrl=jdbc:mysql://localhost:3306/swtest?rewriteBatchedStatements=true & amp;allowMultiQueries=true

5) Connect to mysql to create the application library swtest

# mysql -uroot -p
mysql> create database swtest;

Create the application library swtest in the configuration file in mysql, otherwise an error will be prompted

com.zaxxer.hikari.pool.HikariPool - 574 [main] ERROR [] - HikariPool-1 - Exception during pool initialization.java.sql.SQLSyntaxErrorException: Unknown database 'swtest'

6) Start Skywalking service
Enter the bin directory and execute the startup.sh file to start the SkyWalking platform

cd ../bin
./startup.sh

If the startup is successful, see the log output:

2023-11-04 16:06:18,455 - com.linecorp.armeria.common.util.SystemInfo - 237 [main] INFO [] - hostname: tango-db01 (from /proc/sys/kernel/hostname)
2023-11-04 16:06:20,036 - com.linecorp.armeria.server.Server - 797 [armeria-boss-http-*:12800] INFO [] - Serving HTTP at /0:0:0:0:0 :0:0:0%0:12800 - http://127.0.0.1:12800/

Check the swtest library again. Many tables have been created.

mysql> show tables;
 +------------------------------------------------- ------- +
| Tables_in_swtest |
 +------------------------------------------------- ------- +
| alarm_record |
| alarm_record_tag |
| browser_app_error_rate |

7) Visit the UI page, the port is 18080: http://192.168.112.121:18080/

4. Skywalking full-link monitoring implementation
4.1 SpringBoot application integrated Skywalking monitoring
4.1.1 Environment preparation

1) Create a table in mysql and insert data

mysql> use sw_mysql;
Database changed
mysql> CREATE TABLE `sw_tb` (
    -> `id` int(11) NOT NULL AUTO_INCREMENT,
    -> `username` varchar(50) DEFAULT NULL,
    -> `password` varchar(50) DEFAULT NULL,
    -> PRIMARY KEY (`id`)
    -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8
    -> ;
Query OK, 0 rows affected, 2 warnings (0.03 sec)

mysql> insert into sw_mysql.sw_tb(`username`,`password`) values('Zhang San','AAA'),('Li Si','BBB'),(' Wang Wu','CCC');
Query OK, 3 rows affected (0.00 sec)
Records: 3 Duplicates: 0 Warnings: 0

mysql> select * from sw_mysql.sw_tb;
 + ---- + ---------- + ---------- +
| id | username | password |
 + ---- + ---------- + ---------- +
| 1 | Zhang San | AAA |
| 2 | John Doe | BBB |
| 3 | Wang Wu | CCC |
 + ---- + ---------- + ---------- +
3 rows in set (0.00 sec)
4.1.2 Create SpringBoot project in Eclipse

1) Add the dependency package that introduces Skywalking in the pom.xml file

 <dependency>
            <groupId>org.apache.skywalking</groupId>
            <artifactId>apm-toolkit-trace</artifactId>
            <version>8.3.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.skywalking</groupId>
            <artifactId>apm-toolkit-log4j-2.x</artifactId>
            <version>8.3.0</version>
        </dependency>
<dependency> <!--Introducing log4j2 dependency -->
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>

2) Configure trace

Use apm-toolkit-trace to output traceid information, and modify log4j2.xml to configure the log format. This will write the traceid information into the log for subsequent log collection and centralized analysis.

 <Properties>
        <!-- Formatted output: ?te represents the date, %thread represents the thread name, %-5level: the level displays 5 characters from the left, width %msg: log message, %n is the newline character %logger{36} represents the Logger name Maximum 36 characters -->
        <!--1.File output format-->
        <property name="file_pattern" value="%d{yyyy-MM-dd HH:mm:ss} [%traceId] [%thread] [%-5level] %msg %l%n" />
        <!--2. Console display log format -->
        <!--[%traceId]：Trace id-->
        <!--[%sw_ctx]: Printed as [$serviceName,$instanceName,$traceId,$traceSegmentId,$spanId]: service name, instance name, tracking id, tracking segment id, span id-->
        <property name="console_pattern" value="%red{%d{yyyy-MM-dd HH:mm:ss}} [%traceId] %green{[%thread]} %magenta{[%-5level ]} %cyan{%msg} %l%n"/>
        <!--3.skyWalking collection format-->
        <property name="skyWalking_pattern" value="%msg %l%n"/>

        <!-- Define the path for log storage -->
        <property name="FILE_PATH" value="./log/spring-skywalking/" />
        <property name="FILE_NAME" value="spring-skywalking" />
    </Properties>

3) The control procedure is as follows:

package com.tango.skywalking_mysql.controller;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jdbc.core.BeanPropertyRowMapper;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import com.tango.skywalking_mysql.SkywalkingMysqlApplication;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;

@RestController
@RequestMapping("/demo")
public class DemoController {

private static final Logger logger = LogManager.getLogger(SkywalkingMysqlApplication.class);
    @Autowired
    private JdbcTemplate template;

    @GetMapping("/mysql")
    public String mysql() {
    String result="";
    try {
this.selectById(1);
System.out.println("skywalking-test!");
logger.info("skywalking-test!");
result="MySQL query is normal";
    } catch(Exception e) {
    System.out.println(e);
    logger.error(e);
    result="MySQL query exception";
    }
    \t
        return result;
    }

    public Object selectById(Integer id) {
        return template.queryForObject("SELECT id, username, password FROM sw_tb WHERE id = ?",new BeanPropertyRowMapper<>(Object.class), id);
    }
}

4) Add the following options to the IDE configuration to configure the addresses of the Skywalking agent and service

-javaagent:D:\Skywalking-demo\skywalking-agent\skywalking-agent.jar
-Dskywalking.agent.service_name=skywalking-demo-service
-Dskywalking.collector.backend_service=192.168.112.121:11800

4.1.3 Running SpringBoot service program

1) After running the application, the following information is output, indicating that the agent is successfully started.

INFO 2023-11-04 17:11:44.918 main SkyWalkingAgent : Skywalking agent begin to install transformer ...
Starting application skywalking_mysql
[31m2023-11-04 17:11:52[m [TID: N/A] [32m[main][m [35m[INFO ][m [36mStarting application skywalking_mysql[m com.tango.skywalking_mysql.SkywalkingMysqlApplication.main( SkywalkingMysqlApplication.java:13)

2) At the same time, you can view the traceid information in the log:

[31m2023-11-04 17:17:43[m [TID: e6978740bf3e41bfa6a53760e2d64b8a.44.16988302617370001] [32m[http-nio-18079-exec-1][m [35m[INFO][m [36mskywalking -test! [m com.tango.skywalking_mysql.controller.DemoController.mysql(DemoController.java:29)
skywalking-test!
[31m2023-11-04 17:20:39[m [TID: e6978740bf3e41bfa6a53760e2d64b8a.47.16988304392400001] [32m[http-nio-18079-exec-4][m [35m[INFO][m [36mskywalking-test! [m com .tango.skywalking_mysql.controller.DemoController.mysql(DemoController.java:29)
skywalking-test!

3) Check the 11800 port of the server, there is already a service

[root@tango-DB01 config]# netstat -an|grep 11800
tcp6 0 0 :::11800 :::* LISTEN
tcp6 0 0 192.168.112.121:11800 192.168.112.1:49590 ESTABLISHED

4) Access SpringBoot application service

Each query initiates a business visit: http://192.168.112.1:18079/demo/mysql

4.1.4 Log in to Skywalking to monitor service operation

1) See the new Service on the Skywalking interface: skywalking-demo-service

2) Check the operating performance indicators of the service

3) View the topology of the service. This is an application that accesses the mysql database.

4) View trace information

5) Check the execution status of specific SQL statements

4.2 Python application integrated with Skywalking monitoring
4.2.1 Agent configuration in Python program

1) Introduce Skywalking Agent into the Python program

from skywalking import agent,config

#Configure OAP service information
config.init(agent_collector_backend_services='192.168.112.121:11800', agent_name='skywalking-demo-python')
agent.start()
4.2.2 Run the Python program and monitor the service operation on Skywalking

1) The topology diagram is as shown below, including services and mysql database

2) View Trace information

3) View the specific executed SQL information

4.2.3 Code implementation

1) The complete code implementation is as follows:

# -*- coding: utf-8 -*-

importpymysql
importsys
import time
import codecs
import logging
import base64

from skywalking import agent,config

# Configure logging module
logging.basicConfig(filename='test.log', level=logging.INFO)

def getInfo(sql):
ip="192.168.112.121"
port=3306
user="root"
pwd=base64.decodebytes(b"MTIzNDU2Cg==").strip().decode('utf-8')
dbname="sw_mysql"
\t
info = []
conn = pymysql.connect(host=ip,port=port,user=user,passwd=pwd,database=dbname,charset='utf8')
\t
cursor = conn.cursor()
\t
try:
cursor.execute(sql)
info = cursor.fetchall()
except Exception as e:
print(e)
\t
conn.commit()
\t
cursor.close()
conn.close()
\t
return info

if __name__ == '__main__':
    if sys.version[0] == "2":
        reload(sys)
        sys.setdefaultcoding("utf8")
    
    config.init(agent_collector_backend_services='192.168.112.121:11800', agent_name='skywalking-demo-python')
    agent.start()
    
    exec_sql = "select id,username,password from sw_mysql.sw_tb"
    
    while True:
        get_info = getInfo(exec_sql)
        
        if len(get_info) > 0:
            print(get_info[0])
            print("Success!")
        else:
            print("Error!")
        time.sleep(5)
    agent.stop()

The above is a simple indicator collection implementation based on Skywalking’s full-link tracking. Skywalking is powerful and has topological correlation analysis, distributed tracking and context propagation, alarming and other functions, which are worthy of in-depth study.

References:

  1. https://github.com/apache/skywalking
  2. https://github.com/SkyAPM/document-cn-translation-of-skywalking
  3. https://skywalking.apache.org/docs/skywalking-python
  4. https://blog.csdn.net/weixin_42073629/article/details/106775584