Continuous Profiling of Amazon EKS Container Service to Diagnose Application Performance with Pyroscope

3590b0761c042a08bb69a5cb0a430bcb.gif

The current status of Continuous Profiling

In the observable field, Trace, Log, and Metrics serve as the “three pillars” to help engineers more easily gain insight into the internal problems of applications. However, it is often necessary for developers to drill down into the application to find the root cause of the bottleneck. In the “three pillars” of observability, this information is often collected through logs. However, this method is often very time-consuming and lacks enough details to help developers locate application performance issues.

A more effective method is to use profiling technology. Profiling is a dynamic method of analyzing program complexity that aims to collect application system information at runtime to study system health and locate performance hot spots, such as CPU utilization or the frequency and duration of function calls. Through analysis, you can pinpoint which parts of the application consume the most resources or time, thereby optimizing overall system performance. Profiling technology is generally used in the following forms:

  • System tools: Such as Linux’s strace/perf and Solaris’ DTrace. The use of such tools requires a strong C and operating system foundation, and often requires the ability to understand OS-level system calls;

  • Languages Native: Provided through programming language profiling libraries, such as golang’s net/http/pprof and runtime/pprof. Engineers need to introduce these packages into the program and view and analyze them through specialized tools. Amazon CodeGuru Profiler also provides Java/Python language agents to provide profiling for Java and Python applications;

  • Use eBPF: eBPF Profiling uses the idea of Infrastructure to solve application observation problems. eBPF is a very popular technology in the current Linux kernel. Using eBPF profiling can obtain it from the kernel without modifying the code. Stack traces for the entire system (eBPF is useful for much more than profiling).

It should be noted that when using compiled languages such as Golang/Java/C/C++, the eBPF profiler can obtain very similar information to the non-eBPF profiler. But for interpreted languages such as Python, the runtime stack trace cannot be easily accessed from the kernel, and Languages Native has better results in this scenario. In view of the advantages and disadvantages of Languages Native and eBPF, general commercial products will provide access to both methods at the same time.

On the other hand, the original Profiling information is often difficult to read and understand. To solve this problem, Brendan Gregg, the author of “Systems Performance: Enterprise and the Cloud” (Chinese translation “Top of Performance”) invented FlameGraph (Flame Graph), which analyzes the Visualize application stack traces and durations in Profiling in a layer-by-layer manner to intuitively, quickly and accurately identify the most frequently executed and most resource-consuming code paths. Almost all mainstream profiling tools use FlameGraph for visualization. For interpretation of flame graph, please refer to: Performance Tuning Tool: Flame graph.

  • Performance tuning tool: Flame graph:

    https://www.infoq.cn/article/a8kmnxdhbwmzxzsytlga

e6fd3707de86ee581ab1e52359ab337a.png

Profiling alone is often not enough. In modern application scenarios, immutable infrastructure is widely used. Taking Kubernetes as an example, the application often crashes after a fault occurs, the Liveness Probe check fails, and then the Pod is destroyed, and the new The application Pod will replace the destroyed Pod to provide services. If profiling is not performed in time, the application stack call information will be lost as the Pod life cycle terminates. In addition, for problems such as memory overflow and OOM, it is often necessary to compare profiling data at different times to find the problem. Continuous Profiling adds a time dimension to profiling, helping to locate, debug, and repair performance-related issues by understanding changes in program profiling information over time.

How to use Pyroscope with Amazon EKS

Introduction to Pyroscope Architecture

Pyroscope is a company that provides open source Continuous Profiling services. It was acquired by Grafana in March 2023 and integrated Grafana’s own Phlare into Pyroscope. Similar to the technical implementation of the Trace observable pillar, Pyroscope also supports SDK-Instrumentation and Auto-Instrumentation to produce Profiling data, and uses a combination of Push and Pull to collect data, store and display it. This article takes Pyroscope as an example to demonstrate how to use the Pyroscope tool to implement Continuous Profiling of modern applications and gain insight into the performance of modern applications.

d12333d4920d38c12981984549527b96.png

The deployment of Pyroscope is divided into two parts, Pyroscope Server and Client:

  • The Server part mainly collects, processes, stores, and displays the data reported by the Client, and provides API interfaces to the outside world. You can use Grafana to display Pyroscope Profiling data in the form of a flame graph.

  • The client is a Grafana Agent. The Agent can use eBPF technology to collect Profiling and then push it to the Server, or use Agent pull to collect Profiling data directly from the application. In addition to using Agent, users can also use SDK to push the generated Profiling data directly to the server.

Installing Pyroscope in EKS

To use Pyroscope with EKS, you need to install the Pyroscope service first. The installation steps are as follows:

1. Prepare S3 bucket for Pyroscope persistence

Pyroscope supports using S3 to persist data. Pyroscope uses Thanos’ object store client. Since the document does not specify whether IRSA authentication is supported, you can create a separate IAM User for Pyroscope to access the S3 bucket. It is recommended to keep the user’s AK. /SK to avoid leaks and set fine-grained IAM Policy for the S3 bucket used by Pyroscope. Pay attention to replace in the following template.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Pyroscope",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject",
                "s3:GetObjectTagging",
                "s3:PutObjectTagging"
            ],
            "Resource": [
                "arn:aws:s3:::<YOUR-S3-BUCKET>/*",
                "arn:aws:s3:::<YOUR-S3-BUCKET>"
            ]
        }
    ]
}

Swipe left to see more

2.Use the following command to create a kubernetes namespace for Pyroscope

kubectl create namespace pyroscope

3.Install Pyroscope helm repo

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Swipe left to see more

4. Download the Pyroscope template and adjust the deployment parameters. Pyroscope supports distributed deployment and provides a separate minio as the back-end persistent object storage. In the distributed deployment mode, it is not supported to modify the helm template to use S3 as the long-term persistence layer, so the deployment method of the monolithic architecture is used here.

curl -Lo pyroscope-values.yaml \
https://raw.githubusercontent.com/grafana/pyroscope/main/operations/pyroscope/helm/pyroscope/values.yaml

Swipe left to see more

Pyroscope server is a statefulset stateful application. You can adjust the number of Pyroscope server service instances by configuring replicaCount.

pyroscope:
  replicaCount: 3

If running in production, it is recommended to modify the limit and request of resources used by the Pyroscope server Pod.

resources:
    {}

By default, after the Pyroscope ingester module receives Profiling data, it will retain recent data in memory. When the threshold is reached or exceeds 3 hours, Pyroscope will persist the data to block storage. Since S3 can be used for tiering, pv does not Need to set too large.

persistence:
    enabled: True
    accessModes:
      - ReadWriteOnce
    size: 10Gi
    annotations: {}

When object storage is configured, complete data blocks will be uploaded to S3 for persistence. You can use S3 Intelligent-Tiering to reduce this part of the data persistence cost. Pay attention to adjusting the following S3 configuration parameters according to your own environment.

config: |
    storage:
      backend: s3
      s3:
        region: <YOUR-S3-REGION>
        endpoint: s3.<YOUR-S3-REGION>.amazonaws.com
        bucket_name: <YOUR-S3-BUCKET>
        access_key_id: <YOUR-ACCESS-KEY>
        secret_access_key: <YOUR-SECRET-KEY>

Swipe left to see more

By default, Pyroscope sets up minio object storage for long-term data persistence. Use the following settings to turn off the minio service and use S3 directly:

minio:
  enabled: false

Swipe left to see more

5.After modifying the deployment configuration, execute helm install to install the Pyroscope server

helm -n pyroscope install pyroscope grafana/pyroscope --values pyroscope-values.yaml

Swipe left to see more

The command output is roughly as follows:

NAME: pyroscope
LAST DEPLOYED: Tue Sep 5 09:10:31 2023
NAMESPACE: pyroscope
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thanks for deploying Grafana Pyroscope.


# Pyroscope UI & Grafana


Pyroscope database comes with a built-in UI, to access it from your localhost you can use:


```
kubectl --namespace pyroscope port-forward svc/pyroscope 4040:4040
```


You can also use Grafana to explore Pyroscope data.
For that, you'll need to add the Pyroscope data source to your Grafana instance and configure the query URL accordingly.
See https://grafana.com/docs/grafana/latest/datasources/grafana-pyroscope/ for more details.


The in-cluster query URL for the data source in Grafana is:


```
http://pyroscope.pyroscope.svc.cluster.local.:4040
```


# Collecting profiles.




The Grafana Agent has been installed to scrape and discover pprof profiles endpoint via pod annotations.


As an example, to start collecting memory and cpu profile using the 8080 port, add the following annotations to your workload:


```
profiles.grafana.com/memory.scrape: "true"
profiles.grafana.com/memory.port: "8080"
profiles.grafana.com/cpu.scrape: "true"
profiles.grafana.com/cpu.port: "8080"
```




To learn more supported annotations, read our guide https://grafana.com/docs/pyroscope/next/deploy-kubernetes/#optional-scrape-your-own-workloads-profiles


There are various ways to collect profiles from your application depending on your needs.
Follow our guide to setup profiling data collection for your workload:


https://grafana.com/docs/pyroscope/next/configure-client/

Swipe left to see more

Pay attention to the output information

http://pyroscope.pyroscope.svc.cluster.local.:4040

Required for provisioning Grafana Datasource.

6.Pyroscope uses Grafana for data query and display. By default, Grafana does not support flame graphs. You need to use the following parameters to enable this feature when installing helm.

helm upgrade -n pyroscope --install grafana grafana/grafana \
  --set image.repository=grafana/grafana \
  --set image.tag=main \
  --set env.GF_FEATURE_TOGGLES_ENABLE=flameGraph \
  --set env.GF_AUTH_ANONYMOUS_ENABLED=true \
  --set env.GF_AUTH_ANONYMOUS_ORG_ROLE=Admin \
  --set env.GF_DIAGNOSTICS_PROFILING_ENABLED=true \
  --set env.GF_DIAGNOSTICS_PROFILING_ADDR=0.0.0.0 \
  --set env.GF_DIAGNOSTICS_PROFILING_PORT=6060 \
  --set-string 'podAnnotations.pyroscope\.grafana\.com/scrape=true' \
  --set-string 'podAnnotations.pyroscope\.grafana\.com/port=6060'

Swipe left to see more

7. Grafana installed

Installing Grafana will generate login secret information. The default account is admin. The password needs to be obtained using the following command:

kubectl get secret --namespace pyroscope grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Swipe left to see more

Use the following command to execute port-forward (you can also use a load balancer in the form of ingress or loadbalancer for service exposure) to map the Grafana UI to localhost for access.

export POD_NAME=$(kubectl get pods --namespace pyroscope -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0]. metadata.name}")
kubectl --namespace pyroscope port-forward $POD_NAME 3000

Swipe left to see more

Here, port-forward to the local port 3000, use localhost:3000 in the browser to access Grafana, log in using the admin username and the obtained secret, and select the Grafana Pyroscope type in the Datasource.

9026d7379d2b454a768b661a6156e6f4.png

Fill in the URL:

http://pyroscope.pyroscope.svc.cluster.local.:4040

Save this data source.

c45882a8ac120a8dbdff1b4fb0f8b302.png

In Grafana explorer, select the Pyroscope data source, use the following tags to filter the Profiling data, execute Run query, and you can see that Grafana displays the flame graph information of the pyroscope-0 Pod.

66ace98484633e50af6c5ec68606fee5.png

So far, the Pyroserver server has been built, and only the Profiling flame graph information of the Pyroserver server itself can be accessed. If you want to display the application’s Profiling information, you also need to introduce the SDK into the code for development or install the eBPF agent on the node, and then push the generated Profiling information to the Pyroserver server.

Automatic Profiling using Pyroscope eBPF agent

1.Create agent configuration

The Pyroscope eBPF agent is deployed on each EKS node as a daemonset, uses Kubernetes’ service discovery mechanism to obtain the Pod list, and relabels the collected data by configuring rules. Here, I wrote a configuration with reference to the Grafana agent. Please set the endpoint URL according to your own environment and save the configuration as pyroscope-ebpf-values.yaml.

agent:
  mode: 'flow'
  configMap:
    create: true
    content: |
      discovery.kubernetes "all_pods" {
        selectors {
          field = "spec.nodeName=" + env("HOSTNAME")
          role = "pod"
        }
        role = "pod"
      }
      discovery.relabel "local_pods" {
        targets = discovery.kubernetes.all_pods.targets
        rule {
          action = "replace"
          replacement = "${1}/${2}"
          separator = "/"
          source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"]
          target_label = "service_name"
        }
      }
      pyroscope.ebpf "instance" {
        forward_to = [pyroscope.write.endpoint.receiver]
        targets = discovery.kubernetes.local_pods.targets
      }
      pyroscope.write "endpoint" {
        endpoint {
          url = "http://pyroscope.pyroscope.svc.cluster.local:4040"
        }
      }


  securityContext:
    privileged: true
    runAsGroup: 0
    runAsUser: 0


controller:
  hostPID: true

Swipe left to see more

You can also refer to the configuration manual to modify the configuration according to needs and relabel rules:

https://grafana.com/docs/pyroscope/latest/configure-client/grafana-agent/ebpf/

2.Install Pyroscope eBPF agent

helm install -n pyroscope pyroscope-ebpf grafana/grafana-agent -f pyroscope-ebpf-values.yaml

Swipe left to see more

3. Query the application flame graph generated by eBPF on Grafana

Since istio has been deployed in my environment, the agent converts the profiling data tags after collection. You can directly use the following tags for query:

7debba3a1b21bb4bcbbaa9697fd8275c.png

You can see the flame graph information of istio’s jaeger program:

7625f6235fd6546b24a08361653406be.png

It can be seen that eBPF can continuously collect the profiling information of the jaeger program without modifying the jaeger code, and display the flame graph on Grafana.

Profiling your app using the SDK

In addition to using eBPF for continuous profiling of applications, Pyroscope also supports profiling using the SDK.

This is a sample code for go provided by Pyroscope. The program uses goroutine to continuously run two functions, fastFunction and slowFunction. It calls work in the function and loops 800000000 and 200000000 times respectively to generate profiling information and attach a label for easy query. The generated profiling Data is transferred to PYROSCOPE_ENDPOINT for storage and query display. In addition to golang, Pyroscope also provides corresponding SDKs for other development languages.

package main


import (
    "context"
    "fmt"
    "os"
    "runtime"
    "runtime/pprof"
    "sync"


    "github.com/grafana/pyroscope-go"
)


//go:noinline
func work(n int) {
    // revive:disable:empty-block this is fine because this is an example app, not real production code
    for i := 0; i < n; i + + {
    }
    fmt.Printf("work\
")
    // revive:enable:empty-block
}


var m sync.Mutex


func fastFunction(c context.Context, wg *sync.WaitGroup) {
    m.Lock()
    defer m.Unlock()


    pyroscope.TagWrapper(c, pyroscope.Labels("function", "fast"), func(c context.Context) {
        work(200000000)
    })
    wg.Done()
}


func slowFunction(c context.Context, wg *sync.WaitGroup) {
    m.Lock()
    defer m.Unlock()


    // standard pprof.Do wrappers work as well
    pprof.Do(c, pprof.Labels("function", "slow"), func(c context.Context) {
        work(800000000)
    })
    wg.Done()
}


func main() {
    runtime.SetMutexProfileFraction(5)
    runtime.SetBlockProfileRate(5)
    pyroscope.Start(pyroscope.Config{
        ApplicationName: os.Getenv("SERVICE_NAME"),
        ServerAddress: os.Getenv("PYROSCOPE_ENDPOINT"),
        Logger: pyroscope.StandardLogger,
        AuthToken: os.Getenv("PYROSCOPE_AUTH_TOKEN"),
        TenantID: os.Getenv("PYROSCOPE_TENANT_ID"),
        BasicAuthUser: os.Getenv("PYROSCOPE_BASIC_AUTH_USER"),
        BasicAuthPassword: os.Getenv("PYROSCOPE_BASIC_AUTH_PASSWORD"),
        ProfileTypes: []pyroscope.ProfileType{
            pyroscope.ProfileCPU,
            pyroscope.ProfileInuseObjects,
            pyroscope.ProfileAllocObjects,
            pyroscope.ProfileInuseSpace,
            pyroscope.ProfileAllocSpace,
            pyroscope.ProfileGoroutines,
            pyroscope.ProfileMutexCount,
            pyroscope.ProfileMutexDuration,
            pyroscope.ProfileBlockCount,
            pyroscope.ProfileBlockDuration,
        },
        HTTPHeaders: map[string]string{"X-Extra-Header": "extra-header-value"},
    })


    pyroscope.TagWrapper(context.Background(), pyroscope.Labels("foo", "bar"), func(c context.Context) {
        for {
            wg := sync.WaitGroup{}
            wg.Add(2)
            go fastFunction(c, & amp;wg)
            go slowFunction(c, & amp;wg)
            wg.Wait()
        }
    })
}

Swipe left to see more

Use the following Dockerfile to build the code into an image and save it in ECR:

FROM golang:alpine AS build-env
RUN apk update & amp; & amp; apk add ca-certificates
WORKDIR /usr/src/app
COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -ldflags '-extldflags "-static"'
FROM scratch
COPY --from=build-env /usr/src/app/pyroscope-demo /pyroscope-demo
COPY --from=build-env /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
CMD ["/pyroscope-demo"]

Swipe left to see more

Use the following configuration to deploy the pyroscope-demo application to the EKS cluster, and use environment variables to set the endpoint of the Pyroscope server:

---
apiVersion: apps/v1
Kind: Deployment
metadata:
  name: pyroscope-demo
  labels:
    app:pyroscope-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app:pyroscope-demo
  template:
    metadata:
      labels:
        app:pyroscope-demo
    spec:
      containers:
      - name: pyroscope-demo
        image: "<AWS_ACCOUNT_ID>.dkr.ecr.ap-northeast-1.amazonaws.com/pyroscope-demo:latest"
        env:
        - name: PYROSCOPE_ENDPOINT
          value: "http://pyroscope.pyroscope.svc.cluster.local:4040"
        - name: SERVICE_NAME
          value: "Pyroscope-demo"

Swipe left to see more

After running successfully, query the profiling information of the pyroscope-demo application in Grafana explorer. You can see the running status of the application in the past 5 minutes. The flame graph is a very intuitive chart that displays profiling. The upper and lower layers represent the calling relationship, and the width of the horizontal bar represents the amount of resources occupied. It can be seen that the running time of the main.work function in slowFunction is 3.98 minutes, while that of fastFunction is 59.9s, which almost matches the number of loops executed by the two functions.

923fb7a777445998e34e870b141ffa94.png

Through this example, you can intuitively analyze the resources consumed by the application and find the cause of the performance problem, so as to troubleshoot the problem and optimize the application performance.

Summary

In short, Continuous Profiling is the future of internal performance analysis of modern applications. Combined with the other three pillars of observability, profiling based on eBPF without code intrusion helps customers make it simpler, larger, and more continuous from infrastructure to applications. As well as the involved middleware, we conduct application performance analysis and debugging, helping customers quickly locate problems in key business scenarios by combining logs, indicators, and tracking, and continuously optimize and improve applications.

Reference materials

  • https://www.cncf.io/blog/2022/05/31/what-is-continuous-profiling/

  • https://www.brendangregg.com/flamegraphs.html

  • https://github.com/brendangregg/FlameGraph

  • https://www.infoq.cn/article/a8kmnxdhbwmzxzsytlga

  • https://github.com/grafana/pyroscope

  • https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/

  • https://opentelemetry.io/community/roadmap/

The author of this article

954bc852aa69f326a78ada3cc0b037e7.jpeg

Lin Xufang

Amazon Cloud Technology Solution Architect, mainly responsible for the promotion of Amazon Cloud Technology cloud technology and solutions, has rich practical experience in Container, host, storage, disaster recovery and other directions.

0bd6ac2391c09e767dff9b4263816f22.jpeg

Li Junjie

Amazon Cloud Solutions Architect is responsible for the consulting and architecture design of cloud computing solutions, and is also committed to the research and promotion of containers. Before joining Amazon Cloud Technology, he was responsible for the modernization of traditional financial systems in the IT department of the financial industry. He has extensive experience in the transformation and containerization of traditional applications.

6733d2c41da13b001151049d3e52992b.gif

Stars will not get lost and development will be faster!

Remember to star “Amazon Cloud Developer” after following it

e720a1e16af6688e029d1f875128e69a.gif

I heard, click the 4 buttons below

You won’t encounter bugs!

8bed653d9fc13a51879b93c978684ba0.gif