OpenTelemetry In Java automatic detection Agent configuration

SDK automatic configuration

The SDK’s auto-configuration module is used for basic configuration of the agent, read the documentation to find settings such as configuration export or sampling. Here are some quick links to these documents for configuration options for specific parts of the SDK and agent:

  • Exporters
    • OTLP exporter (both span and metric exporters)
    • Jaeger exporter
    • Zipkin exporter
    • Prometheus exporter
    • Logging exporter
  • Trace context propagation
  • OpenTelemetry Resource and service name
  • Batch span processor
  • Sampler
  • Span limits
  • Using SPI to further configure the SDK

Configure proxy

Agents can use configuration from one or more of the following sources (ordered from highest to lowest priority):

  • system properties

  • environment variable

  • configuration file

  • Properties provided by the AutoConfigurationCustomizer#addPropertiesSupplier() function; use the AutoConfigurationCustomizerProvider SPI

Use environment variable configuration

In some environments, configuration via environment variables is preferred. Any setting that can be configured using system properties can also be configured using environment variables. Many of the settings below include both options, but they apply the following steps to determine the required system Correct name mapping of properties to environment variables:

  • Convert system properties to uppercase

  • Replace all . and – characters with _

For example, otel.instrumentation.common.default-enabled could turn into OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED.

Configuration file

You can provide the path to the proxy configuration file by setting the following property:

System property: otel.javaagent.configuration-file

Environment variable: OTEL_JAVAAGENT_CONFIGURATION_FILE

Description:Path to a valid Java properties file containing proxy configuration

extension

You can enable extensions by setting the following attribute:

System property: otel.javaagent.extensions

Environment variable: OTEL_JAVAAGENT_EXTENSIONS

Description: The extension jar file or path to a folder containing jar files, if pointing to a folder, each jar file in that folder will be treated as a separate, independent extension.

Java agent output log

The agent’s log output can be configured by setting the following properties:

System property: otel.javaagent.logging

Environment variable: OTEL_JAVAAGENT_LOGGING

Description: javaagent log mode, supports the following 3 modes:

  • simple: the agent will print out its logs using the standard error stream, only INFO or higher level logs will be printed,

    This is the default javaagent logging mode.

  • none: The proxy will not output anything – not even its own version.

  • application: The proxy will try to redirect its own logs to the instrumented application’s slf4j logger, this works best for simple one-jar applications that don’t use multiple classloaders, Spring Boot applications are also supported; you can use The log configuration of the instrumented application further configures the javaagent to output logs (eg logback.xml or log4j2.xml), be sure to test whether this mode works for your application before running it in production.

Common detection configuration

Common settings applied to multiple instruments at once.

Peering service name

The peer service name is the name of the remote service making the connection, which corresponds to service.name in the local service resource.

System property: otel.instrumentation.common.peer-service-mapping

Environment variable: OTEL_INSTRUMENTATION_COMMON_PEER_SERVICE_MAPPING

Description: is used to specify a mapping from a hostname or IP address to a peer service, as a comma-separated list pair of =, to which a peer service is added as an attribute to its host or IP The range in which the address matches the mapping.

For example, if set to the following:

1.2.3.4=cats-service,dogs-abcdef123.serverlessapis.com=dogs-api

Then, requests to 1.2.3.4 will have the cats-service property of peer.service and requests to dogs-abcdef123.serverlessapis.com will have the properties of dogs-api .

Database statement parameter cleaning

The agent sanitizes all database queries/statements before setting the db.statement semantic property, all values (strings, numbers) in the query string are replaced with question marks (?),

NOTE: JDBC bind parameters are not captured in db.statement, if you are looking for capturing bind parameters, see the corresponding question.

For example:

  • The SQL query SELECT a from b where password=”secret” will appear as SELECT a from b where password=? in the exported span.

  • The Redis command HSET map password “secret” will appear as HSET map password ? in the exported span.

This behavior is enabled by default for all database instrumentation, disable it with the following attribute:

System property: otel.instrumentation.common.db-statement-sanitizer.enabled

Environment variable: OTEL_INSTRUMENTATION_COMMON_DB_STATEMENT_SANITIZER_ENABLED

Default: true

Description: Enable DB statement cleanup

Capture HTTP request and response headers

You can configure the proxy to capture predefined HTTP request headers as attributes of the span, according to specification conventions. Use the following properties to define the HTTP request headers you want to capture:

System property: otel.instrumentation.http.capture-headers.client.request

Environment variable: OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_CLIENT_REQUEST

Description: A comma-separated list of HTTP header names, the HTTP detection client tool will capture all configured HTTP header values for the header names.

System property: otel.instrumentation.http.capture-headers.client.response

Environment variable: OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_CLIENT_RESPONSE

Description: A comma-separated list of HTTP response header names, the HTTP detection client tool will capture the value of all HTTP response headers with configured response header names.

System property: otel.instrumentation.http.capture-headers.server.request

Environment variable: OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST

Description: A comma-separated list of HTTP request header names, the HTTP server detection tool will capture the value of all HTTP request headers with configured request header names.

System property: otel.instrumentation.http.capture-headers.server.response

Environment variable: OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE

Description: A comma-separated list of HTTP response header names, the HTTP server detection tool will capture the value of all HTTP response headers with configured response header names.

All HTTP client and server instrumentation tools support these configuration options.

Note: The property/environment variable names listed in the table are still experimental and therefore subject to change.

Capture servlet request parameters

You can configure the proxy to capture predefined HTTP request parameters that are processed by the Servlet API as Span attribute values. Use the following attributes to define the servlet request parameters to capture:

System property: otel.instrumentation.servlet.experimental.capture-request-parameters

Environment variable: OTEL_INSTRUMENTATION_SERVLET_EXPERIMENTAL_CAPTURE_REQUEST_PARAMETERS

Description: A comma-separated list of request parameter names

Note: The property/environment variable names listed in the table are still experimental and therefore subject to change.

Capturing telemetry data for consumer messages

You can configure the broker to capture telemetry data in consumer messages by enabling it with the following property:

System property: otel.instrumentation.messaging.experimental.receive-telemetry.enabled

Environment variable: OTEL_INSTRUMENTATION_MESSAGING_EXPERIMENTAL_RECEIVE_TELEMETRY_ENABLED

Default: false

Description:Enable consumer message reception telemetry

Note that this will cause the consumer to start a new trace with only one span connecting it to the producer trace.

Note: The property/environment variable names listed in the table are still experimental and therefore subject to change.

Suppress specific auto-detections

Disable proxy completely

You can disable the agent with -Dotel.javaagent.enabled=false (or with the equivalent environment variable OTEL_JAVAAGENT_ENABLED=false).

Only enable specific detection

You can disable all default automatic detections and selectively re-enable individual detections, which may be required to reduce startup overhead or to have greater control over which detections are applied.

  • Use -Dotel.instrumentation.common.default-enabled=false (or use the equivalent environment variable OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED=false) to disable all instrumentation in the agent.

  • Individually enable a desired instrumentation with -Dotel.instrumentation.[name].enabled=true (or with the equivalent environment variable OTEL_INSTRUMENTATION_[NAME]_ENABLED ), where [name] ([NAME]) is the corresponding instrumentation below name.

Note: Some detections depend on other detections to function properly. When selectively enabling a detection, make sure to enable its dependencies as well. Determining such dependencies is left as an exercise for the user.

Only enable manual detection

You can suppress all automatic detections, but by using -Dotel.instrumentation.common.default-enabled=false, -Dotel.instrumentation.opentelemetry-api.enabled=true, -Dotel.instrumentation.opentelemetry-instrumentation-annotations.enabled=true Manual detection of normal API interactions using @WithSpan is supported.

Suppress specific proxy detection

You can suppress proxy instrumentation for specific libraries with -Dotel.instrumentation.[name].enabled=false (or with the equivalent environment variable OTEL_INSTRUMENTATION_[NAME]_ENABLED ), where name (NAME) is the corresponding instrumentation name, which is the same as The relationship of the library is as follows:

Library/Framework Instrumentation name
Additional methods tracing methods
Additional tracing annotations external-annotations
Akka Actor akka-actor
Akka HTTP akka-http
Apache Axis2 axis2
Apache Camel camel
Apache Cassandra cassandra
Apache CXF cxf
Apache DBCP apache-dbcp
Apache Dubbo apache-dubbo
Apache Geode geode
Apache HttpAsyncClient apache-httpasyncclient
Apache HttpClient apache-httpclient
Apache Kafka kafka
Apache MyFaces jsf-myfaces
Apache Pulsar pulsar
Apache RocketMQ rocketmq-client
Apache Struts 2 struts
Apache Tapestry tapestry
Apache Tomcat tomcat
Apache Wicket wicket
Armeria armeria
AsyncHttpClient (AHC ) async-http-client
AWS Lambda aws-lambda
AWS SDK aws-sdk
Azure SDK azure-core
Couchbase couchbase
C3P0 c3p0
Dropwizard Views dropwizard-views
Dropwizard Metrics dropwizard-metrics
Eclipse Grizzly grizzly
Eclipse Jersey jersey
Eclipse Jetty jetty
Eclipse Jetty HTTP Client jetty-httpclient
Eclipse Metro metro
Eclipse Mojarra jsf-mojarra
Eclipse Vert.x HttpClient vertx-http-client
Eclipse Vert.x Kafka Client vertx-kafka-client
Eclipse Vert.x RxJava vertx-rx-java
Eclipse Vert.x Web vertx-web
Elasticsearch client elasticsearch-transport
Elasticsearch REST client elasticsearch- rest
Google Guava guava
Google HTTP client google-http-client
Google Web Toolkit gwt
Grails grails
GraphQL Java graphql-java
GRPC grpc
Hibernate hibernate
HikariCP hikaricp
Java HTTP Client java-http-client
Java HttpURLConnection http-url-connection
Java JDBC jdbc
Java JDBC DataSource jdbc-datasource
Java RMI rmi
Java Runtime runtime-metrics
Java Servlet servlet
java.util.concurrent executors
java.util.logging java-util-logging
JAX-RS (Client) jaxrs-client
JAX-RS (Server) jaxrs
JAX-WS jaxws
JBoss Logging Appender jboss-logmanager-appender
JBoss Logging MDC jboss-logmanager-mdc
JMS jms
Jodd HTTP jodd-http
JSP jsp
K8s Client kubernetes-client
kotlinx.coroutines kotlinx-coroutines
Log4j Appender log4j-appender
Log4j MDC (1.x) log4j-mdc
Log4j Context Data (2 .x) log4j-context-data
Logback Appender logback- appender
Logback MDC logback-mdc
Micrometer micrometer
MongoDB mongo
Netflix Hystrix hystrix
Netty netty
OkHttp okhttp
OpenLiberty liberty
OpenTelemetry Extension Annotations opentelemetry-extension- annotations
OpenTelemetry Instrumentation Annotations opentelemetry-instrumentation-annotations
OpenTelemetry API opentelemetry-api
Oracle UCP oracle-ucp
OSHI (Operating System and Hardware Information) oshi
Play Framework play
Play WS HTTP Client play-ws
Quartz quartz
R2DBC r2dbc
RabbitMQ Client rabbitmq
Ratpack ratpack
ReactiveX RxJava rxjava
Reactor reactor
Reactor Netty reactor-netty
Redis Jedis jedis
Redis Lettuce lettuce
Rediscala rediscala
Redisson redisson
Restlet restlet
Scala ForkJoinPool scala-fork-join
Spark Web Framework spark
Spring Batch spring-batch
Spring Core spring-core
Spring Data spring-data
Spring JMS spring-jms
Spring Integration spring-integration
Spring Kafka spring-kafka
Spring RabbitMQ spring-rabbit
Spring RMI spring-rmi
Spring Scheduling spring-scheduling
Spring Web spring-web
Spring WebFlux spring-webflux
Spring Web MVC spring-webmvc
Spring Web Services spring-ws
Spymemcached spymemcached
Tomcat JDBC tomcat-jdbc
Twilio SDK twilio
Twitter Finatra finatra
Undertow undertow
Vaadin vaadin
Vibur DBCP vibur-dbcp
ZIO zio

NOTE: When using environment variables, dashes (-) should be converted to underscores (_), for example, to suppress tracing from the akka-actor library, set OTEL_INSTRUMENTATION_AKKA_ACTOR_ENABLED to false.

Suppressing the Span of controllers and views

Some instrumentation (e.g. Spring Web MVC instrumentation) generate SpanKind.Internal spans to capture controller or view execution, these spans can be suppressed using the configuration setting below, without suppressing the entire instrumentation, this also disables instrumentation capturing the parent SpanKind. http.route and associated span name on the Server span.

System property: otel.instrumentation.common.experimental.controller-telemetry.enabled

Environment variable: OTEL_INSTRUMENTATION_COMMON_EXPERIMENTAL_CONTROLLER_TELEMETRY_ENABLED

Default: true

description: enable controller detection

System property: otel.instrumentation.common.experimental.view-telemetry.enabled

Environment variables:

OTEL_INSTRUMENTATION_COMMON_EXPERIMENTAL_VIEW_TELEMETRY_ENABLED

Default: true

Description: enable view detection

Detect span suppression behavior

Some of the libraries instrumented by this agent in turn use some lower-level libraries that also need to be instrumented, often resulting in nested spans containing duplicate instrumentation data. For example:

  • Spans generated by the Reactor Netty HTTP client tool will have duplicate HTTP client spans generated by the Netty tool;

  • Dynamo DB spans generated by AWS SDK instrumentation will have sub HTTP client spans generated by its internal HTTP client library (also instrumented);

  • Spans generated by Tomcat tools will have duplicate HTTP server spans generated by Generic Servlet API tools.

The javaagent prevents these situations by detecting and suppressing nested spans of duplicate telemetry data, the suppression behavior can be configured using the following configuration options:

System property: otel.instrumentation.experimental.span-suppression-strategy

Environment variable: OTEL_INSTRUMENTATION_EXPERIMENTAL_SPAN_SUPPRESSION_STRATEGY

Description: javaagent span suppression strategy. The following 3 strategies are supported:

  • semconv: The proxy will suppress duplicate semantic conventions. This is the default behavior of javaagent.

  • span-kind: The proxy will suppress spans with the same kind (except INTERNAL).

  • none: The agent will not suppress anything at all, we don’t recommend using this option for anything other than debugging purposes as it generates a lot of duplicated telemetry.

For example, suppose we instrument a database client that internally uses the Reactor Netty HTTP client, which in turn uses Netty.

Using the default semconv suppression strategy will result in 2 nested CLIENT spans:

  • CLIENT span has database client semantic properties emitted by database client detection;

  • The CLIENT span has HTTP client semantic properties emitted by the Reactor Netty tool.

Netty detection will be suppressed as it duplicates Reactor Netty HTTP client detection.

Using the suppression strategy span-kind produces only one span:

  • CLIENT span has database client semantic properties emitted by database client detection;

Both Reactor Netty and Netty tools will be suppressed, since they also emit CLIENT spans.

Finally, using the suppression strategy none produces 3 spans:

  • CLIENT span has database client semantic properties emitted by database client detection;

  • The CLIENT span has HTTP client semantic properties emitted by the Reactor Netty tool;

  • The CLIENT span has HTTP client semantic properties emitted by the Netty tools.