SDK automatic configuration
The SDK’s auto-configuration module is used for basic configuration of the agent, read the documentation to find settings such as configuration export or sampling. Here are some quick links to these documents for configuration options for specific parts of the SDK and agent:
- Exporters
- OTLP exporter (both span and metric exporters)
- Jaeger exporter
- Zipkin exporter
- Prometheus exporter
- Logging exporter
- Trace context propagation
- OpenTelemetry Resource and service name
- Batch span processor
- Sampler
- Span limits
- Using SPI to further configure the SDK
Configure proxy
Agents can use configuration from one or more of the following sources (ordered from highest to lowest priority):
-
system properties
-
environment variable
-
configuration file
-
Properties provided by the AutoConfigurationCustomizer#addPropertiesSupplier() function; use the AutoConfigurationCustomizerProvider SPI
Use environment variable configuration
In some environments, configuration via environment variables is preferred. Any setting that can be configured using system properties can also be configured using environment variables. Many of the settings below include both options, but they apply the following steps to determine the required system Correct name mapping of properties to environment variables:
-
Convert system properties to uppercase
-
Replace all . and – characters with _
For example, otel.instrumentation.common.default-enabled could turn into OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED.
Configuration file
You can provide the path to the proxy configuration file by setting the following property:
System property: otel.javaagent.configuration-file
Environment variable: OTEL_JAVAAGENT_CONFIGURATION_FILE
Description:
Path to a valid Java properties file containing proxy configuration
extension
You can enable extensions by setting the following attribute:
System property: otel.javaagent.extensions
Environment variable: OTEL_JAVAAGENT_EXTENSIONS
Description:
The extension jar file or path to a folder containing jar files, if pointing to a folder, each jar file in that folder will be treated as a separate, independent extension.
Java agent output log
The agent’s log output can be configured by setting the following properties:
System property: otel.javaagent.logging
Environment variable: OTEL_JAVAAGENT_LOGGING
Description:
javaagent log mode, supports the following 3 modes:
-
simple: the agent will print out its logs using the standard error stream, only INFO or higher level logs will be printed,
This is the default javaagent logging mode.
-
none: The proxy will not output anything – not even its own version.
-
application: The proxy will try to redirect its own logs to the instrumented application’s slf4j logger, this works best for simple one-jar applications that don’t use multiple classloaders, Spring Boot applications are also supported; you can use The log configuration of the instrumented application further configures the javaagent to output logs (eg logback.xml or log4j2.xml), be sure to test whether this mode works for your application before running it in production.
Common detection configuration
Common settings applied to multiple instruments at once.
Peering service name
The peer service name is the name of the remote service making the connection, which corresponds to service.name in the local service resource.
System property: otel.instrumentation.common.peer-service-mapping
Environment variable: OTEL_INSTRUMENTATION_COMMON_PEER_SERVICE_MAPPING
Description:
is used to specify a mapping from a hostname or IP address to a peer service, as a comma-separated list pair of
For example, if set to the following:
1.2.3.4=cats-service,dogs-abcdef123.serverlessapis.com=dogs-api
Then, requests to 1.2.3.4 will have the cats-service property of peer.service and requests to dogs-abcdef123.serverlessapis.com will have the properties of dogs-api .
Database statement parameter cleaning
The agent sanitizes all database queries/statements before setting the db.statement semantic property, all values (strings, numbers) in the query string are replaced with question marks (?),
NOTE: JDBC bind parameters are not captured in db.statement, if you are looking for capturing bind parameters, see the corresponding question.
For example:
-
The SQL query SELECT a from b where password=”secret” will appear as SELECT a from b where password=? in the exported span.
-
The Redis command HSET map password “secret” will appear as HSET map password ? in the exported span.
This behavior is enabled by default for all database instrumentation,
disable it with the following attribute:
System property: otel.instrumentation.common.db-statement-sanitizer.enabled
Environment variable: OTEL_INSTRUMENTATION_COMMON_DB_STATEMENT_SANITIZER_ENABLED
Default: true
Description:
Enable DB statement cleanup
Capture HTTP request and response headers
You can configure the proxy to capture predefined HTTP request headers as attributes of the span, according to specification conventions. Use the following properties to define the HTTP request headers you want to capture:
System property: otel.instrumentation.http.capture-headers.client.request
Environment variable: OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_CLIENT_REQUEST
Description:
A comma-separated list of HTTP header names, the HTTP detection client tool will capture all configured HTTP header values for the header names.
System property: otel.instrumentation.http.capture-headers.client.response
Environment variable: OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_CLIENT_RESPONSE
Description:
A comma-separated list of HTTP response header names, the HTTP detection client tool will capture the value of all HTTP response headers with configured response header names.
System property: otel.instrumentation.http.capture-headers.server.request
Environment variable: OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST
Description:
A comma-separated list of HTTP request header names, the HTTP server detection tool will capture the value of all HTTP request headers with configured request header names.
System property: otel.instrumentation.http.capture-headers.server.response
Environment variable: OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE
Description:
A comma-separated list of HTTP response header names, the HTTP server detection tool will capture the value of all HTTP response headers with configured response header names.
All HTTP client and server instrumentation tools support these configuration options.
Note: The property/environment variable names listed in the table are still experimental and therefore subject to change.
Capture servlet request parameters
You can configure the proxy to capture predefined HTTP request parameters that are processed by the Servlet API as Span attribute values. Use the following attributes to define the servlet request parameters to capture:
System property: otel.instrumentation.servlet.experimental.capture-request-parameters
Environment variable: OTEL_INSTRUMENTATION_SERVLET_EXPERIMENTAL_CAPTURE_REQUEST_PARAMETERS
Description:
A comma-separated list of request parameter names
Note: The property/environment variable names listed in the table are still experimental and therefore subject to change.
Capturing telemetry data for consumer messages
You can configure the broker to capture telemetry data in consumer messages by enabling it with the following property:
System property: otel.instrumentation.messaging.experimental.receive-telemetry.enabled
Environment variable: OTEL_INSTRUMENTATION_MESSAGING_EXPERIMENTAL_RECEIVE_TELEMETRY_ENABLED
Default: false
Description:
Enable consumer message reception telemetry
Note that this will cause the consumer to start a new trace with only one span connecting it to the producer trace.
Note: The property/environment variable names listed in the table are still experimental and therefore subject to change.
Suppress specific auto-detections
Disable proxy completely
You can disable the agent with -Dotel.javaagent.enabled=false (or with the equivalent environment variable OTEL_JAVAAGENT_ENABLED=false).
Only enable specific detection
You can disable all default automatic detections and selectively re-enable individual detections, which may be required to reduce startup overhead or to have greater control over which detections are applied.
-
Use -Dotel.instrumentation.common.default-enabled=false (or use the equivalent environment variable OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED=false) to disable all instrumentation in the agent.
-
Individually enable a desired instrumentation with -Dotel.instrumentation.[name].enabled=true (or with the equivalent environment variable OTEL_INSTRUMENTATION_[NAME]_ENABLED ), where [name] ([NAME]) is the corresponding instrumentation below name.
Note: Some detections depend on other detections to function properly. When selectively enabling a detection, make sure to enable its dependencies as well. Determining such dependencies is left as an exercise for the user.
Only enable manual detection
You can suppress all automatic detections, but by using -Dotel.instrumentation.common.default-enabled=false, -Dotel.instrumentation.opentelemetry-api.enabled=true, -Dotel.instrumentation.opentelemetry-instrumentation-annotations.enabled=true Manual detection of normal API interactions using @WithSpan is supported.
Suppress specific proxy detection
You can suppress proxy instrumentation for specific libraries with -Dotel.instrumentation.[name].enabled=false (or with the equivalent environment variable OTEL_INSTRUMENTATION_[NAME]_ENABLED ), where name (NAME) is the corresponding instrumentation name, which is the same as The relationship of the library is as follows:
Library/Framework | Instrumentation name |
---|---|
Additional methods tracing | methods |
Additional tracing annotations | external-annotations |
Akka Actor | akka-actor |
Akka HTTP | akka-http |
Apache Axis2 | axis2 |
Apache Camel | camel |
Apache Cassandra | cassandra |
Apache CXF | cxf |
Apache DBCP | apache-dbcp |
Apache Dubbo | apache-dubbo |
Apache Geode | geode |
Apache HttpAsyncClient | apache-httpasyncclient |
Apache HttpClient | apache-httpclient |
Apache Kafka | kafka |
Apache MyFaces | jsf-myfaces |
Apache Pulsar | pulsar |
Apache RocketMQ | rocketmq-client |
Apache Struts 2 | struts |
Apache Tapestry | tapestry |
Apache Tomcat | tomcat |
Apache Wicket | wicket |
Armeria | armeria |
AsyncHttpClient (AHC ) | async-http-client |
AWS Lambda | aws-lambda |
AWS SDK | aws-sdk |
Azure SDK | azure-core |
Couchbase | couchbase |
C3P0 | c3p0 |
Dropwizard Views | dropwizard-views |
Dropwizard Metrics | dropwizard-metrics |
Eclipse Grizzly | grizzly |
Eclipse Jersey | jersey |
Eclipse Jetty | jetty |
Eclipse Jetty HTTP Client | jetty-httpclient |
Eclipse Metro | metro |
Eclipse Mojarra | jsf-mojarra |
Eclipse Vert.x HttpClient | vertx-http-client |
Eclipse Vert.x Kafka Client | vertx-kafka-client |
Eclipse Vert.x RxJava | vertx-rx-java |
Eclipse Vert.x Web | vertx-web |
Elasticsearch client | elasticsearch-transport |
Elasticsearch REST client | elasticsearch- rest |
Google Guava | guava |
Google HTTP client | google-http-client |
Google Web Toolkit | gwt |
Grails | grails |
GraphQL Java | graphql-java |
GRPC | grpc |
Hibernate | hibernate |
HikariCP | hikaricp |
Java HTTP Client | java-http-client |
Java HttpURLConnection |
http-url-connection |
Java JDBC | jdbc |
Java JDBC DataSource |
jdbc-datasource |
Java RMI | rmi |
Java Runtime | runtime-metrics |
Java Servlet | servlet |
java.util.concurrent | executors |
java.util.logging | java-util-logging |
JAX-RS (Client) | jaxrs-client |
JAX-RS (Server) | jaxrs |
JAX-WS | jaxws |
JBoss Logging Appender | jboss-logmanager-appender |
JBoss Logging MDC | jboss-logmanager-mdc |
JMS | jms |
Jodd HTTP | jodd-http |
JSP | jsp |
K8s Client | kubernetes-client |
kotlinx.coroutines | kotlinx-coroutines |
Log4j Appender | log4j-appender |
Log4j MDC (1.x) | log4j-mdc |
Log4j Context Data (2 .x) | log4j-context-data |
Logback Appender | logback- appender |
Logback MDC | logback-mdc |
Micrometer | micrometer |
MongoDB | mongo |
Netflix Hystrix | hystrix |
Netty | netty |
OkHttp | okhttp |
OpenLiberty | liberty |
OpenTelemetry Extension Annotations | opentelemetry-extension- annotations |
OpenTelemetry Instrumentation Annotations | opentelemetry-instrumentation-annotations |
OpenTelemetry API | opentelemetry-api |
Oracle UCP | oracle-ucp |
OSHI (Operating System and Hardware Information) | oshi |
Play Framework | play |
Play WS HTTP Client | play-ws |
Quartz | quartz |
R2DBC | r2dbc |
RabbitMQ Client | rabbitmq |
Ratpack | ratpack |
ReactiveX RxJava | rxjava |
Reactor | reactor |
Reactor Netty | reactor-netty |
Redis Jedis | jedis |
Redis Lettuce | lettuce |
Rediscala | rediscala |
Redisson | redisson |
Restlet | restlet |
Scala ForkJoinPool | scala-fork-join |
Spark Web Framework | spark |
Spring Batch | spring-batch |
Spring Core | spring-core |
Spring Data | spring-data |
Spring JMS | spring-jms |
Spring Integration | spring-integration |
Spring Kafka | spring-kafka |
Spring RabbitMQ | spring-rabbit |
Spring RMI | spring-rmi |
Spring Scheduling | spring-scheduling |
Spring Web | spring-web |
Spring WebFlux | spring-webflux |
Spring Web MVC | spring-webmvc |
Spring Web Services | spring-ws |
Spymemcached | spymemcached |
Tomcat JDBC | tomcat-jdbc |
Twilio SDK | twilio |
Twitter Finatra | finatra |
Undertow | undertow |
Vaadin | vaadin |
Vibur DBCP | vibur-dbcp |
ZIO | zio |
NOTE: When using environment variables, dashes (-) should be converted to underscores (_), for example, to suppress tracing from the akka-actor library, set OTEL_INSTRUMENTATION_AKKA_ACTOR_ENABLED to false.
Suppressing the Span of controllers and views
Some instrumentation (e.g. Spring Web MVC instrumentation) generate SpanKind.Internal spans to capture controller or view execution, these spans can be suppressed using the configuration setting below, without suppressing the entire instrumentation, this also disables instrumentation capturing the parent SpanKind. http.route and associated span name on the Server span.
System property: otel.instrumentation.common.experimental.controller-telemetry.enabled
Environment variable: OTEL_INSTRUMENTATION_COMMON_EXPERIMENTAL_CONTROLLER_TELEMETRY_ENABLED
Default: true
description:
enable controller detection
System property: otel.instrumentation.common.experimental.view-telemetry.enabled
Environment variables:
OTEL_INSTRUMENTATION_COMMON_EXPERIMENTAL_VIEW_TELEMETRY_ENABLED
Default: true
Description:
enable view detection
Detect span suppression behavior
Some of the libraries instrumented by this agent in turn use some lower-level libraries that also need to be instrumented, often resulting in nested spans containing duplicate instrumentation data. For example:
-
Spans generated by the Reactor Netty HTTP client tool will have duplicate HTTP client spans generated by the Netty tool;
-
Dynamo DB spans generated by AWS SDK instrumentation will have sub HTTP client spans generated by its internal HTTP client library (also instrumented);
-
Spans generated by Tomcat tools will have duplicate HTTP server spans generated by Generic Servlet API tools.
The javaagent prevents these situations by detecting and suppressing nested spans of duplicate telemetry data, the suppression behavior can be configured using the following configuration options:
System property: otel.instrumentation.experimental.span-suppression-strategy
Environment variable: OTEL_INSTRUMENTATION_EXPERIMENTAL_SPAN_SUPPRESSION_STRATEGY
Description:
javaagent span suppression strategy. The following 3 strategies are supported:
-
semconv: The proxy will suppress duplicate semantic conventions. This is the default behavior of javaagent.
-
span-kind: The proxy will suppress spans with the same kind (except INTERNAL).
-
none: The agent will not suppress anything at all, we don’t recommend using this option for anything other than debugging purposes as it generates a lot of duplicated telemetry.
For example, suppose we instrument a database client that internally uses the Reactor Netty HTTP client, which in turn uses Netty.
Using the default semconv suppression strategy will result in 2 nested CLIENT spans:
-
CLIENT span has database client semantic properties emitted by database client detection;
-
The CLIENT span has HTTP client semantic properties emitted by the Reactor Netty tool.
Netty detection will be suppressed as it duplicates Reactor Netty HTTP client detection.
Using the suppression strategy span-kind produces only one span:
-
CLIENT span has database client semantic properties emitted by database client detection;
Both Reactor Netty and Netty tools will be suppressed, since they also emit CLIENT spans.
Finally, using the suppression strategy none produces 3 spans:
-
CLIENT span has database client semantic properties emitted by database client detection;
-
The CLIENT span has HTTP client semantic properties emitted by the Reactor Netty tool;
-
The CLIENT span has HTTP client semantic properties emitted by the Netty tools.