Observability: Manual instrumentation of .NET applications using OpenTelemetry

Author: David Hope

In the fast-paced world of software development, especially in the cloud-native world, DevOps and SRE teams are increasingly becoming critical partners in application stability and growth.

DevOps engineers continuously optimize software delivery, while SRE teams act as stewards of application reliability, scalability, and top-level performance. challenge? These teams need a cutting-edge observability solution that includes full-stack insights that allow them to quickly manage, monitor, and correct potential disruptions before they ultimately lead to operational challenges.

Observability in modern distributed software ecosystems is more than just monitoring-it requires unlimited data collection, processing precision, and correlating that data with actionable insights. However, the road to achieving this holistic view is fraught with obstacles, from resolving version incompatibilities to battling restrictive proprietary code.

OpenTelemetry (OTel) will bring the following benefits to users who adopt it:

  • Free yourself from vendor lock-in and ensure best-in-class observability with OTel.
  • View unified logs, metrics, and traces harmoniously unified to provide a complete system view.
  • Improve your application oversight with richer and enhanced tools.
  • Protect your previous detection investments by taking advantage of backward compatibility.
  • Embark on your OpenTelemetry journey with a simple learning curve that simplifies onboarding and scalability.
  • Rely on proven, future-proof standards to increase your confidence in every investment.
  • Explore manual instrumentation for customized data collection to meet your unique needs.
  • Ensure consistent monitoring across tiers using a standardized observability data framework.
  • Decouple development from operations to maximize efficiency from both.

In this article, we’ll take a deep dive into manually instrumenting .NET applications using Docker.

What does this article cover?

  • Manually instrument .NET applications
  • Create Docker images for .NET applications using built-in OpenTelemetry tools
  • Install and run OpenTelemetry .NET Profiler for automatic detection

Prerequisites

  1. Learn about Docker and .NET
  2. Elastic Cloud
  3. Docker installed on your computer (we recommend docker desktop version)

View sample source code

The complete source code, including the Dockerfile used in this blog, can be found on GitHub. This repository also contains the same application, but without instrumentation. This allows you to compare each file and see the differences.

The following steps will show you how to detect this application and run it from the command line or Docker. If you’re interested in a more complete OTel example, check out the docker-compose file here which will show the complete project.

Step-by-step guide

This blog assumes you have an Elastic Cloud account – if not, follow the instructions to get started with Elastic Cloud.

Step 1. Get started

In our demo, we will manually instrument a .NET Core application – Login. The application simulates a simple user login service. In this example, we focus only on tracing because the OpenTelemetry logging tool is currently at mixed maturity, as described here.

The application has the following files:

  1. program.cs
  2. start.cs
  3. telemetry.cs
  4. LoginController.cs

Step 2. Detect the application

The .NET ecosystem presents some unique aspects when it comes to OpenTelemetry. While OpenTelemetry provides its API, .NET leverages its native System.Diagnostics API to implement OpenTelemetry’s tracing API. Pre-existing constructs such as ActivitySource and Activity are appropriately repurposed to conform to OpenTelemetry.

That said, understanding the OpenTelemetry API and its terminology remains critical for .NET developers. It’s essential for gaining complete control over application instrumentation, and as we’ve seen, it also extends to understanding elements of the System.Diagnostics API.

For those who might prefer to use the original OpenTelemetry API instead of the System.Diagnostics API, there is an alternative. OpenTelemetry provides an API shim for tracing that you can use. It enables developers to switch to the OpenTelemetry API, you can find more details about it in the OpenTelemetry API Shim documentation.

By integrating such practices into your .NET applications, you can take advantage of the power that OpenTelemetry has to offer, whether you’re using OpenTelemetry’s API or the System.Diagnostics API.

In this blog, we stick to the default approach and use the activity conventions specified by the System.Diagnostics API.

To manually instrument a .NET application, you need to make changes to each file. Let’s take a look at these changes one by one.

Program.cs

This is the entry point to our application. Here we create an instance of IHostBuilder using default configuration. Notice how we set up the console logger using Serilog.

public static void Main(string[] args)
{
    Log.Logger = new LoggerConfiguration().WriteTo.Console().CreateLogger();
    CreateHostBuilder(args).Build().Run();
}

Startup.cs

In the Startup.cs file, we add OpenTelemetry Tracing using the ConfigureServices method.

public void ConfigureServices(IServiceCollection services)
{
    services.AddOpenTelemetry().WithTracing(builder => builder.AddOtlpExporter()
        .AddSource("Login")
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter()
        .ConfigureResource(resource =>
            resource.AddService(
                serviceName: "Login"))
    );
    services.AddControllers();
}

The WithTracing method supports tracing in OpenTelemetry. We added an exporter for OTLP (OpenTelemetry Protocol), a universal telemetry data transfer protocol. We also added AspNetCoreInstrumentation which will automatically collect traces from our application. This is an extremely important step that is not mentioned in the OpenTelemetry documentation. Without adding this method, instrumentation doesn’t work for my login application.

Telemetry.cs

This file contains the definition of our ActivitySource. ActivitySource represents the source of telemetry activity. It is named after your application’s service name, which can come from a configuration file, constants file, etc. We can use this ActivitySource to start the activity.

using System.Diagnostics;

public static class Telemetry
{
    //...

    // Name it after the service name for your app.
    // It can come from a config file, constants file, etc.
    public static readonly ActivitySource LoginActivitySource = new("Login");

    //...
}

In our example, we create an ActivitySource called Login. In our LoginController.cs, we use this LoginActivitySource to start a new activity when we start the action.

using (Activity activity = Telemetry.LoginActivitySource.StartActivity("SomeWork"))
{
    // Perform operations here
}

This code starts a new activity called SomeWork, performs some actions (in this case, generates a random user and logs in), and then ends the activity. These activities can be tracked and analyzed later to understand the performance of the operation.

This ActivitySource is the basis for manual detection of OpenTelemetry. It represents the source of the activity and provides methods to start and stop the activity.

LoginController.cs

In the LoginController.cs file, we trace the actions performed by the GET and POST methods. We start a new activity SomeWork before starting the operation and dispose of it when completed.

using (Activity activity = Telemetry.LoginActivitySource.StartActivity("SomeWork"))
{
    var user = GenerateRandomUserResponse();
    Log.Information("User logged in: {UserName}", user);
    return user;
}

This will track the time spent on these operations and send this data to any configured telemetry backend via the OTLP exporter.

Step 3. Basic image settings

Now that we have created and instrumented the application source code, it’s time to create a Dockerfile to build and run our .NET login service.

Start with a Dockerfile base layer of the .NET runtime image:

FROM ${ARCH}mcr.microsoft.com/dotnet/aspnet:7.0. AS base
WORKDIR/app
EXPOSE 8000

Here we are setting up the runtime environment of our application.

Step 4. Build the .NET application

This feature of Docker is the best. Here we compile the .NET application. We will use the SDK image. In the bad old days, we used to build on different platforms and then put the compiled code into Docker containers. This way, we have more confidence in replicating our builds from developer desktops to production using Docker.

FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:8.0-preview AS build
ARG TARGETPLATFORM

WORKDIR /src
COPY ["login.csproj", "./"]
RUN dotnet restore "./login.csproj"
COPY . .
WORKDIR "/src/."
RUN dotnet build "login.csproj" -c Release -o /app/build

This section ensures that our .NET code is restored and compiled correctly.

Step 5. Publish the application

Once the build is complete, we will publish the application:

FROM build AS publish
RUN dotnet publish "login.csproj" -c Release -o /app/publish

Step 6. Prepare the final image

Now, let’s set up the final runtime image:

FROM base AS final
WORKDIR/app
COPY --from=publish /app/publish .

Step 7. Entry point settings

Finally, set the entry point of the Docker image as the source of the OpenTelemetry tool, which sets the environment variables required to bootstrap the .NET Profiler and then start the .NET application:

ENTRYPOINT ["/bin/bash", "-c", "dotnet login.dll"]

Step 8. Use environment variables to run the Docker image

To build and run a Docker image, you typically perform the following steps:

Build Docker image

First, you need to build the Docker image from the Dockerfile. Assume that the Dockerfile is located in the current directory and you want to name/tag your image dotnet-login-otel-image.

docker build -t dotnet-login-otel-image .

Run Docker image

After building the image, you can run it using specified environment variables. To do this, use the docker run command with the -e flag for each environment variable.

 docker run \
       -e OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer ${ELASTIC_APM_SECRET_TOKEN}" \
       -e OTEL_EXPORTER_OTLP_ENDPOINT="${ELASTIC_APM_SERVER_URL}" \
       -e OTEL_METRICS_EXPORTER="otlp" \
       -e OTEL_RESOURCE_ATTRIBUTES="service.version=1.0,deployment.environment=production" \
       -e OTEL_SERVICE_NAME="dotnet-login-otel-manual" \
       -e OTEL_TRACES_EXPORTER="otlp" \
       dotnet-login-otel-image

Make sure ${ELASTIC_APM_SECRET_TOKEN} and ${ELASTIC_APM_SERVER_URL} are set in the shell environment, replacing them with the actual values from the cloud as shown below.

Get Elastic Cloud variables

You can copy the endpoint and token from Kibana under the path “/app/home#/tutorial/apm”.

If you have multiple environment variables, you can also use an environment file with docker run –env-file to make the command more concise.

Once you have this program up and running, you can ping the instrumentation service’s endpoint (/login in our example) and you should see the application appear in Elastic APM, as shown below:

It will first track the key metrics of throughput and latency that the SRE needs to focus on.

Digging deeper we can see an overview of all transactions.

Take a look at the specific transactions, including the “SomeWork” activity/span we created in the code above:

There is clearly an outlier here, with one transaction taking over 20 milliseconds. This may be due to CLR warming up.

Summary

With the code instrumentation and Dockerfile bootstrapping the application here, you have transformed a simple .NET application into an application instrumented with OpenTelemetry. This will be a huge help in understanding application performance, tracking errors, and gaining insights into how users interact with the software.

Remember, observability is an important aspect of modern application development, especially in distributed systems. Understanding complex systems becomes easier with tools like OpenTelemetry.

In this blog we discuss the following:

  • How to manually instrument .NET using OpenTelemetry.
  • Our instrumentation application was built and started using standard commands in a Docker file.
  • Using OpenTelemetry and its support for multiple languages, DevOps and SRE teams can easily instrument their applications, instantly understand the health of the entire application stack and reduce mean time to resolution (MTTR).

Since Elastic can support multiple methods of extracting data, whether automatic detection using open source OpenTelemetry or manual detection using its native APM agent, you can focus on a few applications first and then use OpenTelemety to plan a migration to OTel later. Across your applications in a way that best suits your business needs.

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try the detection features I discussed above. I’d love to hear your feedback on your experience using Elastic to learn about your application stack.

The release and timing of any features or functionality described in this article are at the sole discretion of Elastic. Any features or functionality that are currently unavailable may not be delivered on time or at all.