Using Custom Span Attributes in OpenTelemetry

If you are new to OpenTelemetry and confused where to start — Please take a look at my earlier articles for reference — The examples/setup used in this article has been explained well in my earlier blogs. I won’t be going again in to the whole setup in this blog.

1. https://nitin-rohidas.medium.com/introduction-to-opentelemetry-distributed-tracing-part-i-d1dc57d71a96
2. https://nitin-rohidas.medium.com/introduction-to-opentelemetry-distributed-tracing-part-ii-7ddab4a4cf15

In today’s blog , I am going to talk about few customization aspects of OpenTelemetry. You probably already know that the most easiest way to get started with OpenTelemetry is to use the Auto Instrumentation agents which provides zero code telemetry data. However, there will be situations where you might need more control to create your on custom spans or adding business specific information to the tracing data. We will take a use case and then see how to solve for it.

We will take our earlier example of Student and School service. The code can be found in my github repository.

For demonstration purpose , I am going to hit the school service for 5 times to generate the trace data by randomly changing the parameter for the school name for every request.

http://localhost:9001/school-details/xyzschool
http://localhost:9001/school-details/abcschool

In real word scenario, your transactions could be the linked to a CustomerId or an OrderId etc. Imagine you need to search the trace for a particular transaction using CustomerId or OrderId. How will you search a request with some business specific identifiers?

This is when we use the power of customization by adding custom attributes to the existing Auto-generated Spans. The custom attributes can then be mined for getting trace of a specific request.

Attributes

Attributes are key-value pairs that contain metadata that you can use to annotate a Span to carry information about the operation it is tracking.

To use the OpenTelemetry SDK api’s, you need to first add the dependency in the pom file

<!-- https://mvnrepository.com/artifact/io.opentelemetry/opentelemetry-api -->
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-api</artifactId>
        <version>1.16.0</version>
    </dependency>

Let’s add following code in the school service to add a new custom attribute i.e SchoolName in the getStudents() methods.

import io.opentelemetry.api.trace.Span;
.
...
  //Get current Span
  Span span = Span.current();
  //Add custom attributes to Span
  span.setAttribute("SchoolName", schoolname);

Once you make the changes and start the schoolService, you will be able to view the newly added custom attribute in the trace data via Zipkin. Zipkin adds all the attributes as tags in the Zipkin backend database. The Zipkin exporter is responsible for mapping the OTLP attributes data in to Zipkin format. More detailed information can be found here -> https://opentelemetry.io/docs/reference/specification/trace/sdk_exporters/zipkin/

Custom attributes added as tags in Zipkin

We are able to successfully add a custom attribute “ School Name ”to the Span and we can see that it gets stored in Zipkin and can be viewed as well. In the real world scenario, your popular website might be getting thousands of such requests & you would like to use some filters to just look at request where SchoolName = abcSchool.

Ideally, to manage the overall tracing database size, you should be applying appropriate sampling techniques to store only meaningful and important traces. Storing all the trace data could be very costly affair. But more on the management aspects of traces can be handled in a separate article.

Coming back to our example — without any filter applied , I can see a total of 5 requests on the Zipkin UI. To filter only the traces which has SchoolName as ‘abcschool’, we will be using the tagQuery feature of Zipkin UI.

No filter — all traces

Click the ‘+’ button and select the tagquery option from the dropdown and add following query to it and press ‘+’ sign again.

tagQuery=SchoolName=abcschool

Click the RunQuery button to see all traces filtered based on the custom attribute. In my case — I can see the two traces where SchoolName=abcschool.

filter result using tagQuery

Hope you understood this simple concept and can use it for viewing the traces in a more efficient way.