Dinesh Gurumurthy is a Staff Engineer at Datadog and the founding leader of the company’s OpenTelemetry team.
Observability is the critical part of any production-grade system. In today’s era of distributed systems, microservices and cloud-native architectures, understanding how applications behave in production has become more challenging and critical than ever.
AI has made it even more essential, as much of the code is generated using AI tools. A clear insight into the system’s functioning is critical for ensuring the system operates at scale in production. As organizations adopt microservices architectures and cloud-native practices, managing observability data becomes more complex.
OpenTelemetry has emerged as the standard for collecting distributed traces, metrics, logs, profiles and other telemetry signals.
OpenTelemetry has made it possible to manage telemetry data, something that was never possible before. At the core of OpenTelemetry is the OpenTelemetry collector, the vendor-neutral agent that plays a pivotal role in observability architectures. The collector running on our infrastructure enables the ingestion of telemetry data from various sources, transforms or enriches the data and exports it to one or more observability backends. It is extensible and pipeline-driven, with components such as receivers, processors, exporters and connectors that can be composed to suit diverse production needs.
Architecturally, this is possible with OpenTelemetry because of the OpenTelemetry protocol (OTLP) data model. It enables us to collect, transform and export telemetry data using a standard protocol. We can use the OTLP data model for input and output, allowing us to chain OTel collectors in layers.
By chaining the OTel collector, we can create a layer of OTel collectors for processing data. The layer of processing OTel collectors, also known as the gateway layer of collectors, is responsible for processing the telemetry collected from various sources and exporting it to the observability backends.
What A Telemetry Pipeline Is
The gateway Otel collector layer enables us to build a telemetry pipeline, which is nothing but a data management layer for telemetry data. With the help of OpenTelemetry collector components, we can build a directed acyclic graph (DAG) of processing nodes. At each node of the pipeline, we can do any of the following operations:
• Filtering: Removes unnecessary or redundant data to reduce noise and volume.
• Aggregation: Summarizes data to optimize storage and enable trend analysis.
• Transformation: Converts data into standardized formats for consistency.
• Enrichment: Adds contextual information to make data more valuable.
Why You Should Care
Many large enterprises are adopting this model. In most cases, a central team manages the observability vendors and observability. With the gateway model, the central team can manage all aspects of the telemetry data management using the gateway collectors. Some of the common use cases for the central processing layer include:
1. Cost Optimization: Enterprises get to choose how to sample the data and filter the telemetry, making it possible to control the cost. For traces, one of the popular means is tail sampling, which allows sampling the trace data after the trace is complete, making it much better in terms of sampling decisions.
2. Data Transformation: Different observability vendors would have different conventions, making it necessary to transform the data before exporting to the backend.
3. Multiple Destinations: We can route to multiple destinations. A common use case is to use a less expensive backend in non-production environments.
4. Cold Storage: With a gateway collector, we can export the telemetry data to cold storage like S3 for later querying or legal reasons.
Rethinking Observability Strategy
If you’re in a leadership role overseeing platform engineering, infrastructure or cloud transformation, consider these takeaways:
1. Invest in an observability strategy, not just tools. Tools change, but your strategy shouldn’t. Open standards can provide long-term flexibility.
2. Build a centralized gateway layer. Empower a core team to manage observability pipelines. This promotes consistency, governance and cost control across business units.
3. Design for multidestination routing. The ability to route to multiple destinations allows you to choose the observability backends.
4. Stay vendor-agnostic where possible. OpenTelemetry ensures you’re not boxed in by a single platform—something CIOs and CFOs both appreciate when negotiating contracts.
Final Thoughts
OpenTelemetry has become the de facto standard for telemetry collection. With the processing capabilities of the OpenTelemetry collector, we can build a pipeline for processing data, unleashing a new paradigm that gives enterprises control over their telemetry data before sending it to the observability vendors.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
This post was created with our nice and easy submission form. Create your post!
