New Relic Distributed Tracing for Observability

New Relic Distributed Tracing for Observability is a game-changer for modern applications. It provides a unified view of your entire application, from user requests to backend services.

With Distributed Tracing, you can see how different components of your application interact with each other, pinpointing performance bottlenecks and errors with ease. This is particularly useful for microservices-based architectures, where multiple services are involved in completing a single request.

New Relic's Distributed Tracing uses a technology called "span" to track the flow of requests through your application. A span represents a single operation, such as a database query or a service call, and provides detailed information about its execution. This allows you to drill down into specific parts of your application and understand what's happening in real-time.

By using Distributed Tracing, you can reduce the time it takes to identify and resolve issues, improve application performance, and deliver a better user experience.

To get started with New Relic distributed tracing, review the distributed tracing documentation first.

You'll need to deploy or update to the latest APM agent to each service involved in the call path you're interested in. This will make distributed tracing immediately available upon upgrading.

That's it, now you can access the distributed tracing information from New Relic APM and quickly identify slow traces and traces with errors.

To get started with distributed tracing, begin by reviewing the documentation to understand the process. You can find the documentation by following the first step in the getting started guide.

Reviewing the documentation will give you a solid foundation to work with. It's essential to understand the concepts and terminology involved in distributed tracing.

To deploy distributed tracing, you'll need to update the APM agent to the latest version on each service involved in the call path. This can be done by following the second step in the getting started guide.

Upgrading the APM agent will immediately enable distributed tracing on your system. You can then access the distributed tracing information from New Relic APM.

To access the distributed tracing information, you can use the advanced filtering feature to find traces by a combination of attributes. This will help you quickly identify slow traces and traces with errors.

Here are the steps to get started with distributed tracing:

With these steps, you'll be well on your way to getting started with distributed tracing.

Distributed tracing offers unparalleled visibility into your application's inner workings, giving you insights into performance, dependencies, and interactions between different components.

This enhanced visibility helps you understand how services interact and where resources are being used, leading to better scalability, efficient resource utilization, and cost savings in cloud-based environments.

With distributed tracing, you can rapidly identify and resolve performance issues or errors, reducing downtime and improving user experience.

It's like having a superpower that allows you to pinpoint bottlenecks and understand the flow of requests, making it easier to address issues more efficiently.

Distributed tracing acts as a powerful debugging tool, enabling you to trace a specific request or transaction, analyze associated logs, and quickly identify the source of errors or unexpected behaviors.

Collecting and analyzing traces over time allows teams to monitor application performance trends, identifying potential issues before they impact users and enabling continuous improvement and optimization.

This proactive approach contributes to an improved user experience, with faster response times, reduced errors, and seamless interactions increasing customer satisfaction and engagement.

Instrumentation is a crucial step in setting up New Relic Distributed Tracing. You'll need to instrument each service involved in your transactions, which means performing separate installations of the agent for each service.

You can use the PHP agent API to manually instrument applications and services, especially if you're using an unsupported library or have a non-HTTP based distributed system component. This involves modifying your service or application to create or insert distributed tracing data.

To manually instrument applications and services, you'll need to follow a two-step process: modify your service or application to create or insert distributed tracing data, and modify your service or application to accept distributed trace data created by other transactions or requests. This can be done using the PHP agent API.

Distributed tracing is a game-changer for complex systems, providing insights into how requests flow through different components.

It helps diagnose and troubleshoot issues related to latency, errors, and dependencies. This is crucial for understanding the behavior of complex systems.

Optimized resource utilization is a key benefit of instrumentation, which can be achieved through distributed tracing.

Distributed tracing provides a clear picture of how resources are being used, allowing for more efficient allocation and utilization.

Instrumentation is essential for monitoring and optimizing system performance, and distributed tracing is a key tool in this process.

Instrumentation is a crucial step in setting up distributed tracing. To instrument each service, you'll need to install an APM agent for each service involved in your transactions.

There are several ways to instrument your services, including using an APM agent, manual instrumentation with the PHP agent API, and OpenTelemetry instrumentation. You can also leverage a New Relic language agent, similar to OpenTelemetry instrumentation.

To start, click the tile below to begin the installation routine for each service. This will help you instrument each service separately. If some of your services use other languages, simply repeat the installation steps for those languages.

You can manually instrument applications and services with the PHP agent API by modifying your service or application to create or insert distributed tracing data. This involves a two-step process: creating the distributed tracing data and accepting it.

Here are some common methods for accepting distributed tracing data:

In addition to these methods, you can also use the OpenTelemetry instrumentation, which provides a standardized way of instrumenting your services. This includes using the OpenTelemetry Trace Exporter.

To identify services, figure out which services touch your request so you can instrument each of them to send trace data to New Relic.

Here's a summary of the instrumentation methods:

By instrumenting each service, you'll be able to collect valuable data on how requests flow through your distributed system, helping you diagnose and troubleshoot issues related to latency, errors, and dependencies.

To set up the trace observer, you'll need to follow the instructions in Set up trace observer. This involves collecting and analyzing all your traces.

The trace observer is a New Relic AWS-based service that collects and analyzes all your traces. You'll need to follow the instructions to set it up, and then return here with your trace observer information.

Here are the steps to set up the trace observer:

After you've set up the trace observer, you can continue with the next step to configure the agent.

The Kubernetes Cluster Explorer is a powerful tool for visualizing your application's data and deployments.

It provides a unique view of the entire data collected by the Kubernetes integration, allowing you to observe all your data at once.

Automated correlation is a key feature of the visualization capabilities of the New Relic Kubernetes Cluster Explorer.

This means you can quickly identify and investigate performance issues or incidents happening inside your application or microservices.

By using the Kubernetes Cluster Explorer, you can gain a deeper understanding of your application's behavior and performance.

PHP Command Line Programs can see an over-representation in your collection of distributed traces if you run them from the PHP command line.

This is because PHP programs run from the command line are always sampled by the agent's distributed tracer. Depending on the programs you run, these processes could see an over-representation in your collection of distributed traces.

You can opt to disable command line instrumentation to avoid this issue. To do so, you can use the per-directory newrelic.enabled setting in your newrelic.ini files.

To get started with New Relic distributed tracing, you'll need to configure the agent. Infinite Tracing configuration settings include standard distributed tracing plus information about the trace observer, but note that server-side configuration is not available for Infinite Tracing.

For Dapr tracing, the process is a bit different. You can configure Dapr to send traces to New Relic's Trace API using the Zipkin trace format. This requires a New Relic Insights Insert API key to send data to the New Relic Telemetry Data Platform.

To fine-tune your tracing, you can set the trace detail level. This allows you to prioritize spans related to external calls above other spans, which are then recorded in descending order of their duration. If you find that there are too many unimportant spans being reported for PHP function calls, you can reduce the detail of the transaction tracer by setting newrelic.transaction_tracer.detail to 0.

Dapr natively captures metrics and traces that can be sent directly to New Relic.

You'll need a New Relic Insights Insert API key to send data to New Relic Telemetry Data Platform.

To export these metrics and traces, configure Dapr to send the traces to New Relic's Trace API using the Zipkin trace format.

This integration allows you to capture and analyze distributed traces in New Relic.

Transactions are a crucial part of distributed tracing, and understanding how they work can help you configure your system for optimal performance.

A transaction is a logical unit of work in a software application, specifically the function calls and method calls that make up that unit of work.

In the context of application performance monitoring, a transaction often refers to a web transaction that represents activity from the time the application receives a web request to when the response is sent.

Distributed tracing uses transactions in three main ways: analyzing transactions, recording transactions, and describing transactions.

Analyzing transactions allows you to monitor transactions throughout the system to gain insights into its performance, and underperforming services have a knock-on effect on the rest of the services.

Recording transactions helps keep track of lots of transactions by propagating tracing context to other processes and attaching it to transaction data.

Describing transactions measures transactions, providing information such as what transactions took place and how long they lasted.

Here's a breakdown of the three ways distributed tracing uses transactions:

To view your traces, you can generate some traffic in your application so we can capture some traces. After you've located an interesting trace, you can click on the Logs tab (next to Trace details) to see log messages alongside traces in the New Relic UI.

You can refine your search for traces using the Find traces query bar, which is a quick way to narrow your search. You can start typing in the query bar or use the dropdown to create a compound query.

You can filter your span data using a variety of tools, including infinite tracing data, multi-span traces only, error filters, and histogram filters. To use the query bar, you can define spans that have certain criteria, and the search displays traces that contain those spans.

Here are some examples of how to use the query bar:

To view your traces, you can use the New Relic UI. You can see traces for a particular service by following the steps in the View traces section. This will help you identify any issues or bottlenecks in your application.

To view traces that cross accounts, you can bring your logs and tracing details together. This will make troubleshooting easier and faster. You can see log messages alongside traces in the New Relic UI.

If you've enabled logs in context, you can click on the Logs tab next to Trace details to view details related to an individual log. To view details of an individual log, click directly on the message.

New Relic Distributed Tracing provides end-to-end visibility by revealing service dependencies. You can use it to record and visualize requests through complex, distributed systems.

Here are the steps to view your traces:

Filtering your span data can be a powerful tool for troubleshooting and understanding your application's behavior. You can use the Find traces query bar to quickly narrow your search for traces.

The Find traces query bar is a quick way to narrow your search for traces. You can either start typing in the query bar or use the dropdown to create a compound query.

Queries for traces are similar to NRQL, with a few main exceptions: string values don't require quote marks, and the like operator doesn’t require %. For example, you can use either appName = MyApp or appName = 'MyApp'.

Besides the query bar at the top of the page, you can use a variety of filters in the left pane to find traces you're interested in. These filters include Infinite tracing data, Multi span traces only, Error filters, and Histogram filters.

Infinite tracing data: Select this only to show traces related to the Infinite Tracing feature. Multi span traces only: Use this to hide single-span traces. Error filters: Under Errors select the errors to filter by. Histogram filters: Below Errors at the bottom of the left pane, you can click More filters to show histogram filters.

Here are some specific filter options:

To change the filters on the charts, see left-pane filters. The trace scatter plot is a quick way to search for outlying traces, and is available on the opening page of distributed tracing if you turn off the Group similar traces toggle.

Spans are a fundamental concept in distributed tracing, and understanding them is crucial for getting the most out of New Relic's tracing capabilities.

A span represents a single unit of work done by a service, and it's the building block of a distributed trace. Spans are named and timed operations that capture important information about the operation, such as duration and host data.

There are several types of spans, including process entry spans, process exit spans, in-process spans, and client spans. Process entry spans are the first span in the execution of a logical piece of code, while process exit spans are the last span in the execution of a process.

Here's a breakdown of the different types of spans:

Spans are crucial for understanding the flow of requests through your system, and they provide valuable insights into performance and errors. By understanding the different types of spans and how they relate to each other, you can gain a deeper understanding of your system's behavior and make data-driven decisions to improve its performance.

Logging is essential for detailed analysis and forensic examination of specific incidents.

It provides a comprehensive record of application events, errors, and activities. This is crucial for debugging and compliance purposes.

Logging offers granularity at the level of individual events, whereas distributed tracing provides a transaction-level perspective.

This means that logging is perfect for getting into the nitty-gritty details of what's happening within your application.