New Relic Trace: A Comprehensive Guide

New Relic Trace is a powerful tool that helps you visualize and debug complex transactions in your application. It's like having a superpower that lets you see exactly what's happening, step by step.

With New Relic Trace, you can see the entire transaction flow, from the user's request to the database query, and everything in between. This means you can identify bottlenecks and optimize your application for better performance.

New Relic Trace can handle complex transactions with ease, including those with multiple services, databases, and external integrations. This makes it an essential tool for any serious developer or operations team.

New Relic Trace is a powerful tool that helps you gain real-time visibility into the health and performance of your entire application stack. It's an essential part of a unified application performance monitoring (APM) platform.

Distributed tracing is at the core of New Relic Trace, which consolidates records of events that take place across components of a distributed system. This includes tracking the end-to-end path of a request as it travels from service to service.

New Relic Trace provides two core pieces of information: the path a service request takes across a distributed system and the time spent to complete each service request. This helps you pinpoint where failures occur and what causes poor performance in microservices-based architectures.

Here are the key capabilities of New Relic Trace that differentiate it from other tracing tools:

By using New Relic Trace, you can gain a deeper understanding of your application's performance and make data-driven decisions to improve it.

To get started with New Relic Trace, you'll need a New Relic Insights Insert API key, which is required for the integration to send data to the New Relic Telemetry Data Platform. Dapr natively captures metrics and traces that can be sent directly to New Relic using the Zipkin trace format.

For .NET applications, you can use the OpenTelemetry Trace Exporter, which is supported by the New Relic Telemetry SDK. This allows you to leverage the OpenTelemetry implementation for .NET and export traces to New Relic.

You can also use a New Relic language agent, such as the New Relic agent instrumentation for .NET Core, which is part of the Dockerfile. This provides an alternative way to instrument your application and send data to New Relic.

To get started with New Relic, you can sign up for a perpetually free account, which comes with 100 GB/month of free data ingest and unlimited free basic users.

Distributed tracing works by assigning a unique identifier to a request.

This identifier is then propagated across different services involved in processing the request. Each service records information about the request, creating a trace that can be visualized and analyzed.

Using OpenTelemetry instrumentation, such as the New Relic Telemetry SDK, can help you implement distributed tracing in your application.

To get started with configuring and instrumenting, you'll need to meet a few prerequisites. You'll need a perpetually free New Relic account to access the necessary tools and features.

Having a free New Relic account also comes with some generous storage and user limits. You get 100 GB/month of free data ingest and unlimited free basic users.

Configuring Dapr is a crucial step in setting up your application for success. Dapr natively captures metrics and traces that can be sent directly to New Relic.

To export these metrics and traces, you'll need to configure Dapr to send them to New Relic's Trace API using the Zipkin trace format. This is the easiest way to get started with Dapr tracing.

You'll need a New Relic Insights Insert API key to send data to the New Relic Telemetry Data Platform. This API key is essential for the integration to work as expected.

Dapr tracing is a powerful feature that can help you monitor and optimize your application's performance. By following these steps, you can unlock the full potential of Dapr and improve your application's reliability and scalability.

You can use a language agent to instrument your application, just like the New Relic agent instrumentation for .NET Core, which is part of the Dockerfile.

The New Relic agent instrumentation for .NET Core is a great example of how a language agent can be used to instrument your application.

You can also leverage OpenTelemetry instrumentation, which provides a standardized way to instrument your application.

The OpenTelemetry instrumentation is a good choice if you want a flexible and customizable way to instrument your application.

New Relic trace allows you to send all trace data to New Relic using exporters for popular open source monitoring tools like OpenCensus and OpenTelemetry.

You can use an exporter for the OpenTelemetry agent to send trace data to New Relic without interrupting your use of other exporters.

Trace sampling reduces the amount of trace data collected and stored, minimizing overhead associated with distributed tracing.

There are two trace sampling methods: head-based and tail-based.

Head-based sampling randomly selects traces for collection and storage at the beginning of the trace, capturing a representative sample of activity while avoiding storage and performance issues.

Adaptive sampling is a technique applied to head-based sampling, where APM agents adapt the limit on the number of transactions collected based on changes in transaction throughput.

The rate of sampling responds to changes in transaction throughput, so if the previous minute had 100 transactions, the agent would anticipate a similar number of transactions and select 1 out of every 10 transactions to be traced.

New Relic Trace offers a powerful data management and analysis system that helps you understand your application's performance. It collects and stores data from various sources, including your application's code, infrastructure, and user interactions.

New Relic Trace can handle large amounts of data, with a retention period of up to 13 months for most data types. This allows you to analyze and learn from your application's performance over an extended period.

With New Relic Trace, you can create custom dashboards and charts to visualize your data, making it easier to identify trends and areas for improvement. You can also use its powerful query language to extract specific data points and gain deeper insights into your application's performance.

Filtering span data is a crucial step in data management and analysis. You can use the Find traces query bar to quickly narrow your search for traces.

The query bar allows you to start typing in the query bar or use the dropdown to create a compound query. This is a great way to refine your search and get specific results.

Query returns are based on span attributes, not on trace attributes. This means you define spans that have certain criteria, and the search displays traces that contain those spans.

If you use a multi-attribute filter, it is affected by the first attribute selected. This can be a bit tricky, but it's essential to understand how it works.

The query language used in the Find traces query bar is similar to NRQL, but with a few exceptions. For example, string values don't require quote marks, and the like operator doesn’t require %.

Here are some examples of how to use the query bar:

Besides the query bar, you can also use a variety of filters in the left pane to find traces you're interested in. These filters include:

These filters can help you get specific results and make it easier to analyze your data.

The infinite tracing data filter is used to show traces related to the Infinite Tracing feature. The multi-span traces only filter is used to hide single-span traces. The error filters allow you to select the errors to filter by. The histogram filters allow you to drag the sliders to change the values, such as Trace duration.

Here are some key things to keep in mind when using these filters:

A well-designed dashboard is key to making data-driven decisions. A dashboard can be a simple table or a complex visualization, but its purpose is always the same: to provide a clear and concise view of the data.

Dashboards can be used to track key performance indicators (KPIs), such as sales revenue and customer satisfaction scores. These metrics can be displayed in real-time, allowing for quick and easy analysis.

A good dashboard should be user-friendly and easy to navigate. This means using clear and concise labels, and avoiding clutter and unnecessary information.

Data visualization tools like charts and graphs can be used to make complex data more understandable. For example, a bar chart can be used to compare sales revenue between different regions.

Dashboards can be used to identify trends and patterns in the data. By analyzing these trends, businesses can make informed decisions about future investments and resource allocation.

Troubleshooting with New Relic Trace is a breeze, thanks to its robust features. You can troubleshoot performance issues by selecting a segment, which opens up a pane with three separate segment detail pages.

These tabs include the Performance tab, Attributes tab, Details tab, and Logs tab. Each tab provides valuable information to help you identify and fix problems.

The Performance tab gives you a clear view of your segment's performance, while the Attributes tab provides details about your segment's attributes. The Details tab offers a summary of your segment's details, and the Logs tab shows you the logs related to your segment.

Here are the tabs you can access from the segment detail pages:

New Relic Alerts also allow you to set up alerts and notifications based on the data collected from Dapr, Kubernetes, and other services. This way, you can stay on top of any issues that may arise.

Having the right tools at your disposal can make a huge difference when it comes to troubleshooting performance issues. With segment details, you can access a wealth of information to help you get to the bottom of things.

The segment detail pages are divided into four separate tabs: Performance, Attributes, Details, and Logs. These tabs provide a comprehensive view of your segment's performance.

The Performance tab is a great place to start, as it gives you an overview of your segment's performance metrics. You can use this information to identify any potential issues.

The Attributes tab is where you'll find information about the segment's attributes, such as its type and any relevant details.

Segments display date based on the segment type. For example, a datastore segment's details will include the database query.

The Details tab provides a more in-depth look at your segment's details, including any relevant information about its configuration.

The Logs tab is where you'll find a record of any events or errors that have occurred within the segment. This can be a valuable resource when trying to troubleshoot issues.

Here's a summary of the segment detail tabs:

Troubleshooting and monitoring your application's performance can be a daunting task, but understanding the basics of slow spans can help you identify and address issues quickly.

A slow span is a critical component to identify, as it can significantly impact your application's overall performance.

A node marked as a slow span meets the following criteria: the segment is a leaf or exit node, meaning it has no children.

This means that slow spans are typically found at the end of a trace, where the data is being processed or sent to the user.

The segment has an exclusive duration greater than or equal to 25% of the total trace duration.

This is a key indicator that a slow span is present, and it's essential to investigate further to understand the root cause of the issue.

Here are the key characteristics of a slow span:

By understanding these characteristics, you can quickly identify and address slow spans in your application, leading to improved performance and a better user experience.

Alerts are crucial for identifying and resolving issues quickly. All the data collected from Dapr, Kubernetes, or any services that run on top of them can be used to set up alerts and notifications.

You can use this data to receive alerts in your preferred channel of choice. See Alerts and Applied Intelligence for more information.

Having alerts in place helps you stay on top of potential problems before they become major issues.

Backwards compatibility is a game-changer for troubleshooting and monitoring. New Relic APM agents that support W3C Trace Context can accept and emit both the W3C Trace Context header format and the New Relic header format.

This means you can use older and newer agents together without any issues. The new agents will continue to work with older agents, so trace context will be propagated between services.

You can also have services instrumented with something other than New Relic agents, as long as that instrumentation is compliant with W3C Trace Context. This allows you to use any New Relic agent version that supports W3C Trace Context as part of a trace.

In some cases, you may have a mix of older and newer New Relic agents and non-New Relic instrumentation. But don't worry, traces can still be propagated if you always have New Relic agents that are compliant to W3C Trace Context adjacent to the pre-W3C Trace Context New Relic agents.

These New Relic agents will act as a "translator" of the New Relic proprietary trace context, making it all work smoothly.

If you're running Dapr and your applications in a Kubernetes environment, you can enable additional metrics and logs by enabling the New Relic Kubernetes integration.

This integration is a game-changer for visualizing your data and deployments, thanks to the New Relic Kubernetes Cluster Explorer. Automated correlation is part of the visualization capabilities of New Relic, making it easy to dig deeper into performance issues or incidents.

The easiest way to install the New Relic Kubernetes integration is to use the automated installer to generate a manifest, which bundles not just the integration DaemonSets but also other New Relic Kubernetes configurations like Kubernetes events and Prometheus OpenMetrics.

Kubernetes Integration is a powerful feature that allows you to collect and visualize metrics and logs from your applications running in a Kubernetes environment.

You can enable New Relic Kubernetes integration by using the automated installer to generate a manifest, which bundles the integration DaemonSets and other New Relic Kubernetes configurations.

The New Relic Kubernetes Cluster Explorer provides a unique visualization of the entire data and deployments, making it a great starting point to observe all your data and dig deeper into performance issues or incidents.

Automated correlation is part of the visualization capabilities of New Relic, allowing you to quickly identify relationships between different data points and troubleshoot issues more efficiently.

New Relic teamed up with Grafana Labs to make it easy to use the Telemetry Data Platform as a data source for Prometheus metrics. This seamless integration allows you to tap into New Relic's reliability, scale, and security.

You can use Grafana dashboard templates to monitor Dapr system services and sidecars without making any changes. New Relic provides a native endpoint for Prometheus metrics into Grafana.

A datasource can be set up easily, making it simple to start monitoring your metrics. This integration is a game-changer for anyone looking to streamline their monitoring process.