Instrumenting a Node.js Service with OpenTelemetry Auto-Instrumentation

Proprietary APM agents are easy to install and hard to leave. You add a vendor's package to every service, their data format becomes your data format, and your dashboards, alerts and on-call habits all grow around it. When pricing changes or you want a second backend, switching means re-instrumenting every service. That cost is the lock-in.

OpenTelemetry (OTel) separates two jobs that agents bundle together: generating telemetry in your process and analysing it in a backend. Your service emits traces, metrics and logs in the open OTLP format. Where they end up becomes a configuration decision rather than a code change. For Node.js, auto-instrumentation gets you most of the way without touching application code.

This post walks through instrumenting a Node.js service properly: the zero-code route, the ESM trap that silently drops spans, a programmatic setup worth running in production, the defaults that surprise people, and the Collector configuration that turns all of this into something you control. Package names and APIs are current as of @opentelemetry/sdk-node 0.2xx and @opentelemetry/auto-instrumentations-node 0.8x. The JS SDK moves quickly, so check the official docs when you upgrade.

How auto-instrumentation actually works

There's no magic here, just monkey-patching. When the SDK starts, each instrumentation package registers a hook on the module loader. When your code (or a dependency) loads http, express, pg or ioredis, the hook wraps the relevant functions before your code gets the module back. From then on, an incoming request starts a server span, a database query starts a client span, and the active context flows through async calls, so child spans attach to the right parent.

Two consequences follow:

• The SDK must load before anything it instruments. If express is imported before the hook is installed, it isn't patched, and nothing warns you. That's why setup happens on the Node command line (--require or --import), not inside app.js.
• CommonJS and ESM are patched by different mechanisms. require() is intercepted with require-in-the-middle. Native import needs a loader hook built on import-in-the-middle. Forget the loader hook and ESM-imported libraries are silently left uninstrumented.

The meta-package @opentelemetry/auto-instrumentations-node bundles instrumentations for the common stack: http/https, undici (Node's built-in fetch), Express, Koa, NestJS, GraphQL, gRPC, pg, mysql2, MongoDB, Mongoose, Redis and ioredis, several messaging clients, the AWS SDK, and more. Check its README for the current list. Fastify, for example, now ships its own @fastify/otel plugin instead of a bundled instrumentation.

The zero-code route (CommonJS)

If your service is CommonJS, the fastest path is two packages and a flag:

npm install --save @opentelemetry/api @opentelemetry/auto-instrumentations-node

export OTEL_SERVICE_NAME="checkout-api"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4318"
export OTEL_TRACES_EXPORTER="otlp"
export OTEL_METRICS_EXPORTER="otlp"
export OTEL_LOGS_EXPORTER="none"
export OTEL_NODE_RESOURCE_DETECTORS="env,host,os,process"
export OTEL_NODE_DISABLED_INSTRUMENTATIONS="net,dns"

node --require @opentelemetry/auto-instrumentations-node/register app.js

If you can't change the start command (a npm start script or a base image entrypoint, say), set NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register" instead.

Everything is configured through standard OTEL_* environment variables, which is a big part of the appeal. The same variables work in Java, Python and Go, so a platform team can set them once in a Helm chart. Instrumentation names in OTEL_NODE_ENABLED_INSTRUMENTATIONS and OTEL_NODE_DISABLED_INSTRUMENTATIONS drop the @opentelemetry/instrumentation- prefix, so net,dns means @opentelemetry/instrumentation-net and @opentelemetry/instrumentation-dns.

The limitation: you can't pass per-instrumentation options (such as "ignore health checks") through environment variables. Once you need those, move to a programmatic setup.

ESM: the trap that silently drops spans

If your package.json has "type": "module" or your entry point is .mjs, swapping --require for --import isn't enough. Without the loader hook, the result is confusing: you still get HTTP server spans, because http is patched another way, but they're named just GET with no route, and the Express middleware and handler spans are missing entirely. Nothing errors. The trace just looks thin.

The fix is to install the OTel loader hook. With the zero-code entry point:

node --experimental-loader=@opentelemetry/instrumentation/hook.mjs \
     --import @opentelemetry/auto-instrumentations-node/register \
     app.js

This works, but recent Node versions print a warning that --experimental-loader may be removed in favour of module.register(). The cleaner approach is to call module.register() yourself at the top of a small instrumentation file, which you'll want anyway for configuration.

A programmatic setup for production

Install the SDK alongside the auto-instrumentations:

npm install --save @opentelemetry/api @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/resources @opentelemetry/semantic-conventions

Then create instrumentation.mjs:

import { register } from 'node:module';
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { resourceFromAttributes } from '@opentelemetry/resources';
import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from '@opentelemetry/semantic-conventions';

// Install the ESM loader hook so `import` statements can be patched.
register('@opentelemetry/instrumentation/hook.mjs', import.meta.url);

const sdk = new NodeSDK({
  resource: resourceFromAttributes({
    [ATTR_SERVICE_NAME]: 'checkout-api',
    [ATTR_SERVICE_VERSION]: process.env.APP_VERSION ?? 'dev',
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-http': {
        ignoreIncomingRequestHook: (req) =>
          req.url === '/healthz' || req.url === '/readyz',
      },
      '@opentelemetry/instrumentation-net': { enabled: false },
      '@opentelemetry/instrumentation-dns': { enabled: false },
    }),
  ],
});

sdk.start();

process.once('SIGTERM', async () => {
  try {
    await sdk.shutdown(); // flush buffered spans and metrics
  } finally {
    process.exit(0);
  }
});

Run it with node --import ./instrumentation.mjs app.js. Exporters, endpoint, sampling and protocol still come from the same OTEL_* variables. Only what genuinely needs code lives in the file.

Some notes on the choices above:

• Health checks are ignored at the source. Kubernetes probes can easily be the most frequent request a service receives. They tell you nothing in a trace view, and you pay to store them.
• net and dns are disabled. They add a tcp.connect or lookup span under almost every outgoing call. That's occasionally useful when debugging connection problems, but usually noise. fs is already off by default for the same reason.
• Shutdown flushes buffers. Spans are batched in memory. Exit without sdk.shutdown() and you lose the last few seconds of telemetry, which are often the seconds you care about during a crash or deploy. If your app already handles SIGTERM (to drain HTTP connections, say), call sdk.shutdown() from that handler as the last step instead of adding a second listener.

Adding the spans auto-instrumentation can't see

Auto-instrumentation covers I/O at library boundaries. It can't see that a slow request spent most of its time in your pricing logic. Add manual spans where the business logic matters. They nest under the auto-generated spans automatically:

import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('checkout-api');

export async function priceCart(cart) {
  return tracer.startActiveSpan('priceCart', async (span) => {
    try {
      span.setAttribute('cart.items', cart.items.length);
      return await computeTotals(cart);
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      span.end();
    }
  });
}

Application code depends only on @opentelemetry/api. If the SDK isn't loaded (in unit tests, for example), these calls are cheap no-ops.

Defaults that surprise people

Turn on OTEL_LOG_LEVEL=debug once in a non-production environment and read the output. It shows which instrumentations were applied, which exporters were chosen, and what's failing. Three things tend to come up.

Metrics and logs exporters default to OTLP. If OTEL_METRICS_EXPORTER and OTEL_LOGS_EXPORTER are unset, NodeSDK creates OTLP exporters for them too. With no collector listening, export attempts fail and retry, and shutdown can stall for several seconds while it tries to flush. Set every signal explicitly, using none for the ones you don't collect.

Cloud resource detectors make network calls. By default, all resource detectors run, including AWS, GCP, Azure and Alibaba. On a laptop or in a different cloud, they probe metadata endpoints such as 169.254.169.254 and metadata.google.internal and log the failures. Set OTEL_NODE_RESOURCE_DETECTORS to what you actually run on, for example env,host,os,process,container,aws.

The zero-code entry point installs a SIGTERM listener. The register module shuts the SDK down on SIGTERM, but it doesn't exit the process. Node drops its default exit-on-SIGTERM behaviour as soon as any listener exists, so a service with no shutdown handler of its own can keep running after SIGTERM until the orchestrator sends SIGKILL. Make sure your app handles SIGTERM and exits.

Traces that cross service boundaries

Single-service traces are useful. Distributed traces are where OTel pays off. The HTTP and undici instrumentations automatically inject a traceparent header into outgoing requests and extract it from incoming ones, so a request that passes through a gateway, this service and a downstream API shows up as one trace with one trace ID. By default the SDK uses the W3C Trace Context and Baggage propagators. That matters when services in other languages, or a vendor's load balancer, need to join the same trace. For how the headers work and where propagation breaks (message queues, async jobs, proxies that strip headers), see implementing W3C Trace Context across distributed services.

The Collector: where you actually take control

You can point the SDK straight at a vendor's OTLP endpoint. For anything beyond a single service, put an OpenTelemetry Collector in between. Your services always talk to otel-collector:4318, and everything else (filtering, sampling, enrichment, which backends receive data) becomes collector configuration your platform team owns.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 15
  filter/noise:
    error_mode: ignore
    trace_conditions:
      - span.attributes["url.path"] == "/healthz"
      - span.attributes["url.path"] == "/metrics"
  batch: {}

exporters:
  otlp_http/primary:
    endpoint: https://otlp.vendor-a.example.com
    headers:
      api-key: ${env:VENDOR_A_API_KEY}
  otlp_http/evaluation:
    endpoint: https://otlp.vendor-b.example.com
    headers:
      authorization: Bearer ${env:VENDOR_B_TOKEN}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, filter/noise, batch]
      exporters: [otlp_http/primary, otlp_http/evaluation]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp_http/primary]

That config sends every trace to two backends at once. This is how you evaluate a new vendor, or migrate to one, without touching a single service. It also drops noise centrally, for services you can't easily redeploy. The filter processor is in the contrib distribution (otelcol-contrib). Older Collector releases use otlphttp instead of otlp_http, and traces: span: instead of trace_conditions, so check the README for the version you deploy.

Sampling deserves a deliberate decision too. The simplest lever is head sampling in the SDK, set through environment variables:

export OTEL_TRACES_SAMPLER="parentbased_traceidratio"
export OTEL_TRACES_SAMPLER_ARG="0.1"   # keep 10% of new traces; follow the parent's decision otherwise

The parentbased_ prefix matters: downstream services respect the upstream decision, so you don't end up with traces that have holes in them. If you need to keep every error and every slow request, which head sampling can't guarantee, look at tail sampling in the Collector. It buffers whole traces before deciding, which needs more memory and a load-balancing tier in front of it.

What auto-instrumentation won't do for you

• It doesn't make your telemetry meaningful. You get spans for every query and request, but not "which customer tier", "which feature flag" or "which pricing rule". Add business attributes and a handful of manual spans.
• It isn't free. Patching, context propagation and span export cost some CPU and memory per request. The cost is usually modest for I/O-bound services, but measure it under your own load before and after, especially on hot paths with many small spans.
• It can capture things you didn't mean to. URLs with tokens in query strings, database statements with literal values, and headers can all end up in attributes. Review what the database and HTTP instrumentations record, and scrub at the Collector if needed.
• It only covers supported libraries. An in-house RPC client or an unusual driver will show up as a gap in the trace. That's your cue for a manual span or a custom instrumentation.

Rollout checklist

• Load the SDK before the app: --require for CommonJS, --import plus the loader hook (or module.register()) for ESM.
• Check that framework spans actually appear: in an Express app, route-named server spans and handler spans. A trace of bare GET spans means the hook didn't load.
• Set OTEL_SERVICE_NAME and a version attribute on every service.
• Set OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER and OTEL_LOGS_EXPORTER explicitly, using none where you don't collect a signal.
• Limit OTEL_NODE_RESOURCE_DETECTORS to the platforms you actually run on.
• Disable noisy instrumentations (net, dns), and ignore health and readiness probes.
• Flush on shutdown, and make sure the process still exits on SIGTERM.
• Send data to a Collector, not straight to a vendor, so filtering, sampling and backend choice stay in configuration.
• Choose a sampling strategy before the bill chooses one for you.

When not to bother: a short-lived script, a single-process tool with no downstream calls, or a service whose vendor agent already gives you everything and that you have no plans to move. Everywhere else, the case for OTel is the one this post started with. The instrumentation belongs to you, and your backend is just a configuration setting.