Datadog Deep Intuition

1. One-Sentence Essence

Datadog is a tag-indexed time-series and event store with a server-side processing pipeline bolted onto the front, so the real product isn’t “dashboards” — it’s the decoupling of what you collect from what you pay to keep queryable, governed by a tagging model that turns every metric and log into a slice-able, correlatable dimension.

If you remember one thing: in Datadog, the unit of cost and the unit of query are not the same thing. Almost every confusing bill, every “why can’t I search this log,” and every “why is this metric so expensive” traces back to that single split. Hold onto it — the rest of this document is essentially that sentence unfolded.

---

2. The Problem It Solved

Before tools like Datadog, observability was three separate worlds that didn’t talk to each other. You had a metrics system (Graphite, StatsD, later Prometheus) that stored numbers over time. You had a logging system (the ELK stack — Elasticsearch, Logstash, Kibana — or Splunk) that stored text. And you had an APM tool (New Relic, AppDynamics) that traced requests. Three tools, three query languages, three bills, three login screens. When production broke at 3am, you’d see a CPU spike in Graphite, then manually go to Kibana, manually reconstruct the same time window, manually guess which host, and hope you found the log line that explained the spike. The context-switching cost real minutes during incidents, and minutes are what SEV-1s are measured in.

The deeper problem was economic. Logging systems charged you to index everything — Elasticsearch had to build inverted indices on every field of every log to make it searchable, and that indexing is expensive in CPU, disk, and RAM. So teams were forced into a terrible choice: either index everything and pay enormous bills, or filter logs before sending them and permanently lose data you didn’t know you’d need. You’d drop your DEBUG logs to save money, then have an incident that DEBUG logs would have explained, and they were gone forever.

Datadog’s two foundational bets were: (1) unify the three pillars under one tagging model so a service:checkout tag means the same thing on a metric, a log, and a trace, making correlation a click instead of a manual hunt; and (2) decouple ingestion from indexing — take in all your data cheaply, process and enrich it server-side, and let you decide afterward, dynamically which slices are worth the expensive indexing step. They branded the second bet “Logging without Limits™,” and “Metrics without Limits™” for the metrics equivalent. That decoupling — point (2) — is the single most important architectural idea in the whole platform, and it’s why your Section 1 essence is what it is.

This document is mostly about the logs and metrics pipelines specifically, because that’s where the ingestion → parsing → remapping → indexing machinery lives and where most of the operational and cost decisions get made.

---

3. The Concepts You Need

You can’t reason about Datadog without this vocabulary. These terms recur constantly; the rest of the document leans on them by name.

The universal glue: tags

• Tag — a key:value label (env:prod, service:checkout, region:us-east-1) attached to telemetry. Tags are the organizing principle of all of Datadog. They’re how you filter, group, and correlate across metrics, logs, and traces. A tag is not decoration; it’s the dimension along which everything is sliced.
• Unified Service Tagging — the convention of always setting three specific tags: env, service, and version. Datadog’s correlation features (jump from a metric to the traces to the logs for the same service) are wired to expect these three. Skipping them is the single most common reason a team’s Datadog “doesn’t correlate well.” Set them everywhere.
• Cardinality — the number of distinct combinations of tag values. host with 5 values and env with 2 values, if independent, is up to 10 combinations. Cardinality is the hidden cost driver: it determines how many time-series exist (see custom metrics) and how heavy your queries are. High-cardinality tags (user IDs, request IDs, timestamps-as-tags) are the classic way people blow up their bill without realizing it.

Metrics vocabulary

• Metric — a named numeric measurement over time: system.cpu.user, checkout.orders.count. Stored as (timestamp, value) points, each rounded to the nearest second.
• Time-series — one metric plus one specific combination of tag values. system.cpu.user{host:web-1} and system.cpu.user{host:web-2} are two distinct time-series of the same metric. A custom metric is billed per time-series, i.e., per unique metric-name + tag-value combination — this is the definition that makes cardinality matter.
• Metric submission type — how you send a metric: COUNT, GAUGE, RATE, HISTOGRAM, DISTRIBUTION. This is distinct from the in-app type Datadog stores it as. The mapping is not always one-to-one (a submitted HISTOGRAM explodes into several stored GAUGE/RATE metrics), and getting this wrong is a top-three beginner mistake. Covered fully in Sections 4 and 7.
• DogStatsD — Datadog’s extended version of the StatsD protocol, and the daemon (bundled in the Agent) that receives custom metrics over UDP/Unix socket. It aggregates locally then flushes to Datadog every ~10s. “Fire over UDP and forget” is its whole design philosophy — your app never blocks waiting on it.

Logs vocabulary

• Reserved attribute — a small set of special fields Datadog treats as structural: timestamp (the official date), status (severity: INFO/WARN/ERROR), host, service, message, plus ddsource and trace IDs. These drive core UI behavior and get mapped during preprocessing (Section 6).
• Attribute — any structured field on a log (http.status_code, duration_ms, user.email). Either present in JSON or extracted by parsing.
• Pipeline — an ordered chain of processors with a filter at the front. A log that matches the filter runs through the processors in sequence. Pipelines themselves run in sequence too. This is where parsing and remapping happen.
• Processor — a single transform step inside a pipeline: a Grok parser, a remapper, a date/status/message remapper, a GeoIP lookup, etc. (full list in Section 4).
• Grok parser — the processor that turns unstructured text ("john accessed 1.1.1.1") into structured attributes ({user: "john", ip: "1.1.1.1"}) using %{MATCHER:NAME} rules.
• Remapper — a processor that reassigns an existing field to a reserved attribute or renames it. The status remapper says “treat my level field as the official severity.” Remappers move/rename; they don’t extract. (Parsers extract; remappers remap. Keep these straight.)
• Facet — an attribute or tag you’ve promoted so it appears in the left-hand filter panel of the Log Explorer with autocomplete and basic analytics. Critically: you do not need a facet to search a value — facets are a UI/analytics convenience, not a storage requirement. This trips up nearly everyone.
• Measure — a facet on a numeric attribute (e.g. duration), which unlocks range filters and aggregations like p95.
• Index — a named bucket of retained, queryable logs defined by a filter query, with its own retention period and daily quota. This is the expensive tier.
• Exclusion filter — a rule inside an index that drops (or samples) a subset of matching logs from indexing — while still letting them be processed, turned into metrics, archived, and viewed in Live Tail.
• Live Tail — a real-time stream of logs after processing but before indexing. It shows everything you ingest, indexed or not. Your window into excluded data.
• Log-based metric — a metric generated from logs at ingest time (e.g. count of 5xx logs, p95 of a duration attribute). Crucially generated before exclusion filters run, so you keep the statistics even on logs you throw away.
• Archive / Rehydration — routing ingested logs to your own cloud storage (S3, GCS, Azure) cheaply, and later pulling a time-and-query slice back into an index on demand. Your “we dropped it but might need it for a compliance audit” safety net.

Don’t worry if these don’t all click yet. Sections 4–6 walk them in the order data actually flows, and that ordering is what makes them stick.

---

4. The Distilled Introduction

This section is the compressed version of the tutorials. I’ll walk metrics first (simpler), then logs (richer), in the order you’d actually encounter them.

4.0 — Getting data in: the Agent

Almost everything starts with the Datadog Agent, a lightweight Go process you run on every host (or as a DaemonSet — one per node — in Kubernetes). It collects ~75–100 system metrics every 15–20 seconds, runs checks (integration plugins for Postgres, NGINX, Redis, etc.), receives custom metrics via the embedded DogStatsD server, optionally collects logs, and optionally runs an APM agent for traces. It buffers locally and forwards to Datadog over HTTPS. The Agent never stores anything long-term — it’s a collection-and-forward hub, not a database.

Install is a one-liner per platform; the only required config in datadog.yaml is your API key and your site (e.g. datadoghq.com for US1, datadoghq.eu for EU — this matters, the sites are separate data residencies and you can’t query across them). The one config line worth setting on day one:

# datadog.yaml — host-level tags attached to EVERY metric, log, trace,
# and event from this host. This is your cheapest, highest-leverage config.
tags:
  - env:prod
  - team:payments

In containers, you don’t write files — you use Autodiscovery: the Agent watches for containers and reads pod annotations / container labels to know how to collect from each one. Note a sharp edge: Autodiscovery identifies containers by name, not image.

4.1 — Metrics: the five types and why they’re not interchangeable

This is the part people skip and then suffer for. When you send a metric, you pick a submission type, and that choice determines what graphs and math are even possible later.

• GAUGE — a snapshot. “Disk is 60% full right now.” The last value in the flush interval wins. Use for things that rise and fall and where “the current value” is meaningful: memory, queue depth, active connections, temperature.
• COUNT — a tally over the interval. “47 orders this interval.” Use for events you add up. In DogStatsD you increment it; Datadog stores it and you query it with sum.
• RATE — a count normalized per second. COUNT and RATE are the same concept (variation over time) with different normalization; you can flip between them at query time with .as_count() and .as_rate().
• HISTOGRAM — you send many raw values during an interval and the Agent computes aggregates (max, median, avg, 95th percentile, count) and ships those. Use for measuring distributions of something happening on one host: request durations on this server. Gotcha that bites everyone: one submitted histogram becomes ~5 separate stored metrics (.max, .median, .avg, .95percentile, .count), each billed separately. (Section 7.)
• DISTRIBUTION — like a histogram, but the Agent ships the raw values and Datadog computes percentiles server-side, globally across all hosts, using a clever data structure (DDSketch). This is the one you want for “p99 latency of my service” (not per-host), because percentiles don’t average — you can’t take the p99 of five hosts by averaging their p99s, and distributions are how Datadog solves that correctly.

Rule of thumb: GAUGE for “what is it now,” COUNT for “how many,” DISTRIBUTION for “what’s the p95/p99 across my whole service,” HISTOGRAM only when you specifically want per-host Agent-side aggregation.

Sending a custom metric via DogStatsD is one line. The wire format is name:value|type|@sample_rate|#tags:

from datadog import statsd
statsd.increment('checkout.orders', tags=['env:prod', 'payment:card'])  # COUNT
statsd.gauge('checkout.cart.size', 12, tags=['env:prod'])                # GAUGE
statsd.distribution('checkout.latency_ms', 87, tags=['env:prod'])        # DISTRIBUTION

It fires over UDP to the local Agent on port 8125 and returns instantly — if the Agent is down, your app doesn’t even notice. The Agent aggregates and flushes every 10s.

4.2 — Querying metrics: every query is time-then-space aggregation

This is the mental model for the whole graphing experience (dashboards, monitors, and notebooks share it). Every metric query is aggregated twice:

1. Time aggregation (rollup). Datadog can’t draw 4,320 points (a day at 15–20s resolution) on a 300-pixel graph, so it buckets points in time and combines each bucket into one value. Default combine is avg. As you zoom out, buckets get wider and granularity drops — this is why graphs visually “smooth out” as you widen the window. You override with .rollup(max, 60) etc.
2. Space aggregation. A metric reported by 50 hosts is 50 time-series. To draw one line you must combine across the tag-space — avg:system.cpu{*} averages all hosts; avg:system.cpu{*} by {region} gives you one line per region.

So a full query reads avg:system.cpu.user{env:prod} by {host} = space-aggregator : metric {filter} by {grouping}. Internalize “time then space” and the query editor stops being mysterious.

4.3 — Logs: the journey from raw line to searchable event

Now the richer pipeline. A log enters Datadog and goes through a fixed sequence of stages. I’ll name them here and detail the mechanics in Section 6, because the order is the whole point:

Ingest → Preprocess (JSON only) → Pipelines & Processors → Generate log-based metrics → Exclusion filters → Index.

Walking it as a practitioner:

Collection. Logs come from the Agent tailing files / container stdout, or directly via the HTTP intake API, or forwarders (Lambda, Fluentd, etc.). Send JSON if you possibly can — Datadog parses JSON automatically and you skip most of the parsing pain below.

Preprocessing (JSON logs). Before any pipeline runs, Datadog looks for reserved attributes in your JSON and maps them: it finds status or level and sets the official severity; finds timestamp/@timestamp/date and sets the official date; finds host/hostname and sets the host; finds message and uses it as the searchable body. If your fields have nonstandard names, you edit the preprocessing config to add them to the precedence list. (One reserved attribute is special: host is locked in during preprocessing and can never be remapped afterward — Section 7.)

Pipelines. Your log is tested against each pipeline’s filter (e.g. source:nginx). If it matches, the pipeline’s processors run in order. The headline processor is the Grok parser, which turns the unstructured message into attributes:

# Grok rule: name the rule, then describe the line with %{MATCHER:attribute}
access %{ipOrHost:network.client.ip} %{notSpace:http.ident} \[%{httpdate:date}\] \
"%{word:http.method} %{notSpace:http.url}" %{number:http.status_code}
# turns: 1.1.1.1 - [10/Oct/2024:..] "GET /cart" 200
# into:  {network.client.ip:"1.1.1.1", http.method:"GET", http.url:"/cart", http.status_code:200}

Then come the remappers — the “remapping” you asked about. After extracting fields you normalize them: a status remapper points Datadog at your severity field; a date remapper picks the real timestamp; a service remapper sets the service; a plain remapper renames clientip/client_ip/clientIP from five different services all to the one standard network.client.ip. Other processors: GeoIP (IP → country/city), category (bucket values into a new field, e.g. status code → OK/warning/error), lookup (map a code to a human label via a reference table), arithmetic, URL parser, user-agent parser.

Don’t reinvent the wheel: Datadog ships ~250+ integration pipelines for common sources (NGINX, Postgres, Java, etc.). Tag your logs with the right source and the matching pipeline installs and parses automatically. You clone-and-edit if you need changes.

Log-based metrics. Before anything gets dropped, you can define metrics off the log stream: “count of status:error by service,” “p95 of the duration attribute.” These keep flowing even for logs you’re about to exclude.

Exclusion filters & indexes. Finally the log hits an index (matched by filter, first index wins). The index has a retention period and a daily quota, and exclusion filters that drop or sample high-volume low-value logs (e.g. “index 100% of 4xx/5xx, but only 5% of 200s”). Excluded logs still went through processing, still became metrics, still go to your archive, and still show in Live Tail — they’re just not sitting in the expensive searchable index.

4.4 — Searching, dashboarding, alerting (the UI loop)

Once logs are flowing and structured, the daily loop is: Log Explorer to search and pivot, facets to filter quickly, dashboards to make views durable, monitors to get paged. Search syntax: bare words match the message (timeout); @attribute:value matches structured attributes (@http.status_code:500); reserved attributes skip the @ (service:checkout status:error); Booleans AND/OR/-; wildcards service:web*. Click any field in a log’s side panel to filter on it or promote it to a facet. Group and visualize to turn raw logs into timeseries or top-lists, then “Export to Dashboard.” Monitors wrap the same query in a threshold and a notification. All of this is Section 9’s quick reference and Section 4.2’s “time then space” model applied to logs.

---

5. The Mental Model

Four ideas. Internalize these and you can predict Datadog’s behavior without the docs.

Core Idea 1 — Ingestion and indexing are separate stages with separate prices, and processing happens between them.

This is the master key. Data comes in cheaply (ingestion), gets enriched server-side (processing), and only then do you decide what to make expensively queryable (indexing). For logs it’s literally “Logging without Limits”; for metrics it’s “Metrics without Limits” (ingest the metric with all its tags, but only pay to index the tag combinations you actually query).

This predicts:

• You can fix a broken parser or change what you index without touching your applications — it’s all server-side, retroactive from “now” forward.
• Log-based metrics survive even when the underlying logs are excluded, because metric generation sits before exclusion in the pipeline. (You keep the trend even after throwing away the raw data.)
• During an incident you can disable an exclusion filter and instantly get full visibility on logs you were sampling — no app redeploy, no config push to servers.
• Your bill has two knobs per data type (ingest volume and index/retention), and the expensive one is almost always indexing/retention. Cost optimization = “ingest broadly, index narrowly.”

Core Idea 2 — Tags are the schema, and cardinality is the cost.

There is no table schema in Datadog. The “schema” is the set of tags you attach. Every distinct combination of tag values on a metric is a separate billed time-series; every facet on a log is a tag/attribute you chose to make pivotable.

This predicts:

• Adding a high-cardinality tag (user ID, request ID, raw URL with query params) to a custom metric can multiply your time-series count — and thus your bill — by thousands. The cost “scales with your most granular tag.”
• Correlation across metrics/logs/traces just works when (and only when) the same tags are present on all three — which is the entire reason Unified Service Tagging (env/service/version) exists.
• “Group by X” is only possible if X is a tag/facet. If you didn’t tag it, you can’t slice by it. Tagging is a decision made at emit time (mostly) that constrains every future question you can ask.

Core Idea 3 — Every query is time-aggregation then space-aggregation, always.

There’s no such thing as an un-aggregated metric query in Datadog. The points you see are local aggregates, not raw submissions. The query language is fundamentally “combine over time, then combine over the tag-space.”

This predicts:

• Graphs smooth out as you zoom out — wider time buckets, more averaging. A spike visible at 1h resolution can vanish at 1-week resolution. (Use max rollup if you’re hunting spikes.)
• Percentiles are dangerous to aggregate: avg of per-host p99s ≠ true p99. This is why DISTRIBUTION metrics exist — they preserve enough structure (DDSketch) to compute globally-correct percentiles.
• COUNT/RATE metrics need sum-based aggregation to be meaningful; the UI auto-appends .as_count() and disables interpolation for them, because averaging counts or filling gaps would lie to you.

Core Idea 4 — The log pipeline order is fixed, sequential, and the source of most surprises.

Ingest → preprocess → pipelines (in order) → metrics → exclusion → index. Each stage’s output is the next stage’s input. Pipelines run top-to-bottom; processors within a pipeline run top-to-bottom; the first matching index wins; the first matching exclusion filter wins.

This predicts:

• You cannot filter a pipeline on an attribute that the pipeline itself extracts — the filter is evaluated before the processors run. (You’d need a nested pipeline or an earlier pipeline.)
• Reordering pipelines can silently change results downstream, because a remap in pipeline 2 changes the input to pipeline 3.
• A field you extract late can’t be used by something that runs early. When “my parsed attribute isn’t available where I expect,” it’s almost always an ordering problem.
• Preprocessing runs before everything, which is why host (assigned there) can’t be changed later, and why nonstandard JSON timestamp fields need fixing in preprocessing, not in a regular pipeline.

---

6. The Architecture in Plain English

Let’s narrate the two journeys end to end.

A custom metric’s journey

Your app calls statsd.distribution('checkout.latency_ms', 87, tags=['env:prod','region:eu']). The DogStatsD client serializes that into a tiny UDP datagram (checkout.latency_ms:87|d|#env:prod,region:eu) and fires it at 127.0.0.1:8125. Your app moves on immediately — UDP is fire-and-forget, so even if the Agent is overwhelmed or down, your request path never blocks or errors.

Inside the Agent, the DogStatsD server receives the datagram. It doesn’t forward each point individually — that would be absurdly chatty. Instead it aggregates in memory over a 10-second flush interval, enriching each metric with origin tags (which container/pod sent it, plus the host-level tags from datadog.yaml). Meanwhile the Agent’s Collector is independently running checks every 15s, gathering system and integration metrics. Everything converges on the Forwarder, which buffers payloads in memory (so a brief network partition doesn’t lose data — though if the buffer fills, oldest points are dropped to bound memory) and ships them over HTTPS to your Datadog site.

Server-side, Datadog stores the points as time-series keyed by metric-name + tag-set, rounded to the second. For a DISTRIBUTION, it stores the DDSketch structure so it can later compute any percentile across any tag grouping. When you open a graph, the backend (a) scans for time-series matching your {filter}, (b) applies time rollup to bucket points to your screen resolution, (c) applies space aggregation to combine series per your by {...}, then (d) applies any functions (anomaly detection, arithmetic, smoothing). What you see is the output of that four-step pipeline — never the raw submissions.

Where does state live? Nowhere on the Agent (it’s a forwarder). It lives in Datadog’s time-series store, indexed by tags. That’s the whole system.

A log’s journey

A log line is born — say NGINX writes 1.1.1.1 - [10/Oct/2024:13:55:36] "GET /cart" 200 1043 to a file, or your Java app emits a JSON object to stdout. The Agent (with logs_enabled: true) tails it and ships it over an encrypted TCP connection to the intake endpoint. On intake, the log inherits host-level tags automatically — this is why a log “knows” its region and env without you putting them in the log body.

At intake, preprocessing runs (for JSON). Datadog scans the reserved-attribute precedence lists: it grabs your level field as official status, your @timestamp as official date, your host field as official host (permanently — this is the one-way door), your message field as the searchable body. Non-JSON logs skip this and arrive as a raw message string awaiting a Grok parser.

Now the log enters the pipeline gauntlet. It’s tested against pipeline 1’s filter; if it matches source:nginx, the NGINX pipeline’s processors fire in sequence — Grok parser cracks the message into http.method, http.url, http.status_code, network.bytes; a date remapper pins the timestamp; a status remapper maps the status code; a GeoIP processor turns the client IP into a country. The (now-enriched) log moves to pipeline 2, then 3, and so on. Each pipeline sees the cumulative result of the prior ones. Nested pipelines let you do coarse-then-fine routing (filter by team:backend, then a nested pipeline per language).

Just before retention decisions, log-based metrics are computed off the enriched stream. Then the log reaches the indexes. It’s tested against each index’s filter; it lands in the first index it matches. That index’s exclusion filters decide whether to keep it searchable — maybe it’s a status:info 200 and the filter samples those at 5%, dropping this one from the index. Dropped or not, the log was already (a) processed, (b) turned into any matching metrics, (c) eligible for the archive (routed to your S3/GCS bucket), and (d) visible in Live Tail’s real-time stream. Only the indexed copy — the expensive, fully-searchable-for-N-days copy — is gated by exclusion.

When you search the Log Explorer, you’re querying the indexed logs. When you watch Live Tail, you’re seeing the post-processing stream regardless of indexing. When you rehydrate, you pull a query-slice out of cold archive storage back into a temporary searchable index. Three different views onto data at three different stages and price points — exactly what Core Idea 1 predicts.

---

7. The Things That Bite You

Each of these connects to a mental-model idea. They’re the first-6-months bugs.

1. One HISTOGRAM metric is secretly five (or more) billed metrics. You’d expect request.latency (histogram) to be one custom metric. It’s not — the Agent emits .max, .median, .avg, .95percentile, .count as separate stored metrics, each multiplied by your tag cardinality (Core Idea 2). Teams get a surprise bill and don’t understand why. Handle it: know the multiplier, trim histogram_aggregates in datadog.yaml to only what you use, and prefer DISTRIBUTION when you want service-wide percentiles anyway.

2. Submitting the wrong metric type quietly corrupts your data forever. Send a counter as a GAUGE and “total requests today” becomes meaningless (you get the last value, not the sum). The data submitted before you fix the type behaves incorrectly even after you change the in-app type (Core Idea 3). Handle it: decide the type deliberately at the start; if you must change it, often cleaner to submit a new metric name.

3. host is decided in preprocessing and can never be remapped. Every other reserved attribute has a remapper; host does not. A stray host/hostname/syslog.hostname field in your JSON will hijack the log’s host (Core Idea 4 — preprocessing runs first and is final), misattributing it and breaking host-tag inheritance. Handle it: don’t emit a host field unless you mean it; fix it in the preprocessing config, not a pipeline.

4. You think you need a facet to search — you don’t. New users create facets for everything, hit the soft limit (~1000 recommended max), and slow things down. Facets are for the filter panel and analytics; full-text and @attribute:value search work on any value in your indexed logs without a facet. Handle it: only facet attributes you frequently filter or aggregate on. (Core Idea 2: facets are chosen dimensions, not free.)

5. A pipeline can’t filter on an attribute it extracts. You write a pipeline filtered on @http.status_code:500, but that attribute is produced by the Grok parser inside that same pipeline, so the filter (evaluated first) never matches (Core Idea 4). Handle it: filter on something available at entry (source:nginx), parse inside, and use nested pipelines for finer routing.

6. Logs older than 18 hours are silently dropped at intake. If your date remapper points at a wrong/misparsed timestamp, or a timezone is off, logs can be back-dated past the 18h window and vanish — no error, just missing. (Core Idea 4: the date is set early and gates everything.) Handle it: watch datadog.estimated_usage.logs.drop_count; verify date parsing with the in-UI sample tester; remember timestamps are stored UTC and displayed local.

7. Graphs lie when you zoom out (benignly). A latency spike at 1h resolution disappears at 1-week resolution because the default avg rollup smooths it into the bucket (Core Idea 3). People conclude “we had no incident.” Handle it: use .rollup(max, ...) when hunting for spikes; know that what you see is always a local aggregate.

8. High-cardinality tags are a silent bill bomb. Tagging a custom metric with user_id or a unique request_id can create millions of time-series. The cost scales with the most granular tag (Core Idea 2). Handle it: never tag metrics with unbounded values; use logs/traces for high-cardinality detail and Metrics without Limits to drop tag dimensions you don’t query.

9. Exclusion filters are first-match and easy to over-broaden. If a log matches several exclusion filters, only the first applies; and a too-broad free-text exclusion can drop far more than intended, especially with short search strings matching unexpectedly. Handle it: order filters carefully, prefer attribute-based filters over free-text, and use the Log Pipeline Scanner to trace exactly which pipeline/processor/filter touched a given log.

10. @ is reserved in log search. You can’t use @ as a literal in a message search — it’s the attribute-search prefix. Special characters generally aren’t searchable in the raw message; you must parse them into an attribute first. Handle it: parse, then search the attribute.

---

8. The Judgment Calls

The decisions experienced users navigate. There’s a real tradeoff in each.

1. HISTOGRAM vs DISTRIBUTION for latency. Histogram aggregates Agent-side per host (cheaper compute, but per-host percentiles you can’t correctly combine, and a 5× metric multiplier). Distribution ships raw and aggregates server-side (correct global percentiles, threshold queries, but you pay ingestion on the raw values). Experienced choice: DISTRIBUTION whenever you care about service-level p95/p99 across hosts — which is almost always for user-facing latency. Histogram only when per-host Agent-side aggregation is specifically what you want and volume is high.

2. COUNT vs GAUGE for event-ish things. Both can “work” for something like queue size processed. Signal: if the question is “how many total over time,” it’s a COUNT (sum-able); if it’s “what’s the level right now,” it’s a GAUGE. When in doubt, COUNT is more flexible because you can always derive a rate from it, but you can’t recover a sum from a gauge.

3. Index everything vs aggressive exclusion. Indexing all logs gives instant full searchability (great during a SEV) but is the dominant cost (~$1+/million events). Experienced choice: the 80/20 rule — index 100% of errors/warnings and a sample of high-volume success logs; archive everything; keep log-based metrics on the excluded stream so you don’t lose trends. Crank exclusions down (or off) during incidents.

4. Standard index vs Flex Logs vs Archive. Standard index = fast, frequent querying, day-to-day troubleshooting, shorter retention. Flex Logs = cheaper storage for high-volume, longer-retention, occasionally-queried logs (10B+/month, 30+ days). Archive = cheapest, your own bucket, query only via rehydration. Signal: query frequency and latency tolerance. Hot troubleshooting → standard; “might need it for an audit in 6 months” → archive; “large volume, query sometimes” → Flex.

5. JSON logging vs text + Grok. Text logging + a Grok parser is flexible and requires no app change, but parsing is brittle, costs pipeline CPU, and breaks when log formats drift. JSON is parsed automatically, robust, and cheaper to process. Experienced choice: emit JSON from the application wherever you control the code; reserve Grok for third-party/legacy sources you can’t change.

6. Integration pipeline vs custom pipeline. The 250+ prebuilt pipelines are battle-tested and auto-maintained but read-only. Signal: if your source is standard (NGINX, Postgres, a known language logger), use the integration pipeline and only clone-and-edit if you have genuinely custom fields. Don’t hand-roll a parser for a format Datadog already supports.

7. Where to set tags — Agent host tags vs app code vs pipeline remapper. Host-level tags in datadog.yaml are cheap and blanket everything from a host. App-level (Unified Service Tagging) is precise and travels with the service. Pipeline remappers fix things after the fact. Signal: infrastructure facts (env, team, region) → host/Agent; service identity (service, version) → app code; cleanup/normalization of inconsistent third-party fields → remapper. Don’t use remappers to do work that belongs at emit time.

8. Distribution percentiles: enable globally-accurate percentiles or not. Turning on percentile aggregations for a distribution unlocks p99/threshold/stddev queries but increases the indexed custom-metric count. Signal: enable on the handful of latency/SLI metrics where p99 actually drives decisions; leave it off on the long tail.

9. Monitor evaluation: avg vs max vs min vs sum over the window. avg smooths noise (fewer false pages, but misses brief spikes); max catches any spike (sensitive, noisier); sum is for counted events. Signal: page on avg for sustained-degradation SLOs, max for “if it ever crossed this line” safety conditions, sum for error-count thresholds. And avoid .rollup() inside monitor queries — the rollup buckets align to UNIX time, not your evaluation window, so you can alert on a partial bucket; delay evaluation if you must use one.

10. Simple alert vs multi-alert. Simple aggregates all sources into one alert (quiet, but you lose per-entity detail). Multi-alert fires per group (per host/service/device — actionable, but potentially noisy at scale). Signal: multi-alert when the recipient needs to know which entity broke (disk filling on which device); simple alert for aggregate health where one notification suffices.

11. Log-based metric vs just indexing the logs. A log-based metric is permanent, cheap, and survives exclusion — but it’s lossy (you can’t drill into individual events later). Signal: if you only need the trend/count long-term (e.g. “5xx rate over months”), make it a metric and exclude the raw logs. If you need to investigate individual occurrences, you need them indexed.

---

9. The Commands/APIs That Actually Matter

Grouped by task. Datadog is mostly a UI, but these are the syntaxes and primitives you reach for constantly.

DogStatsD metric submission (in your app)

statsd.increment('orders.count', tags=['env:prod'])        # COUNT — tally events
statsd.gauge('queue.depth', 42, tags=['env:prod'])         # GAUGE — current level
statsd.distribution('req.latency_ms', 87, tags=['svc:api'])# DISTRIBUTION — global percentiles
statsd.histogram('render.ms', 12, tags=['host:w1'])        # HISTOGRAM — per-host aggregates

Raw wire format if you ever hand-roll it: name:value|type|@sample_rate|#tag:val,tag2.

Metric query syntax (dashboards, monitors, notebooks)

avg:system.cpu.user{env:prod} by {host}        # space-agg : metric {filter} by {group}
sum:orders.count{*}.as_count()                 # treat as raw counts, not rates
avg:system.disk.free{*}.rollup(max, 60)        # custom time bucketing (catch spikes)
p99:trace.web.request.duration{service:api}    # percentile on a distribution

Reach for functions via the Σ button: .as_count()/.as_rate(), .rollup(), anomaly/outlier detection, per_second(), diff(), smoothing, timeshift (week-over-week comparisons).

Log search syntax (Log Explorer / monitors / dashboard widgets)

service:checkout status:error                  # reserved attrs need no @
@http.status_code:[500 TO 599]                 # @ for structured attributes; ranges work
timeout AND -@http.method:GET                  # bare word hits message; - negates; AND/OR
service:web*                                    # wildcard
@network.client.ip:CIDR(10.0.0.0/8)            # CIDR matching on IP attributes
*:searchterm                                   # full-text across all attributes (not just message)

Grok parsing rules (inside a Grok processor)

# %{MATCHER:attribute_name:optional_filter}
myrule %{word:user.name} id:%{integer:user.id} on %{date("MM/dd/yyyy"):date} %{ipv4:client.ip}

Workhorse matchers: notSpace (greedy-but-stops-at-space — your default), word, integer, number, ipv4/ip/ipOrHost, date(...), regex(...) for anything custom, data (lazy .*? — use sparingly, it causes timeouts on long logs). Use helper rules to reuse patterns, the .* star trick to build a rule one attribute at a time, and always paste a sample log to test against in the UI.

The pipeline configuration surface (Logs → Pipelines)

The key processors, by job: Grok Parser (extract from text), Remapper (rename/move fields), Status/Date/Service/Message Remapper (assign reserved attributes), GeoIP Parser (IP → location), Category Processor (bucket values), Lookup Processor (code → label via reference table), Arithmetic/URL/User-Agent parsers. The Pipeline Scanner traces which pipeline+processor touched any given log — your debugging tool.

Index & cost controls (Logs → Pipelines → Indexes)

Each index: a filter query, a retention period, a daily quota (+ warning threshold), and exclusion filters (drop or sample by %). The Log Management – Estimated Usage dashboard (auto-created) shows ingested vs indexed vs excluded volumes — your cost cockpit.

Building durable views

From the Explorer: Save View (query + columns + time + facets), Generate Metric (log → metric), Export to Dashboard (graph → widget). Dashboards support template variables ($env, $service) for dynamic scoping. Monitors wrap any of these queries with thresholds, avg/max/min/sum evaluation, simple/multi alerting, and template variables in the notification ({{host.name}}, {{value}}) routed to Slack/PagerDuty/email/etc.

---

10. Actually Using It: The End-User Workflow

Everything above is about how the machine works and how data gets in. This section is about what you actually do once someone hands you a configured Datadog and says “go.” This is the daily loop: explore → visualize → make it durable → get paged → investigate. It leans entirely on the mental models from Section 5 — especially “time-then-space aggregation” (Core Idea 3) and “tags are the schema” (Core Idea 2) — so if a query behaves unexpectedly here, that’s where the answer lives.

10.1 — The one workflow underneath everything: explorer → graph → widget

Here’s the thing that makes Datadog click: the Metrics Explorer, a dashboard widget, a notebook cell, and a monitor are all the same query editor wearing different hats. Learn the editor once and you’ve learned all four. The editor is the same “space-aggregator : metric {filter} by {grouping}, then functions” structure everywhere. So the natural flow is:

1. Explore in the Metrics Explorer (for metrics) or Log Explorer (for logs) — this is your scratchpad, no commitment, just poke at data.
2. When a graph answers a question worth keeping, Export to Dashboard (metrics) or Export to Dashboard / Generate Metric (logs).
3. When a graph is worth alerting on, lift the same query into a monitor.
4. When you’re investigating a live problem, pull graphs into a notebook to narrate the story.

You almost never build a monitor or a complex widget from a blank query. You explore until the graph is right, then promote it. Internalize that and you stop fighting the tool.

10.2 — Building a graph: the editor, step by step

Open any graph editor (say, add a Timeseries widget to a dashboard). You’ll configure, in order:

• The metric — type to search (system.cpu.user, trace.web.request.duration). Don’t know the name? Browse the Metrics Explorer or Metrics Summary first.
• from (the filter / space scope) — tags that narrow which time-series you pull: env:prod, service:checkout. Empty = everything ({*}). This is where template variables plug in (below).
• avg by (the grouping / split) — split the result into one line per value of a tag: by {host} gives a line per host, by {region} one per region. This is space aggregation made visible.
• The space aggregator — avg/sum/min/max/p95 — how to combine the series in each group. (Remember: averaging per-host p99s is a lie; use a distribution and p99 here.)
• Functions (the Σ button) — applied last: .as_count(), .rollup(max, 60), anomaly detection, per_second(), week-over-week timeshift, arithmetic across two metrics.

Two refinements you’ll use constantly: formulas (graph a / b * 100 — e.g. error rate = errors ÷ total requests) and the JSON tab (every widget has one; copy a query from a colleague’s dashboard, or paste one Datadog support gives you). When a graph looks wrong, the first diagnostic question is always: is this a time-aggregation artifact (zoomed out, spike smoothed by avg-rollup) or a space-aggregation artifact (wrong aggregator, missing group-by)? That single question (Core Idea 3) resolves most “the graph looks weird” confusion.

10.3 — The widget vocabulary (pick the right picture)

Datadog has ~15+ widget types; you’ll use six 90% of the time. The skill isn’t knowing all of them — it’s matching the widget to the shape of the question:

• Timeseries — change over time. The default, the one you reach for first. “How has latency moved this week?” Display sub-types matter: lines for gauges/rates, bars for counts (a bar is a discrete bucket-total, which is what a count is), area for stacked composition.
• Query Value — one big number. “What’s the current error rate / active users / today’s order count?” Great for the top strip of a dashboard. Add conditional formatting (red if > threshold) to make it glanceable.
• Top List — ranked bars. “Top 10 endpoints by p95 latency,” “noisiest services by log volume.” The fastest way to answer “which thing is worst right now.”
• Table — multi-column, multi-metric per row. “For each service: request count, error rate, p95 — sorted by errors.” The workhorse for service-overview rows.
• Heatmap — distribution across many sources over time (density shown by color). “How is latency distributed across all hosts?” — surfaces bimodal behavior a single avg line hides.
• Log Stream — live list of actual log events embedded in a dashboard. Pair a timeseries of status:error counts with a log stream of those errors directly beneath it — the count tells you something’s wrong, the stream tells you what.

Supporting cast: Distribution (one-moment histogram), SLO (error-budget status — see 10.7), Note/Free Text (markdown headers and runbook links to structure the board), Geomap, Service Map. Don’t overthink the long tail; master the six.

Taste tell: a beginner dashboard is fifteen identical line graphs. An experienced one mixes a Query-Value strip on top, a Top List and Table for “what’s worst,” Timeseries for trends, and a Log Stream for context — because each question has a shape, and the widget matches it.

10.4 — Building a dashboard that doesn’t rot

A dashboard is just a saved arrangement of those widgets. Anyone can drop ten graphs on a grid; the difference between a dashboard people use and one that rots is a few deliberate choices.

Start from a template, not a blank page. Datadog ships out-of-the-box dashboards for every integration (Kubernetes, Postgres, Redis, RUM…). Clone one and edit. You can also Cmd+C/Cmd+V individual widgets between dashboards — steal the graphs you like.

Layout note (historical baggage you’ll still hear): older Datadog had two dashboard kinds — Timeboards (all widgets share one time range; built for correlated time-based debugging) and Screenboards (free-form, per-widget timeframes, mixed media). The current unified Dashboard is a responsive grid that combines both — you can still set individual widgets to their own timeframe for side-by-side comparison. When you see “Timeboard/Screenboard” in old docs or a colleague’s vocabulary, that’s what they mean; just pick “Dashboard.”

Template variables are the single highest-leverage feature and the thing beginners skip. They turn one dashboard into hundreds. Define $env, $service, $region as variables at the top; make every widget’s from filter listen to them (env:$env.value, service:$service.value). Now a dropdown re-scopes the whole board from prod to staging, or from checkout to search, instantly. The rule of thumb: start with $env before you place a single widget — adding variables later means editing every query by hand. The URL encodes the selection (&tpl_var_env=prod), so a scoped view is shareable as a link, and you can Save View to bookmark a specific combination.

Structure for the eye, macro → micro. Use Groups (collapsible containers) and Tabs to section a large board, and Note widgets as headers. Order top-to-bottom the way you’d actually troubleshoot: health summary (Query Values) at the top, then “which component” (Top Lists / Tables), then deep diagnostics (Timeseries, Log Streams) below. A good dashboard reads like a diagnostic flowchart, not a data dump.

Two genuinely different dashboard purposes (don’t conflate them): an overview/status board (high-level KPIs, lives on a TV, answers “are we healthy?”) and a runbook/investigation board (dense, scoped by template variables, walks a responder through “it’s broken — here’s every graph you need, in order”). The best runbook dashboards start as a scratchpad during one incident and accrete graphs as you learn the system.

10.5 — Log analytics: graphing and dashboarding logs (the other half)

Everything in 10.2–10.4 assumed you were graphing metrics. But logs are a co-equal dashboarding source, and the workflow has one fundamental difference you must internalize: with metrics you graph time-series that already exist; with logs you compute numbers out of raw events at query time. A metric like system.cpu arrives pre-aggregated. A pile of log events is just text until you tell Datadog “count these, grouped by that” — the aggregation is the thing you’re building. This is “time-then-space aggregation” (Core Idea 3) applied to events instead of points, and once it clicks, the log query editor and the metric query editor feel like the same tool.

The mechanic: “group into” + “aggregate” + “measure”. In the Log Explorer, switch from the List view to an aggregate visualization, and the query editor sprouts a group-into control. You have three ways to group:

• Fields — the everyday one. Group by one or more facets (status, service, @http.status_code). This is the “group by” you know from metrics.
• Patterns — cluster logs with similar message text into patterns (e.g. all “connection timeout to host X” lines collapse into one row regardless of which host). The fastest way to find your noisiest error and the single most useful log-analytics feature for triage. Based on a 10,000-log sample.
• Transactions — stitch a sequence of related logs (a user session, a request crossing microservices) into one logical unit by a shared ID.

Then you pick what to compute per group:

• Count — how many logs. (count of status:error by service.)
• Unique count (cardinality) — distinct values of a facet. (unique count of @user.id = how many distinct users hit this.)
• A measure’s statistic — for a numeric attribute, choose avg/min/max/sum/p50/p75/p90/p95/p99. This is where the question “what’s the p95 of @duration by endpoint?” gets answered — directly from logs, no metric required. (Remember from Section 3: a measure is a facet on a numeric attribute; you need the field typed as a measure to compute stats on it.)

Picking the visualization mirrors the metric widgets, with log-specific dimension limits worth knowing:

• Timeseries — the count/unique-count/measure over time, optionally split by up to 4 facets. Display rule: bars for counts and unique counts, lines for statistical aggregations (a count is a discrete bucket-total → bar; a p95 is a continuous statistic → line). This rule is the same one from 10.3, and getting it wrong is a giveaway of inexperience.
• Top List — ranked groups, but logs cap this at 1 grouping dimension. “Top 15 customers by unique sessions,” “top 10 URLs by error count.”
• Table — up to 4 dimensions, multiple measures per row, multiple queries. The richest log view: “top 10 availability zones, and within each the top 10 versions, by error count, with unique host count alongside.”
• Tree Map / Pie — share-of-whole (percentage breakdown by service, etc.).

Multiple queries and formulas work just like metrics: add query b, and write a formula across them — e.g. error rate from logs as (count of status:error) / (count of *), or the ratio of enterprise-tier to premium-tier cart IDs. The catch: to combine queries in a formula they must be grouped by the same field. Functions (the Σ button — smoothing, clamp_min, arithmetic, etc.) apply here too.

Getting it onto a dashboard — two paths, and they bridge straight back to 10.4:

1. Export from the Explorer. Build the aggregation you want in the Log Explorer, then Export → Add to Dashboard (the old “Export to Timeboard”). The graph lands as a widget carrying its query.
2. Build in the widget directly. Add a Timeseries / Top List / Table / Query Value widget on the dashboard, and at the top of the editor set the data source to Logs instead of Metrics. Now you get the same group-into/aggregate/measure controls inside the widget. This is how you put metrics, logs, and APM on the same dashboard — even the same graph (a metric line and a log-derived line overlaid), which is the whole point of the unified platform.

Once it’s a log widget, template variables work identically — $env/$service re-scope your log graphs the same way they re-scope metric graphs — and so does the “View related logs” pivot: click a spike on a log-analytics timeseries and jump straight into the Log Explorer scoped to that exact slice and time. (That pivot is Core Idea 2 in action — shared tags make it one click.)

The crucial caveat — aggregations only run on indexed logs. Log analytics queries operate on the searchable index, so anything dropped by an exclusion filter (Core Idea 1) is invisible to a log-analytics graph. If your dashboard count looks suspiciously low, you’re probably aggregating a sampled index. Three ways out: temporarily disable the exclusion filter, rehydrate from archive for a historical window, or — the durable answer — use a log-based metric instead (Section 4.3): because metric generation runs before exclusion, a log-based metric captures the true count even on logs you never index, and it’s retained at 10s granularity for 15 months.

The decision that matters: log widget vs log-based-metric on a dashboard. Use a live log widget when you want full flexibility, ad-hoc grouping, and the ability to pivot to the raw events — and the data is indexed. Use a log-based metric (graphed as a normal metric widget) when you need a permanent KPI, want it to survive exclusion/sampling, need long retention, or the volume is so high that querying raw logs every dashboard refresh is slow or costly. Rule of thumb: dashboards people stare at all day → log-based metric; investigation boards you build during an incident → live log widgets. Watch cardinality on the metric exactly as in Section 7 #8 — don’t group a log-based metric by user_id.

10.6 — Monitors: the full lifecycle, not just “set a threshold”

A monitor is the same query plus a threshold plus a notification plus a lifecycle. Beginners set the threshold and stop; the lifecycle config is what separates a useful alert from pager spam. Walking the creation flow (Monitors → New Monitor):

Pick the type. Metric (the common one), Log, Anomaly (alert when a metric deviates from its learned normal pattern — good for things with daily/weekly seasonality where a static threshold can’t work), Forecast (alert before you cross a line — “disk will fill in 3 days”), Outlier (one host behaving unlike its peers), Composite (combine other monitors), and more.

Define the query — exactly the editor from 10.2. The preview graph updates live and draws your threshold as a line, so you can see how often you’d have paged historically.

The evaluation settings that actually matter (this is the part people miss):

• Evaluation window — the lookback the monitor aggregates over (“the last 5 minutes”). Too short = flappy; too long = slow to fire.
• Evaluation aggregator (avg/max/min/sum over the window) — the judgment call from Section 8 #9. avg for sustained-degradation SLOs, max for “if it ever crossed this line,” sum for error counts (and .as_count() metrics must use sum).
• Recovery threshold — set this separately from the alert threshold to prevent flapping. Alert at >90% CPU but only recover below 80%, so a metric oscillating around 90 doesn’t spam you with alert/recover/alert/recover.
• Evaluation delay — wait N seconds before evaluating, because some data arrives late. Datadog recommends ~15 min (900s) for cloud-provider metrics (AWS/Azure backfill them), and ~60s when using a division formula so both series are complete. Forgetting this is a top cause of false No Data alerts.
• require full window / no-data handling — decide whether missing data is an alert or a shrug.

Simple vs multi-alert (Section 8 #10): multi-alert fires per group — group by host gives you one alert per failing host with {{host.name}} in the message — so the page tells the responder which entity broke. Simple alert collapses everything into one notification. Multi-alert for “which one,” simple for aggregate health.

Write the notification like a human will read it at 3am. The message body supports markdown, template variables ({{host.name}}, {{value}}, {{#is_alert}}…{{/is_alert}} conditional blocks so the alert text differs from the recovery text), and @-mentions to route it: @pagerduty-sre, @slack-payments-alerts, @oncall@example.com. Put the runbook link and the likely causes right in the message — the alert should tell the responder what to do, not just that something happened. For org-scale routing, notification rules auto-route based on monitor tags so you’re not hand-editing recipients on 500 monitors.

The lifecycle after it’s live: monitors transition OK → Warn → Alert → OK (Warn is an optional softer threshold). When you need to silence alerts for planned work, use Downtime — and scope it (service:web-store) rather than muting globally, so maintenance on one service doesn’t blind you to real problems elsewhere. Downtime silences notifications but doesn’t stop state transitions. The Triggered Monitors page is your incident-time view: every currently-firing monitor, mutable/resolvable in bulk per group.

10.7 — SLOs: turning monitors into error budgets

Once you have monitors, an SLO (Service Level Objective) wraps them into a target like “99.9% of requests succeed over a rolling 30 days.” Datadog computes the SLI (good time ÷ total time) and shows your remaining error budget — the headroom before you breach. Two flavors: monitor-based (built on an existing monitor’s OK/ALERT history) and metric-based / time-slice (computed directly from a metric, finer granularity, no monitor to maintain). The point of an SLO isn’t the dashboard widget — it’s that “are we spending error budget too fast?” is a far better operational question than “is this one graph red right now?” Put an SLO widget on your overview board and you’ve turned reliability into something you can manage on a budget instead of react to.

10.8 — Notebooks: the investigation and narrative tool

A Notebook is interleaved prose and live graphs — think a Jupyter notebook for observability. Two real uses: incident investigation (drop in the graphs as you find them, write what you concluded next to each, so the timeline tells a story) and postmortems / runbooks (a durable narrative document with live data that anyone can re-run later). When you’re debugging something genuinely confusing, a notebook beats a dashboard because it’s sequential — it captures the reasoning, not just the final picture.

10.9 — The worked example: “checkout latency is up,” end to end

Abstract principles don’t build muscle memory. Here’s the whole loop on one realistic incident, in actual clicks, so you can see the sections connect.

The page fires. Your phone shows: [ALERT] checkout p95 latency > 800ms on env:prod — a multi-alert monitor, so it names the scope. (That’s 10.6 working: multi-alert + template variables put the which in the message.)

Orient on the dashboard. You open the linked checkout runbook dashboard and set $env:prod (10.4 template variables). The Query-Value strip shows error rate normal but p95 latency red. The Top List “p95 by endpoint” shows POST /checkout/pay is the outlier — not the whole service, one endpoint. (Right widget for “which one is worst”: 10.3.)

Find the time. On the latency Timeseries you switch the rollup to max because the default avg was smoothing the spike (Core Idea 3 / Section 7 #7) and you want to see exactly when it started. It jumped at 14:05.

Pivot to logs. You click the spike → View related logs. Datadog opens the Log Explorer already scoped to service:checkout at 14:05 (this is Core Idea 2 — the shared tags make the pivot one click). You add @http.url:/checkout/pay status:error to the search (10.1 search syntax). A wall of payment gateway timeout logs, all tagged @payment.provider:acme.

Confirm the cause. You group the logs by @payment.provider in the Log Explorer’s analytics view (10.5) — acme is 100% of the timeouts, the other providers are fine. Root cause located: the Acme payment gateway is slow, and /checkout/pay blocks on it.

Capture and communicate. You spin up a Notebook (10.8), paste the latency graph and the log-volume-by-provider graph, write two sentences of conclusion, and drop the link in the incident channel. You schedule a scoped Downtime on provider:acme checks (10.6) so the redundant pages stop while you work, without muting the rest of checkout.

Prevent the recurrence. After mitigation, you make this faster to catch next time: from the Log Explorer you Generate Metric on checkout.payment.timeouts by provider (logs → metric, survives exclusion — Core Idea 1), add a Top List of it to the runbook dashboard, and create an Anomaly monitor on per-provider timeout rate so you’re warned before it hits user-facing latency next time.

Every tool you touched — monitor, dashboard, template variable, explorer, related-logs pivot, notebook, downtime, generate-metric — is one of the pieces from the sections above, and they connected because the tags lined up (Core Idea 2) and the ingest/index split kept the data available (Core Idea 1). That’s the whole platform working as designed.

---

11. How It Breaks

The recurring failure modes and the debugging reflex for each.

“My logs aren’t showing up in the Explorer.” Most common causes, in order: (a) they are ingested but excluded from the index — check Live Tail; if they’re there, it’s an exclusion filter or index-filter issue. (b) Timestamp dropped them — back-dated >18h; check datadog.estimated_usage.logs.drop_count and your date parsing. (c) Wrong index/quota — check if a daily quota was hit. First move: open Live Tail. If the log is in Live Tail but not the Explorer, it’s an indexing problem; if it’s not in Live Tail, it’s a collection/ingestion problem.

“My log is ingested but not parsed (no attributes).” The Grok parser didn’t match, or the wrong pipeline caught it. First move: the Pipeline Scanner — it shows you exactly which pipeline and which processors fired on a sample log. Then test your Grok rule against the real sample in the processor UI (it shows match/no-match). Remember: if no rule matches, the message is left untouched and passed on, silently.

“A parsed attribute exists but I can’t filter a pipeline on it.” Ordering. The pipeline filter ran before the processor that creates the attribute (Core Idea 4). Fix: filter on an entry-time field; restructure with nested pipelines.

“My custom metric bill exploded.” Cardinality. First move: the Metrics Summary page — find the metric, look at its tag cardinality and number of time-series. Hunt for an unbounded tag (user/request ID, raw URL). Use Metrics without Limits to drop tag dimensions you don’t query, and fix the emission if you can.

“A histogram metric is using way more custom metrics than expected.” The 5×-per-aggregate multiplier (Section 7 #1). Fix: trim histogram_aggregates/histogram_percentiles in datadog.yaml.

“My monitor shows No Data.” Often a .rollup() misaligned with the evaluation window, or a group that stopped reporting, or a filter matching nothing. First move: run the monitor’s query in the Explorer/metrics editor over the same window; check rollup vs evaluation-window alignment; add an evaluation delay.

“Logs are attributed to the wrong host.” A host/hostname/syslog.hostname field in the JSON hijacked it during preprocessing (Section 7 #3). Fix: in the preprocessing config, or stop emitting the field.

The general debugging reflex, in order:

1. Live Tail — is the data even arriving, post-processing?
2. Pipeline Scanner — what touched this log, and in what order?
3. Metrics Summary — for metric cardinality/type/interval issues.
4. Estimated Usage dashboard — for volume/cost/drop anomalies.
5. Agent status (datadog-agent status on the host) — is collection healthy at the source?

---

12. The Downsides / Disadvantages

Honest accounting. None of these are dealbreakers by themselves, but you’re signing up for all of them.

1. The pricing model is a genuine operational hazard, not just “expensive.” Cost is the shadow of Core Idea 1’s flexibility: with separate, multiplying knobs (ingest volume, indexed events, retention, custom-metric cardinality, per-product SKUs for APM/RUM/Synthetics/etc.), the bill is emergent from thousands of independent decisions made by engineers who don’t see the price tag. What it costs you: surprise bills measured in real money — a single mis-tagged high-cardinality metric or a forgotten DEBUG log firehose can add thousands a month. When it’s a dealbreaker: tight-budget shops without anyone owning cost governance. What people think mitigates it but doesn’t: “we’ll just watch the usage dashboard” — by the time it shows the spike, you’ve often already been billed; the real fix is preventing high-cardinality emission and aggressive exclusion by default, which requires discipline most orgs don’t impose until after the first scary invoice.

2. The cardinality trap is structural, not a bug you can outgrow. Because tags are the schema (Core Idea 2) and custom metrics bill per time-series, the very flexibility that makes Datadog powerful is the thing that bankrupts the careless. There is no “safe default” that protects you; the platform will happily ingest a million-series metric and bill you for it. Cost: permanent vigilance — every new tag is a cost decision, forever.

3. Vendor lock-in is deep and compounding. Your dashboards, monitors, pipelines, parsers, and the DogStatsD calls embedded in your application code are all Datadog-specific. The tagging conventions permeate everything. Cost: migrating off (to Grafana/Prometheus/OTel) is a multi-quarter project, and partial — features like distribution percentiles, anomaly detection, and the unified correlation don’t have clean equivalents. The more you adopt, the higher the exit cost climbs. OpenTelemetry instrumentation hedges this somewhat, but the moment you use Datadog-proprietary features you’re re-locked.

4. “Logging without Limits” trades a storage problem for a configuration problem. You no longer agonize over what to collect, but you now must continuously manage pipelines, indexes, exclusion filters, and quotas — and in a large org, multiple teams editing these creates conflicts, redundant processing, and gaps (the exact problem the Pipeline Scanner exists to debug). Cost: the complexity didn’t vanish; it moved into a sprawling server-side config surface that needs governance, RBAC, and ideally Terraform — which itself has “unexplored complexity” around pipeline ordering.

5. The submission-type / in-app-type distinction is a footgun that corrupts data silently. Because a metric’s type shapes what queries are valid (Core Idea 3) and the wrong choice misbehaves retroactively and going forward, a junior engineer picking GAUGE instead of COUNT can quietly invalidate a business metric, and nobody notices until a dashboard looks wrong weeks later. Cost: the failure is silent and delayed — the worst kind.

6. Pipeline order-dependence makes log processing fragile at scale. The fixed sequential model (Core Idea 4) means a change in one pipeline can break a downstream one, and “first match wins” for both indexes and exclusion filters means innocent-looking reorderings have non-local effects. Cost: changes to shared pipelines need care and testing; a careless edit can silently stop parsing for an unrelated team’s logs.

7. Sampling means your indexed logs are not the ground truth. Aggressive exclusion (the cost-saving you’re pushed toward) means the Log Explorer shows a sample, not everything. Counts in the Explorer can mislead; the true counts live in log-based metrics or the archive. Cost: a cognitive tax — you must always remember “is this view sampled?” — and grouped/analytics queries over sampled data can quietly mislead.

8. It’s a lot of surface area to hold in your head. Not “hard to learn” in the trivial sense — the breadth is structurally large: metrics, logs, traces, RUM, synthetics, profiling, security, plus the config surfaces for each. Cost: nobody on a small team is an expert in all of it; you’ll have blind spots, and the product keeps adding pillars.

---

13. The Taste Test

What separates someone who gets Datadog from someone cargo-culting it.

Tagging. Bad: inconsistent, ad-hoc tags (environment:prod here, env:production there, Env:PROD elsewhere), high-cardinality tags on metrics, no version. Good: religious Unified Service Tagging (env/service/version) everywhere, a documented tag taxonomy, infrastructure tags set at the Agent, high-cardinality detail kept in logs/traces not metrics. You can spot a mature setup instantly: filtering and correlation “just work” because the tags line up across pillars.

Logging format. Bad: unstructured text logs and a forest of brittle hand-written Grok rules with data matchers everywhere. Good: structured JSON emitted by the app, reserved attributes mapped cleanly, Grok reserved for third-party sources, notSpace preferred over greedy data, sample logs left as comments in every parsing rule.

Indexing strategy. Bad: one catch-all index, everything indexed, no exclusion filters, no archive, a terrifying bill. Good: multiple indexes segmented by value (errors retained long, success logs sampled), exclusion filters with sampling, archive to S3 for everything, log-based metrics capturing trends on excluded streams. The 80/20 rule visibly applied.

Metric types. Bad: everything is a GAUGE; per-host percentiles averaged together in a dashboard (mathematically meaningless). Good: DISTRIBUTION for service-level latency percentiles, COUNT for tallies queried with sum/.as_count(), types chosen deliberately at emit time.

Monitors. Bad: threshold monitors on avg that either never fire or page constantly; one giant monitor for 200 hosts; .rollup() inside the query causing flapping No-Data. Good: aggregator chosen to match intent (max for spike-safety, avg for sustained SLOs, sum for counts), multi-alert grouped by the actionable entity, evaluation delays set, recovery thresholds to prevent flapping, template variables in the notification so the page tells you which host.

Dashboards. Bad: static, hardcoded to one service, a wall of identical line graphs. Good: template variables for $env/$service, a high-level overview that drills into a runbook-style investigation board, mixed metrics + log analytics + traces on one screen so correlation is a glance.

The single fastest tell: look at one log’s side panel. If service, env, version are all present and the message is cleanly parsed into typed attributes, you’re looking at someone who understands the platform. If it’s a raw string with a random host override and no service tag, you’re not.

---

14. The Cheat-Sheet

Fast lookup for the things you’ll forget. Skim the rest of the doc once; keep this section.

Vocabulary in one line each

Term	What it is
Tag	key:value label; the universal filter/group/correlate dimension
Unified Service Tagging	always set env, service, version — correlation depends on it
Cardinality	count of distinct tag-value combinations = number of time-series = cost
Time-series	one metric + one tag-combo; the billed unit for custom metrics
Reserved attribute	timestamp/status/host/service/message — structural, mapped in preprocessing
Attribute	any structured log field; Facet = an attribute promoted to the filter panel
Pipeline / Processor	filtered, ordered chain of transforms; processor = one step
Remapper	renames/reassigns a field (vs Grok parser, which extracts fields)
Index	retained, searchable bucket of logs (the expensive tier)
Exclusion filter	drops/samples logs from an index (still processed, metered, archived, in Live Tail)
Log-based metric	metric generated from logs before exclusion — survives dropped logs
Live Tail	real-time stream of all logs post-processing, indexed or not

The log pipeline order (memorize this): ingest → preprocess (JSON reserved attrs) → pipelines & processors → log-based metrics → exclusion filters → index First matching index wins; first matching exclusion filter wins; pipelines & processors run top-to-bottom.

Metric types → when: GAUGE = current level · COUNT = tally (query with sum) · RATE = count/sec · HISTOGRAM = per-host Agent-side aggregates (becomes ~5 metrics) · DISTRIBUTION = global server-side percentiles (use for service p95/p99).

Query anatomy: space_aggregator:metric{filter} by {grouping} then .functions(). Every query is time-aggregated (rollup) then space-aggregated, always.

Metric query examples

avg:system.cpu.user{env:prod} by {host}
sum:orders.count{*}.as_count()
avg:system.disk.free{*}.rollup(max, 60)     # use max to catch spikes
p99:trace.web.request.duration{service:api}
(a / b) * 100                                # formula: error rate

Log search syntax

service:checkout status:error        # reserved attrs: no @
@http.status_code:[500 TO 599]       # @ for attributes; ranges
timeout AND -@http.method:GET        # message term, negation, AND/OR
service:web*                         # wildcard
*:searchterm                         # full-text all attributes

Grok rule shape: rulename %{MATCHER:attr.name:filter} — workhorse matchers notSpace, word, integer, number, ipv4/ip, date(...), regex(...); avoid greedy data.

Widget → question: Timeseries = trend · Query Value = one number now · Top List = what’s worst · Table = per-entity rows · Heatmap = distribution across sources · Log Stream = the actual events.

Log analytics (graphing logs): switch List → aggregate view; group into Fields / Patterns / Transactions; compute count / unique-count / measure-stat (incl. p50–p99); bars for counts, lines for stats; Top List = 1 dimension, Timeseries/Table = up to 4. Put on a dashboard via Export → Add to Dashboard, or a widget with data source = Logs. Aggregations run on indexed logs only — for dropped/sampled logs use a log-based metric (survives exclusion, 15-month retention) graphed as a normal metric. Dashboards stared at all day → log-based metric; incident boards → live log widget.

Monitor settings that prevent pager pain: set a recovery threshold below the alert threshold (anti-flap); set evaluation delay (~900s for cloud metrics); pick the evaluation aggregator (avg sustained / max any-spike / sum counts); use multi-alert to get {{host.name}} in the page; route with @-mentions; silence planned work with scoped Downtime, never global mute.

Template variable syntax: define $env, make widget filters listen with env:$env.value. Start with $env before placing widgets.

Debug reflex, in order: Live Tail (is data arriving?) → Pipeline Scanner (what touched this log?) → Metrics Summary (cardinality/type) → Estimated Usage dashboard (volume/cost/drops) → datadog-agent status (collection healthy?).

The five usage-watch metrics: datadog.estimated_usage.logs.drop_count (logs dropped at intake), .truncated_count (oversized), plus ingested/indexed/excluded volumes on the auto-created Log Management – Estimated Usage dashboard.

---

15. Where to Go Deeper

• Datadog official docs — the “Log Configuration” and “Metrics” sections (docs.datadoghq.com/logs/log_configuration/, /metrics/). Unusually good for vendor docs; the Pipelines, Parsing, Processors, and Metric Types pages are the canonical source for everything in Sections 4–6. Read the Parsing page in full before writing your first Grok rule.
• “Introducing Logging without Limits” (Datadog blog, 2018) and the “Guide to Log Management Indexing Strategies” (Datadog architecture center). The first explains why the ingest/index split exists; the second is the practical playbook for indexes, exclusion filters, and the 80/20 rule. Read when you’re about to design your indexing strategy.
• Datadog Agent architecture docs + the DataDog/datadog-agent repo on GitHub. For when you want to know what’s actually happening on the host — Collector, Forwarder, DogStatsD internals (the flush interval, batching, string interning). Read when you’re debugging collection or curious about the UDP path.
• “Datadog Metrics: The Core Concepts for Success” (Nicolas Narbais, Medium). The clearest outside explanation of cardinality and why distributions matter at scale. Read after you’ve hit your first cardinality surprise.
• The Datadog Learning Center (learn.datadoghq.com) — free hands-on courses with real trial accounts, specifically the Log Querying & Analytics and Log Indexes paths. The best way to actually internalize the UI loop. Do these instead of watching a YouTube tutorial.
• The in-app Pipeline Scanner and the Estimated Usage dashboard. Not reading material — tools to go play with on real data. An afternoon tracing your own logs through the scanner teaches Core Idea 4 better than any doc.

---

16. The Final Verdict

Here’s the honest take after all of that. Datadog is the best-integrated observability platform on the market, and that integration is real, not marketing — the moment you have env/service/version set consistently and you can jump from a CPU spike to the traces to the exact log lines in two clicks without changing tools or re-typing a time window, you understand why people pay what they pay. The unification of metrics, logs, and traces under one tagging model is the thing it gets profoundly right, and the ingest/index decoupling is a genuinely elegant answer to a problem (the ELK-era “index everything or lose it forever” dilemma) that used to have no good answer. Watching a raw log line get parsed, enriched, turned into a metric, and selectively indexed is watching a well-designed machine work.

What it costs you is, bluntly, money and vigilance — and those two are linked. The same flexibility that makes it powerful makes the bill an emergent property of a thousand uncoordinated engineering decisions, and the platform offers you no guardrail by default. You will, at some point, get a scary invoice traceable to a high-cardinality tag or a log firehose nobody excluded. The lock-in is deep enough that by the time the bill hurts, leaving is a project, not a switch. And the configuration complexity that “Logging without Limits” introduced is real: you traded a storage problem for a governance problem, and governance problems need an owner.

So who should reach for it: teams that have outgrown single-tool setups and genuinely need correlated observability across pillars, that have (or will assign) someone who owns cost governance and tagging conventions, and for whom engineer-minutes during incidents are worth more than the subscription. Who shouldn’t: a tiny team on a tight budget monitoring a handful of services — Prometheus + Grafana + Loki will do 80% of it for the cost of the VMs, and you won’t wake up to a five-figure surprise.

What you should now believe: Believe that tags are the schema and cardinality is the cost — that one belief prevents most of the pain. Believe that the ingest/index split is the key to both the platform’s power and its bill. Don’t believe that “collect everything” is free; it’s cheap to ingest and expensive to keep queryable, and conflating the two is how budgets die. When someone says “Datadog is too expensive,” what they almost always mean is “we indexed everything and never tagged with cost in mind” — which is a discipline failure, not a pricing failure, though Datadog is happy to let you make it.

The hard-won line, the one to quote back to a colleague: In Datadog, every tag you add is a question you can ask later and a bill you pay forever — so tag like you’ll be reading the invoice, because you will.

---

The ideas are mine. The writing is AI assisted