Cloud computing

Know about a problem before your customers do: monitoring with Datadog

What to watch first, how to write alerts people act on, how to keep the monitoring bill under control, and when you do not need Datadog at all.

By Mac Milan ·

Many businesses find out their system is down from a customer phone call. Monitoring changes the order: you hear about it first, and usually with enough detail to fix it. Datadog is one of the tools we use for this.

Three kinds of signal

  • Metrics are numbers over time: requests, errors, response time, CPU and memory.
  • Logs are the lines your app writes as it works.
  • Traces follow one request through every service and database query it touched, with the time spent in each.

Datadog keeps all three in one place and links them. From a spike on a chart you can open the slow requests behind it, and from a request you can open the log lines it wrote.

From a slow page to its cause
  1. 01AlertThe response time alarm fires and sends a message to the team.
  2. 02DashboardA chart shows which service slowed down and when it started.
  3. 03TraceOne slow request shows where the time went, for example a single database query.
  4. 04LogsThe log lines from that request give the exact error or input.
  5. 05FixThe cause is fixed, and the same chart confirms it.

What to watch first

You do not need hundreds of charts. Start with four questions.

  • Is it up? A scheduled check loads your site or API from outside, the way a customer would.
  • Is it failing? The share of requests that end in an error.
  • Is it slow? The response time of the slowest 5% of requests, not the average. The average hides the customers who are having a bad time.
  • Is it running out of room? CPU, memory, disk space and database connections.

Alerts someone will act on

An alert that fires every day gets ignored, and then the real one is missed. Each alert should say what is wrong, how bad it is and where to look. Problems that hurt customers right now go to a person immediately. Everything else goes into a list that is reviewed during working hours. An alert that never leads to action is changed or removed.

Keeping the cost under control

Datadog charges by how much you monitor and how much data you send it. Left alone, that grows with every new service. These are the controls we set from the start.

  • Keep the logs that help and drop the noise, such as health check lines and debug output.
  • Keep every trace of an error or a slow request, and only a sample of the normal ones.
  • Store data for a shorter time where a long history is not needed.
  • Review the usage each month, the same way as the AWS bill.

Do you need Datadog at all?

Not always. For one small app, alarms and a dashboard in AWS CloudWatch can be enough, and CloudWatch is already part of your AWS account. Datadog earns its cost when several services talk to each other and the hard part is finding which one caused the problem.

Either way, we add the measuring code to your app with OpenTelemetry, an open standard. Your app is then not tied to one monitoring vendor, and you can change tools later without rewriting it.

Related service: Cloud computing

Have a project in mind?

Tell us what you want to build or fix. We will reply with a clear next step and a quote.

Request a quote