Missing time series in PromQL: why your success rate disappears during an outage

A service that returns nothing but errors looks exactly like a service with no traffic in Grafana. The line stops, the alert stays quiet, and the next morning you hear about the outage from a customer. The cause is not a broken data source, it is the way PromQL joins two time series. This article shows the mistake on a dashboard built over Spring Boot metrics, then the correction, and finally both as Terraform so it does not get lost again with the next click.

The night nobody noticed anything
The first attempt: the query everyone writes first
Why the line stops: how PromQL joins two series
The reflex that makes it worse: 1 minus the error rate
The zero series: taking the missing labels from the denominator
Why vector(0) is not enough here
The same mistake in the alert, only more expensive
The limit: when the denominator disappears too
Neighbouring mistake 1: success is not the same as SUCCESS
Neighbouring mistake 2: the panel goes empty when you zoom in
The complete query
All of it as code: the naive Terraform version first
One query, two targets: the correction in Terraform
The dashboard as a resource
The alert rule in Cloud Monitoring
Triggering the outage before it arrives on its own
Pinning the rule down with a test
The patterns at a glance
When a missing series is not a problem
FAQ
Conclusion
Sources

The night nobody noticed anything

The setup is ordinary: a few Spring Boot services in a Kubernetes namespace called shop-prod, each with spring-boot-starter-actuator and the Micrometer bridge to Prometheus. Three of them matter here, catalog, checkout and payment. For operations there is a Grafana dashboard with a panel showing the success rate per service, and an alert that fires below 99 percent.

At 02:14 payment loses the connection to its database. The service keeps running, the pods are healthy, the readiness probe is satisfied. It just answers every request with an HTTP 500.

In the morning the dashboard shows a quiet night. The lines for catalog and checkout run flat along the top at 100 percent. The line for payment ends at 02:14 and never comes back. The alert never fired. The outage was discovered at 08:40 by a customer who called because he could not pay.

The monitoring did not stay quiet because nothing was happening. It stayed quiet because everything was broken. Both states look the same as long as the query is built naively.

The first attempt: the query everyone writes first

This is what the panel looked like. Successful requests divided by all requests, grouped by service:

sum(rate(http_server_requests_seconds_count{namespace="shop-prod", status!~"5.."}[5m])) by (job)
/
sum(rate(http_server_requests_seconds_count{namespace="shop-prod"}[5m])) by (job)

That reads correctly, and during normal operation it is. http_server_requests_seconds_count is the counter behind the Micrometer timer http.server.requests, which Spring Boot increments for every request it serves. rate() turns it into requests per second, and sum(...) by (job) aggregates across all pods of a service. The filter status!~"5.." lets through everything that is not a server error.

Grouping by job is not an accident. Prometheus assigns its own instance label to each pod while scraping. Without sum ... by (job) the panel would get one line per pod, and with three replicas the statement about the service would already be diluted before the actual mistake even strikes.

That mistake lies elsewhere, and it only shows once payment returns nothing but 500s.

Why the line stops: how PromQL joins two series

In PromQL a time series exists only as long as it has data points. It does not fall to zero, it stops. After roughly five minutes without a new data point it counts as stale and no longer appears in any result.

From 02:14 onwards not a single request for payment arrives with a status that does not begin with 5. So there is no series left for the numerator above the fraction bar. Below it there still is, because the 500s keep being counted there.

In a division Prometheus does not simply compute numbers. For every series on the left it looks for a series on the right with an identical label set. A result only appears where both sides match. If one side is missing, no result appears, and not a result with the value zero, but none at all.

Before 02:14                        After 02:14
Numerator:   payment -> 42.0        Numerator:   (no series)
Denominator: payment -> 42.0        Denominator: payment -> 41.8
Match:       payment -> 1.0         Match:       (no series)

For the graph that means: no series, no line. Grafana cannot know that a total outage hides behind the gap. It receives the same empty answer as for a service that simply has no traffic at night.

Here lies the expensive confusion: an outage and a quiet evening look the same.

The reflex that makes it worse: 1 minus the error rate

Once you have understood that, the first idea is usually this one:

1 - (
  sum(rate(http_server_requests_seconds_count{namespace="shop-prod", status=~"5.."}[5m])) by (job)
  /
  sum(rate(http_server_requests_seconds_count{namespace="shop-prod"}[5m])) by (job)
)

Instead of counting the successes, this version counts the errors and inverts the result. During an outage it does work, because the 500 series exists then.

The mistake has only moved. As long as everything is healthy there is not a single request with a 5xx status, so the series in the numerator does not exist, so no result appears. Now the line is gone during normal operation and only shows up once the first error occurs.

Of the two, that is the worse variant. A panel that only displays something when there is trouble feels broken in everyday use, and nobody looks at a broken panel after two weeks.

Both versions fail at the same point: one side of the division can disappear, and the result disappears with it.

The zero series: taking the missing labels from the denominator

The solution is to put a substitute series next to the numerator that steps in when the real one is missing. That substitute has to meet two conditions: its value must be zero, and it must carry exactly the labels you group by.

The denominator supplies both, once you multiply it by zero:

(
  sum(rate(http_server_requests_seconds_count{namespace="shop-prod", status!~"5.."}[5m])) by (job)
  or
  sum(rate(http_server_requests_seconds_count{namespace="shop-prod"}[5m])) by (job) * 0
)
/
sum(rate(http_server_requests_seconds_count{namespace="shop-prod"}[5m])) by (job)

The or operator works per label set, not per expression. For every service the left side supplies, it takes the left side. For every service only the right side knows about, it takes the right one. As long as payment has successful requests, the real value wins. If the service fails, the zero remains.

What matters is where the substitute series comes from. It stems from the same metric with the same grouping, so it inevitably carries the same labels. The division finds its partner again, and the result is a line that falls to zero instead of vanishing.

As a side effect this also handles a case people rarely think about: a service that is newly deployed and whose first requests all fail. Without a zero series it never appears in the panel, with it it sits at zero right away.

Why vector(0) is not enough here

Many answers give or vector(0) as the solution, and for a single stat panel that is correct. Not for this panel.

sum(rate(http_server_requests_seconds_count{namespace="shop-prod", status!~"5.."}[5m])) by (job)
or vector(0)

vector(0) produces a series without any label. So it cannot step in for payment, because it does not know that payment exists. It simply sits in the result as an extra nameless row, and in the division it finds no partner with a matching label set. The panel gains one more series and still has no line for the failed service.

vector(0) answers the question of whether there is any result at all. The zero series from the denominator answers the question of which services exist and how each individual one is doing.

If you want to stay with vector(0) for other reasons, you have to hand it the labels yourself:

label_replace(vector(0), "job", "payment", "", "")

That works, but it requires every service name to be hard-coded in the query. The next new service will be missing, and nobody will notice. This is why the route via the denominator is the more robust one: it knows the services because it has just counted them itself.

The same mistake in the alert, only more expensive

On a dashboard the missing series costs a gap in the graph. In an alert rule it costs the alarm.

sum(rate(http_server_requests_seconds_count{namespace="shop-prod", status!~"5.."}[5m])) by (job)
/
sum(rate(http_server_requests_seconds_count{namespace="shop-prod"}[5m])) by (job)
< 0.99

This rule cannot fire during a total outage. The comparison < 0.99 filters series: whatever is left becomes an alarm. If there is no series, nothing is left, and Prometheus reads an empty result as all clear. The more complete the outage, the more reliably the rule stays silent.

That is what makes this mistake particularly unpleasant. It does not hide in an edge case, it strikes when it counts. A service with sporadic errors fires dutifully. A service that is completely gone does not.

With the zero series in the numerator the series survives, falls to zero and therefore sits below the threshold. The rule fires.

The limit: when the denominator disappears too

The zero series saves the case where a service answers and returns errors while doing so. It does not save the case where it stops answering altogether.

If all pods of payment are terminated, there is nobody left to count anything. Prometheus finds no target and the metric disappears entirely. Then the denominator is gone too, and no zero series can be derived from it. The panel faces the same emptiness again, one level deeper.

For that you need a second rule that does not ask for a value but for existence:

absent(
  sum(rate(http_server_requests_seconds_count{namespace="shop-prod", job="payment"}[5m]))
)

absent() returns a result precisely when the inner expression returns none. The price is that the service name has to be hard-coded in the query, because an absence cannot be grouped by labels that do not currently exist.

In practice that means: for the services whose outage really hurts there is a rule by name. For everything else the success rate with a zero series carries the load. Confusing the two leaves you either with a rule that stays silent in an emergency, or with a list of service names nobody maintains.

A third route goes through the up metric that Prometheus writes for every target itself. It tells you whether scraping worked and therefore reacts earlier than any business figure. In exchange it says nothing about whether the service gives sensible answers. Only all three together produce a complete picture.

Neighbouring mistake 1: success is not the same as SUCCESS

Alongside status, Micrometer supplies a second label that seems to simplify things. outcome condenses the status code into a category, and reaching for it is tempting:

sum(rate(http_server_requests_seconds_count{namespace="shop-prod", outcome="SUCCESS"}[5m])) by (job)
/
sum(rate(http_server_requests_seconds_count{namespace="shop-prod"}[5m])) by (job)

That reads more cleanly than the regex and measures something other than what most people expect. outcome knows five values, and SUCCESS stands exclusively for 2xx. Everything else drops out of the numerator: INFORMATIONAL for 1xx, REDIRECTION for 3xx, CLIENT_ERROR for 4xx and SERVER_ERROR for 5xx.

That makes every redirect count as a non-success. An endpoint that redirects to the result page after a POST drags the rate down although it does what it is supposed to do. The same applies to 304 Not Modified, which is entirely normal for a client with a working cache. A dashboard that suddenly shows 88 percent after a deployment, without a single error having occurred, often has this cause.

The right filter targets what you actually want to exclude:

sum(rate(http_server_requests_seconds_count{namespace="shop-prod", outcome!="SERVER_ERROR"}[5m])) by (job)

This version is also more robust than status!~"5..", because it does not depend on status always holding a three-digit number. On aborted connections a non-numeric value can show up there depending on the environment, and a regex on 5.. will not match it.

That leaves the question of how to treat 4xx. Counting them as a success is defensible, because an invalid request is not a failure of the service. It should still be a conscious decision, because broken authentication produces 401s in bulk and stays invisible in this count. If you want to see that separately, build a second panel on outcome="CLIENT_ERROR" instead of overloading the success rate.

Neighbouring mistake 2: the panel goes empty when you zoom in

The third finding in the same dashboard concerns the range in square brackets. Several panels contained:

rate(http_server_requests_seconds_count{namespace="shop-prod"}[$__interval])

$__interval is the spacing between two points that Grafana calculates from the time window and the panel width. Over seven days that is several minutes and everything looks fine. Zoom in to one hour and the value shrinks to a few seconds. But rate() needs at least two data points inside the window, otherwise it cannot form a slope. Once the window gets smaller than the spacing between two measurements, the query returns nothing.

The tricky part is the timing. The mistake never shows while building the dashboard, because there you usually look across days. It shows the moment somebody zooms into the last few minutes during an incident, and that is when the panel is supposed to display something.

The correct form is:

rate(http_server_requests_seconds_count{namespace="shop-prod"}[$__rate_interval])

$__rate_interval is defined as max($__interval + scrape interval, 4 * scrape interval) and is therefore never too small. The value for the scrape interval comes from the Min step field of the query, failing that from the Scrape interval setting of the data source, whose default is 15 seconds.

There is a pitfall here that renders the switch useless: if the data source assumes 15 seconds while the metrics are actually scraped once per minute, the macro calculates with a resolution that is too fine and the window stays too small. Switching to $__rate_interval does not help then. So before switching, check what is configured in the data source and at what interval data actually arrives.

The complete query

All three corrections together, this is the version that goes into the panel:

(
  sum(rate(http_server_requests_seconds_count{namespace="shop-prod", outcome!="SERVER_ERROR"}[$__rate_interval])) by (job)
  or
  sum(rate(http_server_requests_seconds_count{namespace="shop-prod"}[$__rate_interval])) by (job) * 0
)
/
sum(rate(http_server_requests_seconds_count{namespace="shop-prod"}[$__rate_interval])) by (job)

Three things differ from the first attempt: the filter via outcome instead of a regex on status, the zero series behind the or, and $__rate_interval instead of $__interval. The query is longer for it and reads more awkwardly. In exchange it shows a zero during an outage instead of nothing.

All of it as code: the naive Terraform version first

A dashboard built by hand in the UI is the second half of the problem. The correction above lives exactly until somebody duplicates the panel, swaps the namespace and leaves the zero series behind.

So move it into Terraform. The obvious first draft looks like this, and it has a flaw:

terraform {
  required_providers {
    grafana = {
      source = "grafana/grafana"
    }
    google = {
      source = "hashicorp/google"
    }
  }
}

locals {
  success_rate_query = <<-EOT
    (
      sum(rate(http_server_requests_seconds_count{namespace="shop-prod", outcome!="SERVER_ERROR"}[$__rate_interval])) by (job)
      or
      sum(rate(http_server_requests_seconds_count{namespace="shop-prod"}[$__rate_interval])) by (job) * 0
    )
    /
    sum(rate(http_server_requests_seconds_count{namespace="shop-prod"}[$__rate_interval])) by (job)
  EOT
}

One query, defined in one place, usable by both dashboard and alert. That is the right thought and still implemented wrongly.

$__rate_interval is a Grafana macro. Grafana replaces it with a concrete value when running the query. Cloud Monitoring does not know it. If the same string travels into an alert rule, it contains a range the parser cannot resolve and the rule is broken. In the best case terraform apply already fails, in the worse case a rule sits in the cloud that never evaluates.

On top of that comes the hard-coded namespace, three times inside the same string. When copying it for the staging environment, one of the three occurrences gets forgotten and the query mixes two environments.

One query, two targets: the correction in Terraform

Both problems dissolve with a template of placeholders that gets filled twice, differently:

locals {
  namespace = "shop-prod"

  # %[1]s is the namespace, %[2]s the range for rate().
  # The range is the only place where dashboard and alert are allowed
  # to differ, because $__rate_interval exists only in Grafana.
  success_rate_template = <<-EOT
    (
      sum(rate(http_server_requests_seconds_count{namespace="%[1]s", outcome!="SERVER_ERROR"}[%[2]s])) by (job)
      or
      sum(rate(http_server_requests_seconds_count{namespace="%[1]s"}[%[2]s])) by (job) * 0
    )
    /
    sum(rate(http_server_requests_seconds_count{namespace="%[1]s"}[%[2]s])) by (job)
  EOT

  success_rate_dashboard = format(local.success_rate_template, local.namespace, "$__rate_interval")
  success_rate_alert     = format(local.success_rate_template, local.namespace, "5m")
}

format() with the indices %[1]s and %[2]s lets you insert the same value several times without passing it several times. The namespace therefore sits in exactly one place.

Two details that tend to bite on the first attempt. First: in HCL, ${ starts an interpolation. The curly braces of the label selectors are harmless because no dollar sign precedes them, and $__rate_interval is harmless too because no curly brace follows the dollar sign. Second: as soon as format() is involved, every percent sign in the template is a format verb. This query contains none, a query with percentages in the legend text does.

The dashboard as a resource

resource "grafana_folder" "shop" {
  title = "Shop Production"
  uid   = "shop-prod"
}

resource "grafana_dashboard" "success_rate" {
  folder    = grafana_folder.shop.uid
  overwrite = true

  config_json = jsonencode({
    uid      = "shop-success-rate"
    title    = "Shop, success rate per service"
    timezone = "browser"
    time     = { from = "now-24h", to = "now" }

    panels = [
      {
        type    = "timeseries"
        title   = "Success rate per service"
        gridPos = { h = 10, w = 24, x = 0, y = 0 }

        fieldConfig = {
          defaults = {
            unit = "percentunit"
            min  = 0
            max  = 1
            custom = {
              spanNulls   = false
              lineWidth   = 2
              fillOpacity = 5
            }
            thresholds = {
              mode = "absolute"
              steps = [
                { color = "red", value = null },
                { color = "green", value = 0.99 },
              ]
            }
          }
          overrides = []
        }

        targets = [
          {
            refId        = "A"
            expr         = local.success_rate_dashboard
            legendFormat = "{{job}}"
            interval     = "1m"
          },
        ]
      },
    ]
  })
}

Two settings in there matter more than they look.

spanNulls = false makes sure Grafana does not bridge real gaps. Set to true, the display draws a straight line across a period without data, and the outage disappears visually a second time, this time in the rendering instead of the query. A panel meant to show the emergency must not interpolate.

interval = "1m" is the Min step field from the UI. It sets the lower bound for the resolution and is therefore the value $__rate_interval calculates with. If the metrics are scraped once per minute, this is where that belongs, otherwise switching to $__rate_interval stays ineffective.

The alert rule in Cloud Monitoring

The rule gets the same query, only with a fixed range:

resource "google_monitoring_alert_policy" "success_rate" {
  display_name = "Success rate below 99 percent (shop-prod)"
  combiner     = "OR"

  conditions {
    display_name = "Success rate below threshold"

    condition_prometheus_query_language {
      query               = "(${trimspace(local.success_rate_alert)}) < 0.99"
      duration            = "300s"
      evaluation_interval = "60s"
      alert_rule          = "ShopSuccessRateLow"
      rule_group          = "shop-prod"
    }
  }

  alert_strategy {
    auto_close = "1800s"
  }
}

resource "google_monitoring_alert_policy" "payment_no_data" {
  display_name = "payment stopped reporting data entirely"
  combiner     = "OR"

  conditions {
    display_name = "No time series present"

    condition_prometheus_query_language {
      query               = "absent(sum(rate(http_server_requests_seconds_count{namespace="${local.namespace}", job="payment"}[5m])))"
      duration            = "600s"
      evaluation_interval = "60s"
      alert_rule          = "ShopPaymentNoData"
      rule_group          = "shop-prod"
    }
  }

  alert_strategy {
    auto_close = "1800s"
  }
}

Notes on the fields that cost time the first time round:

evaluation_interval has to be a positive multiple of 30 seconds. The API rejects other values, and the error message from terraform apply does not spell that out particularly clearly.

duration is the time the condition has to hold continuously before the alarm moves from pending to firing. Without a value it is zero, and then a single spike already triggers. Five minutes is a workable starting point for a success rate, because a rate() over five minutes reacts slowly anyway.

The parentheses around the inserted query are not decoration. The template ends with a division, and the comparison is meant to apply to the whole result, not to the denominator alone. Arithmetically the division binds more tightly anyway, but a query whose correctness depends on knowing operator precedence is a query that breaks during the next rework. trimspace() removes the trailing newline the heredoc leaves behind.

The second rule covers the case from the section on the limit. Its duration is deliberately higher, because a deployment can briefly cause no data to arrive, and nobody should be woken up at night for that.

Triggering the outage before it arrives on its own

The actual mistake in this story was not the query. The actual mistake was that nobody ever produced the state the dashboard was built for.

For a Spring Boot service a filter behind a property is enough, one that does not even become a bean during normal operation:

package com.example.shop.outage;

import jakarta.servlet.Filter;
import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.ServletRequest;
import jakarta.servlet.ServletResponse;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;

import java.io.IOException;

@Component
@ConditionalOnProperty(name = "outage.active", havingValue = "true")
class OutageFilter implements Filter {

    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
            throws IOException, ServletException {

        String path = ((HttpServletRequest) request).getRequestURI();

        // The actuator has to stay reachable, otherwise the metric disappears
        // completely and the test checks the wrong case.
        if (path.startsWith("/actuator")) {
            chain.doFilter(request, response);
            return;
        }

        HttpServletResponse http = (HttpServletResponse) response;
        http.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
        http.getWriter().write("Outage active");
    }
}

The exception for /actuator is the part you forget on the first try, and it decides whether the test says anything at all. If the filter chokes the metrics endpoint too, Prometheus can no longer scrape anything. Then the metric disappears completely and you are observing the case from the section on the limit instead of the case this is about. Both look the same in the panel but have different causes and different remedies.

The outage is switched on through the property, without a new image:

kubectl set env deployment/payment OUTAGE_ACTIVE=true -n shop-prod

Then wait five minutes and look at the dashboard. With the old query the line breaks off. With the new one it falls to zero, and the alert moves from pending to firing after the configured five minutes. You revert it with the same command and OUTAGE_ACTIVE=false.

If you want to manage entirely without extra code, take the database away from the service instead. That comes closest to the nightly incident, because it produces the errors the service really generates in an emergency. In exchange it is harder to dose, and in a shared environment other people notice.

Pinning the rule down with a test

For the correction to survive a rework, it belongs in a test. promtool brings everything needed and does not require a running Prometheus.

First the rule in its own file, shop-alerts.yml:

groups:
  - name: shop-prod
    rules:
      - alert: ShopSuccessRateLow
        expr: |
          (
            sum(rate(http_server_requests_seconds_count{namespace="shop-prod", outcome!="SERVER_ERROR"}[5m])) by (job)
            or
            sum(rate(http_server_requests_seconds_count{namespace="shop-prod"}[5m])) by (job) * 0
          )
          /
          sum(rate(http_server_requests_seconds_count{namespace="shop-prod"}[5m])) by (job)
          < 0.99
        for: 5m

Plus the test, shop-alerts-test.yml:

rule_files:
  - shop-alerts.yml

evaluation_interval: 1m

tests:
  - interval: 1m
    input_series:
      # Ten minutes healthy, after that no data point arrives for the successes.
      - series: 'http_server_requests_seconds_count{namespace="shop-prod", job="payment", outcome="SUCCESS", status="200"}'
        values: '0+60x10 _ _ _ _ _ _ _ _ _ _'
      # From minute 10 onwards only the server errors are counted.
      - series: 'http_server_requests_seconds_count{namespace="shop-prod", job="payment", outcome="SERVER_ERROR", status="500"}'
        values: '0x10 0+60x10'

    promql_expr_test:
      - expr: |
          (
            sum(rate(http_server_requests_seconds_count{namespace="shop-prod", outcome!="SERVER_ERROR"}[5m])) by (job)
            or
            sum(rate(http_server_requests_seconds_count{namespace="shop-prod"}[5m])) by (job) * 0
          )
          /
          sum(rate(http_server_requests_seconds_count{namespace="shop-prod"}[5m])) by (job)
        eval_time: 20m
        exp_samples:
          - labels: '{job="payment"}'
            value: 0

    alert_rule_test:
      - eval_time: 20m
        alertname: ShopSuccessRateLow
        exp_alerts:
          - exp_labels:
              job: payment

The value syntax is compact and unfamiliar on first reading. 0+60x10 produces eleven points starting at zero and increasing by 60 each, so a counter at 60 requests per minute. The underscore stands for a missing data point, and that is what this is about: from minute 10 the success series delivers nothing more while the error series starts running.

You run it with:

promtool test rules shop-alerts-test.yml

The promql_expr_test is the more important of the two blocks. It checks that at the 20 minute mark a sample with the label job="payment" still exists at all, and that its value is zero. That is the property the naive query does not have. Take the zero series out again and the test fails with an empty result set, well before somebody depends on it at 02:14.

The call fits into any pipeline and needs neither a cluster nor a database. It is the cheapest part of this entire article and the one that lasts longest.

The patterns at a glance

Case	Wrong	Right
Success rate, service returns only errors	`success / total`	`(success or total * 0) / total`
Success rate, thought backwards	`1 - (errors / total)`	the same zero series, otherwise the line is missing while healthy
Fallback without grouping	`or vector(0)`	`or <denominator> * 0`, because only that carries the labels
Fallback with fixed names	`label_replace(vector(0), ...)`	workable, but every new service has to be added by hand
Service gone completely	success rate alone	additionally `absent(...)` as its own rule
Defining success	`outcome="SUCCESS"`	`outcome!="SERVER_ERROR"`, otherwise 3xx counts as an error
Several pods per service	selector without grouping	`sum(...) by (job)`, otherwise one line per instance
Range for `rate()`	`[$__interval]`	`[$__rate_interval]`, plus Min step matching the interval
Gaps in the graph	`spanNulls = true`	`spanNulls = false`, otherwise the gap gets painted over
Query in two places	once for Grafana, once retyped for the alert	one template, two fillings via `format()`

When a missing series is not a problem

Not every gap needs closing, and anyone who fits or ... * 0 to every query buys new problems.

With a metric that has many possible label values, the zero series produces a line for each of them, including the ones that have done nothing for weeks. Break the success rate down by uri instead of job and you have the problem immediately: a tidy panel turns into thirty lines at zero, one for every endpoint nobody has called since the last release. Here the gap is the more honest presentation.

The same holds for counters that deliberately run rarely, a nightly reconciliation for instance. There the absence of the series is the correct statement. A zero would claim that it was measured that nothing happened. In fact nothing was measured. That is a difference, and in a panel somebody reads at three in the morning it is an important one.

The question that settles it is this: is the absence of data a normal state here, or a finding? If it is normal, the gap stays. If it is a finding, it has to become visible, either as a zero or through absent().

FAQ

Why does a Prometheus series not simply fall to zero?
Because Prometheus does not know that it should fall. A counter exists only as long as something writes it. Once data points stop arriving, the series counts as stale after about five minutes and disappears from every result. A zero would be a statement about reality, and Prometheus cannot make one when nobody is measuring any more.

What exactly does or do in PromQL?
It works per label set, not per expression. For every combination of labels the left side supplies, the left side applies. All label sets that appear only on the right get added. This is why or works as a fallback only if the right side carries the same labels as the left.

Is or vector(0) not enough?
For a stat panel showing a single number, yes. For a panel grouped by service, no, because vector(0) has no labels and therefore cannot step in for any particular service. It appears as an additional nameless series and changes nothing about the missing line.

How do I spot that an alert rule has this problem?
The test question is: assume the monitored condition occurs completely. Does the series still exist? For every rule whose expression contains a division or a filter on a label value, the question is worth asking. You can answer it reliably with a promtool test in which the series ends in underscores.

Should I filter on status or on outcome?
On outcome, but by exclusion rather than inclusion. outcome!="SERVER_ERROR" means exactly what a success rate is supposed to mean. outcome="SUCCESS" only captures 2xx and therefore counts every redirect as a failure.

Why does $__rate_interval achieve nothing with some data sources?
Because the macro calculates with the scrape interval stored in the data source or in the Min step field. If it says 15 seconds while the metrics are scraped once per minute, the calculated window stays too small. The value has to match the actual interval, otherwise the switch changes nothing.

Do I have to rebuild every panel now?
No. What is affected are expressions in which one side of a division or a comparison can disappear, typically through a filter on a label value such as the status code. A plain sum(rate(...)) without such a filter does not have the problem.

Conclusion

A dashboard is only tested once somebody has produced the outage and checked whether it shows up. Before that you only know it looks good during normal operation, and that is the one situation in which nobody is looking.

The three corrections in this article cost half an hour together: the zero series from the denominator, the filter via outcome instead of a regex, and $__rate_interval with a matching Min step. The fourth measure is the one that holds the rest together, and it costs the least: switch the outage on, wait five minutes, look.

The concrete next step: take the alert rule that matters most to you and write a promtool test in which the monitored series stops halfway through the run. If the rule does not fire, you have found one.

Sources

Spring Boot, metrics in the actuator: docs.spring.io/spring-boot/reference/actuator/metrics.html
Prometheus, unit testing for alerting rules: prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules
Prometheus, operators and vector matching: prometheus.io/docs/prometheus/latest/querying/operators
Grafana, introducing $__rate_interval in Grafana 7.2: grafana.com/blog
Google Cloud, PromQL-based alerting policies: docs.cloud.google.com/monitoring/promql/promql-in-alerting
Terraform, Grafana provider: registry.terraform.io/providers/grafana/grafana
Terraform, google_monitoring_alert_policy: registry.terraform.io/providers/hashicorp/google

All queries, Terraform examples and code snippets in this article are my own.