Promql

PromQL

PromQL is a nested functional language for selecting and aggregating Prometheus time series. Every query is an expression tree: selectors at the leaves, functions and operators at the nodes, a single root value (scalar, instant vector, range vector, or string) at the top. Type discipline makes queries correct; cardinality awareness makes them fast.

Authoritative reference for writing PromQL queries — dashboards, alerts, recording rules, exploration. The companion backend/prometheus skill covers instrumentation (metric types, naming, labels, exporters). When both apply, use backend/prometheus for emitting metrics and this skill for querying them.

References

• Selectors and data types — [${CLAUDE_SKILL_DIR}/references/selectors-and-types.md] Expression types, instant and

range vector selectors, label matchers (=, !=, =~, !~), time durations, offset and @ modifiers, staleness, instant vs range queries, string and float literals

• Operators — [${CLAUDE_SKILL_DIR}/references/operators.md] Arithmetic, comparison, logical/set, vector matching

(on, ignoring, group_left, group_right), aggregation operators, by/without, fill modifiers, operator precedence

• Functions — [${CLAUDE_SKILL_DIR}/references/functions.md] Full function catalog: counter family, gauge

functions, classic and native histograms, *_over_time, existence checks, label manipulation, math, time, type conversion, sorting, trigonometric

• Native histograms — [${CLAUDE_SKILL_DIR}/references/native-histograms.md] Native vs classic detection,

histogram_quantile across both forms, native-only functions (histogram_avg, histogram_count, histogram_sum, histogram_fraction, histogram_stddev, histogram_stdvar), aggregation without le, NHCB semantics, trim operators, migration patterns, function compatibility matrix, gotchas, annotations

• Subqueries — [${CLAUDE_SKILL_DIR}/references/subqueries.md] Syntax, resolution and alignment, nested subqueries,

when to use subqueries vs recording rules, pitfalls

• Optimization and pitfalls — [${CLAUDE_SKILL_DIR}/references/optimization.md] Cardinality awareness, common

pitfalls (rate-of-gauge, aggregate-then-rate, averaging ratios, averaging summary quantiles), counter resets, staleness, rate window sizing, native histograms, diagnostics

• Recording and alerting rules — [${CLAUDE_SKILL_DIR}/references/recording-alerting-rules.md] Rule file structure,

level:metric:operations naming, ratio aggregation patterns, alert syntax, for/keep_firing_for, templating, alerting best practices, anti-patterns

Read the relevant reference before proceeding.

---

Data Types

Every expression has one type. Functions and operators require specific input types — type mismatch is the most common cause of "query parses but returns nothing".

• instant vector — set of series, one sample per series at one timestamp. Default result of metric selectors,

aggregations, most functions.

• range vector — set of series, multiple samples per series across a time window. Produced only by range vector

selectors (metric[5m]) and subqueries. Cannot be graphed directly — must pass through a function returning an instant vector.

• scalar — single numeric value, no labels. Used as numeric arguments and in arithmetic.
• string — function arguments only; cannot be a query result.

Function type rules:

• rate, irate, increase, delta, idelta, deriv, predict_linear, *_over_time — range vector → instant

vector

• histogram_quantile, histogram_fraction, histogram_* — instant vector (histogram or bucket series) → instant

vector

• Aggregation operators (sum, avg, max, topk, ...) — instant vector → instant vector
• scalar() — instant vector → scalar; vector() — scalar → instant vector

Selectors

Instant Vector Selectors

http_requests_total                                # all series with this metric name
http_requests_total{job="api"}                     # filtered by label
http_requests_total{job="api", method="GET"}       # multiple matchers — AND
http_requests_total{status=~"5.."}                 # regex match (RE2, fully anchored)
http_requests_total{method!="OPTIONS"}             # negative match
{__name__=~"http_.*"}                              # regex on metric name via __name__

Label matchers: = exact, != not equal, =~ regex match, !~ negative regex.

Regex is fully anchored — =~"foo" matches ^foo$, not arbitrary substrings. Use =~".*foo.*" for substrings.

Required selectors: at least one matcher must not match the empty string. {job=~".*"} is illegal; {job=~".+"} is valid.

Range Vector Selectors

Append [duration] to a selector:

http_requests_total[5m]                            # last 5 minutes of samples
http_requests_total{job="api"}[1h]

Duration units: ms, s, m, h, d, w, y. Combine longest-first: 1h30m, 12h34m56s. Floats with units are not allowed (1.5h is invalid — use 1h30m).

Modifiers

• offset — time-shift into the past (or future with negative). Must immediately follow the selector.

``promql http_requests_total offset 5m rate(http_requests_total[5m] offset 1w) # rate one week ago ``

• @ — pin evaluation to an absolute time. start() and end() resolve to the range query's

bounds.

``promql http_requests_total @ 1609746000 rate(http_requests_total[5m] @ end()) ``

Staleness: instant vector selectors return the most recent sample within the lookback window (default 5 minutes). Series with no sample in the window disappear from the result.

Full selector reference, lookback delta tuning, instant-vs-range-query semantics: [${CLAUDE_SKILL_DIR}/references/selectors-and-types.md].

Operators

Arithmetic and Comparison

+, -, *, /, %, ^ — IEEE 754 arithmetic. The metric name is dropped from any vector arithmetic result.

==, !=, >, =, ` 100 # only series whose latest value exceeds 100 httprequeststotal > bool 100 # all series, value is 1 if > 100 else 0

### Logical/Set Operators

Operate on label-set membership, ignoring sample values:

- `vector1 and vector2` — intersection
- `vector1 or vector2` — union
- `vector1 unless vector2` — complement

```promql
up{job="prom"} or up{job="node"}                       # union
metric and on(device) (other_metric == 0)              # constrained intersection

Vector Matching

The biggest source of "no results returned". Default rule: two elements match if they have exactly the same label set (after the metric name drops). When operands have different label sets, match keywords are required:

• on(labels) — match only on these labels; ignore all others
• ignoring(labels) — match on all labels except these

Group modifiers for many-to-one (arithmetic, comparison, trigonometric only — not set operators):

• group_left(extra_labels) — many on the left, one on the right; copy extra_labels from right to result
• group_right(extra_labels) — symmetric

Canonical patterns:

# Error ratio: errors carry an extra `code` label; ignore it to match
errors:rate5m{code="500"} / ignoring(code) requests:rate5m

# Many-to-one: same denominator for every code
errors:rate5m / ignoring(code) group_left requests:rate5m

# Join info-style metric: bring `version` label into result
node_filesystem_avail_bytes * on(instance, job) group_left(version) node_exporter_build_info

Aggregation

Instant vector → instant vector with fewer elements.

• sum, avg, min, max — reduce across dimensions
• count — number of series; group — value 1 per group
• topk(k, v) / bottomk(k, v) — k largest/smallest; preserves labels
• stddev, stdvar — population statistics
• quantile(φ, v) — quantile across series
• count_values(label, v) — distribution of values

by vs without:

• by (labels) keeps only the listed labels
• without (labels) keeps everything except the listed labels

Prefer without for normal aggregations — preserves job, instance, and friends. Use by for a known output dimension set (typically recording rules).

sum without (instance) (rate(http_requests_total[5m]))    # collapses instance, keeps job/method/status
sum by (job) (rate(http_requests_total[5m]))              # collapses everything except job

Operator Precedence (highest to lowest)

1. ^
2. *, /, %, atan2
3. +, -
4. ==, !=, =, >
5. and, unless
6. or

Left-associative except ^ (right-associative). Use parentheses when mixing comparison and logical operators.

Full operator catalog, fill modifiers, detailed vector matching examples: [${CLAUDE_SKILL_DIR}/references/operators.md].

Key Functions by Category

Counters — rate, irate, increase

The three most important PromQL functions. All take a range vector, return an instant vector, handle counter resets automatically.

• rate(v[5m]) — per-second average rate over the range, extrapolated. Use for alerts and slow counters.
• irate(v[1m]) — instant rate from the last two samples. Graphing volatile counters only; never alerts (flaps).
• increase(v[1h]) — total increase over the range; syntactic sugar for rate(v) * range_seconds. Use for

human-readable totals.

Composition rule — rate() first, then aggregate:

sum(rate(http_requests_total[5m]))             # CORRECT
rate(sum(http_requests_total)[5m])             # WRONG — counter resets disappear into the sum

Applies to all counter functions (rate, irate, increase), gauge functions (delta, deriv, predict_linear), and *_over_time functions.

Window sizing: range must contain ≥2 samples. Rule of thumb: 4× the scrape interval, minimum 1m. With 15s scrapes the standard default is [5m].

Gauges — delta, idelta, deriv, predict_linear

For gauges only. Never apply rate() to a gauge.

delta(cpu_temp_celsius[2h])                                  # absolute change over 2h
deriv(node_memory_MemAvailable_bytes[5m])                    # smoothed per-second change
predict_linear(node_filesystem_avail_bytes[1h], 4*3600) [:[]]

Use when a range-vector function needs the output of another function or aggregation:

max_over_time(rate(http_requests_total[5m])[1h:])             # peak 5m-rate over the last hour
quantile_over_time(0.99, rate(errors[1m])[1d:])               # 99th percentile of per-minute error rates over a day
deriv(rate(http_requests_total[5m])[30m:])                    # is the rate accelerating?

Performance: subqueries re-run the inner expression at every resolution step. A [1d:1m] subquery runs the inner 1,440 times. For expensive inner expressions or repeated use, convert to a recording rule.

Avoid subqueries inside recording rules. Split into two recording rules.

Full syntax, alignment rules, subquery vs recording rule guidance: [${CLAUDE_SKILL_DIR}/references/subqueries.md].

Common Query Idioms

# Request rate by job
sum by (job) (rate(http_requests_total[5m]))

# Error ratio — aggregate numerator and denominator separately
  sum(rate(http_requests_total{status=~"5.."}[5m]))
/
  sum(rate(http_requests_total[5m]))

# P99 latency by handler (classic histogram)
histogram_quantile(0.99, sum by (handler, le) (rate(http_request_duration_seconds_bucket[5m])))

# Top 5 CPU consumers
topk(5, sum by (app) (rate(process_cpu_seconds_total[5m])))

# Disk-full prediction
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24*3600)  2 * 3600

Pitfalls — Quick Reference

Positive rules when writing or reviewing queries:

• Apply rate() to counters only. Use delta(), idelta(), or deriv() for gauges. rate() on a gauge silently

produces nonsense.

• rate() first, then aggregate. sum(rate(x[5m])), never rate(sum(x)[5m]). Counter resets are per-series.
• Aggregate ratios correctly. Sum numerator and denominator separately, then divide. Never avg(ratio) or

avg(avg).

• Don't average summary quantiles. avg(metric{quantile="0.95"}) is statistically invalid. Use histograms for

cross-instance aggregation.

• Preserve le when aggregating classic histogram buckets. sum by (job, le) or sum without (instance).
• Range ≥ 4× scrape interval, minimum 1m. Avoid rate(x[1m]) with 30s+ scrape intervals.
• rate() (not irate()) for alerts. irate() is for graphing volatile counters.
• without over by for most aggregations — preserves useful labels like job automatically.
• offset must immediately follow the selector. sum(x offset 5m) is correct; sum(x) offset 5m is invalid.
• Filter early. Push label matchers into the innermost selector.

Full pitfall catalog with diagnostics: [${CLAUDE_SKILL_DIR}/references/optimization.md].

Recording and Alerting Rules

Recording Rule Naming — level:metric:operations

instance:http_requests:rate5m                   # per-instance rate (no aggregation)
job:http_requests:rate5m                        # rolled up to per-job
job:http_errors_per_requests:ratio_rate5m       # ratio, per job
job:request_latency_seconds:mean5m              # average latency from Summary

• level — labels remaining in the output (instance, job, cluster, instance_path, etc.)
• metric — metric name; strip _total after rate()
• operations — operations applied, newest first. Omit _sum when other ops present; mean replaces rate for

sum/count averages; ratio for divisions named with _per_

Recording rule examples and ratio aggregation patterns: [${CLAUDE_SKILL_DIR}/references/recording-alerting-rules.md].

Alerting Rule Essentials

- alert: HighErrorRatio                   # CamelCase community convention
  expr: |
      sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
    /
      sum by (job) (rate(http_requests_total[5m]))
    > 0.05
  for: 10m                                # must be true for the entire window before firing
  keep_firing_for: 5m                     # keep firing this long after expression becomes false
  labels:
    severity: page
  annotations:
    summary: "High error rate on {{ $labels.job }}"
    description: "Error ratio is {{ $value | humanizePercentage }} for {{ $labels.job }}"
    runbook_url: "https://wiki.example.com/runbooks/{{ $labels.job }}"

Alerting principles:

• Alert on symptoms, not causes. User-visible latency, error rates — not internal-only metrics.
• Page on one stack layer. If user latency is fine, don't page on a slow sub-component.
• Every page must be actionable. Nothing to do → delete the alert.
• Match for: to the rate window. rate(x[5m]) with for: 30s is contradictory.

Full alerting best practices, template variables, anti-patterns: [${CLAUDE_SKILL_DIR}/references/recording-alerting-rules.md].

Application

Writing PromQL queries:

• Apply conventions silently — don't narrate each rule.
• Wrap counters in rate() or increase() before any further operation.
• Aggregate before joining; filter at the innermost selector.
• Prefer without over by when removing a noise dimension.
• For histograms, prefer native when available; otherwise preserve le in aggregations.

Writing recording rules:

• Follow level:metric:operations naming.
• Build a hierarchy: per-instance rule first, then aggregate up. Strip _total after rate().
• Aggregate ratios by computing numerator and denominator separately.
• No subqueries inside recording rules — split into two rules.

Writing alerting rules:

• CamelCase alert names; severity labe

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: xobotyi
• Source: xobotyi/cc-foundry
• License: MIT

Install and usage instructions live in the source repository linked above.