See what each request did — inside the app.

Lookout is Rainier’s opt-in diagnostics dashboard. Every request, job, console command and scheduled task becomes a batch: the queries it ran, the cache it touched, the calls it made out, the jobs it queued, the mail it sent, the failures it reported, and how long each took. Open /_lookout and look.

Built from the same spans the telemetry feature exports — no application code opens anything for it.

Read the Lookout docs Observability

The Lookout dashboard for the sample app: slowest routes, slowest queries, most frequent exceptions, slowest jobs and recent failures.

Two lines to turn it on

A Cargo feature and a variable. A typo in any LOOKOUT_* variable fails the boot, naming it — a misconfigured deployment should stop, not record nothing.

Cargo.toml

[dependencies]
rainier-framework = { git = "https://github.com/safewords/rainier-framework.git", rev = "…", features = ["lookout"] }

.env

LOOKOUT_ENABLED=true
# memory (default) · database · cache
LOOKOUT_STORE=cache
LOOKOUT_SLOW_QUERY_MS=100
LOOKOUT_SLOW_REQUEST_MS=1000

What it records

One batch per unit of work, its entries in order. The span each entry comes from is catalogued in the framework.

request
query
cache
http_client
job
mail
notification
event
scheduled_task
command
exception
log
model
view
broadcast
session
auth_attempt
n_plus_one
slow
custom

Slow thresholds per watcher: queries at 100 ms, requests, jobs and outbound HTTP at 1 s — with per-pattern overrides.
N+1 detection: five or more queries with the same sanitised SQL in one batch, taking 50 ms or more together, add an n_plus_one entry naming the query.
Trace ids on every batch, so a batch found in Lookout can be looked up in your tracing backend, and the other way round.

Up close

Select an image to see it full size.

Lookout timeline of one POST /api/posts/{post}/publish request with its queries, event, broadcasts, queued job, notification, view and mail. — One request, everything it caused: three queries, the PostPublished event, its broadcast, the queued job, the notification and the mail it rendered, on one timeline.

Lookout timeline for GET /posts/{post} with a query flagged as slow. — A query over the slow threshold is recorded twice: as the query, and as a slow entry naming the threshold it crossed, so it shows up on the Slow page without searching.

Lookout exceptions page grouping failures by fingerprint with counts. — Exceptions are grouped by fingerprint (kind and message), so four validation failures are one row with a count, and boot-time configuration warnings sit beside request failures.

Lookout status page listing every watcher with its state and slow threshold. — Every watcher can be switched and sampled separately; the slow thresholds (requests 1 s, queries 100 ms) are the defaults Lookout ships with.

Scoping & sampling

Decided in two stages, so excluded work costs as little as possible.

When work starts, its route (name, else pattern, else path) and whether jobs, commands and tasks are watched decide whether anything is buffered — before the handler runs.
When it ends, the guard and the user — known only after authentication — decide whether to keep it.
Sampling per kind of work and per watcher. Failures ignore both: nothing reported is lost to scoping or sampling.
Outside local, a batch is kept when it has an exception, something slow, a failed job or a monitored tag — production keeps the interesting batches and lets the rest go.

src/bootstrap.rs

use rainier_framework::lookout::config::{Patterns, Scope, WatcherConfig};
use rainier_framework::lookout::EntryType;

Rainier::new(".")
    .with_lookout(|lookout| {
        lookout
            .scope(Scope {
                routes: Patterns { include: vec![], exclude: vec!["health".into()] },
                jobs: true,
                commands: true,
                scheduled_tasks: true,
                ..Scope::default()
            })
            .watch(EntryType::Cache, |w| WatcherConfig { sample_rate: 0.1, ..w })
            .keep_batch_if(|batch| batch.has_exception || batch.duration_us > 250_000)
    })

Private by default

The defaults record what the framework’s spans carry, and those leave values out: a query with its bindings as ?, a cache operation without its key, a mail without its recipients. Anything that identifies a person is opt-in.

request_headers: Request headers — with authorization, cookie, set-cookie, x-api-key and the CSRF headers redacted.
request_body_on_exception: The request body, only on a batch with an exception, with password, token, secret, card_number and their kin redacted.
cache_keys: Cache keys.
mail: Subjects and recipients.
user_id: The authenticated user’s id — needed for user scoping and “top users”.

Add your own names to the redaction lists; values over 2 KB are truncated.

Storage

memory: This process, the last 1000 batches.; For: Development and tests. Per pod on Kubernetes.
database: Lookout’s own four tables, on a connection you name. Opt-in migrations; the same tables on SQLite, D1, MySQL/MariaDB and Postgres.; For: Anything shared and durable.
cache: A ring buffer of the last LOOKOUT_CAPACITY batches in a cache store you name, written under a cache lock.; For: Redis in production, with no migrations.

Nothing prunes on its own: schedule lookout:prune. lookout:status and lookout:clear complete the set.

What it costs

Off, nothing. A disabled Lookout binds nothing, mounts nothing and adds no layer or middleware. Without the feature, none of it is compiled.
On, nothing waits for it. The request path appends to an in-memory buffer; a finished batch goes to a bounded channel with try_send. When the channel is full the batch is dropped and counted — never the request held up.
A background writer drains the channel every second, or sooner when 500 batches are waiting. Its own store traffic is marked internal, so it does not record itself.
Spans below RUST_LOG are the honest cost: Lookout needs info spans whatever the log filter says, so with Lookout on they are created for it alone.

Who can see it

Mounted outside every middleware group, at LOOKOUT_PATH. With APP_ENV=local it is open. Anywhere else, the viewer must be authenticated by the dashboard’s guard and pass the viewLookout gate — and everyone else gets the router’s own 404, the same bytes as a path that never existed.

It denies without with_lookout_viewers, without the bindings, with an unknown guard, and without the ability defined.
The one route that changes anything — clearing — is a DELETE that also needs an X-Lookout: 1 header, which a cross-site form cannot send.
The dashboard is a prebuilt Vue app compiled into the binary, served with a strict Content-Security-Policy, X-Frame-Options: DENY and no-store.

src/bootstrap.rs

// Outside APP_ENV=local, only a viewer the guard authenticates and the
// gate lets through sees the dashboard. Everyone else gets the router's
// own 404 — the same bytes as a path that never existed.
Rainier::new(".")
    .with_lookout_viewers::<User>()
    .with_instance_arc(auth)          // AuthManager<User>
    .with_instance(
        Gate::<User>::new().define_simple("viewLookout", |user: &User| user.is_admin),
    )

Limits

Written down, so you can plan around them.

Spans outside a unit of work — a query at boot, a cache read in a background task — are not recorded. Only failures get a batch of their own.
The memory store is per process: on several replicas, use database or cache to see them together.
The database store has no transactions; a write that fails half-way can leave a batch missing entries.
A console command’s own batch is written only if the process calls lookout::shutdown (or flush) after the console returns.