Skip to content

Observability & Metrics

RAVEN exposes a Prometheus metrics endpoint and ships with pre-built Grafana dashboards that give you a real-time view of your routing security posture.

Prometheus

Enabling the Metrics Endpoint

Add to your raven.yaml:

outputs:
  prometheus:
    listen: ":9595"
    path: "/metrics"

Verify it is working:

curl http://localhost:9595/metrics | grep raven

Prometheus Scrape Configuration

Add RAVEN to your prometheus.yml:

scrape_configs:
  - job_name: raven
    static_configs:
      - targets: ["localhost:9595"]
    scrape_interval: 15s

Note

If RAVEN is running in a container or WSL and Prometheus is running in Docker, use the Docker bridge gateway IP instead of localhost. Typically 172.17.0.1 on Linux:

    static_configs:
      - targets: ["172.17.0.1:9595"]

Available Metrics

Route counts by security posture: raven_routes_total{posture="secured", afi="ipv4"} raven_routes_total{posture="origin-only", afi="ipv4"} raven_routes_total{posture="path-suspect", afi="ipv4"} raven_routes_total{posture="path-only", afi="ipv4"} raven_routes_total{posture="unverified", afi="ipv4"} raven_routes_total{posture="origin-invalid", afi="ipv4"}

Same metrics available with afi="ipv6".

raven_routes_total counts Adj-RIB-In pre-policy routes only: what your routers received, before import policy. Routes your routers send are counted separately (see Adj-RIB-Out routes), so a sum over raven_routes_total never mixes the two directions.

Total route table size: raven_route_table_size # Total Adj-RIB-In pre-policy routes in the table

BMP session health: raven_bmp_session_state{router="edge1"} # 1=up, 0=down raven_bmp_messages_total{router="edge1", msg_type="route_monitoring"} raven_bmp_peer_state{router="edge1", router_addr="192.0.2.1", peer="198.51.100.1", peer_distinguisher=""} # 1=established, 0=down

router is the router's BMP sysName, or its address when it sent none.

Label change on raven_bmp_peer_state

raven_bmp_peer_state gained two labels: router_addr, the source address of the router's BMP session, and peer_distinguisher, the peer's VRF route distinguisher ("" for a peer in the global instance, for example 65000:100 for one in a VRF). The same peer address on two routers, or in two VRFs of one router, is now two series instead of one series that both overwrote. Dashboards and alerts that aggregate this metric with by (router, peer) keep working but merge VRFs; add peer_distinguisher to the grouping to keep them apart. Recording rules that match on the exact label set need updating.

RTR cache health: raven_rtr_session_state{cache="localhost:3323"} # 1=connected, 0=disconnected raven_rtr_vrp_count{cache="localhost:3323"} # VRPs loaded raven_rtr_aspa_count{cache="localhost:3323"} # ASPA records loaded raven_rtr_last_sync_seconds{cache="localhost:3323"} # Unix time of the last completed sync (End of Data) raven_rtr_last_serial_change_seconds{cache="localhost:3323"} # Unix time of the last sync that changed the serial raven_rtr_data_expired{cache="localhost:3323"} # 1 if no sync completed within the Expire Interval raven_rtr_protocol_version{cache="localhost:3323"} # Negotiated RTR version (ASPA needs 2)

ASPA verification

raven_aspa_validations_total{procedure="upstream", source="config-asn", state="Invalid"}

A counter of ASPA verifications, incremented each time RAVEN verifies a route: on ingest and again on every revalidation pass. Labels:

Label Values
procedure upstream, downstream
source default, config-address, config-asn: where the procedure came from (see validation.aspa-peers)
state Valid, Invalid, Unknown, Unverifiable

Use it to confirm your aspa-peers mapping took effect, and to see which procedure produces your Invalid results. A jump in procedure="upstream", source="default", state="Invalid" after a new transit session comes up usually means a provider is missing from aspa-peers and is being verified with the wrong procedure. Adj-RIB-Out routes are not ASPA-verified and are not counted. Because revalidation counts too, read the rate, not the absolute value:

sum by (procedure, source, state) (rate(raven_aspa_validations_total[5m]))

Revalidation

When an RTR sync changes VRPs or ASPA records, RAVEN revalidates the routes the change can affect (see Validation Engine).

raven_revalidation_duration_seconds{mode="incremental"} # Histogram: wall time of each pass raven_revalidation_routes_total{mode="incremental", result="checked"} raven_revalidation_routes_total{mode="incremental", result="changed"}

mode is incremental for a pass driven by an RTR delta and full for a pass over the whole table, which happens after a Cache Reset, a new RTR session or a session-ID change. result="checked" counts routes revalidated; result="changed" counts those whose ROV, ASPA or posture result changed. The duration is wall time from start to finish, and the pass is interleaved with BMP ingest, so it is not time ingest was blocked.

Watch these to catch revalidation falling behind. An incremental pass touches only the routes the delta can affect, so it should be far shorter than a full pass; a run of full passes means the cache keeps resetting the session. Each pass also logs re-validation complete with mode, candidates, checked, changed and took.

histogram_quantile(0.95, sum by (le, mode) (rate(raven_revalidation_duration_seconds_bucket[1h])))

BMP ingest health

raven_bmp_rejected_connections_total{reason="not_allowed"} # Source outside bmp.allowed-routers raven_bmp_rejected_connections_total{reason="bad_remote_addr"} # Source address could not be read raven_bmp_unsupported_nlri_total{afi="25", safi="70"} # MP_REACH/MP_UNREACH not decoded, by AFI/SAFI

raven_bmp_rejected_connections_total counts BMP connections RAVEN closed at accept, before reading anything, because of bmp.allowed-routers. Any increase means something other than your routers reached the BMP port; check the WARN log line BMP connection rejected for the source address. It stays at 0 when allowed-routers is empty, because nothing is rejected.

raven_bmp_unsupported_nlri_total counts MP_REACH_NLRI and MP_UNREACH_NLRI attributes for address families RAVEN does not decode, anything other than IPv4 and IPv6 unicast (for example, L2VPN EVPN is AFI 25, SAFI 70). Those routes are not in the route table. A steady rate tells you a router is sending address families you may want to stop exporting over BMP; RAVEN counts them so they are not dropped silently. afi and safi are decimal.

Malformed UPDATEs

raven_bmp_malformed_updates_total{action="treat_as_withdraw", reason="origin"}

A counter of malformed BGP UPDATEs received over BMP, which RAVEN handles under RFC 7606 the way your router does (see Malformed BGP Input). Labels:

Label Values
action The strongest RFC 7606 class the UPDATE's errors call for: attribute_discard, treat_as_withdraw or session_reset
reason The error that called for it: as_path, attr_length, attr_flags, origin, next_hop, med, local_pref, communities, originator_id, cluster_list, large_communities, missing_mandatory, as4_path, as4_aggregator, duplicate_attr, nlri, mp_reach, mp_reach_next_hop, mp_unreach, duplicate_mp or update_length

attribute_discard carries only as4_path, as4_aggregator and duplicate_attr. session_reset carries nlri, mp_reach, mp_reach_next_hop, mp_unreach, duplicate_mp, update_length, and any attribute reason found in an UPDATE that announces no prefixes. Each malformed UPDATE counts once, under its strongest error.

On a healthy network this stays at 0. Alert on any session_reset: it means a peer sent something your router answers by resetting the BGP session, and RAVEN's view of that peer may be incomplete until the session comes back.

sum by (action, reason) (increase(raven_bmp_malformed_updates_total[1h]))

Adj-RIB-Out routes

raven_adj_rib_out_routes_total{posture="origin-only", afi="ipv4"}

The number of Adj-RIB-Out pre-policy routes (RFC 8671): what your routers send their peers, before outbound policy, by posture and address family. It is kept apart from raven_routes_total so that received and sent routes never add up into one number. Adj-RIB-Out routes get ROV only; their ASPA state is always Unverifiable, because ASPA verification applies to routes received from a neighbor. Use it to spot origin-invalid routes your network is advertising.

Zero-valued series at startup

These counters and histograms export every label combination at 0 from startup, so an alert can tell "none yet" from "not exported": raven_bmp_rejected_connections_total, raven_revalidation_duration_seconds, raven_revalidation_routes_total, raven_aspa_validations_total, raven_bmp_malformed_updates_total (every action and reason pair it can carry), the raven_global_check_* counters once a provider is enabled, and raven_rtr_anomaly_total per cache in raven rtr monitor. raven_bmp_unsupported_nlri_total appears only once an unsupported family arrives, because its label values are not known in advance.

RTR anomaly detection:

Emitted by the adaptive anomaly detector in raven rtr monitor when it is run with --prometheus. severity is high or medium. See CLI Reference → raven rtr monitor.

raven_rtr_anomaly_total{cache="localhost:3323", severity="high"} # Count of anomalies, by severity raven_rtr_anomaly_last_timestamp{cache="localhost:3323"} # Unix timestamp of most recent anomaly

External global-visibility correlation:

Emitted when the Event Engine's global-correlate action runs, which requires external.ripestat.enabled: true. result is the consensus verdict: match, divergent, local_only or inconclusive. See Configuration → External Correlation.

raven_global_check_total{source="ripestat", result="match"} # Count of correlations, by source and verdict raven_global_check_latency_seconds # Histogram of correlation latency (live lookups only) raven_global_check_cache_hits_total{source="ripestat"} # Correlations served from the in-process cache raven_global_check_rate_limited_total{source="ripestat"} # Lookups suppressed by the local rate limiter

When a provider is enabled, RAVEN exports every result series of raven_global_check_total, and the cache-hit and rate-limited counters, at 0 from startup, so a series exists before the first correlation.

raven_global_check_latency_seconds covers only correlations that actually went to the provider. Cache hits and rate-limited lookups make no network round-trip, so their near-zero timings are excluded — otherwise a warm cache would pull p50/p95 toward zero and mask the provider latency degradation you would want to alert on. Those two are counted on raven_global_check_cache_hits_total and raven_global_check_rate_limited_total instead, so the activity stays visible.

A rate-limited lookup is a local policy decision, not a failure, so it does not appear on raven_global_check_total at all. A cache hit does: it produced a real verdict.

Cache hit ratio:

rate(raven_global_check_cache_hits_total[5m])
/
rate(raven_global_check_total[5m])

Note

raven check global runs in its own short-lived CLI process, so its correlations are never scraped. These metrics reflect daemon-side correlations only.

Useful PromQL Queries

Percentage of routes that are origin-invalid:

raven_routes_total{posture="origin-invalid", afi="ipv4"}
/
sum(raven_routes_total{afi="ipv4"}) * 100

Alert: origin-invalid routes above threshold:

groups:
  - name: raven
    rules:
      - alert: RAVENOriginInvalidHigh
        expr: raven_routes_total{posture="origin-invalid"} > 100
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High number of origin-invalid routes detected"

      - alert: RAVENPathSuspect
        expr: raven_routes_total{posture="path-suspect"} > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Path-suspect routes detected — possible route leak"

      - alert: RAVENRTRDataExpired
        expr: raven_rtr_data_expired == 1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "RAVEN RTR data has expired — RPKI validation may be outdated"

      - alert: RAVENBMPRejectedConnection
        expr: increase(raven_bmp_rejected_connections_total{reason="not_allowed"}[10m]) > 0
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "A source outside bmp.allowed-routers tried to open a BMP session"

      - alert: RAVENBMPSessionDown
        expr: raven_bmp_session_state == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "RAVEN BMP session is down"

      - alert: RAVENRTRAnomaly
        expr: increase(raven_rtr_anomaly_total{severity="high"}[10m]) > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "RAVEN detected a high-severity RTR sync anomaly"

      - alert: RAVENGlobalDivergence
        expr: increase(raven_global_check_total{result="divergent"}[10m]) > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Global BGP view disagrees with RAVEN's local origin — likely a propagated hijack"

      - alert: RAVENGlobalCheckInconclusive
        expr: |
          increase(raven_global_check_total{result="inconclusive"}[30m])
          /
          increase(raven_global_check_total[30m]) > 0.5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "More than half of global-visibility correlations are failing — check RIPEstat reachability"

Global correlation verdict rate:

sum by (result) (increase(raven_global_check_total[1h]))

A high local_only rate is worth investigating on its own: those prefixes exist in your RIB but no route collector carries them, which points at leaks or misconfiguration rather than hijacks.

Grafana

Importing the Pre-built Dashboard

RAVEN ships with a Grafana dashboard in the repository at lab/grafana-dashboard.json.

Via Grafana UI:

  1. Go to Dashboards → Import
  2. Click Upload JSON file
  3. Select lab/grafana-dashboard.json from the RAVEN repository
  4. Set the Prometheus datasource
  5. Click Import

Via Grafana API:

curl -X POST \
  -H "Content-Type: application/json" \
  -d @lab/grafana-dashboard.json \
  http://admin:admin@localhost:3000/api/dashboards/import

Dashboard Panels

The pre-built dashboard includes:

Security Posture Overview

  • Route count by posture (time series) — watch for spikes in origin-invalid
  • Posture distribution (pie chart) — your network's overall security health
  • Origin-invalid routes table — prefix, peer, origin ASN, matched VRP
  • Path-suspect routes table — prefix, peer, AS_PATH, failing hop

RTR Cache Health

  • VRP count over time — should be stable, drops indicate validator issues
  • ASPA count over time
  • Cache sync latency
  • Session state — alert panel turns red if cache goes down

RTR Anomaly Detection

  • Total RTR Anomalies — raven_rtr_anomaly_total, broken down by severity
  • Time Since Last Anomaly — driven by raven_rtr_anomaly_last_timestamp

BMP Session Health

  • Messages per second by router
  • Route count per peer
  • Session up/down events

Global BGP Visibility Correlation

The last row on the dashboard, after RTR Anomaly Detection. It is fed by the Event Engine's global-correlate action, so it stays empty until external.ripestat.enabled: true and a rule using that action fires. See Configuration → External Correlation.

  • Global Consensus Results — rate(raven_global_check_total[5m]) by result. Colour-matched to the security posture palette: match green, divergent red, local_only amber, inconclusive grey.
  • Global Consensus — Current Totals — one stat per verdict, straight off the raven_global_check_total counters. Reads 0 rather than "No data" before the first correlation runs, so a verdict appearing is visible as a change.
  • RIPEstat Query Performance — cache hits (raven_global_check_cache_hits_total), rate-limited calls (raven_global_check_rate_limited_total) and live queries (raven_global_check_latency_seconds_count) on one graph. This is the panel to watch when tuning cache-ttl and rate-limit-per-min: rising rate-limited calls mean the budget is too tight for your event volume.
  • RIPEstat Query Latency p50/p95 — quantiles over raven_global_check_latency_seconds, in seconds. A p95 spike means RIPEstat is slow or unreachable, not that RAVEN is busy.

This row answers the question the posture panels cannot. When a prefix turns up as origin-invalid, the verdict here tells you how far the announcement actually spread. local_only ticking up on Current Totals means no route collector carries the prefix at all — the announcement never left your network, which points at a leak, a misconfiguration or a lab injection rather than a hijack the internet has accepted. divergent is the opposite and the more serious reading: the world sees a different origin than you do, so the bogus announcement has propagated. Global Consensus Results shows the same verdict as a rate over time, so you can see when it started.

Note

The latency panels deliberately do not filter by {source="ripestat"}. raven_global_check_latency_seconds is registered as a plain histogram rather than a histogram vector, so it carries no source label — adding the matcher would select no series and leave both panels empty.

Tip

lab/demo-master.sh setup imports the dashboard through the Grafana API with overwrite: true, so the demo lab picks this row up automatically — no manual import step. See Lab → Overview.

Running Grafana with Docker

For the demo lab or a quick local setup:

docker run -d \
  --name grafana \
  -p 3000:3000 \
  -e GF_SECURITY_ADMIN_PASSWORD=admin \
  grafana/grafana:latest

Then import the dashboard as described above and point the Prometheus datasource at your RAVEN metrics endpoint.

Memory Profiling (pprof)

RAVEN can expose Go's runtime profiler for diagnosing memory growth, CPU hot spots or goroutine leaks on a running daemon. It is off by default and runs on its own listener, separate from the API server and the Prometheus endpoint.

Enabling the pprof Endpoint

Add to your raven.yaml and restart RAVEN:

debug:
  pprof:
    listen: "localhost:6060"   # empty or absent = disabled (default)

RAVEN logs pprof endpoint started with the bound address when it is up. An idle endpoint costs nothing: profiling work happens only while a profile is being requested, and a CPU profile adds a few percent of overhead for the seconds it runs.

Warning

Keep pprof on loopback. The endpoint is unauthenticated, and heap profiles expose process memory contents. If the listen address binds anything other than 127.x or ::1 (including :6060, which binds every interface), RAVEN logs a warning at startup. To profile a remote host, leave it on localhost and tunnel in: ssh -L 6060:localhost:6060 raven-host.

Diagnosing Memory Growth

Live heap — what is holding memory right now:

go tool pprof http://localhost:6060/debug/pprof/heap

At the (pprof) prompt, top ranks functions by in-use memory and list <func> shows the allocating lines. To find what is growing, save a heap profile now and another a few hours later, then diff them:

curl -s -o heap-1.pb.gz http://localhost:6060/debug/pprof/heap
# ...hours later...
curl -s -o heap-2.pb.gz http://localhost:6060/debug/pprof/heap
go tool pprof -base heap-1.pb.gz heap-2.pb.gz

Allocations since start — what is churning memory:

go tool pprof http://localhost:6060/debug/pprof/allocs

This counts every allocation since the process started, including memory already freed. High churn with flat in-use memory points at GC pressure rather than a leak.

Goroutines — whether something is accumulating:

curl -s http://localhost:6060/debug/pprof/goroutine?debug=1

Goroutines are grouped by identical stack, with a count for each. A count that keeps climbing between runs, for example per-session or per-subscriber goroutines outliving their peer, is a leak in its own right. Each leaked goroutine also pins whatever memory it references.

Note

RSS reported by the OS can sit well above the Go heap in use: the runtime returns freed memory to the OS lazily. Compare the heap profile's in-use total against go_memstats_heap_inuse_bytes on the Prometheus endpoint before concluding that memory is leaking outside the heap.

Other endpoints follow the standard Go layout: /debug/pprof/ lists every profile, /debug/pprof/profile?seconds=30 is a CPU profile, and /debug/pprof/trace?seconds=5 is an execution trace. Delta profiles (?seconds= on heap or allocs) are not supported; use -base as shown above.