Health

Returns the node's operational status and the typed condition catalog that explains it.

Method: GET /health

Parameters: None

Returns: JSON object containing the overall status, network/version information, validator identity (validator mode), and the full condition catalog

HTTP status: 503 when status is unavailable; 200 for every other state.

Example Request:

curl -X GET http://localhost:9153/health

Example Response:

{
  "status": "healthy",
  "network": "bradbury-phase2",
  "node_version": "v0.6.0",
  "protocol_version": "0.3.0",
  "genvm_version": "v0.0.18",
  "validator_address": "0x1234567890abcdef1234567890abcdef12345678",
  "validator_moniker": "my-validator",
  "conditions": {
    "chain.rpc":                { "state": "healthy", "occurrences": 0, "consecutive_failures": 0, "since": "2026-09-01T08:00:03Z", "updated_at": "2026-09-01T10:31:12Z", "last_ok": "2026-09-01T10:31:12Z" },
    "chain.websocket":          { "state": "healthy", "occurrences": 0, "consecutive_failures": 0, "since": "2026-09-01T08:00:03Z", "updated_at": "2026-09-01T10:31:12Z", "last_ok": "2026-09-01T10:31:12Z" },
    "chain.sync":               { "state": "healthy", "occurrences": 0, "consecutive_failures": 0, "since": "2026-09-01T08:02:14Z", "updated_at": "2026-09-01T10:31:12Z", "last_ok": "2026-09-01T10:31:12Z" },
    "consensus.ban":            { "state": "healthy", "occurrences": 0, "consecutive_failures": 0, "since": "2026-09-01T08:00:03Z", "updated_at": "2026-09-01T10:31:12Z", "last_ok": "2026-09-01T10:31:12Z" },
    "consensus.stake":          { "state": "healthy", "occurrences": 0, "consecutive_failures": 0, "since": "2026-09-01T08:00:03Z", "updated_at": "2026-09-01T10:31:12Z", "last_ok": "2026-09-01T10:31:12Z" },
    "consensus.operator_funds": { "state": "healthy", "occurrences": 0, "consecutive_failures": 0, "since": "2026-09-01T08:00:03Z", "updated_at": "2026-09-01T10:31:12Z", "last_ok": "2026-09-01T10:31:12Z" },
    "consensus.prime":          { "state": "healthy", "occurrences": 0, "consecutive_failures": 0, "since": "2026-09-01T08:04:41Z", "updated_at": "2026-09-01T10:31:12Z", "last_ok": "2026-09-01T10:31:12Z" },
    "genvm.llm":                { "state": "healthy", "occurrences": 0, "consecutive_failures": 0, "since": "2026-09-01T08:00:31Z", "updated_at": "2026-09-01T10:29:56Z", "last_ok": "2026-09-01T10:29:56Z" },
    "genvm.web":                { "state": "healthy", "occurrences": 0, "consecutive_failures": 0, "since": "2026-09-01T08:00:31Z", "updated_at": "2026-09-01T10:29:56Z", "last_ok": "2026-09-01T10:29:56Z" },
    "node.disk":                { "state": "healthy", "occurrences": 0, "consecutive_failures": 0, "since": "2026-09-01T08:00:03Z", "updated_at": "2026-09-01T10:31:12Z", "last_ok": "2026-09-01T10:31:12Z" },
    "node.snapshot_age":        { "state": "healthy", "occurrences": 0, "consecutive_failures": 0, "since": "2026-09-01T08:00:03Z", "updated_at": "2026-09-01T10:31:12Z", "last_ok": "2026-09-01T10:31:12Z" },
    "webdriver.render":         { "state": "healthy", "occurrences": 0, "consecutive_failures": 0, "since": "2026-09-01T08:00:03Z", "updated_at": "2026-09-01T10:31:12Z", "last_ok": "2026-09-01T10:31:12Z" }
  }
}

Response Fields:

status (string): the most urgent state across the catalog — unknown, healthy, suspect, warning, degraded, or unavailable
network (string): The network the node is connected to
node_version, protocol_version, genvm_version (string): build versions
validator_address (string, validator mode only): The validator's address
validator_moniker (string, validator mode only): The validator's on-chain moniker; reported alongside the address even when empty
conditions (object): the full catalog. Every entry is always present — this is not a list of what is broken.
- state (string): current observed health; use this field to determine whether the capability is working.
- occurrences (number): accepted failed observations since process start
- consecutive_failures (number): current failure streak; zero after recovery
- since, updated_at, last_ok (string, optional): ISO 8601 lifecycle timestamps
- reason (string, present only when not healthy): stable, low-cardinality cause code
- impact (string, present only when not healthy): what the fault does to this node
- action (string, present only when not healthy): what the operator should do
- details (object, present only when not healthy): typed fields explicitly allowlisted by the catalog entry. Raw provider payloads, prompts, credentials, URLs, and stack traces are rejected.

severity is not returned by this endpoint. Alert priority belongs to the alerting policy; clients that previously read conditions.*.severity should use state for current health.

States

State	Meaning	Alerted
`unknown`	The node never observed this condition.	no
`healthy`	The last observation succeeded.	no
`suspect`	One failure, not yet confirmed. It may clear by itself.	no
`warning`	The capability works, and it is confirmed sub-optimal.	yes
`degraded`	The failure is confirmed. The capability does not work.	yes
`unavailable`	The node cannot do its duty at all.	yes

Roll-up precedence, most urgent first: unavailable, degraded, warning, suspect, unknown, healthy.

unknown is not on that ladder. It is only ever the state a condition starts in, so it means "not observed yet", never "faulty". A GenVM module rests there until a transaction needs one; node.snapshot_age rests there on a node that takes no snapshots. Letting either gate the endpoint would report unknown for a node where nothing is wrong.

If no condition has been observed at all, status is unknown — nothing observed is not the same as nothing wrong.

Only unavailable returns 503.

A condition can never exceed the liveness cap its catalog entry declares. The GenVM conditions are capped at degraded, so a dead LLM provider returns 200: that node is still booted, synced and voting — it votes Timeout on the transactions it cannot execute, which is the designed behaviour rather than an outage. Failing the endpoint would pull a working validator out of a load balancer.

503 means the node cannot make progress or cannot act. Only four conditions can return it:

Condition	Reason	503 when
`chain.rpc`	`chain_rpc_unreachable`	The HTTP endpoint has failed every check for 30 seconds. The node cannot submit a duty, read a receipt or read its nonce.
`chain.websocket`	`websocket_disconnected`	The WebSocket endpoint has failed every check for 30 seconds. The node receives no new chain heads and no contract events.
`chain.sync`	`sync_blocked`, `sync_block_unreadable`	The chain head is above the synced block and the synced block has not moved for 5 minutes. See below.
`node.disk`	`disk_space_critical`	2% or less of the disk is free. The node cannot write new blocks or receipts.

A shorter transport failure is suspect, then degraded (HTTP 200). The checks run when a client calls this endpoint, so the 30 seconds are measured in time, not in checks.

consensus.ban and consensus.operator_funds are capped at warning. A banned node, and a node whose operator balance is below the critical amount (reason operator_balance_critical), still syncs and serves; leaving rotation or a restart fixes neither.

consensus.prime and node.snapshot_age are capped at warning.

consensus.stake is capped at warning: below the minimum self-stake the node keeps syncing, and only the consensus contracts do not select the validator for duties. The reason is self_stake_pending when a pending self-stake deposit covers the gap (it becomes stake at the validator's prime in a later epoch), and self_stake_below_minimum otherwise.

WebDriver render check

webdriver.render runs the same fresh render probe as doctor, using webdriver_host from the configured GenVM's genvm-module-web.yaml. It requires HTTP 200, Resulting-Status: 200, and exactly Hello world! from the diagnostic page, within 10 seconds. It bypasses WebDriver's cached /healthcheck response. A failure reports degraded with reason webdriver_check_failed and safe failure, http_status and page_status details. The endpoint still returns HTTP 200: a browser outage does not stop the node syncing and voting.

Allow time for the ten-second render deadline when setting a client timeout. task devtools:nodes:health defaults to 15 seconds per request so a browser timeout can return its diagnosis instead of appearing as an unreachable node.

genvm.web continues reporting the broader web module's boot and transaction results. A successful direct browser check does not clear a web-module failure.

For genvm.start_manager: false, the manager owns the browser configuration. The node leaves webdriver.render unknown instead of probing an unused local root. Doctor likewise reports the browser as not checked; run it on the node that owns the manager configuration. Observed genvm.web results still apply.

Example: the LLM provider is dead — HTTP 200

{
  "status": "degraded",
  "conditions": {
    "genvm.llm": {
      "state": "degraded",
      "reason": "llm_request_failed",
      "impact": "This validator cannot execute a transaction that needs an LLM. It votes Timeout, and the consensus contracts may penalize a Timeout vote.",
      "action": "Check the affected LLM provider service and payment or billing status, restore access, or configure a working fallback. If credentials are changed through .env, restart the node.",
      "occurrences": 2,
      "consecutive_failures": 2,
      "since": "2026-09-01T10:30:45Z",
      "updated_at": "2026-09-01T10:31:12Z",
      "last_ok": "2026-09-01T04:12:03Z"
    }
  }
}

Example: the validator is banned — HTTP 200

{
  "status": "warning",
  "conditions": {
    "consensus.ban": {
      "state": "warning",
      "reason": "validator_banned",
      "impact": "The consensus contracts do not select this validator for a round. The node earns nothing.",
      "action": "Wait for the stated epoch. If the ban is permanent, restake with a new validator.",
      "details": { "until_epoch": "4182", "permanent": "false" },
      "occurrences": 1,
      "consecutive_failures": 1,
      "since": "2026-09-01T09:14:02Z",
      "updated_at": "2026-09-01T10:31:12Z",
      "last_ok": "2026-09-01T09:13:52Z"
    }
  }
}

chain.sync — 503 only when the sync is blocked

Depth is not a fault. A node syncing from genesis is millions of blocks behind and healthy while its synced block advances. A quiet chain is not a fault either: when the chain head is at the synced block there is nothing to sync, however long no block arrives.

The sync is blocked when the chain head is above the synced block and the synced block has not moved for 5 minutes. The node reads the chain head with a request (eth_blockNumber), not only from its head subscription, so a subscription that stopped delivering heads is also seen. The 5 minutes start when there is a block to sync: a failure at a block, an unreadable block, or the first check that saw the head above the synced block.

Reason	State	HTTP	When
`sync_lagging`	`warning`	200	1–100 blocks behind for 30 s, or the sync cursor cannot be read
`sync_catching_up`	`degraded`	200	More than 100 blocks behind, at any depth
`sync_block_unreadable`	`degraded`	200	The node cannot read the logs of a block, for less than 5 minutes (`details`: `unreadable_block`, `unreadable_seconds`)
`sync_blocked`	`unavailable`	503	The sync is blocked (`details`: `chain_head`, `blocked_seconds`; `failing_block`, `failures`, `failing_seconds` when the sync loop failed at a block)
`sync_block_unreadable`	`unavailable`	503	The sync is blocked below a block the node cannot read (`details`: as `sync_blocked`, plus `unreadable_block`, `unreadable_seconds`)

The node never skips a duty because it is behind. It votes Timeout only on a transaction that reads state it has not synced; a new contract deploy still runs.

cURL Example with Headers:

curl -X GET http://localhost:9153/health \
  -H "Accept: application/json" \
  -v

balance metrics