Test and troubleshoot

Before you start

Use a bench device and its logs. Record command-gate checks during Commands, step 4. Run the Lab checks first, then keep Test with Balto on a bus open. If skipped, use a separate test registration with a new serial.

1. Add recovery to the device process

At startup: load credentials, initialize hardware, start the command server, and announce. Run telemetry, heartbeat, and capability refresh independently. Re-announce address, port, or capability changes.

• Add network and driver timeouts; a failed sensor must not stop other reporting.
• 401: reload rotated credentials or repair enrollment.
• Heartbeat/telemetry 404: re-announce. A recreated registration needs adoption again.
• Announcement response with status: "retired": an operator removed the registration. Device traffic does not restore it. An operator must call POST /payloads/:id/restore and adopt it again.
• 429/503 or connection failure: retry with bounded exponential backoff and jitter. Honor Retry-After. Bound the telemetry queue and define retention.
• failed > 0: some samples were not stored, but individual failures are not identified. Reconcile or retry uncertain samples; retries may create duplicates. Do not discard the whole buffer on HTTP 200 alone.
• Keep the last accepted manifest until its replacement is accepted. Reject commands for hardware that is no longer available.
• Restart automatically after reboot; handle credential renewal and address changes. Store credentials outside the executable/image.
• Resolve uncertain command outcomes using the request journal and measurements before issuing another command.

2. Run the acceptance checks

Test	Pass condition
Announce/re-announce	Discovered initially; same UUID and refreshed manifests on repeat.
Identity	Invalid tokens fail; another credential cannot publish under the serial.
Telemetry	Sent values read back with matching timestamps and stored counts.
Invalid samples	Wrong types/ranges and times over five minutes ahead are rejected; valid samples still pass.
Missing sensor	New samples omit the reading; other fields continue. Older UI values may remain.
Command gates	Discovered and adopted-but-disabled registrations both refuse commands.
mTLS	Untrusted clients cannot invoke hardware.
Command validation	Unknown commands, missing parameters, and invalid values never reach hardware.
Duplicates	Concurrent delivery executes once; reboot meets the promised deduplication behavior.
Driver failure	Accurate rejected/failed outcome, including partial changes.
Command effect	Matching acknowledgment and measured result.
Lost reply	Uncertain outcome is reconciled without repeating an unsafe operation.
Restart/address change	Automatic re-announcement and reporting recovery.
Credential rotation	New device tokens work; revoked tokens fail. Expired or untrusted TLS certificates fail command authentication.
Outage/overload	Bounded buffering and successful recovery.

3. Check each test with Balto

Test	Balto check
Announce/re-announce	balto payloads list --bus <bus>: the same id, and the new manifest.
Telemetry	balto trace <payload> --up --every: one row per message, with its values.
Command gates	balto payloads send while discovered, then while disabled: both refused.
Command effect	balto payloads audit <payload> --request-id ID, then balto trace --up.
Restart	Restart the payload during balto trace <payload> --follow: it announces again and reports within one interval.

Problems and their messages: Troubleshooting.

Record software version, environment, and test results. For support, provide the serial, UTC timestamp, request ID, your command log lines for that ID, and redacted error. Exclude credentials and sensitive parameters.

---

← Test with Balto on a bus · Next: Go to flight →