Appearance
Troubleshooting
Find the step that fails, then the message you see. Messages are shown as MC, the bus, or Balto print them.
For support, send the serial, a UTC timestamp, the request ID, your command log lines for that ID, and the error with credentials removed.
Payload Lab
Messages as Payload Lab and balto lab print them.
| Message | Meaning |
|---|---|
Install the Lab runtime first / no lab runtime is installed | Install the lab-runtime file Urban Sky sent you: balto lab runtime install ./lab-runtime. |
the lab runtime did not start in 15 s / the lab runtime stopped before it started: … | The runtime file is damaged or built for another computer. Ask Urban Sky for a new one, then install it again. |
enter the payload's address, like 127.0.0.1:8090 / "…" is not an address | Give a host and port, with no path. |
the bus dials the device API over http, not https | The bus reads the device API over plain HTTP. Serve :8090 without TLS. |
| Nothing listens there. Start your payload, or check the port. | The Lab reached the host, but nothing listens on that port. |
| It took the connection but never answered. | Your route accepted the connection and hung. Answer each request in under a second. |
does not parse: … under Its capabilities pass the bus's lint | GET /api/v1/capabilities is not valid JSON, or not the document shape. See Serve the capability document. |
| A lint refusal, such as a code and its detail | The bus would refuse that part of your document. The Lab shows the fix with it. |
acme.temp_c: declared number, got string "warm" | A key arrived with another type than you declared. Send the declared type, or change the declaration. The bus drops the key as wrong_type. |
acme.temp_c: declared -80 to 60, got 99 | The value is outside the declared range or enum. The bus drops the key as out_of_range. |
acme.extra: sent, but not declared in telemetry_fields | Declare the key, or stop sending it. The bus drops it as undeclared. |
| bus refused before sending: PAYLOAD_REJECTED: … | The bus checked the command against your capability document and refused it. It never reached your payload. The reason names the parameter and its range. |
No log lines yet. Push NDJSON to …/api/v1/payload/logs. | Nothing reached the Lab's ingest. Set USKY_BUS_INGEST_URL to the URL in Ingest for this payload. |
| Logs and metrics accepted with no drops fails | Over the ingest limits. The check passes again after a minute with no drops. See Limits. |
fault switches are for the built-in payloads; stop your own payload to test recovery | Break it works on the Example sensor and Mock Harrier only. |
Mock Harrier does not push logs; flood logs from the Example sensor | Use the Example sensor for the log-flood fault. |
… is not a Payload Lab report | balto lab verify was given another JSON file. |
report signature does not match its contents | The report file changed after Balto signed it. |
Bench
See Set up a test bench: If it fails for balto pair, balto up, DHCP and MTU problems.
Announce and telemetry
| Symptom | Check |
|---|---|
Device request 401 | Device token, revocation or rotation, API URL. |
Announce 403 on the serial | Another credential owns this serial. Ask Urban Sky to resolve it. |
Announce 400 | Names, enum values, address, unique fields, and limits. See Manifest rules. |
Announce 400: payload names itself as its gateway | identity.parentSerial is your own serial. Set it to the bus's serial. |
Telemetry 404 | Announce first. Check the serial and the organization of the token. |
Telemetry 200 but no data | Response counts, rejection reasons, timestamps. |
| Telemetry missing, no error | GET /payloads/:id/ingest-rejections for the last seven days. |
| Fresh heartbeat, stale values | Your sampling loop and sensor reads. |
Not in balto payloads list --bus <bus> | Same organization's device token. parentSerial is the bus's serial, as balto payloads list prints the bus. |
Status retired | An operator removed the registration. Restore it and adopt it again. |
Commands
Device API. These come from the bus, before it sends anything to your payload:
| Message | Meaning and fix |
|---|---|
The payload computer does not declare '<command>' | Your capability document has no such command. The bus re-reads the document every 30 seconds. |
'<command>' belongs to no module the payload computer serves | The command's module is not in modules, or the module name is reserved. |
The <module> module's commands are not allowed on this device | The bus's PAYLOAD_LAN_MODULES does not name your module. Ask Urban Sky. |
'<command>' has a parameter this bus cannot type | Use number, integer, boolean, string, or a string enum. |
MC answers a command it could not deliver with 502 and Command delivery failed (<reason>): <message>. The reason tells you whether the command may have run.
| Message | Meaning and fix |
|---|---|
403 | The payload is not adopted, commands are off, or the user lacks the permission or role. Check balto payloads list, then balto payloads adopt and enable-commands. |
400 | The parameters do not match the stored command manifest: names, types, ranges. Re-announce if you changed the manifest. |
link_target_unsupported: ops knows no payload-LAN address for <serial> | MC found the bus, but no lease for your payload on it. Check that the payload has a DHCP lease from the bus and that address is that lease. Wait one bus report and retry. |
Payload gateway is not adopted | The bus row named in your parentSerial is discovered. An operator adopts the bus with balto payloads adopt <bus serial>. |
Payload has no owning bus network path | parentSerial does not name a Husky bus that MC knows in your organization. Fix parentSerial. |
Payload <serial> declares SHARED_VPN but MC has no VPN transport configured (TAILSCALE_SOCKS5_PROXY unset) — refusing to dial <addr> directly | MC did not find a bus route, and fell back to the old shared VPN, which this environment does not run. Set parentSerial to the bus's serial. |
409: Payload <serial> has no bus-link path, and its organization reaches buses over the bus link only. Attach it to a bus that is on the link: ops then dials it at its payload-LAN address through that bus | MC found no bus for the payload, or the bus has no live link. Set parentSerial to the bus's serial, and check that the bus is online. |
409: bus_image_too_old | The bus software has no payload command route. Ask Urban Sky to update the bus. |
requires mTLS but MC has no client cert configured, or 502: mtls_unavailable | You announced SECURITY_TRANSPORT_MTLS, and MC has no client certificate here. Use SECURITY_TRANSPORT_VPN behind a bus, or ask Urban Sky. |
502: unroutable | Not sent. Check the route messages above and the network mode. |
502 with PAYLOAD_UNREACHABLE | The bus could not open a connection to your command server. Check the server listens on the payload-LAN interface and port you announced. |
400 with PAYLOAD_REJECTED | Your command server answered with an error. Read its message and your log for the request ID. |
409 with DEVICE_BUSY | A command with the same request ID is still running. Wait, then read the result with balto payloads audit --request-id. |
502: bad_response | Your server's answer is not a JSON acknowledgment with the same request ID and a known state. |
502: outcome_unknown | The command may have run. Check the device state and your log for the request ID. Retry with the same request ID. |
| TLS failure | CA trust, certificate hostname or IP, validity, client identity. |
| Accepted, no effect | Your request journal, the driver result, and the measured value in balto trace <payload> --up. |
An operator's click, and no command in balto payloads audit | The operator's client refused it. Ask Urban Sky for the client's blocked log. |
| No device log line for a request ID | The command did not reach the device. Send Urban Sky the request ID. |
Logs and metrics
| Message from the bus | Fix |
|---|---|
Connection refused or timeout on port 4318 | Check BUS_ADDRESS: 10.55.0.1 on a cloud bench bus, 192.168.8.1 on a Husky bus. |
403: This address is not a payload device on this bus | Send from your payload-LAN address, not through NAT or another interface. |
413: Request body is over <n> bytes | Send smaller batches. PayloadTelemetry does this for you. |
429 with Retry-After | You are over the rate. Wait that many seconds. PayloadTelemetry paces and waits for you. |
More: Send logs and metrics.
Video
| Symptom | Check |
|---|---|
| No video in the Balto Payloads tab | MC pulls SRT from your announced address, and cannot pull it through the bus link. Ask Urban Sky for your route. |
| Video plays locally, not in MC | Codec (H.264 or H.265 in MPEG-TS) and the route. |
| No camera footprint on the map | KLV tags in the stream. See Camera metadata (KLV). |