NOTE: Updates are happening daily but these notes are updated once every two weeks.
2026-09-07
journalctl -o json, which replaces any field where the name, an = and the value together reach 4096 bytes with null unless -a is passed. So a message of about 4088 bytes or more arrived as "MESSAGE": null — the entry was stored, the priority and unit and hostname were stored, and the text was gone. Stack traces, tracebacks, verbose SQL errors and pretty-printed API payloads all clear 4 KB without trying, which made this worst for exactly the lines worth keeping. Nothing reported it: journalctl exits 0 and the server accepted the entry. The agent now passes -a. If you ship the journal yourself with a journalctl -o json | curl pipeline, add -a to it too. To find out whether it has been happening to you, run SELECT COUNT(*) FROM logs WHERE json_valid(msg) AND json_type(msg, '$.MESSAGE') = 'null' against the source2026-08-24
host:port, credentials (leave the username blank for a relay that needs none), a From address and one or more comma-separated To addresses. Port 465 connects with implicit TLS; other ports upgrade to STARTTLS when the server offers it. Like every channel it has a Test button, so you can confirm delivery before pointing alerts at it2026-08-21
2026-08-10
promhttp_metric_handler_requests_total, which reports one series per HTTP status code — are now stored as the sum of their series each scrape. Previously only one arbitrary series was kept, the rest failed with a UNIQUE constraint warning in the logs on every scrape, and because a scrape stopped at the first error other metrics from the same source could be dropped with them. Charts for an affected metric will show a one-time step up to the summed value2026-08-07
json_extract raises an error the first time it meets a line that is not JSON, and rows found before that point had already been returned — so a query over a source containing one plain-text line quietly returned a truncated result set and no error. Add WHERE json_valid(msg) to skip those linesFROM logs query produced an empty file with no errorstate=fail on schedule was flagged as “no longer checking in” shortly after its failure alert — it had not stopped checking in, it was checking in with failures. That second message is now suppressed while a failure is outstanding, and returns once the job succeeds and then goes quiet for realstate=fail now notifies you within a minute. Previously the failed run was recorded and shown in red on the job page, but nothing was sent — you found out when the successful check-in failed to arrive, which takes about 12 minutes for an hourly job and about 5 hours for a daily one. The notification carries the status_code and message from the ping. A job that fails every run does not message you every run: the first failure notifies, then it stays quiet until the job completes successfully or a day passesstatus_code, message and repeated metric=name:value parameters are stored with the ping — previously the documentation showed these but the server read only state and dropped the restPOST or HEAD are now recorded. The cron page has always said all three methods work, but only GET was routed — a POST or HEAD ping was silently rejected2026-07-06
2026-07-05
BACKUP_S3_ENDPOINT environment variable)appdb/ and logs/{source-token}/ prefixes, and automatic cleanup of old backups only touches those prefixes so other objects in a shared bucket are left alone2026-04-17
2026-04-16
2026-04-15
2026-04-11
? to see all shortcuts. Navigate results with j/k, expand with Enter, focus search with /, run queries with Ctrl+Enter, paginate with n/p, and open the time range picker with t.2026-04-10
2026-04-09
2025-12-21
2025-02-13
2025-01-09
2024-12-08
2024-12-06
2024-12-02
2024-12-01
2024-11-26
2024-11-23
2024-11-22
2024-11-20
2024-11-19
2024-11-18
2024-11-17
2024-11-16
2024-11-15
2024-11-14
2024-11-13
2024-11-03
2024-10-26
2024-10-24
2024-10-23
2024-10-19
2024-10-19
2024-10-05
2024-10-03
2024-10-02
2024-08-01
2024-07-25
2024-07-19
2024-07-13
2024-04-30
2024-04-28
2024-03-19
2024-03-16
2024-03-06
2024-03-01
2024-02-22
2024-01-29
2024-01-22
2023-11-21
2023-03-26
2023-03-19
2023-03-18
2023-03-16
2023-03-15
2023-03-14
2023-03-06
2023-03-05
2023-02-29
2023-02-28
2023-02-23
2023-01-27
2023-01-09
2023-01-08
2023-01-06
2023-01-03
2023-12-23
2023-12-18
2023-11-30
2023-11-28
2023-11-26
2023-11-20
💌 Get notified on new features and updates