How to send logs to a server with curl

Introduction

You have a script that prints useful things. You want those things somewhere you can search.

The smallest possible answer is curl. No agent, no config file, no daemon. One command at the end of a pipe, and the line is on the log server.

This guide covers posting a single line, posting a whole file, compressing it, authenticating two ways, and the four mistakes that make it look like it worked when it did not.

The short answer

One line:

curl -X POST -d 'backup finished in 418s' \
    https://logs.example.com/api/v1/log/YOUR-SOURCE-TOKEN

A whole file, one row per line:

curl -X POST --data-binary @/var/log/backup.log \
    https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN

The rest of this page is why those are different commands, and what goes wrong.

Two endpoints, and why you want the second one

Central Logging has one endpoint for a single line and one for many.

	Endpoint	Body	Becomes
One line	`POST /api/v1/log/{token}`	The line	One row, body stored verbatim
Many lines	`POST /api/v1/ingest_logs/{token}`	Newline-separated lines	One row per line

The difference is not just a matter of scale. The single-line endpoint stores whatever you send it as one row, newlines and all. It never splits. Post a ten-line file to it and you get one row containing ten lines.

The bulk endpoint splits the body on newlines and stores one row per line. Blank lines are skipped. Leading and trailing whitespace is trimmed.

So a 900-line file becomes 900 rows in one request. Calling the single-line endpoint 900 times gets you the same rows and 900 TCP connections, and it is slow enough to notice on a busy host.

Use ingest_logs unless you genuinely have one line.

The response tells you how many rows it took:

{"logs_ingested": 900}

Compare that number to what you sent. It is the only confirmation you get.

A note on when the rows actually appear. Both endpoints answer before the write lands. Single lines are held in memory and written when a thousand have piled up or ten seconds have passed, whichever comes first, so a lone line from a script shows up within about ten seconds rather than instantly. Bulk posts go on a queue and land shortly after. If you post and immediately query, give it a moment.

`-d` and `--data-binary` are not the same thing

This is the mistake that costs the most time.

curl -d @file strips newlines. It was designed for form bodies, where newlines are not meaningful. Post a 900-line file with -d and the server receives one enormous line, and stores it as one row.

--data-binary @file sends the bytes as they are.

# wrong: 900 lines arrive as 1 row
curl -X POST -d @/var/log/backup.log https://logs.example.com/api/v1/ingest_logs/TOKEN

# right
curl -X POST --data-binary @/var/log/backup.log https://logs.example.com/api/v1/ingest_logs/TOKEN

The {"logs_ingested": 1} in the response is the tell. If you sent a file and got back 1, this is why.

For a single line -d is fine, because there are no newlines to lose.

Piping a command straight in

@- reads standard input, so a command’s output can go to the server without touching disk:

/usr/local/bin/backup.sh 2>&1 | curl -X POST --data-binary @- \
    https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN

2>&1 matters. Half the useful output of a failing script is on stderr, and a pipe only carries stdout.

One caveat on this shape: curl buffers the whole body before sending, so nothing arrives until the command exits. A script that runs for an hour ships an hour of logs in one request at the end, and a script killed by the OOM killer ships nothing at all. That is fine for a nightly backup. It is not a live feed.

If you need the lines to arrive as they happen, run the agent instead — see sending journalctl logs to a central server.

Authenticating with a header instead of the URL

The token in the URL is convenient and it ends up in shell history, in ps output while curl runs, and in any proxy access log along the way.

The bulk endpoint also accepts the token as a bearer token, with no token in the path:

curl -X POST --data-binary @/var/log/backup.log \
    -H "Authorization: Bearer $CL_TOKEN" \
    https://logs.example.com/api/v1/ingest_logs

Read the token from a file or an environment variable that came from one:

CL_TOKEN="$(cat /etc/central-logging.token)"

Use -H @- style redirection or a config file if you want it off the command line entirely:

curl -X POST --data-binary @/var/log/backup.log \
    -K /etc/curl-logging.conf \
    https://logs.example.com/api/v1/ingest_logs

where /etc/curl-logging.conf is mode 0600 and contains:

header = "Authorization: Bearer abc123..."

The single-line endpoint takes the token in the path only.

Compressing the body

Large batches compress well — plain text logs routinely go to a tenth of their size.

gzip -c /var/log/backup.log | curl -X POST --data-binary @- \
    https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN

The server detects gzip by sniffing the first bytes of the body, not by reading a header. So this works with no Content-Encoding header at all, and adding one changes nothing. Send it if it makes you happy; it is not what makes the decompression happen.

Do not gzip a single short line. You will spend more bytes on the gzip header than you save.

Four ways this looks like it worked when it did not

1. curl exits 0 on an HTTP error

By default curl prints the server’s error body and exits 0. In a cron job that means a failed upload looks exactly like a successful one.

-f makes curl fail the exit status on a 4xx or 5xx. -sS keeps it quiet while still printing real errors:

curl -fsS -X POST --data-binary @- \
    https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN \
  || echo "log upload failed" >&2

Use -fsS in every unattended script. It is the difference between a silent gap in your logs and a line in your mail.

2. A bad token returns 400, not 401

An unknown source token is answered with HTTP 400. The bulk endpoint says invalid user token and the single-line endpoint says {"error": "invalid log source"}, so do not match on the message either.

If you are checking specifically for a 401 to detect an auth problem, you will never see one. -f catches it whatever the code is, which is the argument for using -f rather than checking codes by hand.

3. A line over 1 MB is shortened, not stored whole

One line is stored whole up to 1 MB. A longer line is cut to fit and gets a marker on the end:

...[truncated by centrallogging: line exceeded 1048576 bytes]

Nothing is dropped. The rest of the batch arrives, and the response tells you it happened:

{"logs_ingested": 4, "lines_truncated": 1}

This is worth checking in a script, because a shortened line still counts as ingested. A JSON log line with a large payload is the usual cause, followed by base64 blobs and stack traces that something has joined into one line.

sent=$(awk 'NF{n++} END{print n+0}' /var/log/backup.log)
resp=$(curl -fsS -X POST --data-binary @/var/log/backup.log \
    https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN)
got=$(printf '%s' "$resp" | sed -n 's/.*"logs_ingested": *\([0-9]*\).*/\1/p')
cut=$(printf '%s' "$resp" | sed -n 's/.*"lines_truncated": *\([0-9]*\).*/\1/p')

[ "$sent" = "$got" ] || echo "sent $sent lines, server took $got" >&2
[ "$cut" = "0" ] || echo "$cut line(s) were over 1 MB and were shortened" >&2

awk rather than wc -l because blank lines are skipped on the way in, and because a file with no trailing newline makes wc -l one short.

To find shortened lines after the fact, search for the marker:

SELECT id, datetime("timestamp", 'unixepoch') AS at, length(msg) AS bytes
FROM logs
WHERE msg LIKE '%truncated by centrallogging%'
ORDER BY "timestamp" DESC;

Older releases behaved much worse here: a line of 65,536 bytes or more ended the split, so that line and every line after it in the same request were thrown away behind an HTTP 200. If you are running a build from before 2026-09-14, the logs_ingested check above is the only way to see it, and upgrading is the fix.

4. The timestamp is when the server received the line

Nothing parses a timestamp out of your log line. Every row gets the server’s clock at the moment it arrived.

For a script that posts at the end of its run, that means every line in the batch carries the same arrival time and sorts in file order rather than in event order. Usually that is close enough. When it is not, put your own timestamp in the line as JSON and sort on that instead:

printf '{"ts":"%s","msg":"backup finished"}\n' "$(date -Is -u)"

Then in a query:

SELECT json_extract(msg, '$.ts') AS ts, json_extract(msg, '$.msg') AS message
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$.ts') IS NOT NULL
ORDER BY ts DESC
LIMIT 50;

Two things about that query are load-bearing.

The json_valid(msg) guard belongs in WHERE, not around the SELECT. A source holding a mix of JSON and plain-text rows will otherwise error out on the first plain line. The IS NOT NULL is what keeps rows that are valid JSON without a ts field — journald entries, say — out of the result instead of showing them as blanks.

The -u on date is load-bearing too. ORDER BY on an ISO-8601 string is a text sort, which only matches time order while every row carries the same UTC offset. Ship everything in UTC and the sort is correct; ship local times from hosts in two countries and it is not.

There is more on this in querying logs with SQL.

A cron job that ships its own output

Putting the pieces together. This runs a job, keeps both streams, ships them, and says something if the shipping fails:

#!/bin/sh
set -u
TOKEN="$(cat /etc/central-logging.token)"
URL="https://logs.example.com/api/v1/ingest_logs"

out="$(/usr/local/bin/backup.sh 2>&1)"
rc=$?

printf '%s\n' "$out" | gzip -c | curl -fsS -X POST --data-binary @- \
    -H "Authorization: Bearer $TOKEN" "$URL" >/dev/null \
  || echo "log upload failed" >&2

exit "$rc"

exit "$rc" at the end matters: the script’s own exit status should be the job’s, not curl’s. Otherwise a successful upload hides a failed backup.

Note that this holds the whole output in a shell variable, which is the right trade for a backup script and the wrong one for anything that prints megabytes.

Telling the job’s health apart from the job’s output

Shipping the output answers “what did it say”. It does not answer “did it run at all”, because a job that never starts prints nothing and a pipe that ships nothing looks identical to a job with a quiet night.

That is a separate ping, on a separate endpoint:

0 3 * * * /usr/local/bin/backup.sh && curl -fsS "https://logs.example.com/p/YOUR-API-KEY/nightly-backup?state=complete"

No ping, and the job is reported late on its own learned schedule. Monitoring cron jobs covers the whole setup, and where cron job output goes covers why cron’s stdout does not reach the journal on its own.

When curl is the wrong tool

curl is a good fit for a script that runs, prints, and exits. It is a bad fit for a log file that grows forever.

For a file that keeps being appended to, you want something that remembers its position across restarts, retries a failed send, and does not re-send what it already sent. That is what an agent is for. clagent does it for the systemd journal, and rsyslog does it for files — see sending rsyslog logs to a central server.

Rolling your own with tail -f | curl is the version of this that looks fine for a week. It loses everything buffered when the process dies, it starts from the end of the file on restart, and it has no idea whether the last send succeeded.

Sending journalctl logs to a central server — the agent version, for logs that keep arriving.
Where cron job output goes — why the job you just wrote prints into the void.
Querying logs with SQL — what to do with the lines once they land.
Getting JSON output from journalctl — if the lines you are posting come from the journal.
Reading Nginx access logs — a common thing to be posting a file of, and how to read it once it lands.
Sending logs — the endpoint reference.

Try it

The free download is the complete product, not a trial. Install it, create a source, and the token in these commands is the one it gives you.

💌 Get notified on new features and updates

Only sent when a new version is released. Nothing else.