Your JSON is fine. Your batch size isn't.

Three teams push the same 50,000 events a second through Kafka. One of them pays roughly $4,355 a month for the privilege. Another pays $762. Same events, same fields, same business. The only thing that changed is how the bytes were packed before they hit the socket.

ntail meters compressed bytes on the wire — for data in, data out and storage — so packing isn't an aesthetic choice here, it's a line on the invoice. We went looking for the biggest lever. We assumed it was the serialization format: rip out JSON, put in Protobuf, watch a third of the bill fall off.

It isn't. We measured it on 50,000 real events, and the format turned out to be close to a rounding error — in one case it moved the bill in the wrong direction. Here's what actually matters, and the bit that genuinely surprised us.

One event, four ways

We needed a real message rather than a tidy invented one, so we took a GH Archive hour — every public GitHub event from 15:00 UTC on 15 January 2024 — and pulled out the PushEvents. There are 169,030 of them in that hour, 63.3% of all traffic. This one is 1,403 bytes:

{
  "id": "34834890257",
  "type": "PushEvent",
  "actor": {
    "id": 41898282,
    "login": "github-actions[bot]",
    "display_login": "github-actions",
    "gravatar_id": "",
    "url": "https://api.github.com/users/github-actions[bot]",
    "avatar_url": "https://avatars.githubusercontent.com/u/41898282?"
  },
  "repo": {
    "id": 421419351,
    "name": "jobara/platform",
    "url": "https://api.github.com/repos/jobara/platform"
  },
  "payload": {
    "repository_id": 421419351,
    "push_id": 16645589756,
    "size": 2,
    "distinct_size": 1,
    "ref": "refs/heads/release-please--branches--dev--components--accessibility-exchange/platform",
    "head": "bcdc58f9cfa9b4565219199e03184a517080676e",
    "before": "90afb51a127bf546fc3ba5050e722b78aacb742c",
    "commits": [
      {
        "sha": "4fa21232eecbf54f5f027ee65c3c61eebc6341a7",
        "author": {
          "email": "jobara@users.noreply.github.com",
          "name": "Justin Obara"
        },
        "message": "fix: dashboard breaks with Invitation notification and site is using a signed language locale (resolves #2054) (#2100)",
        "distinct": false,
        "url": "https://api.github.com/repos/jobara/platform/commits/4fa21232eecbf54f5f027ee65c3c61eebc6341a7"
      },
      {
        "sha": "bcdc58f9cfa9b4565219199e03184a517080676e",
        "author": {
          "email": "41898282+github-actions[bot]@users.noreply.github.com",
          "name": "github-actions[bot]"
        },
        "message": "chore(release): release 2.0.0",
        "distinct": true,
        "url": "https://api.github.com/repos/jobara/platform/commits/bcdc58f9cfa9b4565219199e03184a517080676e"
      }
    ]
  },
  "public": true,
  "created_at": "2024-01-15T15:00:45Z"
}

It's a good specimen: nested objects, an array of objects inside them, a couple of long URLs, and the kind of field names a real API actually ships. Here is the thing to notice, because the whole post turns on it — the strings "sha", "author", "email", "message", "distinct" and "url" appear once per commit. Add a third commit and you pay for all six field names again. JSON is self-describing, and it re-describes itself on every single record, forever.

CBOR is the obvious first fix: binary JSON. Same data model, same self-describing structure, but the syntax — braces, quotes, colons, commas — collapses into type tags. (MessagePack does the same job and lands within a few bytes; treat them as interchangeable for this discussion.) Our event drops to 1,255 bytes, about 10% off. Look at the first bytes and you can see why it isn't more:

a7 62 69 64 6b 33 34 38 33 34 38 39 30 32 35 37 64 74 79 70 65 69 50 75 73 68 45 76 65 6e 74
 .  b  i  d  k  3  4  8  3  4  8  9  0  2  5  7  d  t  y  p  e  i  P  u  s  h  E  v  e  n  t

There they are: id, type. CBOR made the punctuation cheaper. It did nothing about the key names, because it can't — without a schema, the keys are the data.

Protobuf takes the other road. You declare the shape up front and the field names never travel at all; a number does the job instead.

message Commit {
  string sha      = 1;
  Author author   = 2;
  string message  = 3;
  bool   distinct = 4;
  string url      = 5;
}

Those numbers are the contract — rename sha to commit_sha and every existing consumer carries on working, because it was only ever reading field 1. Our event drops to 993 bytes, 29% off, and the wire goes properly opaque:

08 91 fc c8 e2 81 01 12 09 50 75 73 68 45 76 65 6e 74 1a 8f 01 08 aa a2 fd 13 12 13 67 69 74
 .  .  .  .  .  .  .  .  .  P  u  s  h  E  v  e  n  t  .  .  .  .  .  .  .  .  .  .  g  i  t

Avro goes furthest. The schema travels separately, fields are positional, and even the field numbers disappear: 956 bytes, 32% off. It is also the format most Kafka shops already have a registry for.

So far the story is the one everybody tells. Schemas are smaller. JSON is fat. Four formats, sorted by size, exactly as expected:

Ship Protobuf, save a third. Except that none of those numbers is what you get billed for.

Kafka doesn't compress your messages

It compresses batches. This is the single most under-appreciated fact about Kafka's pricing behaviour, so it's worth being precise. A producer accumulates records into a RecordBatch, which looks like this on the wire:

+-- RecordBatch ---------------------------+
| baseOffset, length, crc, attributes,     |    61 bytes, never compressed
| timestamps, producerId, recordCount...   |    (the broker reads these)
+------------------------------------------+
| record 1 | record 2 | ... | record N     |    compressed as ONE block
+------------------------------------------+

The 61-byte header stays in the clear so a broker can read offsets and timestamps without unpacking anything. Everything after it — all N records — is handed to the codec as a single buffer.

Which means the compressor doesn't see your message. It sees a hundred of your messages, stacked. And a hundred stacked JSON records contain the string "avatar_url" a hundred times, which is the most compressible thing imaginable. How many records end up in that buffer is set by batch.size and linger.ms, and almost nobody touches them.

The result we didn't expect

We encoded all 50,000 events in each format, framed them into batches exactly as a Kafka producer does — zigzag varints, null keys, per-record headers, the 61-byte batch header on top — and compressed each batch with each of the five codecs compression.type accepts. Wire bytes per event, at a batch depth of 100:

Read the zstd column again. JSON, at 213.0 bytes per event, is smaller than Protobuf at 223.7.

The format that was 50% fatter uncompressed comes out 5% leaner on the wire. That isn't a fluke, it's the whole mechanism: JSON compressed 5.44×, Protobuf only 3.46×. Protobuf's saving comes from deleting redundancy — the field names — and redundancy is precisely what a compressor deletes for free. Having done that work by hand, Protobuf hands zstd a dense, high-entropy buffer with very little left to squeeze. JSON's bloat is pure repetition, and repetition is what these algorithms were built to annihilate.

Put uncharitably: a team that migrated from JSON to Protobuf to save bandwidth spent a quarter of engineering doing, by hand and badly, a job zstd was already doing for nothing.

The lever that actually moves the bill

If the format is worth a few percent, what's worth real money? The same 50,000 events, zstd throughout, varying only how many records land in a batch:

JSON goes from 607.6 bytes per event to 189.0. That's 3.2×, from one setting, with no code change and no schema registry. The entire spread between the best and worst format at depth 100 is 8%.

Notice too that at a batch depth of one, the expected order returns — Avro's 450.7 does beat JSON's 607.6, because with a single record there's no cross-record repetition to exploit and you're back to arguing about field names. Every format-comparison benchmark that measures one message at a time is measuring the case Kafka almost never puts you in.

"Sure, but your events are all identical"

Fair challenge, and the obvious one. Fifty thousand consecutive PushEvents from a single hour are about the friendliest corpus a compressor could ask for — same schema, and consecutive events may well come from the same repositories. So we re-ran it on two harder corpora: the same PushEvents sampled across the whole hour and shuffled, which destroys any burst clustering; and a genuine firehose of all nine event types shuffled together, where a 400-byte WatchEvent sits next to a 30 KB PullRequestEvent.

Shuffling costs about 8% at depth 100 — real, but nothing like the effect we're claiming. And the mixed firehose compresses better, not worse: its bigger, more deeply nested payloads carry even more structural repetition. The batch-depth effect survives everywhere, and the honest caveat is narrow: if your topic carries genuinely high-entropy payloads — already-compressed images, encrypted blobs, random identifiers and little else — none of this applies to you, and you should be metering rather than trusting anyone's blog post, ours included.

While you're in there: use zstd

The codec column has one unambiguous answer. Against gzip, zstd produced smaller batches (213.0 vs 224.3 bytes per event) and was faster in both directions — 184 MB/s vs 117 MB/s compressing, and 2,537 MB/s vs 540 MB/s decompressing. There is no trade-off to weigh. If you are on gzip, you are paying more for less.

snappy and lz4 are the genuine trade: roughly 7× faster to compress than zstd, at 44% more bytes. That's a real choice if your producers are CPU-bound. Turning zstd up isn't: level 11 bought 3% over the default level 3 for a large multiple of the CPU. Leave it alone.

Where the format does matter: CPU

None of this makes serialization format irrelevant. It relocates the argument. Time to encode and decode one event:

JSON decodes 13× slower than Avro. And decoding is the asymmetric cost: a message is encoded once and decoded by every consumer group that reads it. At a 2× fan-out you pay it twice; at ten consumer groups, ten times. That is the real bill for JSON, and it lands on your compute, not your streaming invoice.

So the rule of thumb we'd actually defend: batching is a bill decision, format is a CPU decision. They're separate arguments and they deserve separate meetings.

Choosing a format for the other reasons

Schema evolution is where the schema'd formats earn their keep, and it has nothing to do with bytes. Protobuf's field numbers mean a rename is free and a deletion is a decision you make once, deliberately, by retiring a number forever. Avro's writer/reader resolution lets an old consumer read new data and a new consumer read old data, with defaults filling the gaps. JSON's answer to "we added a field" is that everyone finds out in production.

Against that: you can read JSON off a topic with your eyes at three in the morning. That is worth more than most architecture documents admit, and it is the single best argument for keeping it.

What this means for your bill

ntail meters compressed bytes on the wire — data in, data out and storage, all three. So a change in packing lands three times: on what you write, on what every consumer reads, and on what sits in retention. Taking the measured numbers above at 50,000 events a second with a 2× consumer fan-out:

The gap between the top row and the highlighted one is 5.7×, and crossing it required changing two producer settings. The gap between JSON and Protobuf on the same settings is $38 a month, in Protobuf's favour on CPU and against it on bytes.

A European company, on our own European infrastructure

ntail is operated by Factual Tech AB, a company registered in Sweden. Customer data is processed and stored in the EU by default — other regions, including the US, available on request — on infrastructure we operate ourselves. For teams with stricter residency or sovereignty requirements, we can also run a dedicated cluster fully off the public cloud, in a specific region of your choice. See how we handle security for the full detail.

So what should you actually do

In order of how much they're worth, on our numbers:

1. Check your batch depth before anything else. If linger.ms is 0 and batch.size is the 16 KB default while your records are ~1 KB, you're leaving the largest single saving on the table. Raising it trades a little delivery delay for a lot of bytes — a real trade-off, and yours to make.
2. Move off gzip to zstd. Smaller and faster, both directions. There's no downside to weigh.
3. Leave the compression level alone. The default is where the curve flattens.
4. Pick your format on decode cost, schema evolution and debuggability — because that is what it actually decides. If you're moving to Protobuf or Avro, move for the contract and the CPU, and treat the bytes as a rounding error.
5. Then measure your own data. Ours is one hour of GitHub events. Yours isn't.

The harness that produced every number here is in our repo, along with the pinned dataset hash, so you can point it at your own events and argue with us. If you'd rather we did it, send us a sample — or go and model your workload against the rate card, where the sliders are already denominated in the compressed bytes we bill on.