Apache Kafka: Denial of service via unbounded memory allocation when decoding compressed record batches

Summary

When a broker decodes a compressed record batch, the per-record decoder reads a declared body size from the decompressed stream and allocates a buffer of that size before validating it against the bytes available or any configured limit. The size is attacker-controlled, and the batch CRC does not constrain it, so a small (around 90-byte), CRC-valid compressed produce request can declare a body size of up to about 2 GiB and force the broker to attempt that allocation, raising OutOfMemoryError. The record size limit (max.message.bytes), the request size limit, and client/broker quotas bound only the compressed wire size, not the declared decompressed size.

Impact:

An authenticated principal with Write on a topic can crash broker request-handler threads (reached when a batch's codec differs from the topic's compression.type and records are re-encoded) and any consumer that fetches the record, denying service. On brokers started with -XX:+ExitOnOutOfMemoryError, a common hardening, a single crafted request terminates the broker JVM. More seriously, a crafted record  is accepted at ingest and written to disk; when the log-cleaner later reads it on a compacted topic, the OutOfMemoryError (an Error, not an Exception) terminates the cleaner thread, which is not restarted. With the default of a single cleaner thread, log compaction then stops for the entire broker, including the internal __consumer_offsets and __transaction_state topics, and the condition persists across restarts until the offending record is removed. The impact is availability only; no data is disclosed or altered.

Mitigation:

Upgrade to Apache Kafka 4.2.2, 4.3.2, or 4.4.0, and set max.decompressed.message.bytes (at the broker and/or topic level) to a bounded value appropriate for your workload. Upgrading alone is not sufficient: for backward compatibility this configuration defaults to an effectively unlimited value (about 2 GiB), so a broker remains exposed until the limit is set. Choose a value at or slightly above the largest legitimate decompressed record your applications produce, and comfortably below the broker heap divided by the number of request-handler and cleaner threads. Until you can upgrade and configure the limit, restrict Write access to trusted producers; quotas and max.message.bytes do not mitigate this issue.

Resolution:

From Apache Kafka 4.2.2, 4.3.2, and 4.4.0: the broker validates a record's declared decompressed body size against max.decompressed.message.bytes before allocating, on the produce-validation, log-compaction, and timestamp-lookup paths, and rejects oversized or negative sizes with InvalidRecordException. Operators must set this configuration to a bounded value to be protected, as its default is effectively unlimited.