Understanding Kafka’s Cleanup Policies: Delete, Compact, and Combination 🧹🔄

Apache Kafka is a powerful distributed streaming platform that handles trillions of events a day. A crucial aspect of managing Kafka topics is understanding the cleanup policies: Delete, Compact, and how to combine them. Let’s dive in!

The Delete policy is the default cleanup policy in Kafka. It dictates that old log segments are discarded when they reach their retention limit, either by time or size. This policy is straightforward and ensures that the disk space is not overwhelmed by old data.

𝐓𝐡𝐞 𝐂𝐨𝐦𝐩𝐚𝐜𝐭 𝐏𝐨𝐥𝐢𝐜𝐲 📦

The Compact policy, on the other hand, retains only the latest value for each key within a log segment. This means that even if a key has been updated multiple times, only the most recent value is kept. It’s ideal for scenarios where the complete history of updates for a key isn’t necessary.

𝐂𝐨𝐦𝐛𝐢𝐧𝐢𝐧𝐠 𝐃𝐞𝐥𝐞𝐭𝐞 𝐚𝐧𝐝 𝐂𝐨𝐦𝐩𝐚𝐜𝐭 🤝

Kafka allows you to set both Delete and Compact policies together. When combined, the log segments are first compacted, and then the delete policy is applied based on retention settings. This hybrid approach offers the benefits of both policies.

𝐃𝐞𝐟𝐚𝐮𝐥𝐭 𝐏𝐨𝐥𝐢𝐜𝐲 𝐚𝐧𝐝 𝐇𝐨𝐰 𝐭𝐨 𝐂𝐡𝐚𝐧𝐠𝐞 𝐈𝐭 🔧

As mentioned, the default policy is Delete. However, you can change the cleanup policy for a topic when creating it with the kafka-topics tool using the --config option, or alter it later with the kafka-configs tool.

Key Points to Remember 🔑

𝐃𝐞𝐟𝐚𝐮𝐥𝐭 𝐏𝐨𝐥𝐢𝐜𝐲: 𝐃𝐞𝐥𝐞𝐭𝐞 🗑️

Compact Policy: Retains the latest message per key 📦

Combination: Offers benefits of compaction and controlled retention 🤝

Changing Policy: Use Kafka’s topic and config tools 🔧

Remember, choosing the right cleanup policy is essential for efficient Kafka management and ensuring data integrity. 🚀

𝐃𝐨𝐜𝐮𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧: 𝐜𝐥𝐞𝐚𝐧𝐮𝐩.𝐩𝐨𝐥𝐢𝐜𝐲 🥼

This config designates the retention policy to use on log segments. The “delete” policy (which is the default) will discard old segments when their retention time or size limit has been reached. The “compact” policy will enable log compaction, which retains the latest value for each key. It is also possible to specify both policies in a comma-separated list (e.g. “delete,compact”). In this case, old segments will be discarded per the retention time and size configuration, while retained segments will be compacted.

Type:list

Default:delete

Valid Values:[compact, delete]

Server Default Property:log.cleanup.policy

Importance:medium

#kafka #cleanup #policy #delete #compact

Doc:https://docs.confluent.io/platform/current/installation/configuration/topic-configs.html