Q:

Kafka, commitSync on success

I am trying to figure out if I need to turn on manual commit using KafkaConsumer.commitSync(Map offsets)
When I consume records I need to ensure they get processed by my service, I can't afford to loose them, but if my service throws a bunch of errors or blows up I don't want to move the offset on until I know they have been processed.
Does commitSync seem like the method I need to call?
If I didn't use commitSync and instead I used automatic offset commit, is there potential for me to loose events when a service blows up while trying to process a few Kafka events.  In other words when using auto offset commit at what point are the offsets committed?

A:

If you don't use KafkaConsumer::commitSync and rather chose autocommit mechanism, you will not lose any message, but you can process same message/s several time, ex If you process message and save result somewhere and your application blow up (before saving offset be autocommit), than after restart your application will start processing from last committed offset and it can cause that same message will be reprocessed several time.
The same scenario can also happen if you will use KafkaConsumer::commitSync, but in that case manually managing commit you can commit after processing each message and at most you will reprocess only one message. Drawback of that is lower throughput.

