#apache-kafka

That can be managed with quotas

but that was client level issues

i had to lower max fetch bytes

yeah, if you say "give it all to me" you can have a bad time.

why doesnt the protocol set a max # of messages?

The Akka client was notoriously bada about that.

or why doesnt it allow i mean

the fetch protcol only has max bytes

<--- not a Kafka dev

You can also specify max batch size on fetch

is this new to 2.0.0?

as ive recently read protocol docs and couldnt find that

unless your referencing size in bytes

im asking about # of messages

because we have tons of small messages, so my issue was i can easily pause when i have work in queue, but that initial request was returning so much when lag gets high

Hmm, I'm sure I found something a while back where you could configure the max number of messages to receive...

maybe your client can do that and it buffers before sending you the next

Can't find it now

This is just the Java client, so the docs are just https://kafka.apache.org/documentation/ Can't find it now

Anyway, home

how do you handle errors in consumer? keep retrying?

saml: depends on the error, thats going to be a personal case solution

wouldn't crashing and restarting be fine? what other ways to handle are there?

i guess it might keep restarting on a bad log record

redriving

(i.e. writing the "bad" record to a topic where it can be examined later

)

If you use Akka and choose the crash solution, every crash will result in losing a bunch of events by default, since the default behavior of Akka is to commit optimistically

geekatcmu, then you're processing log out of sync

yep

collecting few log records and replying them later is not feasible in my usecase

but it directly relates to the question you asked, and expand's on Aikar's answer.

99% of the time, the only correct answer is "it depends"