#kafka-python

Aha, I remember what it was

We were passing around lists, and had to send_messages(*msgs) them

planning to merge https://github.com/mumrah/kafka-python/pull/213 then tag 0.9.2 and publish to pypi

So what are you thinking for the high level consumer?

A similar thing to the java high level consumer?

start with a similar interface

pass in a list of topics w/ config, get back a list of iterators

either 1 background thread that multiplexes non-blocking connections

or try to manage a bunch of threads, 1 per broker connection

if we dont have to worry about multi-consumer rebalancing, the first approach is probably easier

hmm

The second sounds annoying for not a lot of benefit. Why would multi-consumer rebalancing affect that?

i think getting a good multi-threaded or multi-process solution would require solving the rebalancing problem

but yea we could also solve rebalancing without requiring basic consumer to be multi-threaded

also I looked at those benchmarks that were posted on threaded v. multiprocess performance

wow

0.9.1 and 0.9.2 available on pypi!

Rockin

Nicely done

So what do you think of the thread vs process performance discussion?

seems pretty clear that we need to switch to threads there

Or at least make them available

problems raised: worse performance; blocks higher level multiprocess code

well so his point was wrt producer side

i think i have a PR in the works for that too :)

Duuude, you're a monster lately

I started building a retry mechanism last night (for something related to kafka-python) and ended up adding a lru cache to @memoize()

:-/

ultimately i think the async producer design just needs to be changed from relying on a special "STOP" message, to using a shared Event() instead

yeah

that's how we tackle it here

re thread v. multiprocess; both support a shared Event()

code should be almost identical

have you looked at multiprocess.threadpool?

that'd make the code pretty much identical

will have to investigate

I don't know if it's actually possibel

possible*

It may also disable higher level multiprocessing