#celery

hi all, is there any known problems with celery when there are lots of jobs piled up in queues

I have over 1M+ jobs piled up

and they seem to be consumed in a slow pace

but I can't see any task in this queue with success or failure

in flower

cagdas: see log from earlier today (or yesterday, depending on where you are). steven and asksol talk about -0fair

ok

here is what I have found

we are using batching with 200

but the value was 4

so it was never working

the question is

now only one worker seems to be busy

amoing 32 others

what should I set my CELERYD_PREFETCH_MULTIPLIER

to make a 200 batch work properly?

cagdas: i'm not i understand your problem correctly, but, the multiplier doesn't look like a problem here to me

*i'm not sure

Have a scenario for a company with 5 facilities and each one with a broker. Does Celery allow routing between brokers?

Patrick99, you can create multiple Celery instances, for each broker. Answering your question: no, using a single Celery instance it is not allowed

The thing is that a client can't directly connect to a broker from another branch

so they would have to submit it to the local queue and then it would have to routed from a specific worker to the other broker

Patrick99, i think you should not think about celery possibilities on that level, but about broker availabilities. For example, rabbitmq can do that: https://www.rabbitmq.com/distributed.html

i know was just attached to the ease of deployment of Redis as it can just run as another service on one of our servers

so i was hopeing to make it work with Redis

seems like for rabbitmq i would need another virtual machine per branch

Thanks for your help malinoff. Opened a ticket on Github. Maybe somebody else ran into a similar issue already https://github.com/celery/celery/issues/2298

On another note has anybody tried out ZeroMQ with Celery? The code is in there but there are no docs