#celery

Hello people! I'm looking for way to close result queues in rabbitmq after I fetched the result. Can't find a way to do that through Celery, any idea? thanks

The question was asked on the mailing list but no answer http://thread.gmane.org/gmane.comp.python.amqp.... and asked many million times around the internet

I don't understand why there isn't an answer

Is it a limitation from the way celery interfaces with rabbitmq...?

Hi, I have a moderately sized celery deployment -- 7 different repositories talking to each other over celery + rabbit. one problem I've found is if we don't deploy in just the rightorder, the queues might not be configured correctly, and then messages sent from one server to another might get lost. has anyone found a solution to this?

is there a "standard" way to pool postgres connections between workers?

Hey guys, what are some good resources on pool implementations (prefork, processes, eventlet, etc) and use cases? My email: me@evgg.cc

evg1, source code

Is there a way to delete the finished tasks after a max number has been reached?

Nivix, max number of what?

For example if a worker executes a task every minute, after <MAX_TASK_RESULTS> number has been reached, then every minute thereafter the worker will delete the oldest task result and save the newest result?

I tried setting the app.conf.CELERY_MAX_CACHED_RESULTS to 100, but in the Flower GUI one worker still saves 100+ task results

Nivix, which backend you use?

malinoff, I use RabbitMQ

Each worker has a direct queue

Nivix, amqp or rpc?

malinoff, amqp

Nivix, so they're stored in rabbitmq and CELERY_MAX_CACHED_RESULTS is not related

do you actually retrieve the results?

e.g. do you call res.get() somewhere?

Oh..I thought the CELERY_MAX_CACHED_RESULTS was somehow related to the result storage.

It just caches results on the caller side

I do not call res.get() in my code, thought Flower probably does. (since it shows the result)

though*

no, it definitely shouldn't, because the result won't be discoverable

so it just pops it from the queue without acking

note that i don't really know about flower, i just *think* that it should work as i described

Well, one of my tasks is a GET request to a webpage. The task parses the page and returns the parsed information.

I can see the results in Flower, so I am assuming it retrieves the results.

yes, after that celery puts the message in a result queue

Ah.

flower just inspects the queue

Gotcha. So I guess I should look into RabbitMQ queue length limits?

That result queue is getting very large.

OK, new question. RabbitMQ shows 340 queues (and increasing) but I only have 5 workers running each listening to a single queue. Why are queues being generated?

hi guys

Nivix, a new queue is generated for each task request

anyone here knows how use celery and beat in diff instances?

I'm using heroku...and wanna use one instance per process (celery and beat)

Nivix, this is how amqp backend works, it simply tries to work as result storage. You can switch to rpc backend if you don't need to query results from different processes

I might need to change..generating a new queue for every task will become resource intensive quickly

Nivix: I believe the queues go away. do you need results from your celery tasks, or are you doing fire-and-forget messages?

gchristensen, They are not necessary, though I would prefer to keep them (for debug purposes). How would I setup a "fire-and-forget" system?

Nivix: you can set ignore_result=True