#celery

I'm trying to figure out how to queue tasks in Celery, launched from Flask. I have the task happening via taskname.apply_async(args=args, queue=celery) but despite this it seems that when multiple tasks come in one soon after another, they both start, instead of the second task waiting for the first.

how do i force tasks to wait for the current task to complete first?

oh i see: app.config['CELERYD_CONCURRENCY'] = 1

hi all..

i've got celery tasks that never get out of the PENDING state. they complete with all the results properly

the tasks complete successfully, but yet are in a PENDING state. how do i debug/fix this? t = AsyncResult(task_id); t.status === 'PENDING'

Is it normal for Celery to run the old version of a task's code even after service restart?

If the task has been loaded onto the queue by then that is

With the old version of the code

So a task is registered on the queue with the old code, and then I change the task's code and restart the Celery service. Should it run the older version or the newer one?

Whenever the task is executed that is.

VaticanCameos: are you sure this task isn't executed _before_ you restart the service?

malinoff: The task was registered on the queue last Wednesday with the ETA of this Monday morning. The task's code was changed in the meanwhile and the service restarted twice.

VaticanCameos: stale .pyc files maybe?

malinoff: If that really is the case then there's no way of preventing this issue from happening, short of knowing that this is the cause.

And if we delete these supposedly stale .pyc files and restart Celery, it should use the newer code yes? So it isn't Celery's behaviour that is suspect, it is Python's bytecode.

yes, if the issue is with stale .pyc, it should help

Turns out clearing the pyc didn't help, Celery is still sending emails according to the older code.

malinoff: Clearing the pyc files didn't help, it still seems to be using the older version of the code.

Okay no, that's something else, gimme a few minutes.

Hi! I'm using celery to distribute rather big workloads (in the order of half a million jobs) to complete over several days. However, simply submitting ~500k jobs to the queue at once seems like a bad idea as the broker (rabbitmq in my case) will need to store an excessive amounts of info in the queue. Some sort of rate limiting submissions seems to be

needed, so I wonder what is the recommended practice of doing so? Is this something that needs to be done on the application side, or are there mechanisms in celery to block submissions until the queue size has reached some acceptable size?