#celery

I'm having some problems with Celery refusing to process tasks. Take a look: http://pastie.org/pastes/9278664/text

This is with this file: https://github.com/synthead/pidrator/blob/testi...

Am I doing something wrong, or is this a bug?

Can I use celery inspect to debug this somehow?

tasks that depend on each other being processed by the same queue?

dpn`: what do you mean by depend on each other?

dpn`: I start chains in async using delay, if that's what you mean

Synthead, for example, WateringCycle starts actuate relay on a countdown

dpn`: is that bad?

so WateringCycle can not finish until actuate relay finishes

but if the queue is full of tasks that are waiting for other tasks to finish, actuate relay will never run

and wateringcycle will never complete

causing a deadlock in your queue

I'm not 100% sure on this - as I've just looked at the code and logs

but I've definitely experienced this before

dpn`: can WateringCycle (or Celery tasks in general, for that matter) handle more than one task at a time?

i would have two queues - one that only runs watering cycle tasks and another that runs everything else

yes - the other alternative is to up the concurrency level on the celery pool you have

though you may reach the ceiling there when you add more sensors to your system or something

dpn`: in this project, there will be 8 sensors and 8 relays

dpn`: would it be unreasonable to bump up the concurrency level for this?

Synthead, i think that'd be fine - it's more your memory usage you want to be careful about

since these tasks aren't particularly CPU bound

dpn`: yeah, the most intensive thing they'll be doing is using django models

FWIW it's not uncommon to have multiple celery pools

at work we have like 6 of them

dpn`: how would that work?

or literally run the celery worker command twice with the options/names you need

then in your project config you can route certain tasks to certain workers

dpn`: oh, I see

dpn`: can the concurrency level be dynamic (jobs start as needed)?

Synthead, yes there is an autoscaler option - I've had horrible experiences with it though

might as well try it

dpn`: I added CELERYD_CONCURRENCY = 20 to my settings.py as an experiment and Celery still stops processing tasks

dpn`: if you were to change the program flow a bit so that settings wouldn't have to be changed, how would you write it?

to isolate the problem test, I wouldn't run tasks in chain as subtasks

just run them directly from that task.. ie no delay/apply_async

and see if it still happens

dpn`: so instead of ActuateRelay.delay(), run ActuateRelay()?

yeah

dpn`: what about the tasks with the countdown keyword?

i suspect they are the main offenders in this case

so definitely

dpn`: would chain(a.si(), b.si()).apply() (not .apply_async()) be ok for this test?

I'm seeing a lot of IndexError exceptions :/ http://pastie.org/pastes/9281796/text

could someone help me with this? https://gist.github.com/anonymous/e41dad636d261...

the task seems like it keeps replacing my last variable with the one that's before it

the merge_snapshots task should take 3 arguments, but it's only using 2 for some reason, and the command it is executing is only using the first two given arguments, but using the second argument in place of the third

if that makes sense

my brain hurts from trying to figure out wtf is going on

the command that the subprocess calls works perfectly fine on the command line and via interpreter

celery is going to drive me crazy, man

I've spent so many late nights fidgeting with it :/

hello

Synthead, I said move them out of chain

don't make tasks depend on each other

then work up from there

is there a known issue with celery and django 1.6s persistent database connections (CONN_MAX_AGE)?

when set celerybeat tasks are not executed

no tasks are executed

what is the current recommended way of reusing database connections with django/celery?