#celery

heya

Just getting in to work here in Tokyo

Redis is my broker, I know that might not be well tested in 3.1rc

installed rabiitMQ

still hangs on cold shutdown, after sigterm or sigint

so it's hanging on line 170 of bootsteps.py, join on the shutdown-complete signal... the comment implies i'd only get there if running green, but i'm using default multiprocessinfg

actually it seems that TERM wil close all the workers, and INT will close the main process thereafter

but if you send TERM to main process, it closes subprocs and then hangs on line 170 again

I should mention that this happens only during a long-running process

so not sure if my fix is pull-request worthy, but since the comment says "won't be called unless running green" I did a check on pool_cls.is_green before calling self.blueprint.join from WorkController Blueprint _shutdown function

hi, i am using lates celery+django-celery, i have problems with celery time zone, my server time zone is +3, django timezone setted to this timezone too and works great, but celery uses localtime +3, so all datetimes is future, CELERY_ENABLE_UTC CELERY_TIMEZONE doesn't help

good morning UK! :)

good morning

How you doing?

I'm good thanks, how are you?

alright, werkwerkwerk

are you a dev on celery?

mdan: no

ah

what brings you here then?

i am not either.. yet.. getting closer the deeper in this code i go

we are having issues with gevent leaking memory and mongodb connections. I was just wondering if anyone here has had similar issues? :)

hm.. nope, using redis/rabbitmq and processes instead of gevent so can't help you there

regarding using processes, how lightweight are they?

just starting out. seems pretty lightweight, but our machines have 8 cores

ok

and we plan to consume all the CPU :)

hehe

:)

they mark threading as "minefield" in the docs. does it mean that it's a total minefield or does the minefield tag mean something different?

unsure, you'll have to asksol

are you already using celery in production? what py version?

i've been looking at all the alternatives

including some that are threaded, like https://github.com/coleifer/huey

but I keep coming back to celery....

if you are on more recent version of python concurrent.futures is pretty promising.. there's a backport- but i also like some of the higher-level celery features

and on the theme of lower level, theres http://code.google.com/p/scoop/

i spent a good amount of time playing with rq today, I like it a lot, and it has a compact codebase...

but i don't think it's mature enough for our super demanding application

we are using celery in production and we really like it. the only thing that is wrong is our "async" workers, they use gevent and seem to leak memory and mongodb connections

how long do your tasks tend to be?

not that long, perhaps maximum ~30 seconds

i don't know much about threading ...are you opening and closing your db connections for each task?

i mean, threading in the context of celery. have done tons with threads elsewhere :)

but mostly low level stuff

actually we aren't closing the connections manually. we are not having that issue in the API or in the "non-async" taskworkers. Maybe I should check if that makes any difference

it would be nice to use gevent or eventlet or something similar instead of threading. I'm just wondering what might be causing this issue and if I can find a workaround :)