#celery

the worker doesn't own any results, it updates the state nothing more

asksol, so in a chain, how/where does a result become/trigger the next task? Or does the previous task trigger the next task and the result is independent of that? And the next task has a wrapper to pull the result and pass it as an argument?

I get the feeling that Celery isn't targeting my particular use case. I have complex chains of tasks where each task needs to be sent to a particular worker and then the results get collected. The impression I get is that Celery is really designed for a situation where workers are all relatively generic. Is this accurate? Am I using the wrong tool?

happened again, even though I disabled every cross talk protocol it has

sigh

tcos: first task calls the next and so on, no results involved at all

the workers being similar doesn't matter, it's much easier if they all use the same codebase though

Kaelten: seems unlikely to related to that, it's all passive

to be*

try sending SIGUSR1 to the main process to get a traceback in the logs

tcos: you can enable the environment variable KOMBU_LOG_DEBUG=1 to log how messages are sent

there's a lot of output, so try reducing your problem

asksol kill -USR1 <pid> right?

yeah

no output to console

(you can pip install setproctitle to see type of process in ps listings)

you don't use a logfile?

yeah

huh

see that for the first time in there

I don't think I even have a softtimelimit set

ya not in my config

that's a child process

you sent it a soft time limit

nothign I did does :/

ok found one that's not been restarted

sending that sig to the master doesn't seem to have done anything

use strace/ktrace/dtruss

if the signal doesn't work it's probably stuck at the C level

hrm, never used strace before

not sure how useful that is asksol

does it do stuff, or is it just staying on the same line?

that chunk keeps repeating

fast?

every second or so

the only thing that's not changing is the wait4 line and the epoll_ctl lines

makes me wonder if I should try eventlet or gevent instead of the standard taskpool

what is the output of lsof -p <pid> | grep -v REG

please use pastie.org

is the last lines missing something?

usually would be like localhost:62342->localhost:6379 (ESTABLISHED)

apt01a-live.curse.us:39655->10.104.255.99:amqp (ESTABLISHED)

you have to scroll over inside the paste to see it

ah

looks good to me

not sure why 11 is being unregistered

the other log said you received some ping requests?

do you not receive a reply for those?

yes, from my test earlier

it hasn't received or responded to a ping in hours though

so it stopped receiving them as well

but you get the log?

those are from before it deadlocked

are the control messages still in the queue (sudoi rabbitmqctl list_queues -p your_vhost name messages_ready messages_unacknowledged consumers)

are you using librabbitmq or pyamqp?

librabbitmq iirc

try using pyamqp

rabbitmqctl list_queues -p sloth shows no empty queues

er, shows all empty queues

what about consumers?

?

the last field

consumers has numbers in it

yeah, the number of consumers, are there no unacked messages?

ok seeing this for the host in question

apt01a-live000

and there's no pidbox showing for that host

there is nothing relevant in the rabbit logs?

don't find a reference to the IP address in the log no

seems like rabbitmq thinks the connection is closed already, but the worker does not receive any notification about that

hrm, I'm accessing this over a haproxy

could that be the issue?

sure, if haproxy keep the connection open it will not be able to recover

hrm

easy way to test that

you can set BROKER_HEARTBEAT=30 or so, but then you must use pyamqp://

speaking of pyamqp how do I tell it to use that explicitly

just use pyamqp:// instead of amqp://

do I just use that in the Broker_URL?

yeah

hrm

I restarted all the haproxies

no luck on that one. :/

wonder if the issue is int he hardware load balancer...

use lsof, see if the connection is still open

yup

usually a problem with load balancers, but heartbeats solves that

so there is no broker_heartbeat by default?