#celery

ffffffffffffffff: you just need to specify the correct app...

44UAAAAHL: what signal?

[travis-ci] celery/celery#1975 (master - 2d7904a : Ask Solem): The build was fixed.

[travis-ci] Change view : https://github.com/celery/celery/compare/b9ee0f...

[travis-ci] Build details : http://travis-ci.org/celery/celery/builds/26314143

asksol_ rabbitMQ clients may not be able to reconnect, but qpid clients can

that is going to be a sweet feature for the qpid transport

so it will redeclare the queue if the connection is lost?

either that, or queues that start with the celery event queue prefix will force durable=True within Channel.basic_consume

reconnect works perfectly with that set

the transport shouldn't treat queues specially based on name, it should be decoupled from the application

and how about QoS limits? does it re-enable them?

actually the QoS limits need some work anyway, so I'll have to give that some thought

kombu expects the transport to propagate errors so that the caller can reconnect

it's not only the events queue, it's also publishing to any non-durable queue

that's why we have Producer.publish(..., rertry=True, declare=entitites)

where is that call in the code? I'd like to read around it

from what you are telling me, using retry=True and declare=entities would provide auto-reconnect functionality

the transport doesn't reconnect, the Producer.publish operation does

and the user may have to do so manually if different steps are required

so that may handle publishes that would fail because of a failed connection

but what about consumers

for instance if I start a single celery worker and connect it to rabbitmq

then I stop rabbitmq

I get this traceback

I expected that the error would propogate through kombu to celery and celery would wait and retry

the worker restarts the Consumer blueprint when it sees Connection.recoverable_connection_errors

if the error is not in there, it will crash

why for rabbitMQ wouldn't the error experienced when rabbitMQ shuts down, but celery doesn't not be in recoverable_connection_errors?

basically, if the broker shuts down, celery should just wait

maybe not as default, but at least as an optional feature

that's exactly what it does, it will continue retrying forever

so then why does my worker exit and give the stack trace above when rabbitMQ shuts down?

with rabbitmq

I would guess SystemError is not a recoverable connection error, and that you have an old version of librabbitmq perhaps

librabbitmq==1.5.0

so you're saying this is not the expected behavior

it's the expected behavior if a SystemError is raised

oh wait, there is more to the stack trace

here is the full one

for unexpected errors it's better to crash so that operators can be notified

that is not always true, in our case we want celery workers to just keep on going if the broker bounces

monitoring is another component's responsibility, watching celery run should not be the monitoring point of broker availability

right, but you need to map out the errors

agreed, "all errors" should not be treated this way

I'll have to map them for qpid.messaging as a client

I'm saying unexpected errors, a broker connection error is not unexpected

oh ok

well then we agree perfectly I think

try upgrading to the latest librabbitmq, it's probably fixed

so back to this stack trace with rabbitMQ, Celery observes ""Connection to broker lost Trying to re-establish" and then another ConnectionErorr is raised

ok upgrading now...

it would never have been able to recover from this error, so in that case it would just never recover even if the broker does

right because it's an acutual correctness problem you are saying

with librabbitmq

it's a programming error in librabbitmq, like a SyntaxError they are not treated as a connection error

right

got it

unless the transport adds SyntaxError to the list of recoverable connection errors :)

so how can I have my Qpid transport for kombu indicate the recoverable error set?

hahaha

no no, we're not doing that :-)

before it was Transport.connection_errors and Transport.channel_errors

both would cause the connection to be reestablished

are those still respected?

you say "before..."

now it's recoverable_connection_errrors and recoverable_channel_errors

oh ok

they are used if the new ones are not defined

are these used by kombu or celery?

it sounds like celery

because celery is decided if it needs to re-enter or not

recoverable_channel_errors will close and recreate the channel, recoverable_connection_errors will reestablish the connection

if you only have connections you can just define the latter

it's used by both kombu and celery, usually you will not use them directly but use the Connection.ensure and Connection.autoretry wrappers

and some of the operations support the retry argument (like Producer.publish)

def b(channel): consumer = Consumer(channel, [queue]); consumer.qos(prefetch_count=10); ...,

so upgrading to librabbitmq==1.5.2 caused the behavior with rabbitMQ to be really reliable

Connection.autoretry(b)()

autoretry for functions taking a channel argument, ensure for objects having a channel attribute

asksol_ cool

so you are defining transaction points kinda

is there anywhere in the celery docs that talks about celery's reconnect bahavior for users?

reminds me how cool would it be if the with statement could reexecute the block

we could make a modified ContextManager

that supported re-entry

only http://docs.celeryproject.org/en/latest/usergui...

kombu docs have been on my todo for years

could you show me where you would want this?

I don't think there is any way to re-execute the with statement block

unless you do some AST hacking maybe

there should be a error recover section in the kombu userguide, but not yet written

I may writ eit

write it

an error recovery section

that would be awesome!

the retry policy described in the celery calling guide is the same in kombu

there are also some best practices that are not described anywhere

like always passing connections instead of channels

e.g. x = Producer(connection.channel()); x.publish(retry=True) won't work unless the connection is already established

x = Producer(connection).publish(retry=True) will work and uses the connections default channel, since producer treats the channel lazily

brb, must get food

asksol_ I will try to include those things in my first pass at this