#celery

use pyamqp maybe

That doesnt have timeouts either?

and doesn't use rabbitmq-c

But if it's a network screwup then it will hang in the same way if the network stops?

you can use publisher confirms, though no one else had problems with that on a local network

This is cross AZ on AWS.

Might not meet your regular definition of local :)

I thought you said there were no routers?

None that interfere with tCP

I thought you meant like HA-proxy or something :)

aws definitely interferes, but I geuss it depends on what you use there

Does anyone have any recent experience using SQS with celery?

sqs has been rewritten in kombu/celery master

I'm trying to use celery to publish and subscribe on a queue for image processing jobs. I created a subscriber script which calls worker_main() on the app. It's located in a different repository and completely isolated. Is there a way to call tasks from my publisher using the task name only or do I have to import the same module as used in the subscriber in order to send a message in the queue?

ldiamond: app.send_task(name, args=(), kwargs={}, **options)

but creating dummy tasks is better: @app.task def name(a, b): pass

asksol: I was under the impression the task name had the module name in it.

I'll just create a dummy one then

app.task(name='other_proj.tasks.name') def name(a, b): pass

is app.setup_security(allowed_serializers=[...]) what I should be using to tell celery which serializer to use? (the subscriber supports only json)

app.task(serializer='json', ...)

or set it individually when you call the tasks

Ah, cool

gents

and ladies

gnoze5 :)

good you backtracked on that one

asksol, heheh you caught that...

I was hoping it would go unnoticed...

there's a timestamp, but could've been a very sudden router latency :)

I catch myself saying, or about to write guys often though, no harm done

I feel better

lolol

question

what happens when a worker encounters an unhandled exception?

does it just stop working?

or does it just continue on its merry little way

hi, I noticed there is 1 queue per worker process in rabbitmq

with this name patter celeryev.*

and i noticed that there is quite some traffic on this queues

like 5 pubs per second

strange this is that there is not much going on in celery

run celery -A proj events -d

all shall be clear

are events enabled by default on workers?

i remember a -E option

lot of heartbeat related messages

eg. heartbeat: active=0, clock=525380, freq=2.0, loadavg=[0.05, 0.08, 0.08], local_received=1417120065.56, pid=10397, processed=1994, sw_ident=py-celery, sw_sys=Linux, sw_ver=3.1.12, utcoffset=0

@asksol is that normal to have around 3 pub/s in celeryev queues because of heartbeat? (for workers with -c 2)

worker events are enabled, task events are not enabled by default

a worker sends a heartbeat every 2 seconds

or as specified by the --heartbeat-interval option

ok

just found it weird to see 40 messages/s on rabbitmq

still feels bit weird

it includes the copies generated by broadcast

see the number published in the exchange view I thinj

think

the number received will be higher since it's received by the workers

correct

celeryev is 4/s in 40/s out

and i have 10 worker processes

maybe that’s a silly question but; what’s the hearbeat there for?

clock synchronization and anything else you may want to use the info for

and these are not persistent messages, a small fraction of the total number of messages that can be handled by a worker as long as you don't have hundreds of workers