#celery

http://pastebin.com/K5vZX433 (Celery 3.1.25) - how to prevent this (see pastebin) from happening? I send task only once and as you can see, it is processed by multiple workers at the same time... This is ridiculous and annoying!

URL Found: -> celery1@ec2-54-234-55-209.compute-1.amazonaws.com: OK * {'hostname': 'ce - Pastebin.com

Dejan: fixed the same issue yesterday

blusky, please enlighten me!!

✓ #2599: Long ETA tasks executed multiple times on Redis broker despite setting visibility_timeout https://github.com/celery/celery/issues/2599

I wonder is this the same issue

maybe not

see, i checked -- all these workers are running the same task

are those delayed task, and are you using redis ?

i did send_task() once

and multiple workers started running it

i use all defaults

no fancy exchanges, etc

if you see the pastebin, the '5bf971b7-c2d2-47a1-9e3e-abec6c3c7ab4' task is running on at least 3 workers

and I triggered it just once

can't help you more than I know :/ I'm kind of a newbie with celery and only know how to copy/paste stackoverflox fixes :-)

lolz

i am no better myself

i'm trying to run a celery-4.0.2 using pickle but i always see the error "ContentDisallowed('Refusing to deserialize untrusted content of type pickle (application/x-python-serialize)',)". i configured celery like app.conf.update(task_serializer='pickle', result_serializer='pickle'). what am i missing?

I think pickle is not supported any more

I could be wrong ofc

Dejan: it's supported, but disabled by default

i remember it being deprecated for some time

but what you said, malinoff, makes more sense

:)

it is supported: http://docs.celeryproject.org/en/latest/usergui...

URL Found: Calling Tasks — Celery 4.0.2 documentation

ah, so you just need to enable it...

I use msgpack

for everything

celery/py-amqp#362 (master - 8aa492d : gord chung): The build was fixed.

Change view : https://github.com/celery/py-amqp/compare/78779...

Build details : https://travis-ci.org/celery/py-amqp/builds/188...

[ celery/py-amqp - amqplib fork | ★ 114 stars | 17 open issues ]

mfilipe: celery loads configuration it a quite complicated way, where exactly you do `app.conf.update`?

malinoff, tasks.py that the worker loads

i saw this workaround but i believe that celery should handle this for me: http://stackoverflow.com/questions/27579736/cel...

URL Found: python - Celery is refusing to deserialize content of my custom serialization throwing ContentDisallowed Exception - Stack Overflow

mfilipe: I believe at that point configuration is finalized and you can't really update it anymore

hum... what do you recommend to configure the serialize settings?

before you call update

anywhere

mfilipe: set it in the same module where Celery instance is defined

like this: app = Celery('tasks', ..., task_serializer='pickle', result_serializer='pickle')?

mfilipe: http://docs.celeryproject.org/en/latest/usergui...

URL Found: Application — Celery 4.0.2 documentation

malinoff, Dejan, it worked! thanks guys! :D

Hi. We are considering to use Celery with SQS as a broker and I've been told that "Celery creates randomly named queues for its interim usage and don't clean up them". Can someone confirm if this is still the case with Celery 4.x or is there a simple way to test this?

I wonder whether someone is working on having Celery in conda package repo...

(yes, I know I can simply pip install it inside a conda environment)

mike_k: use rabbitmq, don't use sqs

malinoff, unfortunately there is a bit of admin overhead to set that up and maintain... If you have experience with SQS to share that would be awesome. Otherwise we might look into using Redis, which has it's own shortcomings.

hi! has anyone tried to run Celery with SQS and JSON as a serialization method? it looks like it is not serializing

celery/kombu#1412 (master - 6998986 : Dmitry Dygalo): The build was fixed.

Change view : https://github.com/celery/kombu/compare/e232299...

Build details : https://travis-ci.org/celery/kombu/builds/18888...

[ celery/kombu - Messaging library for Python. | ★ 912 stars | 81 open issues ]