#celery

hi, i’ve a live system that has been using pickle.

i want to it to start using json, but in such a way that new tasks are serialized with json, old ones are processed but it basically stops to use pickle

Off the top of my head, I'd change the queue for the new json format, start new workers on that queue, and leave the old workers running to finish consuming the pickle queue

Hello. I was asked a particular question about celery, that I can't find an answer. How many tasks can you put inside celery queues?. I'm using redis as the backend. Is this determined by the max amount of memory available on redis?

It sounds like the question on optimizing celery, ❝ How much water flows out of the Mississippi River in a day? ❞

jvazquez yeah, you're limited by the storage available in your broker and how fast you're consuming the tasks.

Are you trying to figure out how many tasks you can hold and not process?

no, they just simply asked, how many tasks can you place inside the queues

So, that would be a relation between how much I can process and the amount of memory that I have on redis

Do they mean, different tasks? Like sum, add, mul ?

Yes, different tasks

So that question is "How many tasks can you register" I don't think there's a real limit to that.

beyond typical limits like your program should fit in RAM

Is there a way to get more performant behavior with multiple workers and eta'd tasks? I think right now I'm not properly saturating the workers with tasks because eta'd tasks ignore prefetch/-Ofair settings.

thanks amcm

anyone familiar with celery.contrib.rdb know if you can / how to use rdb.Pdb.runcall() to step through a specific task that doens't have a breakpoint set?

celery/celery#2939 (master - 4d998d1 : Ask Solem): The build was fixed.

Change view : https://github.com/celery/celery/compare/50185a...

Build details : https://travis-ci.org/celery/celery/builds/8817...

celery/celery#2940 (master - a7d3fcf : Ask Solem): The build was fixed.

Change view : https://github.com/celery/celery/compare/4d998d...

Build details : https://travis-ci.org/celery/celery/builds/8818...

hello all. Is it possible to install celery in the local directory where my project will run? or it needs to be installed either globally or inside a virtualenv?

my project will run both in ubuntu 14 and 12, and I noticed that standard deb versions vary from 2.4 to 3

and from what I saw in the doc examples the celery api is different in both versions. Not sure if celery 3 is backward compatible though

this might be silly... but can I have an app with a task defined as noop that is used to publish messages...

and then have the consuming app have the actual implementation?

jamescarr you can publish an unregistered task by name, don't need to fake it like that

ah thank you

Check the docs, it's in there somewhere

So, I'm trying to configure a backend for the first time. Specifically, ironcache. Well, Celery seems to create the cache but I'm not seeing anything being thrown into the cache.

Now I don't have a good environment set up for my worker to succeed on, but I'm still expecting to see a failture since I'm seeing this log message: Task worker.install[15ca23a3-xxxxxxx52f2c3] succeeded in 0.92937989402s: False

Any idea why I wouldn't be seeing any key/values thrown into the cache?

anyone have a "job reporter" they like for Celery? I may just roll my own, but I really just want to see rate of a given job & progress to finish, not sure if that's easy to see with Flower (we have lots of complex workflows). I've used logentries in the past, and could do basic aggregation, but it had a lot of limitations

felixn: Isn't that a Celery backend? I'm not sure, I'm just starting to look into these now.

fxfitz: not sure! maybe see if it works with a local cache, then go from there

felixn: Yeah, so it seems liek the point of a backend is to keep track of current status of your worker.