#celery

I'm assuming it is possible to define a shared task in a separate python file and import that into multiple workflows?

and use the @shared_task decorator

ah yeah it looks like it

sweet action

badloop: Awesome :)

josuebc: yeah celery is checking all the boxes....loving it so far

What is the purpose of GroupResult.completed_count()? The Canvas API guide says, "Return the number of completed subtasks.", but in a contrived simple example, that does not bear out: https://gist.github.com/Quidge/69055b4276bd3180...

lirum: Do you see any errors on the worker? Do you have a result backend setup?

I'm using RPC and no errors on the worker. Both add tasks in that group complete successfully.

I appended some lines onto that Gist showing that as soon as I call r.get(), everything changes and goes to what I should expect. But ...I shouldn't have to call .get().

I believe this is something specific to RPC backend. I just tested the same code with redis as backend and rpc and I can see the difference.

Thank you! Relieving to know I wasn't misunderstanding the docs. I'll look into getting redis.

lirum: I believe the issue is that with RPC the status is not updated unless you do a `.get()`. There's an issue for it but we haven't been able to get to it: https://github.com/celery/celery/issues/4084

@josuebc New to message brokers in general. As I understand it, RPC is a more limited backend capable of passing messages and meta information about those messages ('SUCCESS', 'FAILURE', etc), but is not designed to be a store for task results themselves. Do I have that correctly?

lirum: Yes, it is focused on passing results as messages that way you can implement things with a more event driven approach. One of the limitations right now is that you'll have to do `.get()` in order to introspect a result

I'm using RabbitMQ as my broker. In my celery instance, if I switch `backend='rpc://' to `backend='amqp://`, what happens? The stuff I'm doing seems to work the same. Does this mean that celery runs detection on my system for the running RabbitMQ broker and connects to that?

lirum: The difference is the internal class used. If you select rpc as the protocol celery will use this https://github.com/celery/celery/blob/master/ce... to handle the backend

If you use amqp then it will use this: https://github.com/celery/celery/blob/master/ce...

The only thing is that the amqp result backend is being deprecated. So, there might be other bugs in it

josuebc: Got it. Thanks!

lirum: Np. Btw, solving that issue is on the 4.3 milestone. So, keep an eye on it. We'll hopefully fix it soon

hi all, celery 4.1.1 here :-) Trying to run periodic task through rabbitmq. The task executes well however, task routing is completely ignored :-/ queues are set with celery -Q option. task_routes defined through app.conf.update and the function to execute is decorated with periodic_task() and called with .s().. any hint?

azerty1664: How are you routing your tasks? Are setting a queue on the signature or something like that?

josuebc, lirum: I don't think that's 100% correct. I think you can call `.state` to get the current status without getting the result.

But the RPC backend definitely does some weird things since AMQP isn't really a place to just store information.

hi josuebc, i've posted a clean version of the actual code here: https://pastebin.com/BZfCTjbe

clokep: The thing is RPC. It works using redis or db backend but with RPC the value of state is 'PENDING' until you do `result.get()`. Hopefully it will get fixed for v4.3

josuebc: I disagree.

I'm using that in some code that never calls get().

clokep: Ah, that's interesting. And you are getting the correct valu in state?

Yes.

Which celery version?

It eventually proceeds from PENDING to SUCCESS or FAILURe.

4.1.0, but this code was also used with 4.0.x.

clokep: Are you checking for the result's state on the same producer loop or are you getting the result like `res = AsyncResult(uuid)` ?

Calling state ends up doing message-y things on the network if the result isn't available: https://github.com/celery/celery/blob/6fd0d3543... calls https://github.com/celery/celery/blob/6fd0d3543... which eventually calls into the RPC backend: https://github.com/celery/celery/blob/5af199c94...

josuebc: Same producer loop.

So the producer is listening for all results on tha queue, yes.

Although I'm surprised you can even try to have multiple things getting results from (the same queue) while using the RPC backend, that seems like a recipe for disaster. :)

clokep: Yes, well the idea is to only allow the producer to retreive the result. But I believe you can specify a `reply_to` string and allow som other listener to retrieve the result.

Yep, you can do that sort of thing.

clokep: Are you using only single tasks or also chains and/or groups ?

Single tasks.

Never really got the hang of chains and/or groups.

The design is overly complicated IMO.

And I don't need themf or my application.

clokep: Ah, I think that's where the issue lies. It works for single tasks but not for groups.

Ah, I see.

clokep: Yeap, I try to avoid them as much as possible. Simple is better than complex

Yeah. I could replace some custom code I have with them, but never got around to it. Oh well. :)

Maybe once celery 5 is out and the code-base is readable.

clokep: Hahaha, yeah it can get tricky following the codebase