#celery

gist/bpase

bpaste

https://gist.github.com/saisandeep/3d39eb75aacd... @the_drow

sorry a newbie

I have no idea what Batches is

batches is nothing but a class that denotes that the tasks we are dealing with should be read in batches, i mean if we set the limit as 10, the function will be called after accumulating 100 such tasks and the function takes array of all the tasks arguments and execute on those

@the_drow, i also included in the comment the module location

@the_drow I've also commented the example usage in the gist

I think you can implement this in Canvas

Guest80321, Oh I get what your doing

I never used that feature

You could submit a canvas from that batch task

i'm not sure what a canvas is

Guest80321, Read the docs :)

whats the best way to review the results of a periodic task? particularly I would like to look for something that would be sent back to stdout or stderr on the task

(using this through django)

sry. back at this again. do periodic tasks store results?

v0lksman: not when using the old periodic task decorator, they are ignore_result=True by default

asksol: I went digging for ingore_results rules but couldn't find anything...so if i set that to ignore_results=False it will store them?

I'm pretty sure it will

does that then mean that the task itself will need self passed in? def foo(self): self.results

no, doesn't need to be bound

it can though? that would be handy... :)

asksol: https://dpaste.de/Wo4J

v0lksman: @periodic_task(bind=True, ..)

I don't think there's a `self.results` though

hrm...thought I saw that somewhere...but it errors...will dig

thanks

what do you expect to be in there?

well the problem I'm trying to solve is that I have a task that goes into a state where the task completes without exception but the result can indicate a larger problem in my stack. So I want to check for something in the output of the task and take action accordingly

I can't modify the script as it's part of a 3rd party module so I can only really access things through my task wrapper

my experience so far with Celery has always been a fire and forget background task...so this is the first time I want to try to validate the task execution

so you have a task starting another task, trying to monitor the subtask?

or do you want to check the exception in the task? Wrap it in a try: except block I guess. The final result of the task does not exist until after the task function returns

asksol: no...it's a task running a django management script

so would create a task to look for the results of the periodic_task?

I'm not sure, if the result is there can you not just look at the worker logs?

as in if the result can be sent, the error should've been logged already

yeah what I'm looking for appears in the celeryd.err log I'm logging all task events to

I would maybe have a monitor task to do this in general, but if you're debugging a single issue it's probably better to add some logging or similar

but I was hoping for an easier way to find that error as I have other tasks that run but only this one can cause a problem

management commands could e.g. raise SystemExit

that event should be in the logs

or it could raise an exception that in unpickleable

wrap the task in a try: except: traceback.print_stack()

so your suggesting is a log parser for my celery deamon log?

we have a new log parsing tool in master: celery logtool

but it never raises an exception...well it does when it really fails but I'm looking for this partial failures that don't

if the exception is unpickleable, it could end up being swallowed

it helps if you know the id of the task, so that you can search for it in the logs

I was going to ask if self on the periodic_task would provide that...

so it seems even with bind=True I can't get the task id?