#celery

how does one change beat time for celery ?

Hi! How can I troubleshoot rate_limit configuration? Currently tasks are executed in 2.5-3 times often than I configured.

Hello! For a long running async job not being executed by a worker and only polled by one what is the best practice to run a polling job in a worker task? obviously keeping the worker busy with a for loop+sleep isn't the way to do it. Would it be to submit a new check task from within the worker with an ETA/Countdown?

e.g. - if not job_finished: some_task.apply_async(..., eta=90) in the worker calling itself recursively with a delay

(and obviously something to expire eventually as well)

ponpanderer the "long running async job" is not a Celery task, correct?

@amcm: correct. basically it's an async call, but to an external rest API. the call could take about 5-10 minutes to complete. i don't want to tie up a worker with a while loop. right now I create the job with an eta=300, then poll every 30 seconds in a loop which is messy imo. i'd rather do a quick rest api call to the target service and if not done exit the task but submit a new one eta=60

or something

this will free up a worker to do other stuff and i have no requirement for exact timing, best effort/eta is fine

(to better explain. the external rest API returns it's own job id and I am polling that job id in the celery task)

I'd have a single task, and raise a Retry Exception when the remote API isn't ready. Then continue to the processing when it is. (or launch a new task if that makes more sense)

More sepecifically http://celery.readthedocs.org/en/latest/usergui... to set a custom countdown

ah this is exactly what the type of builtin functionality i assumed i missed in the docs :)

the custom retry delay is perfect. would it reset an expiry time for the job each retry or if i started with eta=300/expiry=600 and then retry delay of 30 would the job still expire in 600?

or is max retries preferred to handled that?

Hmm... I don't know how expiry plays with retry.. I /think/ the expiry time would /not/ get reset when you retry. But I might opt for the max_retries limit instead

I have an instance of flower and making the api call /api/tasks throws a 500. Trying to determine what's causing it, I have tried various logging options such as --logging=debug --log_to_stderr and --log_file_prefix, but I am unable to get flower to produce any logs at all. Is there something I might be missing?

stracing the process only shows calls to gettimeofday() between reading the http request and writing the 500 response

celery/celery#2803 (master - 5efd77f : Ask Solem): The build was fixed.

Change view : https://github.com/celery/celery/compare/e489f3...

Build details : https://travis-ci.org/celery/celery/builds/8264...

celery/celery#2804 (master - fe33f16 : Ask Solem): The build was fixed.

Change view : https://github.com/celery/celery/compare/5efd77...

Build details : https://travis-ci.org/celery/celery/builds/8264...