#celery

similarly to what chain does but I really do not need to pass the result to the next one

Dejan: i do not know, i use celer from times group was not there

so i just use tasks :)

group imho is not ordering tasks. just returns all results at once

really, give up ordering, if possible

that is why i am asking if there is a way to execute tasks in a group or something sequentially

the only way to have something similar is with chain...

i do call tasks from tasks -> but i do not wait for results there

OK, let me explain what exactly I am doing...

I have a very restricted FTP server that only allows two connections in parallel

and I need to download 12 huge files from there

I can't run download all at once

So, I can run only two downloads in parallel

and i need to wait for one to finish in order to do the next one

i hope you do that many times around, or else i would say wget is better idea :)

this download is a part of a large system that later parses the files, produces HDF5 files, parquet files, as well as some CSV files...

and it is fairly frequent

i see now

frankly, this crap about not calling .get() is just wrong

dejan: or you can download those files in one task only?

that takes too long

that is how it was before :)

Dejan: if you will have 10 workers, and all 10 will get a task, where they will .get() -- they all will block...

and no work will be done

or at least provide us with a simple pipeline that is executed sequentially

hmm i never used it like this.

it is not 10 workers that do this

it is a single task

that calls get

that task is executed in a single worker...

nevermind :) thanks for trying to help

i appreaciate it

single tasks -- but you can get more of them in the queue

nope

Dejan: maybe someone other can help here, i never really used http://docs.celeryproject.org/en/latest/usergui...

they never overlap

Dejan: how do you know? i was like you, until all my workers stopped

and i was frantically running around, trying to figure out what happened

i know because I know how i schedule them man...

let's just stop it here please...

:)

Dejan: best way is imho this

make a task which will take list of files to download.

downloads first one and runs antoher task with shorter list...

but make SURE the task will not just fail

(and not continue the work)

do not WAIT for result

would that work for you?

Dejan: late ack could also help. but make sure you will not run tasks 5 times...

or use chain...

but i would rather use just tasks

if you want to download 2 concurrently, when you are starting the download, run 2 tasks, giving each one half of the list

Pfffff... Can I give a group a callback to be called when all tasks in the group are finished?

(either successfully or not)

Dejan: does immutable=True help?

(just skimming scrollback but immutable=True will stop results being passed down a chain)

lllama, i haven't tried this to be honest

i will

thanks