#django

maybe if you need captcha

nvm

i have an existing Performance model and as part of a migration process I am creating lots of Records for each Performance. (Record ---fk---> Performance), each Performance will have over 30,000 Records once the data-migration (which is actually a management command) is completed.

my management command loops over all of the Performances in the system, and creates the ~30k records for that performance. the algorithm is getting progressively slower and slower, because i am performing a filter that is taking longer and longer as more records are being added. here is the code:

I tried to simplify the real structure of my models too much for that explanation.. here is another attempt

----

here is the model: http://dpaste.com/2C76QVD, and here is the management command: http://dpaste.com/2J8BPEF

the problem is that the filter on line 3 of the management command is progressively taking longer and longer (taking over 5minutes once i've processed about 1500 of the DataFile objects.)

f.parse() on line 7 of the management command creates ~30k Record objects, so after the 1500th DataFile has been processed there are 45million Records

and i'm pretty sure this part: .filter(track__record=None) is the bottleneck, is there anything i could do to make this faster?