#scrapy

has anyone used Splash and knows how to pass headers?

response = requests.get(splash_url, params={'url': url, 'timeout': timeout, 'wait': 0.5, 'headers': headers})

this returns http status 400 as soon as I call it

headers is dict

if I omit 'headers', it works

is that not an issue with requests then?

ohhh nvm

i'll do a search

thanks

actually, it looks like with `requests.get` you do mean the requests library. so I don't know

ok i figured it out

had to use POST request to splash render.html page

response = requests.post(splash_url, json={'url': url, 'http_method': 'GET', 'timeout': timeout, 'wait': 0.5, 'headers': headers})

this works

I am thinking of having a pipeline send my Scrapy item to a celery task which would use the Django ORM to create/update a Product model

Should I do it like that or would it be better just to import the django Product model directly into the pipeline and do the create/update there? or via a deferred?

https://github.com/scrapy-plugins/scrapy-django... doesn't work for you?

nyov possibly it would

I'm trying to think about if that would be blocking or what...or if that even matters for my app

Also if I used this package would django post_save signals still be fired?

the RDBMS is blocking, but in reality that's often not an issue unless you're really high quantity

quantity is about 15k rows per day

so prob would be ok

see the caveats section: "DjangoItem is a rather convenient way to integrate Scrapy projects with Django models, but bear in mind that Django ORM may not scale well if you scrape a lot of items (ie. millions) with Scrapy."

disclaimer: I haven't used it. (so I don't know about post_save signals)

I suppose it is worth a try first

if I didn't use this package...then I was thinking to publish a message to celery and have a task pick it up and do the django orm sve

or possibly just importing my django model directly into the pipelines of my project and having a pipeline use the django model directly to save

isn't that what this plugin does?

I think it probably does something close to the second option, yeah

I don't see how it would be different, doing it in the pipeline, but perhaps you can mod it better there

but the first option would be to just shove the items back into celery queue and have that task write them

which might lead to really fast scrape times without scrapy stopping to write out to django

dunno celery much, try it I guess

yeah, will give these options a try and see what works best

thanks for the help

for what it's worth :)

yw