i reading through scrapy docs @ http://doc.scrapy.org/en/latest/topics/settings.html confused "designating settings" section. describe in plain english how go implementing settings? assumed making modification settings.py doesn't seem case. trying download delay getting blocked site scraping. thanks!
well, description bit confusing right: settings.py right file add settings. there 2 settings work parallel:
download_delay , concurrent_requests_per_domain. latter default value 8. override both , should work.
to read more default concurrent settings refer docs: http://doc.scrapy.org/en/latest/topics/settings.html#concurrent-requests
Comments
Post a Comment