Scrapy HTTP Redirect 301 Issue -


i'm attempting scrape website , every child link scrape returning http 301 redirection response no data in response object.

2015-07-18 01:54:52 [scrapy] debug: crawled (301) http://website.example> >(referer: http://website.example/tutorials/education/)

i've turned off following of redirects meta tags in scrapy request (meta={'dont_redirect': true, 'handle_httpstatus_list': [301]}).

with redirects turned on, 301 response returns same url redirect request...

2015-07-18 02:00:52 [scrapy] debug: redirecting (301) http://website.example/tutorials/communications/20364/com-340-entire-course-mass-communication/> http://website.example/tutorials/communications/20364/com-340-entire-course-mass-communication/>

can give me advice how handle http 301 response in order scrape page?

i've tried turning on/off redirect middleware , used anonymous proxy no luck.


Comments