i'm attempting scrape website , every child link scrape returning http 301 redirection response no data in response object.
2015-07-18 01:54:52 [scrapy] debug: crawled (301) http://website.example> >(referer: http://website.example/tutorials/education/)
i've turned off following of redirects meta tags in scrapy request (meta={'dont_redirect': true, 'handle_httpstatus_list': [301]}).
with redirects turned on, 301 response returns same url redirect request...
2015-07-18 02:00:52 [scrapy] debug: redirecting (301) http://website.example/tutorials/communications/20364/com-340-entire-course-mass-communication/> http://website.example/tutorials/communications/20364/com-340-entire-course-mass-communication/>
can give me advice how handle http 301 response in order scrape page?
i've tried turning on/off redirect middleware , used anonymous proxy no luck.
Comments
Post a Comment