i not sure question correct, quite new in scrapy. therefore, comments related topic useful.
my problem have website structure( json document ):
{ 2: { 1: 'http://example.com/1.jpg', 2: 'http://example.com/2.jpg' } ...// bunch of other information } and links related, need direct link images. trying solve task as:
urldata = scrapy.request(url='http://myserver/1.jpg', callback=self.parse_link)
where callback function is:
def parse_link( self, response ): hxs = scrapy.selector.htmlxpathselector(response) data = lacbaclink() link = hxs.select("(//img)[1]/@src").extract() data["link"] = link return data i consider work if able result callback request function in scrapy immediately.
i have tried write code presented documentation:
def parse_page1(self, response): return scrapy.request("http://www.example.com/some_page.html", callback=self.parse_page2) def parse_page2(self, response): # log http://www.example.com/some_page.html self.logger.info("visited %s", response.url) it not work me. if know how solve task using code, please, give me realisation better understanding.
this isn't working code -- read comments added.
def parse_page1(self, response): # here collect information need first page , put in item r = scrapy.request("http://www.example.com/some_page.html", callback=self.parse_page2) r.meta['item'] = item yield r def parse_page2(self, response): item = response.meta['item'] # add need response item yield item # if need parse more pages yield request instead.
Comments
Post a Comment