1.创建爬虫项目，在指定文件夹的命令窗口下运行
scrapy startproject cnblogspider
项目结构如下

2.添加item
代码如下

import scrapy

class CnblogspiderItem(scrapy.Item): # define the fields for your item here like: # name = scrapy.Field() url=scrapy.Field() cimage_urls=scrapy.Field() cimage=scrapy.Field() image_paths=scrapy.Field()

3.创建spider,在创建的项目文件夹命令窗口下执行
scrapy genspider cnblog cnblogs.com
产生爬虫
然后重写start_urls和parse方法
其中Request为scrapy自带的一个类，项目中用到直接实例化
全部代码如下

import scrapy from cnblogspider.items import CnblogspiderItem from scrapy import Request,Selector

class CnblogSpider(scrapy.Spider): name = 'cnblog' allowed_domains = ['cnblogs.com'] # start_urls = ['http://cnblogs.com/qiyeboy/default.html?page=1'] start_urls = ['http://cnblogs.com/qiyeboy/default.html?page=1']

def parse(self, response): # 实现网页的解析 # 首先抽取所有的文章 papers=response.xpath(".//*[@class='day']") # 从每篇文章中抽取数据 for paper in papers: url = paper.xpath(".//*[@class='postTitle']/a/@href").extract_first() # title = paper.xpath(".//*[@class='postTitle']/a/span/text()").extract()[0] # time = paper.xpath(".//*[@class='dayTitle']/a/text()").extract()[0] # content = paper.xpath(".//*[@class='postTitle']/a/span/text()").extract()[0] # print('%s,%s,%s,%s'%url%title%time%content) # print(f'{url}{title}{time}{content}') # item=CnblogspiderItem(url=url,title=title,time=time,content=content) item = CnblogspiderItem(url=url) request=scrapy.Request(url=url,callback=self.parse_body) # 将item暂存 request.meta['item']=item yield request next_page=Selector(response).re(u'<a href="(\s*)">下一页</a>') if next_page: yield scrapy.Request(url=next_page[0],callback=self.parse) def parse_body(self,response): item=response.meta['item'] body=response.xpath(".//*[@class='postBody']") # 提取图片链接 item['cimage_urls']=body.xpath('.//img//@src').extract() yield item

4.构建item pipeline存储下载的图片
因为是存储图片所以要用到MediaPipline中的ImagesPipeline
以及设置settings
4.1 要存储到指定的路径下所以重写get_media_requests(item,info)方法和item_completed(results,items,info)方法
其中get_media_requests方法中要用到item的cimage_urls字段
item_completed方法中要用到item的image_paths字段
具体代码如下：

from itemadapter import ItemAdapter from scrapy.exceptions import DropItem import scrapy from scrapy.pipelines.images import ImagesPipeline

class MyImagesPiplines(ImagesPipeline): def get_media_requests(self, item, info): for image_url in item['cimage_urls']: yield scrapy.Request(image_url) def item_completed(self, results, item, info): image_paths=[x['path'] for ok,x in results if ok] if not image_paths: raise DropItem('item contains no image') item['image_paths']=image_paths return item

4.2设置settings,
4.2.1 因为不能用scrapy初始的user_agent,所以要激活USER_AGENT，并添加自己浏览器的user_agent
代码如下：

USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36'

4.2.2 激活Pipeline
代码如下

```python ITEM_PIPELINES = { # 'cnblogspider.pipelines.CnblogspiderPipeline': 300, 'cnblogspider.pipelines.MyImagesPiplines': 301, }

4.2.3
其他项目为固定项目，根据实际需要设置即可
其中
IMAGES_STORE为文件存储路径字段
IMAGES_URL_FIELD问文件url所在item字段
IMAGES_RESULT_FIELD为文件结果信息所在item所在字段
IMAGES_EXPIRES为文件过期时间（天）
IMAGES_THUMBS制作图片缩略图，并设置缩略图大小
具体代码如下

IMAGES_STORE='D:\\cnblogs' IMAGES_URL_FIELD='cimage_urls' IMAGES_RESULT_FIELD='cimages' IMAGES_EXPIRES=30 IMAGES_THUMBS={ 'small':(50,50), 'big':(270,270), }

5.运行爬虫
在项目所在文件夹命令窗口下运行命令
scrapy crawl cnblog
–