scrapy 请求继承体系

Request |– FormRequest

通过以下请求测试
GET: https://httpbin.org/get
POST: https://httpbin.org/post

get请求

方式：通过Request 发送

import json

from scrapy import Spider, Request, cmdline

class SpiderRequest(Spider): name = "spider_request"

def start_requests(self): url = "https://httpbin.org/get?name=tom" yield Request(url, body=json.dumps({"age": "23"}))

def parse(self, response): print(response.text)

if __name__ == '__main__': cmdline.execute("scrapy crawl spider_request".split())

服务端收到url链接中的参数name，而没有收到body里边的参数age

"args": { "name": "tom" },

post请求

方式一：通过FormRequest 发送

from scrapy import Spider, cmdline, FormRequest

class SpiderFormData(Spider): name = "spider_form_data"

def start_requests(self): url = "https://httpbin.org/post" yield FormRequest(url, formdata={"name": "Tom"})

def parse(self, response): print(response.text)

if __name__ == '__main__': cmdline.execute("scrapy crawl spider_form_data".split())

服务器接收到参数

"form": { "name": "Tom" },

而且headers里边有一个参数

"headers": { "Content-Type": "application/x-www-form-urlencoded", },

方式二：通过Request发送

需要添加参数 method="POST"

import json

from scrapy import Spider, Request, cmdline

class SpiderPost(Spider): name = "spider_post"

def start_requests(self): url = "https://httpbin.org/post" yield Request(url, method="POST", body=json.dumps({"name": "Tom"}))

def parse(self, response): print(response.text)

if __name__ == '__main__': cmdline.execute("scrapy crawl spider_post".split())

1、直接发送post请求，服务器端收到参数data，和json：

"data": "{"name": "Tom"}", "form": {}, "json": { "name": "Tom" },

2、如果添加headers参数:

"headers": { "Content-Type": "application/x-www-form-urlencoded", },

服务器收到参数，form将接收到参数，也就是FormRequest的提交方式

"data": "", "form": { "{"name": "Tom"}": "" }, "json": null,

3、如果添加headers参数:

"headers": { "Content-Type": "application/json", },

服务器端将收到data 和json 参数，和第一个情形一样，不过有时候不加这个请求头参数获取，会请求错误

"data": "{"name": "Tom"}", "form": {}, "json": { "name": "Tom" },

总结

参考
Scrapy Requests and Responses