# 14-分布式爬虫

在上一章中，我们了解了 Scrapy 爬虫框架的用法。这些框架都是在同一台主机上运行的，爬取效率比较有限。如果多台主机协同爬取，那么爬取效率必然会成倍增长，这就是分布式爬虫的优势。

本章我们就来了解一下分布式爬虫的基本原理，以及 Scrapy 实现分布式爬虫的流程。


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://python3webspider.cuiqingcai.com/14-fen-bu-shi-pa-chong.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.