forked from Python3WebSpider/ProxyPool
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
29 changed files
with
429 additions
and
400 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2,4 +2,5 @@ | |
*.pyc | ||
*.db | ||
venv | ||
/.idea | ||
/.idea | ||
*.log |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,52 +1,76 @@ | ||
# ProxyPool | ||
|
||
## 安装 | ||
## Requirements | ||
|
||
### 安装Python | ||
* Docker | ||
|
||
至少Python3.5以上 | ||
or | ||
|
||
### 安装Redis | ||
* Python: >=3.6 | ||
* Redis | ||
* Environment: Virtual Env | ||
|
||
安装好之后将Redis服务开启 | ||
## Run with Docker | ||
|
||
### 配置代理池 | ||
|
||
``` | ||
cd proxypool | ||
```shell script | ||
docker-compose up | ||
``` | ||
|
||
进入proxypool目录,修改settings.py文件 | ||
## Run without Docker | ||
|
||
PASSWORD为Redis密码,如果为空,则设置为None | ||
Here are steps to run ProxyPool. | ||
|
||
#### 安装依赖 | ||
### Install Redis | ||
|
||
``` | ||
pip3 install -r requirements.txt | ||
You need to install Redis locally or get a Redis server firstly. | ||
|
||
Next set Redis environment: | ||
|
||
```shell script | ||
export REDIS_HOST='localhost' | ||
export REDIS_PORT=6379 | ||
export REDIS_PASSWORD='foobar' | ||
``` | ||
|
||
#### 打开代理池和API | ||
Also you can just set the Redis Connection String: | ||
|
||
```shell script | ||
export REDIS_CONNECTION_STRING='redis://[password]@host:port' | ||
``` | ||
python3 run.py | ||
|
||
You can choose one method of above to set Redis environment. | ||
|
||
### Clone ProxyPool | ||
|
||
```shell script | ||
https://github.com/Python3WebSpider/ProxyPool | ||
cd ProxyPool | ||
``` | ||
|
||
## 获取代理 | ||
### Install Requirements | ||
|
||
```shell script | ||
pip3 install -r requirements.txt | ||
``` | ||
|
||
### Run ProxyPool | ||
|
||
利用requests获取方法如下 | ||
You can run all of the processors including Getter、Tester、 | ||
Server: | ||
|
||
```python | ||
import requests | ||
```shell script | ||
python3 run.py | ||
``` | ||
|
||
PROXY_POOL_URL = 'http://localhost:5555/random' | ||
or run with args to run specific processor: | ||
|
||
def get_proxy(): | ||
try: | ||
response = requests.get(PROXY_POOL_URL) | ||
if response.status_code == 200: | ||
return response.text | ||
except ConnectionError: | ||
return None | ||
```shell script | ||
python3 run.py --processor getter | ||
python3 run.py --processor tester | ||
python3 run.py --processor server | ||
``` | ||
|
||
### Usage | ||
|
||
After running the ProxyPool, you can visit | ||
[http://localhost:5555/random](http://localhost:5555/) to access random proxy. |
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.