Skip to content

Commit

Permalink
update structure
Browse files Browse the repository at this point in the history
  • Loading branch information
Germey committed Feb 19, 2020
1 parent 39bcf6b commit d4fca11
Show file tree
Hide file tree
Showing 29 changed files with 429 additions and 400 deletions.
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,5 @@
*.pyc
*.db
venv
/.idea
/.idea
*.log
201 changes: 0 additions & 201 deletions LICENSE

This file was deleted.

80 changes: 52 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,52 +1,76 @@
# ProxyPool

## 安装
## Requirements

### 安装Python
* Docker

至少Python3.5以上
or

### 安装Redis
* Python: >=3.6
* Redis
* Environment: Virtual Env

安装好之后将Redis服务开启
## Run with Docker

### 配置代理池

```
cd proxypool
```shell script
docker-compose up
```

进入proxypool目录,修改settings.py文件
## Run without Docker

PASSWORD为Redis密码,如果为空,则设置为None
Here are steps to run ProxyPool.

#### 安装依赖
### Install Redis

```
pip3 install -r requirements.txt
You need to install Redis locally or get a Redis server firstly.

Next set Redis environment:

```shell script
export REDIS_HOST='localhost'
export REDIS_PORT=6379
export REDIS_PASSWORD='foobar'
```

#### 打开代理池和API
Also you can just set the Redis Connection String:

```shell script
export REDIS_CONNECTION_STRING='redis://[password]@host:port'
```
python3 run.py

You can choose one method of above to set Redis environment.

### Clone ProxyPool

```shell script
https://github.com/Python3WebSpider/ProxyPool
cd ProxyPool
```

## 获取代理
### Install Requirements

```shell script
pip3 install -r requirements.txt
```

### Run ProxyPool

利用requests获取方法如下
You can run all of the processors including Getter、Tester、
Server:

```python
import requests
```shell script
python3 run.py
```

PROXY_POOL_URL = 'http://localhost:5555/random'
or run with args to run specific processor:

def get_proxy():
try:
response = requests.get(PROXY_POOL_URL)
if response.status_code == 200:
return response.text
except ConnectionError:
return None
```shell script
python3 run.py --processor getter
python3 run.py --processor tester
python3 run.py --processor server
```

### Usage

After running the ProxyPool, you can visit
[http://localhost:5555/random](http://localhost:5555/) to access random proxy.
4 changes: 0 additions & 4 deletions importer.py

This file was deleted.

7 changes: 0 additions & 7 deletions proxy provider.txt

This file was deleted.

Loading

0 comments on commit d4fca11

Please sign in to comment.