wescraper 配置使用

Posted by Adobe on August 7, 2016 | - view

wescraper可用于爬取公众号文章。

安装

资料:在国内使用 pip 命令
pact install libxml2-devel libxslt-devel
pact install libffi-devel
pact install openssl-devel
pip install scrapy -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install tornado

命令行抓取

python wescraper/scraper.py account xxx > we.json # 查询liriansu和miawu相关的公众号
python wescraper/scraper.py key-day xxx > we.json # 查询liriansu和miawu相关的文章(一天内)
account
key-all
key-day
key-week
key-mon
key-year

#浏览器查看

python wescraper/server.py –port==8080
http://localhost/account/
http://localhost/key-all/
http://localhost/key-day/
http://localhost/key-week/ http://localhost/key-mon/
http://localhost/key-year/