GitHub - hackersdotmu/legal

1 Branch 0 Tags

Name	Name	Last commit message	Last commit date
Latest commit loganaden Merge pull request #1 from 451hackathon/master Jul 15, 2017 fe4f6f5 · Jul 15, 2017 History 14 Commits
censorship_crawler	censorship_crawler	Add regexp-constrained recursion. closes #1	Jul 15, 2017
.gitignore	.gitignore	write output in json logging format	Jul 15, 2017
LICENSE	LICENSE	Add Daniel Ramsay	Jul 15, 2017
README	README	allow running against arbitrary URL	Jul 15, 2017
first.py	first.py	initial repo	Jun 19, 2017
requirements.txt	requirements.txt	add requirements.txt and README	Jul 15, 2017
second.py	second.py	initial repo	Jun 19, 2017
third.py	third.py	initial repo	Jun 19, 2017

Repository files navigation

Legal Crawler
=============

Installation requirements:

pip -r requirements.txt

Running the crawler

  $ cd censorship_crawler
  $ scrapy crawl 451

Running the crawler against a specific URL:

  $ cd censorship_crawler
  $ scrapy crawl -a url=https://www.example.com 451