GitHub - 451hackathon/legal

This branch is 14 commits ahead of hackersdotmu/legal_crawler:master.

Name	Name	Last commit message	Last commit date
Latest commit dantheta switch collector to https Jul 16, 2017 58af08b · Jul 16, 2017 History 28 Commits
censorship_crawler	censorship_crawler	switch collector to https	Jul 16, 2017
.gitignore	.gitignore	write output in json logging format	Jul 15, 2017
LICENSE	LICENSE	Add Daniel Ramsay	Jul 15, 2017
README	README	allow running against arbitrary URL	Jul 15, 2017
first.py	first.py	initial repo	Jun 19, 2017
requirements.txt	requirements.txt	Added outputting to output.json (finished)#2	Jul 16, 2017
second.py	second.py	initial repo	Jun 19, 2017
third.py	third.py	initial repo	Jun 19, 2017

Repository files navigation

Legal Crawler
=============

Installation requirements:

pip -r requirements.txt

Running the crawler

  $ cd censorship_crawler
  $ scrapy crawl 451

Running the crawler against a specific URL:

  $ cd censorship_crawler
  $ scrapy crawl -a url=https://www.example.com 451