https://www.python.org/downloads/
https://docs.scrapy.org/en/latest/intro/install.html
https://pypi.org/project/itemloaders/
https://pypi.org/project/pymongo/
https://www.geeksforgeeks.org/how-to-install-selenium-in-python/ https://selenium-python.readthedocs.io/installation.html
make sure selenium driver path is in PATH environment variable
https://docs.mongodb.com/v4.0/installation/
in this file, you can select setting you want to use. Defautl value is scrapy01.settings_dev_with_proxy, because I'm in China mainland. Don't forget enabling your own proxy and configure the right server and port.
You can use "scrapy runspider <spider_file.py>." for a single spider. for example: scrapy runspider scrapy01/spiders/douxing/yifymovies.py
data in MongoDB:
downloaded files structure:
