ULTIMATE RESOURCES TO FIND DATA PART-2

WEB SCRAPPING

1.Beautifulsoup  https://www.freecodecamp.org/news/how-to-scrape-websites-with-python-and-beautifulsoup-5946935d93fe/

mechanicalsoup https://analyticsindiamag.com/mechanicalsoup-web-scraping-custom-dataset-tutorial/

2.Scrapy

3.Selenium https://www.freecodecamp.org/news/better-web-scraping-in-python-with-selenium-beautiful-soup-and-pandas-d6390592e251/

4.Request to access data

5.AUTOSCRAPER - https://github.com/alirezamika/autoscraper https://www.youtube.com/watch?v=9BQ353Yu1D0 https://www.analyticsvidhya.com/blog/2021/04/automate-web-scraping-using-python-autoscraper-library/

Scrap Images From E-Commerce Website Using AutoScraper https://www.analyticsvidhya.com/blog/2021/05/scrap-images-from-e-commerce-website-using-autoscraper-library/

FiftyOne Now easier to download and evaluate https://towardsdatascience.com/googles-open-images-now-easier-to-download-and-evaluate-with-fiftyone-615ce0482c02

webbot https://pypi.org/project/webbot/

gazpacho https://github.com/maxhumber/gazpacho

html_scraper_streamlit_app https://www.youtube.com/watch?v=6U5xJ3mXRKA&feature=youtu.be

6.Twitter scraping tool (𝚝𝚠𝚒𝚗𝚝 or tweepy)-https://github.com/twintproject/twint

twitterscraper https://www.youtube.com/watch?v=MpIi4HtCiVk

snscrape extract twitterr data https://github.com/JustAnotherArchivist/snscrape

Scweet A simple and unlimited twitter scraper https://github.com/Altimis/Scweet

GetOldTweets3,GoogleNews

Scrape Twitter for Tweets https://github.com/taspinar/twitterscraper

HAR File Web Scraper https://stevesie.com/har-file-web-scraper https://www.youtube.com/watch?v=LcqVDfueb8g

https://analyticsindiamag.com/complete-tutorial-on-twint-twitter-scraping-without-twitters-api/

https://developer.twitter.com/en/docs

pytrends https://medium.com/nerd-for-tech/scraping-data-from-online-platforms-to-enhance-time-series-forecasts-6eec3c68636d

Scraping Instagram -instaloader https://thecleverprogrammer.com/2020/07/30/scraping-instagram-with-python/

Instascrape

Scraping Reddit using Python Reddit API Wrapper (PRAW)

Scrape Wikipedia wikipedia https://www.thepythoncode.com/article/access-wikipedia-python

patang - Scrape Product details from eCommerce Sites with Puppeteer and DOM String https://www.youtube.com/watch?v=3sgxRmyOuXs

Download Wikipedia https://www.wikidata.org/wiki/Wikidata:Main_Page https://www.youtube.com/watch?v=hC1rY4lRY0s https://towardsdatascience.com/an-efficient-way-to-read-data-from-the-web-directly-into-python-a526a0b4f4cb

Web Scraping to Create a CSV File https://thecleverprogrammer.com/2020/08/08/web-scraping-to-create-csv/

Amazon Web Scraper, Amazon Auto Scraper

7.urllib

8.pattern

9.Octoparse Easy Web Scraping https://www.octoparse.com/

ParseHub https://www.parsehub.com/ https://analyticsindiamag.com/parsehub-no-code-gui-based-web-scraping-tool/

Apify https://apify.com/

obsei https://github.com/lalitpagaria/obsei

Diffbot https://analyticsindiamag.com/diffbot/

Trustpilot,webhose,scrapingbot

lxml https://lxml.de/index.html#introduction

ScrapingBee https://analyticsindiamag.com/scrapingbee-api/

Scrape HTML tables https://www.youtube.com/watch?v=6U5xJ3mXRKA&feature=youtu.be or pd.read_html

requests-html https://github.com/kennethreitz/requests-html

newspaper https://github.com/codelucas/newspaper https://www.youtube.com/watch?v=Hfry5XnISyc

newspaper3k: https://newspaper.readthedocs.io # easily extract text from articles

newscatcher https://github.com/kotartemiy/newscatcher https://www.youtube.com/watch?v=pHzOuizZq4I

patang (extract product details) https://github.com/tejazz/patang

lisc https://github.com/lisc-tools/lisc

pandas(read_html)

wget,curl,parsehub,webhouse,octoparse,scraping bot,scraping bee,Common,Content Grabber,Docparser,Scraper API,Import.io,Altair Monarch,WebAutomation.io,WebScraper.io

Crawl Crawly https://crawly.diffbot.com/

HTML basics for web scraping,Web Scraping with Octoparse,Web Scraping with Selenium

10-best-web-scraping-tools https://www.scraperapi.com/blog/the-10-best-web-scraping-tools/

https://www.kdnuggets.com/2021/02/6-web-scraping-tools.html

https://analyticsindiamag.com/complete-learning-path-to-web-scraping-with-all-major-tools/ https://towardsdatascience.com/6-web-scraping-tools-that-make-collecting-data-a-breeze-457c44e4411d

https://towardsdatascience.com/6-web-scraping-tools-that-make-collecting-data-a-breeze-457c44e4411d https://www.kdnuggets.com/2018/02/web-scraping-tutorial-python.html

https://www.octoparse.com/ https://github.com/tirthajyoti/pydbgen https://www.mozenda.com/ https://www.mockaroo.com/ https://lionbridge.ai/ https://www.mturk.com/ https://appen.com/

11.GoogleImageCrawler,google_images_download,bing_image
If you like to explore more dataset then visit my github repositoryhttps://github.com/achuthasubhash/Complete-Life-Cycle-of-a-Data-Science-Project