THE BEST SERVER MEMORY UPGRADES™
It is an e-commerce website, where users can buy server’s Memory and Storage devices from brands like HP, Dell, Lenovo, Cisco, etc.
Using Python we particularly do scraping of memory and server information from these brands through their official or other websites. Performing some normalisation with data and importing it to the database and then viewing it on the dashboard. Also, we have built some API to import data to the database and API’s to integrate with Frontend and automated this scraping and importing flow to the database using Apache Airflow.
Frameworks we using:
Scraping/mining data – Scrapy, Scrapy with selenium
Automate scraping and import data – Apache Airflow
Database – PostgreSQL
Data normalisation – Pandas