Data Collector
Collect data from 3k+ servers, store in Database, display in User Interface
About this project

The Data Collector app effectively collects internal data from the 3k+ company owned servers, stores it in a database, and displays the data in a searchable User Interface.

My former manager desired information about the servers his team was responsible for. Specifically, he wanted to know what exist in particular servers and what its purpose was. There was no easy way to find this information except to contact his "go-to" person to research and report which normally takes about an hour for each server.

This Data Collector app enables the team and manager to research servers as fast as one can type. Since the server data is stored in a database as well as in friendly searchable User Interface, it will take anyone just minutes to find what they're looking for.

Architecture

There are three main parts to this client - server - database application.

The first part is the server, which is responsible for collecting server data which includes the server details such as its name, location, operating system, database, applications installed, etc.

The second part is the MySQL database, which holds the server details.

Finally, an internal API was used to concurrently hit the database to obtain all the servers and its data to produce a searchable table within an internal Web Application.

This application is hooked up with a CRON job which checks daily for servers that are newly deployed, modified, removed, etc. and update the database as needed.

Technologies
  • Python
  • Flask
  • MySQL
  • SQLAlchemy
  • HTML5
  • CSS3