Automation - Perl / Python - Processing Workflow
I started this project initially because I wanted to scrape specific attributes and data from one of the distributed systems I work with. After completing my initial tool for extracting the data that I wanted to assess I seen the potential for the application to be of use for introducing some configuration management and automation.
During my investigation for this project, I wanted to see if I could further optimise the process by allowing tasks to be run in parallel by containerising the application and letting it run in a docker environment. Unfortunately, time constraints prevented this part of the project from being realised although I was able to successfully containerise a large portion of the application, this included creating the initial container, installing the application into the container, configuring the database to work within the container environment, configuring the automation scripts to run within the container and for the container to be kept in a repository.
To facilitate this task, a series of Perl modules were created to provide the areas of functional blocks required this initially consisted of:
In order to execute the automation, a small script was created that imported the relevant Perl modules and triggered the correct methods for the batch processing task.
In order to view the results, monitor the progress, and configure the processing workflow and line queue a python application was created making use of the PyQT modules to represent the information in a desktop application.
This application offered a line by line view (for each imported file), a survey wide view - for all files in each survey, and a task view to show the progress of the workflows.
Some additional configuration tools were add in to facilitate the configuration of a processing flow, and line queue so that a batch processing task could be configured and then triggered at a later stage.
The GUI application and the Perl application were separate processes so the GUI application was not required for the automation function to be executed.
The application itself has been successfully deployed to help the development team reach their next milestone and by re-running the automatic tasks regularly it is possible to ensure the results and behaviour of the application are as expected.
The application was successfully deployed to AWS infrastructure and used as part of a SaaS solution to present progress of the development effort to one of our clients.