Data Strategy for Automation Hub Platform
Black Basil contributed to the Automation Hub console for providing real time information about the workflow transactions along with more insights and analytics
About Element5
The company was founded in 2019 by four post-acute care industry veterans distressed by the sheer amount of time spent on manual tasks. This company is working to eliminate 200,000 manual hours a year spent on administrative work.
Challenge
The current Automation Hub (UI) for workflow status and transactions view is lacking key
features, like:
- No facility for user actions like pause, resume, stop at the transaction level
- Human in the loop is not present
- Information analytics are not available
- Lack of observability at granular level of the transaction like at what stage it is and status etc..
- Data is not stored in optimised form and cannot be leveraged for other purposes like analysis, AI/ML (Lack of canonical data layer)
- The system is not scalable and cannot sustain ever growing scale.
Solution
Automation Hub is used for real-time view of the workflow data and transactions lifecycle
BB provided right guidance on building the long term data strategy by following the Data
Lakehouse architecture using Snowflake.
- Revamped the whole UI/UX
- The webapp was built using React.js and Node.Js as backend
- The reporting dashboard was developed using PowerBI
- The services were deployed on Kubernetes (AWS)
- Terraform and Jenkins were used for infrastructure provisioning and deployment pipelines
- SonarQube for code quality
- Transformed the Data strategy and introduced Lakehouse architecture using Snowflake. The lakehouse has 3 layers :
- Transformed the Data strategy and introduced Lakehouse architecture using Snowflake. The lakehouse has 3 layers :
- Raw Layer
- Canonical Layer
- Business Layers
- API layer
- Analytics layer : Star Schema for Visualization and Slicing and Dicing on PowerBI
Tech Stack :
Key Results:
- System is now able to sustain huge load without any impact on the latency.
- Clean data layers and more fit for purpose layers can be built on top of that
- Analytics with granular slice and dice is possible now
- Data platform for all data needs is there now providing seamless experience to all workflow teams
- Better data governance and data quality
- Features like masking, obfuscation are implemented ensuring better security and compliance, eventually building more trust with customers
- Hierarchical role based access control
- More adoption as UX attracts many clients and sales becomes easier.
Black Basil played a key strategic role in shaping up our new age automation platform's architecture and collaborated well with our architects, product manager, technical leads and developers ensuring the platform is designed and delivered well with the right engineering practices. They provided thought leadership in DevOps, QA Automation and Agile delivery to ensure that Element5 emerges as an 'Elite DevOps performer' based on DORA metrics. They also provided the right guidance on building our long-term data strategy by following the Data Lakehouse architecture. Their tech leadership is never hesitant even in having code pairing sessions with our developers and are always ready to roll up their sleeves whenever required
Murali Vivekanandan
Co-founder and CTO, Element5
