Integrating reporting into the heart of this SaaS Tool.
Objective
Build complex multi-tenancy infrastructure for client reporting.
Obstacle
An expensive data stack, non-standard requirements and bespoke reporting.
Outcome
Over $300k saved in annual data stack costs and an overhauled self-service system deployed.
Background
SimplifyVMS is a provider of vendor management systems for temporary staffing needs with clients like EasyJet, Mariott and PWC. Its platform offers extreme flexibility to clients, enabling efficient management of recruitment services for contingent workers. The team wanted to enhance the reporting function offered to clients by providing them with 100% customisation of their own dashboards and the ability to drag-and-drop bespoke reports. This meant setting up multi-tenancy infrastructure and bespoke modelling pipeline for each individual client – without costs ballooning. They turned to 173tech with this big technical challenge…
Challenges
Costly Data Infrastructure: SimplifyVMS’s existing reporting infrastructure relies on Databricks and EC2, costing approximately $30,000 per month. The complexity of the architecture results in inefficiencies, making it difficult to scale. Moreover, the infrastructure requires substantial maintenance, with frequent memory issues arising in Kafka and Databricks workers.
High Manual Intervention: Client onboarding takes months due to the need for manual code creation. Each client has a separate Git branch, requiring individual updates for new metrics and features. Additionally, the lack of automation in data reporting leads to increased manual work and redundancy in code, delaying insights for clients.
Solution
Redesigned Data Architecture: To address these challenges, SimplifyVMS explored multiple data architecture options, including Redshift, Databricks, Snowflake and Clickhouse. The selected solution focuses on implementing federated schemas in Redshift to enable real-time data access, introducing dbt for automated data transformation to reduce manual intervention, leveraging Airflow as an orchestrator to streamline ETL processes and developing a multi-tenant schema strategy to support custom client requirements efficiently.
Enhanced Data Processing and Governance: The new solution introduces a structured data modelling layer to simplify reporting and minimise the need for manual SQL joins. Additionally, governance policies ensure secure access controls and data segregation per client, improving compliance and security.
The Proof Is In The Numbers...
$330k+
75%
6
Implementation
Infrastructure Setup: Redshift was set up as the core reporting warehouse. dbt was integrated to automate data transformations. A federated schema system was established to provide seamless data access. Automated workflows for new client onboarding were developed, reducing setup time from months to a few days. Materialised views were implemented to optimise query performance and reduce reporting lag. Data refresh rates improved from one hour to near real-time, enhancing decision-making capabilities for clients.
Knowledge Transfer: We worked closely with SimplifyVMS’s internal team to enhance their understanding of data architecture, analytics, and best practices. By providing guidance on common data challenges, we empowered them to troubleshoot issues effectively, optimise their data workflows, and implement scalable solutions.”
We collaborated with SimplifyVMS to enhance their internal team’s expertise in data architecture and analytics. Through targeted training and hands-on support, we helped them resolve data integration challenges, optimise reporting workflows, and implement scalable solutions tailored to their needs.
Future-Proof: Through a strategic overhaul of its data architecture, SimplifyVMS achieved a scalable, cost-effective and automated reporting system. The new infrastructure enables the company to onboard clients faster, reduce costs and provide faster insights.
