Landauer - Talend
HealthcareLandauer achieves 50% infrastructure cost reduction and 80% lower latency by migration to AWS OpenSearch
About Landauer - Talend
LANDAUER is the leading global provider of technical and analytical services to determine occupational and environmental radiation exposure and the leading domestic provider of outsourced medical physics services.
The company provides dosimetry services to about 1.8 million individuals globally and is used by 78% of United States hospitals. The company offers tools and support for organizations with potential exposure to ionizing radiation, helping them achieve their radiation safety goals.
LANDAUER’s innovations in radiation safety continue to shape the industry in dosimetry and medical physics.
The Challenge
Landauer faced several daunting challenges in optimizing their ETL workflow. First and foremost, they needed to maintain the order of processing to ensure that parallel iterations could be processed at a time.
Additionally, Landauer needed to transition away from Talend and build a new workflow using AWS Step Functions. This was a significant undertaking, requiring a thorough understanding of the new system and its capabilities. The new workflow required to be able to handle the demands of Landauer’s high-volume data processing needs, all while optimizing the cost of the workflow and providing automation, speed, and reliability.
Given these complex challenges, Landauer needed a partner with deep expertise in cutting-edge serverless frameworks to help them save costs and scale better. That’s where AntStack came in!
Our Goals
The primary objective of the project was to help Landauer transition from their existing ETL process using Talend to a more cost-effective, reliable, and efficient workflow using AWS Step Functions.
The project aimed to accomplish the following goals:
- Optimize the workflow cost while delivering top-notch automation, speed, and reliability.
- Ensure the new system can handle the demands of high-volume data processing requirements.
- Provide parallel processing for packages and runs to reduce the time for data processing.
- Automate the entire process from development to deployment while still providing the ability to alert in case of failures.
- Maintain the order of processing to ensure that independent data iteration could be processed at the same time.
- Create a more scalable and flexible architecture that would sustainable for future growth and expansion.
Our Impact
From Talend To Serverless.
LANDAUER’s pre-implementation system was expensive and limited in processing speed. The Wrapper Package acted as an orchestrator that processed packages sequentially, causing hefty delays.
The new architecture powered by SQS, Lambda, Step Functions, and DynamoDb brought substantial improvements to their ETL process. Step-Functions brilliantly orchestrated the flow of processing, while DynamoDb securely and efficiently stored in-process data.
Additionally, the backend infrastructure was seamlessly deployed through a robust CI/CD pipeline using AWS SAM and GitHub Actions.
Robust Alert Mechanism
AntStack integrated an alert mechanism using the existing framework, DataDog, to detect and respond to any failures in the ETL process. DataDog offers comprehensive attributes, including real-time alerting, anomaly detection, and machine learning-based algorithms that can automatically dispatch automated alerts to the team when the error rate exceeds a specific threshold. LANDAUER’s new infrastructure empowers them to swiftly address any issue, proactively take action and resolve processing errors in the first instance, much before they escalate into a larger problem.
Parallel Data Processing
LANDAUER previously relied on Talend data pipelines running on clusters, a linear system that required waiting for each run to successfully complete before moving on to the next. However, with AntStack’s new architecture built on limitless scale principles, the company has been able to implement batch processing, resulting in substantial speed improvements. With parallel runs into the picture, LANDAUER has significantly reduced the time required to complete their data runs, leading to greater efficiency and precision in their business operations.
Cost Effective Scalability
LANDAUER made the strategic decision to migrate from a stateful instance-based workflow to an event-driven serverless model with AntStack. With AntStack, they could use cutting-edge serverless frameworks with Lambda functions and pay only for what was being consumed.
They could scale up and down, resulting in significant cost savings for the company while also improving their ability to handle large volumes of data in a highly scalable and swift fashion.
A startup within the organisation
After successfully completing two projects with AntStack, LANDAUER continued to modernize its IT systems and create a more resilient and scalable infrastructure. AntStack acted as a true startup within the organization, filling in when LANDAUER’s team lacked the bandwidth and serverless expertise to progress their systems. The AntStack team’s customized engineering solutions surpassed LANDAUER’s goals and expectations, ensuring seamless migration to AWS serverless services.
Results that speak volumes
AntStack crafted and engineered a remarkably efficient serverless infrastructure in a mere 45 days, an impressive feat accomplished over a month ahead of schedule.
- The updated architecture includes a robust alert mechanism in case of failures, improving the system’s overall reliability.
- The workflow is fully automated, from data staging to processing and deployment, reducing manual intervention and errors.
- The AntStack team enabled faster processing of packages, resulting in shorter processing times.
- The new serverless frameworks and resource optimization helped reduce the cost of the workflow, making it more efficient and cost-effective.
In their words
Landauer
Team
In their words
Landauer
Team
