Voir en français
Menu

Data

Speeding up scientific result analysis on AWS and Snowflake

The context

A European contract research organisation produces very large volumes of experimental results for its clients, upstream of new drug discovery. The value of that work is not only in producing the data, but in how fast the delivery teams can analyse it and hand it back.

The bottleneck was not scientific, it was computational: processing launched by hand, files moving from one workstation to another, and analyses so slow that people hesitated to re-run them. That hesitation is the real cost, because it reduces the number of hypotheses a team allows itself to explore.

What we did

We industrialised the chain, from the arrival of the results to the way they are handed back.

From experimental results to reporting, through one warehouseExperimental results land on S3 alongside the compound library and other sources. Their arrival triggers the processing pipeline through Lambda, with no manual launch. Processed data is loaded into a single Snowflake warehouse, which every analysis reads from. That one warehouse is then consumed three ways: Power BI dashboards, a client portal served through CloudFront behind WAF, and ad-hoc queries by the delivery teams.From raw results to reportingAssay resultsinstrument outputCompound librarystructures, metadata…further sourcesArrival on S3 triggers the pipeline — Lambda, no manual launchSnowflake warehouse — one queryable reference setPower BIdashboardsClient portalCloudFront and WAFDelivery teamsad-hoc queries
The pipeline starts because data arrived, not because someone remembered to start it. And the three read channels are three views of one warehouse, which is what makes the freshness of a figure a property of the system rather than a question to ask around.
  • Processing industrialised into reproducible pipelines on AWS, triggered automatically when new data arrives rather than run on demand by someone who has to remember. A file landing on S3 is the event that starts everything else.
  • A Snowflake warehouse so analyses run against a single queryable reference set, instead of scattered extracts whose freshness nobody can vouch for.
  • Reporting for the delivery teams and, beyond them, for their clients: Power BI dashboards for recurring follow-up, and a client portal served through CloudFront behind WAF, with access rights and deliverable state held in DynamoDB. A portal that exposes the data of competing clients to one another is not allowed to be approximate about isolation.
  • Similarity search across the compound library. Structures and results are turned into embeddings and indexed in OpenSearch alongside the keyword fields. One query then answers “close to this structure, and assayed after that date”, which no filter can express. That is the question a chemist actually asks, and until then it was answered from memory.
  • A data model designed so that adding a new experiment type does not require rebuilding the processing chain.
Similarity search across the compound libraryStructures and results from the compound library are turned into embeddings computed on Lambda. Those vectors are indexed in OpenSearch alongside the keyword fields, so one query combines similarity and exact filters. The result is a ranked list of comparable compounds handed back to the project team, rather than a list of rows a chemist has to sift through.Searching the library by similarityCompound librarystructuresEmbeddingscomputed on LambdaOpenSearch indexvector and keywordSimilar compoundsranked for the team
Similarity is the question a chemist actually asks, and it is not expressible as a filter. Keeping vectors and keyword fields in the same index is what lets one query answer "close to this structure, and assayed after that date" without two round trips.

The outcome

Analyses that took days take minutes. That is not only a time saving: it changes how people work, because re-running an analysis with different parameters becomes unremarkable instead of something you have to justify.

The delivery teams spend their time interpreting results rather than operating the machinery that produces them.

All case studies

Talk to us directly

Write to us directly. No salesperson, no qualification call. One of us two replies, within 24 hours.

contact@onescale.io

We reply within 24 hours, and it is one of us who replies.

Based in Lyon, working with clients in France, across Europe and internationally.