Data Processing and Curation
The Problem
Many industries lack standardisation, especially when it comes to technical product information. You only have to look at the biomedical research and computer technology industries to see that every manufacturer, supplier and reseller all format and provide their product data differently. Coupled with the inherent complexity of the data itself; references to 3rd party databases, detailed product specifications using complex units and measurements, lengthy blocks of text containing multiple points of data etc; the challenge for any company in these worlds to present this information in a clear and understandable way to an end user can be overwhelming.
The cost in man-power and software solutions for any company of any size to develop an in-house solution can often be out of the question, and attempting to put pressure on the original data creators (often the manufacturers/developers of the products) to provide a standardised format for the data that is suitable to your needs can often be seen as unreasonable, especially considering how many other parties the data is often shared with.
Our Solution
This is where TDI can come in. Through years of experience in dealing with this kind of data, TDI has developed a wide ranging knowledge base and suite of custom developed software to efficiently manage, curate, standardise and format complex product data to our clients needs.
Regardless of how complex the data is at the start and how particular the client needs the information to be formatted, we have a proven track record in taking on the project, taking entire portfolios of product data – in some cases in the order of millions of lines – and working with them to assess their needs, provide insightful advice on the use of the data, agree on a standardised format, and then develop new or harness existing solutions to standardise and subsequently manage the data on behalf of the client.
Our solutions have been developed with modularity and adaptability in mind, as the clients’ needs change and grow, so does our technology. We are always looking for newer ways of working to both benefit our clients strict timeframes, and adapt to changes in their respective industries. For example, thanks to our efficient solutions, when the Covid-19 pandemic struck, our clients in the biomedical research industries were able to get the complex data for newly developed coronavirus research products up and online in a fraction of the time it would have otherwise taken, allowing end user researchers to look for the products they needed as soon as they wanted.
We have developed advanced scripting using Python in particular that can be adapted and used to process complex data in any way needed that can output thousands of products per second. Our scripts can reference external or internally curated databases to standardise all sorts of data sets with examples including, industry application types, species and/or conjugation names and medical publication and gene/protein databases.
© Copyright 2024. All rights reserved