Sponsored by Informatica
Data integration for open table formats
Open table formats can unlock huge benefits for IT leaders looking to get value out of their data
Any modern enterprise generates reams of data, which can be difficult to correctly store, process, and use without the right approach.
This has become even more important in the face of AI. Before any leader can begin implementing AI across their workflow, they must first ensure they’ve prepared data at the right scale – and across its entire lifecycle.
Data integration, in which data from across a business is combined into a single stream, can be a huge help here. But where do you start with this? And where do open data formats fit into this mix?
In this episode, in association with Informatica, Jane and Rory are joined by Eric Ho, senior director product management at Informatica, to discuss data integration at scale and explore the evolution of open table formats for modern data architectures and data warehousing.
Highlights
“So there's now more data, there are more data sources additionally, and more data formats that need to be supported in terms of structured, semi-structured, as well as unstructured data. Now, with the data being in the cloud, it makes it even more important that the data is secure and private and the quality of the data needs to be proper, otherwise with poor data quality, poor decisions will be made.”
“Open table formats are essentially specialized formats, which actually manage and organize data in a lakehouse architecture. And what it essentially is, is it's the logical data structures created via metadata to manage and maintain and address the physical data objects that are in the lakehouse architecture.”
“Informatica supports open table formats in multiple ways. At the end of the day, we want to ensure we support every single permutation of open table formats, but we support the different file formats, the different storage components, as well as the different catalogs. When I say the different file formats, we support Iceberg, Delta, as well as Hudi on the roadmap. In terms of storage, we support all hyperscalers, AWS, Microsoft Fabric, OneLake, as well as [Azure Data Lake Storage] (ADLS) gen two, as well as Google Cloud Storage.”
“So what open table formats are designed specifically to do is to be able to leverage processing engines to be able to process large amounts of data, like Spark, and make it available at a convenient cost. Another example or use case, why folks consider using open table is because they might need to audit their records. So for example, if you need to audit your invoices or expenses, every single update of open table formats creates a snapshot in time, so snapshots can be retrieved easily in the lakehouse to be able to answer the audit questions.”
Footnotes
- Structured vs unstructured data management
- Only a handful of generative AI projects make it into production – here’s why
- A quarter of firms still don’t have a formal data strategy – and it’s hampering AI adoption
- What is Delta Lake in Databricks?
- Why your business needs data protection policies
Subscribe
- Subscribe to The IT Pro Podcast on Apple Podcasts
- Subscribe to The IT Pro Podcast on Spotify
- Subscribe to the IT Pro newsletter
- Join us on LinkedIn
Sign up today and you will receive a free copy of our Future Focus 2026 report - the leading resource for IT decision-maker insight on priorities and investment areas in AI, security and more.
ITPro is a global business technology website providing the latest news, analysis, and business insight for IT decision-makers. Whether it's cyber security, cloud computing, IT infrastructure, or business strategy, we aim to equip leaders with the data they need to make informed IT investments.
For regular updates delivered to your inbox and social feeds, be sure to sign up to our daily newsletter and follow on us LinkedIn and Twitter.
-
TfL takes Oyster photocard systems offline following breachNews Security experts warn the Oyster photocard breach could lead to follow-up attacks on victims
-
Irish credit unions hit by data breach after IT provider compromisedNews Wellington IT reportedly delayed alerting customers by multiple weeks
-
Upgrading mission-critical services with AI, without causing disruptionPodcast NHS 24 CIO Ann-Marie Gallacher explains how the service was able to modernize without jeopardising critical patient care
-
What UK tech can learn from a decentralized governmentPodcast RSM head of technology industry, Ben Bilsland, talks to ITPro about the state of the UK's startup ecosystem
-
SPECIAL EDITION: TD SYNNEX talks about attaching services to productsSPONSORED PODCAST
-
SPECIAL EDITION: TD SYNNEX talks MSP Evolve™Sponsored podcast
-
Google's Spirit Airlines auction & TrendAI insider threat reportPodcast Google's Spirit Airlines auction & TrendAI insider threat report
-
Why doesn't more data produce better results?ITPro Podcast Riverbed CIO Fernando Castanheira talks about data and how businesses can use it better
-
HPE Discover & Pure AcceleratePodcast Live from Las Vegas for HPE and Everpure's annual conferences
-
The future of employment and gentlemanly hackersFeature Will the next IT jobs be in data center maintenance?