<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
     xmlns:content="http://purl.org/rss/1.0/modules/content/"
     xmlns:dc="http://purl.org/dc/elements/1.1/"
     xmlns:dcterms="http://purl.org/dc/terms/"
     xmlns:media="http://search.yahoo.com/mrss/"
     xmlns:atom="http://www.w3.org/2005/Atom"
     xmlns:cf="https://www.futureplc.com/rss/content-flags"
>
    <channel>
                    <atom:link href="https://www.itpro.com/feeds/tag/deep-learning" rel="self" type="application/rss+xml" />
                            <title><![CDATA[ Latest from ITPro in Deep-learning ]]></title>
                <link>https://www.itpro.com/data-analytics/big-data/deep-learning</link>
        <description><![CDATA[ All the latest deep-learning content from the ITPro team ]]></description>
                                    <lastBuildDate>Wed, 20 Sep 2023 14:23:27 +0000</lastBuildDate>
                            <language>en</language>
                                <item>
                                                            <title><![CDATA[ Dell Technologies and Starburst announce collaboration on new data lakehouse platform and query engine ]]></title>
                                                                                                <dc:content><![CDATA[ <p>Dell Technologies and data analytics platform Starburst have announced a new partnership, with the intention of building an advanced data lakehouse solution for better oversight and control of enterprise data.</p><p>The initiative, announced at Big Data London, will use Dell’s storage expertise and Starburst’s engine to allow on-demand access to decentralized data.</p><p>Customers will then be able to federate and activate data around this lakehouse from a single point of access, which the firms hope will enable more detailed data analysis and for customers to have more oversight of training for <a href="https://www.itpro.com/strategy/28181/what-is-ai"><u>artificial intelligence (AI)</u></a> and <a href="https://www.itpro.com/strategy/28071/what-is-machine-learning"><u>machine learning (ML)</u></a> systems.</p><p>Data lakehouses are a model that has arisen in the past few years that combine the structured and unstructured information stored in data warehouses and data lakes. They are particularly useful for performing responsive searches on raw data.</p><p>“Dell Technologies is on a journey to a data lakehouse architecture,” said Joe Steiner, CTO of unstructured data solutions at Dell Technologies.</p><p>“We have big plans, and step one on our journey is a common query engine, and that&apos;s what we&apos;re doing with Starburst.</p><p>“For far too long our customers, like you, have been bound by the limitations of proprietary databases, data lakes, and <a href="https://www.itpro.com/cloud/33424/four-business-benefits-of-cloud-data-warehousing"><u>data warehouses</u></a>. My personal feeling is that&apos;s going to come to an end. An open ecosystem is emerging, and my customers want these open ecosystem capabilities.</p><p>“We&apos;re co-engineering solutions, and we&apos;re going to deliver some incredible capabilities very soon."</p><p>Rick DeMare, global business development leader at Starburst said that his firm’s engine will “sit on top” of Dell’s data lakehouse, with the aim of giving customers warehouse-like speed over all the forms of data contained within. This will also allow customers to federate and activate their data across the lakehouse from a single point of access.</p><div  class="fancy-box"><div class="fancy_box-title">RELATED RESOURCE</div><div class="fancy_box_body"><figure class="van-image-figure "  ><div class='image-full-width-wrapper'><div class='image-widthsetter' ><p class="vanilla-image-block" style="padding-top:56.25%;"><img id="ZaT3vdNiPeXtbnQaXub8sX" name="Equipping the Future Workspace Technology that Lifts Collaboration.jpg" caption="" alt="Equipping the Future Workspace: Technology that Lifts Collaboration" src="https://cdn.mos.cms.futurecdn.net/ZaT3vdNiPeXtbnQaXub8sX-1920-80.jpg" mos="" link="" align="" fullscreen="" width="" height="" attribution="" endorsement="" class="pinterest-pin-exclude" /></p></div></div><figcaption itemprop="caption description" class=""><span class="credit" itemprop="copyrightHolder">(Image credit: Dell)</span></figcaption></figure><p class="fancy-box__body-text"><em>Create the ideal hybrid workplace that will keep you competitive.</em></p><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/business/the-future-of-business/equipping-the-future-workspace-technology-that-lifts-collaboration">DOWNLOAD FOR FREE</a></p></div></div><p>On average, Starburst says this approach can help customer systems go 90% faster and reduce the cost of ownership by 53%. In spite of its new partnership with Dell, Starburst will maintain its vendor-agnostic approach and to this end is committed to upholding open file and table formats across its systems.</p><p>DeMare also rejected the use of the phrase ‘single source of truth’ used by competitors, describing it as a “single source of lies”.</p><p>"It&apos;s a mess, it&apos;s never been more of a mess,” he said.</p><p>DeMare cited a report by S&amp;P Global Market Intelligence which found that on average, firms now maintain 5.4 copies of data between their cloud environments and on-premise data.</p><p>He criticized solutions that bill themselves as new technologies but are in effect just data silos, including data lakes, and argued that existing ‘single source of truth’ architecture produces monolithic, closed systems that are expensive to scale.</p><p>Easier access to data lakehouses could work to address CIO concerns over cloud complexity. A recent study by Dynatrace found that 47% of CIOs were in favor of more lakehouse structures, to enable greater use of <a href="https://www.itpro.com/technology/artificial-intelligence/slack-says-automation-can-save-every-employee-a-month-of-work-per-year"><u>automation</u></a>.</p><h2 id="prepping-for-ai-and-reducing-cio-strain">Prepping for AI, and reducing CIO strain</h2><p>Steiner and DeMare made the announcement on the keynote stage at Big Data London 2023, an event this year dominated by strategies and solutions aimed at organizing business data for use in AI and ML applications.</p><p>The <a href="https://www.itpro.com/technology/artificial-intelligence/are-we-in-the-middle-of-a-generative-ai-bubble"><u>explosion of interest in generative AI</u></a>, in particular, has put new demands on data teams. Large language models (LLMs) require vast swathes of curated data to function optimally, which requires firms to have a good grip on both structured and unstructured data, and oversight of which data is being used for AI systems to ensure privacy, security, and safety is upheld.</p><p>At Dell Technologies World 2023, Dell Technologies global CTO John Roese <a href="https://www.itpro.com/technology/artificial-intelligence/dell-cto-ai-nothing-compared-to-the-oncoming-quantum-storm"><u>told </u><u><em>ITPro</em></u></a> that curation of data was the most important factor for making any LLM work correctly. </p><p>Dell’s own effort to remove non-inclusive language such as ‘whitelist’ and ‘blacklist’ from its content repository, for example, would allow for an AI to be trained on the firm’s internal code without fears of unwanted biases appearing in output.</p><p>Roese also pointed to the fact that <a href="https://www.itpro.com/network-internet/29791/what-is-an-artificial-neural-network"><u>neural networks</u></a> can make faster connections between data that is unlabelled, as human labels may be seen as unnecessary or arbitrary. In this regard, Dell and Starburst’s data lakehouse could have an advantage over competitors in that it allows firms to quickly draw together data in a variety of forms.</p><p>"If everybody can access the data the same way, then they can have the fuel that they need to start working on their generative AI products,” said DeMare.</p><p>Recent <a href="https://www.itpro.com/technology/artificial-intelligence/ai-security-tools-see-mounting-investment-as-businesses-scramble-to-mitigate-generative-ais-issues"><u>Gartner research</u></a> presented evidence that IT teams at many organizations are concerned about the risks of passing their data through public AI systems run by hyperscalers such as <a href="https://www.itpro.com/cloud/cloud-computing/369875/microsoft-adds-chatgpt-to-azure-openai-support-cloud-services"><u>Azure OpenAI</u></a>, and that many firms are weighing the safety of running their own on-premise AI models against the far lower costs of public AI.</p><p>DeMare claimed Starburst and Dell&apos;s project can help companies to find and manage sensitive data to ensure they have controls over what is and is not given over to public AI firms.</p><p>"Maybe you don&apos;t want to share all your data with a hyperscaler, which is a general requirement of those generative AI tools."</p> ]]></dc:content>
                                                                                                                                            <link>https://www.itpro.com/business/business-strategy/dell-technologies-and-starburst-announce-collaboration-on-new-data-lakehouse-platform-and-query-engine</link>
                                                                            <description>
                            <![CDATA[ A senior figure at Starburst described current single source of truth repositories as “a mess” ]]>
                                                                                                            </description>
                                                                                                                                <guid isPermaLink="false">3BCJj9HbAjmQGTft27DrfD</guid>
                                                                                                <enclosure url="https://cdn.mos.cms.futurecdn.net/UXNf8tggj3Ahpd5bDXTkp3-1920-80.jpg" type="image/jpeg" length="0"></enclosure>
                                                                        <pubDate>Wed, 20 Sep 2023 14:23:27 +0000</pubDate>                                                                                                                                <updated>Thu, 19 Nov 2026 15:05:19 +0000</updated>
                                                                                                                                            <category><![CDATA[Deep Learning]]></category>
                                                    <category><![CDATA[Data and Analytics]]></category>
                                                    <category><![CDATA[Big Data]]></category>
                                                                                                <author><![CDATA[ rory.bathgate@futurenet.com (Rory Bathgate) ]]></author>                    <dc:creator><![CDATA[ Rory Bathgate ]]></dc:creator>                                                                                    <dc:source><![CDATA[ https://cdn.mos.cms.futurecdn.net/Vep5JogbPhduK7R6CUWAm6-320-70.jpg ]]></dc:source>
                                                                <dc:description><![CDATA[ &lt;p&gt;Rory Bathgate was the Features and Multimedia Editor at ITPro until 2026, overseeing all in-depth content and case studies. Throughout his time at ITPro, Rory  charted the rise in popularity of generative AI and specifically companies such as Microsoft, OpenAI, and Google. &lt;/p&gt;&lt;p&gt;Alongside this, he delved into increasing calls for ethical and responsible AI as global legislators circle the technology, as well as the latest in mobile networking technology, from 5G mmWave to the 3G sunset and how it will affect businesses.&lt;/p&gt;&lt;p&gt;He  provided coverage from high-profile tech conferences such as Dell Technologies World, SuiteWorld, and VMware Explore Europe. His on-the-ground coverage  included live blogs, extensive daily coverage of the most significant announcements, analysis pieces, and podcast.&lt;/p&gt;&lt;p&gt;Rory was the co-host of the ITPro Podcast alongside Jane McCallion, where he swapped a keyboard for a microphone to discuss the latest learnings in tech.&lt;/p&gt;&lt;p&gt;Rory has an MA in Eighteenth-Century Studies from King’s College London, as well as a BA in English and American Literature from the University of Kent. He joined ITPro in 2022 as a graduate, after four years in student journalism.&lt;/p&gt;&lt;p&gt;In his free time, Rory enjoys photography and video editing, and can often be found at the cinema or reading a good science fiction paperback.&lt;/p&gt; ]]></dc:description>
                                                                                                                                <cf:isSponsored>false</cf:isSponsored>
                <cf:hasAffiliateLinks>false</cf:hasAffiliateLinks>
                <cf:isPaid>false</cf:isPaid>
                                                                                                                                                                                    <media:content type="image/jpeg" url="https://cdn.mos.cms.futurecdn.net/UXNf8tggj3Ahpd5bDXTkp3-1920-80.jpg">
                                                            <media:credit><![CDATA[Rory Bathgate/Future]]></media:credit>
                                                                                                                                                                                                                                    <media:description><![CDATA[The Starburst logo hanging from the roof of Olympia exhibition centre in London during Big Data London]]></media:description>                                                            <media:text><![CDATA[The Starburst logo hanging from the roof of Olympia exhibition centre in London during Big Data London]]></media:text>
                                <media:title type="plain"><![CDATA[The Starburst logo hanging from the roof of Olympia exhibition centre in London during Big Data London]]></media:title>
                                                    </media:content>
                                                    <media:thumbnail url="https://cdn.mos.cms.futurecdn.net/UXNf8tggj3Ahpd5bDXTkp3-1920-80.jpg" />
                                                                                                                                                                    <content:encoded >
                            <![CDATA[
                            <article>
                                <p>Dell Technologies and data analytics platform Starburst have announced a new partnership, with the intention of building an advanced data lakehouse solution for better oversight and control of enterprise data.</p><p>The initiative, announced at Big Data London, will use Dell’s storage expertise and Starburst’s engine to allow on-demand access to decentralized data.</p><p>Customers will then be able to federate and activate data around this lakehouse from a single point of access, which the firms hope will enable more detailed data analysis and for customers to have more oversight of training for <a href="https://www.itpro.com/strategy/28181/what-is-ai"><u>artificial intelligence (AI)</u></a> and <a href="https://www.itpro.com/strategy/28071/what-is-machine-learning"><u>machine learning (ML)</u></a> systems.</p><p>Data lakehouses are a model that has arisen in the past few years that combine the structured and unstructured information stored in data warehouses and data lakes. They are particularly useful for performing responsive searches on raw data.</p><p>“Dell Technologies is on a journey to a data lakehouse architecture,” said Joe Steiner, CTO of unstructured data solutions at Dell Technologies.</p><p>“We have big plans, and step one on our journey is a common query engine, and that&apos;s what we&apos;re doing with Starburst.</p><p>“For far too long our customers, like you, have been bound by the limitations of proprietary databases, data lakes, and <a href="https://www.itpro.com/cloud/33424/four-business-benefits-of-cloud-data-warehousing"><u>data warehouses</u></a>. My personal feeling is that&apos;s going to come to an end. An open ecosystem is emerging, and my customers want these open ecosystem capabilities.</p><p>“We&apos;re co-engineering solutions, and we&apos;re going to deliver some incredible capabilities very soon."</p><p>Rick DeMare, global business development leader at Starburst said that his firm’s engine will “sit on top” of Dell’s data lakehouse, with the aim of giving customers warehouse-like speed over all the forms of data contained within. This will also allow customers to federate and activate their data across the lakehouse from a single point of access.</p><div  class="fancy-box"><div class="fancy_box-title">RELATED RESOURCE</div><div class="fancy_box_body"><figure class="van-image-figure "  ><div class='image-full-width-wrapper'><div class='image-widthsetter' ><p class="vanilla-image-block" style="padding-top:56.25%;"><img id="ZaT3vdNiPeXtbnQaXub8sX" name="Equipping the Future Workspace Technology that Lifts Collaboration.jpg" caption="" alt="Equipping the Future Workspace: Technology that Lifts Collaboration" src="https://cdn.mos.cms.futurecdn.net/ZaT3vdNiPeXtbnQaXub8sX-1920-80.jpg" mos="" link="" align="" fullscreen="" width="" height="" attribution="" endorsement="" class="pinterest-pin-exclude" /></p></div></div><figcaption itemprop="caption description" class=""><span class="credit" itemprop="copyrightHolder">(Image credit: Dell)</span></figcaption></figure><p class="fancy-box__body-text"><em>Create the ideal hybrid workplace that will keep you competitive.</em></p><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/business/the-future-of-business/equipping-the-future-workspace-technology-that-lifts-collaboration">DOWNLOAD FOR FREE</a></p></div></div><p>On average, Starburst says this approach can help customer systems go 90% faster and reduce the cost of ownership by 53%. In spite of its new partnership with Dell, Starburst will maintain its vendor-agnostic approach and to this end is committed to upholding open file and table formats across its systems.</p><p>DeMare also rejected the use of the phrase ‘single source of truth’ used by competitors, describing it as a “single source of lies”.</p><p>"It&apos;s a mess, it&apos;s never been more of a mess,” he said.</p><p>DeMare cited a report by S&amp;P Global Market Intelligence which found that on average, firms now maintain 5.4 copies of data between their cloud environments and on-premise data.</p><p>He criticized solutions that bill themselves as new technologies but are in effect just data silos, including data lakes, and argued that existing ‘single source of truth’ architecture produces monolithic, closed systems that are expensive to scale.</p><p>Easier access to data lakehouses could work to address CIO concerns over cloud complexity. A recent study by Dynatrace found that 47% of CIOs were in favor of more lakehouse structures, to enable greater use of <a href="https://www.itpro.com/technology/artificial-intelligence/slack-says-automation-can-save-every-employee-a-month-of-work-per-year"><u>automation</u></a>.</p><h2 id="prepping-for-ai-and-reducing-cio-strain">Prepping for AI, and reducing CIO strain</h2><p>Steiner and DeMare made the announcement on the keynote stage at Big Data London 2023, an event this year dominated by strategies and solutions aimed at organizing business data for use in AI and ML applications.</p><p>The <a href="https://www.itpro.com/technology/artificial-intelligence/are-we-in-the-middle-of-a-generative-ai-bubble"><u>explosion of interest in generative AI</u></a>, in particular, has put new demands on data teams. Large language models (LLMs) require vast swathes of curated data to function optimally, which requires firms to have a good grip on both structured and unstructured data, and oversight of which data is being used for AI systems to ensure privacy, security, and safety is upheld.</p><p>At Dell Technologies World 2023, Dell Technologies global CTO John Roese <a href="https://www.itpro.com/technology/artificial-intelligence/dell-cto-ai-nothing-compared-to-the-oncoming-quantum-storm"><u>told </u><u><em>ITPro</em></u></a> that curation of data was the most important factor for making any LLM work correctly. </p><p>Dell’s own effort to remove non-inclusive language such as ‘whitelist’ and ‘blacklist’ from its content repository, for example, would allow for an AI to be trained on the firm’s internal code without fears of unwanted biases appearing in output.</p><p>Roese also pointed to the fact that <a href="https://www.itpro.com/network-internet/29791/what-is-an-artificial-neural-network"><u>neural networks</u></a> can make faster connections between data that is unlabelled, as human labels may be seen as unnecessary or arbitrary. In this regard, Dell and Starburst’s data lakehouse could have an advantage over competitors in that it allows firms to quickly draw together data in a variety of forms.</p><p>"If everybody can access the data the same way, then they can have the fuel that they need to start working on their generative AI products,” said DeMare.</p><p>Recent <a href="https://www.itpro.com/technology/artificial-intelligence/ai-security-tools-see-mounting-investment-as-businesses-scramble-to-mitigate-generative-ais-issues"><u>Gartner research</u></a> presented evidence that IT teams at many organizations are concerned about the risks of passing their data through public AI systems run by hyperscalers such as <a href="https://www.itpro.com/cloud/cloud-computing/369875/microsoft-adds-chatgpt-to-azure-openai-support-cloud-services"><u>Azure OpenAI</u></a>, and that many firms are weighing the safety of running their own on-premise AI models against the far lower costs of public AI.</p><p>DeMare claimed Starburst and Dell&apos;s project can help companies to find and manage sensitive data to ensure they have controls over what is and is not given over to public AI firms.</p><p>"Maybe you don&apos;t want to share all your data with a hyperscaler, which is a general requirement of those generative AI tools."</p>
                                                            </article>
                            ]]>
                        </content:encoded>
                                                </item>
                                <item>
                                                            <title><![CDATA[ Databricks injects array of AI tools into Lakehouse ]]></title>
                                                                                                <dc:content><![CDATA[ <p>Databricks has launched a raft of new features for its flagship Lakehouse data lake platform, including <a href="https://www.itpro.com/strategy/28181/what-is-ai">artificial intelligence (AI)</a> and natural language improvements as well as <a href="https://www.itpro.com/policy-legislation/data-governance/369834/building-a-data-governance-strategy">data governance</a> features.</p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/business/data-and-insights/what-is-delta-lake">What is Delta Lake in Databricks?</a></p></div></div><p>With Lakehouse AI, customers can develop <a href="https://www.itpro.com/technology/artificial-intelligence-ai/369959/what-is-generative-ai">generative AI</a> applications, including large language models (LLMs), within the platform itself, with tools spanning the breadth of the AI lifecycle. This includes <a href="https://www.itpro.com/strategy/28185/what-is-data-mining">data collection</a>, model development, and monitoring.</p><p>“This gives you control over your intellectual property,” said Databricks CEO Ali Ghodsi during his keynote address at Databricks Data + AI Summit 2023. He added every company in the next five years “will be an AI and data company” and would need the tools to create their own LLMs to be competitive, rather than rely on third-party services.</p><h2 id="what-is-lakehouseiq">What is LakehouseIQ?</h2><p><strong>LakehouseIQ </strong>is the firm’s ‘knowledge engine’ that’s designed to make natural language interfaces, like chat bots, more accurate and relevant to the needs of a particular organization.</p><p>Sitting as a fabric between data and a front-end interface, the tool uses generative AI to understand a business’ unique culture, jargon, and data usage patterns, among other factors.</p><div  class="fancy-box"><div class="fancy_box-title">RELATED RESOURCE</div><div class="fancy_box_body"><figure class="van-image-figure "  ><div class='image-full-width-wrapper'><div class='image-widthsetter' ><p class="vanilla-image-block" style="padding-top:56.25%;"><img id="oFGNRqZmBsfWMRvsmW3rdg" name="Modern storage The answer to multi-cloud complexity_listing.jpg" caption="" alt="Whitepaper cover with title below an image of a cityscape with the sunrise in the background" src="https://cdn.mos.cms.futurecdn.net/oFGNRqZmBsfWMRvsmW3rdg-1920-80.jpg" mos="" link="" align="" fullscreen="" width="" height="" attribution="" endorsement="" class="pinterest-pin-exclude" /></p></div></div><figcaption itemprop="caption description" class=""><span class="credit" itemprop="copyrightHolder">(Image credit: Dell)</span></figcaption></figure><p class="fancy-box__body-text"><strong>Modern storage: The answer to multi-cloud complexity</strong></p><p class="fancy-box__body-text"><em>Innovative organizations need innovative storage to manage and leverage their data no matter where it lives</em></p><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/cloud/cloud-hosting/370149/modern-storage-the-answer-to-multi-cloud-complexity"><strong>DOWNLOAD FOR FREE</strong></a></p></div></div><p>It ingests a vast quantity of internal material, including organizational charts, notebooks, and more, in order to understand how best to answer questions users ask via any chat interface.</p><p>It’s designed to be specific to each business, so natural language interactions are far more accurate and context-sensitive than classic <a href="https://www.itpro.com/networking/27171/what-is-a-chatbot">chat bots</a>. While not a chat bot or <a href="https://www.itpro.com/technology/voice-assistant/367610/future-of-virtual-assistants-lies-in-the-metaverse">virtual assistant</a> itself, it’s designed to improve the reliability and performance of these tools.</p><p>Databricks has made Lakehouse IQ available as an <a href="https://www.itpro.com/application-programming-interface-api/33557/the-api-economy-what-your-business-needs-to-know">API</a>, and the firm plans to integrate this into <a href="https://www.itpro.com/software/28109/what-is-open-source">open source</a> libraries.</p><h2 id="new-ai-features-in-lakehouse">New AI features in Lakehouse</h2><p><strong>Vector Search</strong> gives developers the capacity to improve accuracy of generative AI responses. The features creates vector embeddings based on files found in Unity Catalog — the central hub that gives customers access control, auditing, lineage and data discovery capabilities across their workspaces.</p><p>It creates vector embeddings from Unity Catalog files, and automatically updates them through integrations with Databricks Model Serving. Developers can also add query filters to provide better outcomes.</p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/business/acquisition/databricks-ceo-rebuffs-claims-of-generative-ai-bubble-following-dollar13bn-mosaicml-acquisition">Databricks CEO rebuffs claims of ‘generative AI bubble’</a></p></div></div><p>Improvements to <strong>AutoML</strong> also adds low-code functionality to finetuning LLMs. Any tweaks customers make to their LLMs will be owned by the enterprise, with data kept in-house and not shared with any third parties. This model can also be sahred within an organization with Model Serving, Unity Catalog and MLflow integrations.</p><p>Finally, Databricks has launched several open source models through its marketplace including MosaicML’s MPT-7B and Hugging Face’s Falcon-7B, as well as Stable Diffusion.</p><p>On the LLMOps front, <strong>MLflow AI Gateway</strong> lets customers centrally manage credentials for software as a <a href="https://www.itpro.com/cloud/software-as-a-service-saas/362655/what-is-saas">service (SaaS)</a> models or APIs, and provide access-controlled querying. This also allows for prediction caching to track repeat prompts and rate limiting to reduce costs.</p><p><strong>MLflow Prompt Tools</strong>, meanwhile, is a no-code feature that allows users to compare different outputs from various models based on prompts that are automatically tracked within MLflow.</p><h2 id="new-governance-features-in-lakehouse">New governance features in Lakehouse</h2><p>Also new in Lakehouse is <strong>LakeHouse Federation</strong> capabilities, which has broken down silos between data systems that previously existed on the platform. As a result, businesses are now able to discover, query and govern data across all their systems without the need to copy or move data. </p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/policy-legislation/data-governance/369834/building-a-data-governance-strategy">Building a data governance strategy in 2023</a></p></div></div><p>Query federation comprises catalog and querying functionality to consolidate and map out data asses from different platforms beyond the confines of Databricks, including MySQL, PostgreSQL, Redshift, Snowflake, Azure AQL Database, Azure Synapse, BigQuery, and others. Users can secure, audit and access data from one interface rather than having to move the data between different platforms.</p><p>Additional governance tools in Unity Catalog also lets customers apply access policies on any data asset that’s registered, including tables, rows, columns and tags. This comes ahead of future plans to define data access policies and push them out to other data warehouses.</p><p>The Lakehouse Federation&apos;s capabilities also include giving users a common approach to discovering and exploring both <a href="https://www.itpro.com/big-data-analytics/34532/structured-vs-unstructured-data-management">structured and unstructured data</a> with a single query. Domain-owned data sources for <a href="https://www.itpro.com/business-strategy/28163/what-is-big-data-analytics">analytics</a> and AI use cases are also rapidly exposed, offering faster access to data. Finally, the changes make it possible for users to apply a single permission model for the data estate.</p> ]]></dc:content>
                                                                                                                                            <link>https://www.itpro.com/business/data-and-insights/databricks-injects-array-of-ai-tools-into-lakehouse</link>
                                                                            <description>
                            <![CDATA[ Lakehouse IQ and Lakehouse AI, businesses can build better internal chatbots and create their own LLMs ]]>
                                                                                                            </description>
                                                                                                                                <guid isPermaLink="false">QQuhTAVyN5CCoDdZC9o3nF</guid>
                                                                                                <enclosure url="https://cdn.mos.cms.futurecdn.net/hThy9A2PfEPFvv4T9FqV5X-1920-80.jpg" type="image/jpeg" length="0"></enclosure>
                                                                        <pubDate>Thu, 29 Jun 2023 01:39:50 +0000</pubDate>                                                                                                                                <updated>Thu, 19 Nov 2026 13:16:03 +0000</updated>
                                                                                                                                            <category><![CDATA[Deep Learning]]></category>
                                                    <category><![CDATA[Data and Insights]]></category>
                                                    <category><![CDATA[Data and Analytics]]></category>
                                                    <category><![CDATA[Big Data]]></category>
                                                    <category><![CDATA[Business]]></category>
                                                                                                <author><![CDATA[ keumars.afifi-sabet@futurenet.com (Keumars Afifi-Sabet) ]]></author>                    <dc:creator><![CDATA[ Keumars Afifi-Sabet ]]></dc:creator>                                                                                    <dc:source><![CDATA[ https://cdn.mos.cms.futurecdn.net/EAvwpZggMZ2K5h8s2pTAEm-320-70.jpg ]]></dc:source>
                                                                <dc:description><![CDATA[ null ]]></dc:description>
                                                                                                                                <cf:isSponsored>false</cf:isSponsored>
                <cf:hasAffiliateLinks>false</cf:hasAffiliateLinks>
                <cf:isPaid>false</cf:isPaid>
                                                                                                                                                                                    <media:content type="image/jpeg" url="https://cdn.mos.cms.futurecdn.net/hThy9A2PfEPFvv4T9FqV5X-1920-80.jpg">
                                                            <media:credit><![CDATA[Getty Images]]></media:credit>
                                                                                                                                                                                                                                    <media:description><![CDATA[The Databricks logo on a Macbook Pro]]></media:description>                                                            <media:text><![CDATA[The Databricks logo on a Macbook Pro]]></media:text>
                                <media:title type="plain"><![CDATA[The Databricks logo on a Macbook Pro]]></media:title>
                                                    </media:content>
                                                    <media:thumbnail url="https://cdn.mos.cms.futurecdn.net/hThy9A2PfEPFvv4T9FqV5X-1920-80.jpg" />
                                                                                                                                                                    <content:encoded >
                            <![CDATA[
                            <article>
                                <p>Databricks has launched a raft of new features for its flagship Lakehouse data lake platform, including <a href="https://www.itpro.com/strategy/28181/what-is-ai">artificial intelligence (AI)</a> and natural language improvements as well as <a href="https://www.itpro.com/policy-legislation/data-governance/369834/building-a-data-governance-strategy">data governance</a> features.</p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/business/data-and-insights/what-is-delta-lake">What is Delta Lake in Databricks?</a></p></div></div><p>With Lakehouse AI, customers can develop <a href="https://www.itpro.com/technology/artificial-intelligence-ai/369959/what-is-generative-ai">generative AI</a> applications, including large language models (LLMs), within the platform itself, with tools spanning the breadth of the AI lifecycle. This includes <a href="https://www.itpro.com/strategy/28185/what-is-data-mining">data collection</a>, model development, and monitoring.</p><p>“This gives you control over your intellectual property,” said Databricks CEO Ali Ghodsi during his keynote address at Databricks Data + AI Summit 2023. He added every company in the next five years “will be an AI and data company” and would need the tools to create their own LLMs to be competitive, rather than rely on third-party services.</p><h2 id="what-is-lakehouseiq">What is LakehouseIQ?</h2><p><strong>LakehouseIQ </strong>is the firm’s ‘knowledge engine’ that’s designed to make natural language interfaces, like chat bots, more accurate and relevant to the needs of a particular organization.</p><p>Sitting as a fabric between data and a front-end interface, the tool uses generative AI to understand a business’ unique culture, jargon, and data usage patterns, among other factors.</p><div  class="fancy-box"><div class="fancy_box-title">RELATED RESOURCE</div><div class="fancy_box_body"><figure class="van-image-figure "  ><div class='image-full-width-wrapper'><div class='image-widthsetter' ><p class="vanilla-image-block" style="padding-top:56.25%;"><img id="oFGNRqZmBsfWMRvsmW3rdg" name="Modern storage The answer to multi-cloud complexity_listing.jpg" caption="" alt="Whitepaper cover with title below an image of a cityscape with the sunrise in the background" src="https://cdn.mos.cms.futurecdn.net/oFGNRqZmBsfWMRvsmW3rdg-1920-80.jpg" mos="" link="" align="" fullscreen="" width="" height="" attribution="" endorsement="" class="pinterest-pin-exclude" /></p></div></div><figcaption itemprop="caption description" class=""><span class="credit" itemprop="copyrightHolder">(Image credit: Dell)</span></figcaption></figure><p class="fancy-box__body-text"><strong>Modern storage: The answer to multi-cloud complexity</strong></p><p class="fancy-box__body-text"><em>Innovative organizations need innovative storage to manage and leverage their data no matter where it lives</em></p><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/cloud/cloud-hosting/370149/modern-storage-the-answer-to-multi-cloud-complexity"><strong>DOWNLOAD FOR FREE</strong></a></p></div></div><p>It ingests a vast quantity of internal material, including organizational charts, notebooks, and more, in order to understand how best to answer questions users ask via any chat interface.</p><p>It’s designed to be specific to each business, so natural language interactions are far more accurate and context-sensitive than classic <a href="https://www.itpro.com/networking/27171/what-is-a-chatbot">chat bots</a>. While not a chat bot or <a href="https://www.itpro.com/technology/voice-assistant/367610/future-of-virtual-assistants-lies-in-the-metaverse">virtual assistant</a> itself, it’s designed to improve the reliability and performance of these tools.</p><p>Databricks has made Lakehouse IQ available as an <a href="https://www.itpro.com/application-programming-interface-api/33557/the-api-economy-what-your-business-needs-to-know">API</a>, and the firm plans to integrate this into <a href="https://www.itpro.com/software/28109/what-is-open-source">open source</a> libraries.</p><h2 id="new-ai-features-in-lakehouse">New AI features in Lakehouse</h2><p><strong>Vector Search</strong> gives developers the capacity to improve accuracy of generative AI responses. The features creates vector embeddings based on files found in Unity Catalog — the central hub that gives customers access control, auditing, lineage and data discovery capabilities across their workspaces.</p><p>It creates vector embeddings from Unity Catalog files, and automatically updates them through integrations with Databricks Model Serving. Developers can also add query filters to provide better outcomes.</p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/business/acquisition/databricks-ceo-rebuffs-claims-of-generative-ai-bubble-following-dollar13bn-mosaicml-acquisition">Databricks CEO rebuffs claims of ‘generative AI bubble’</a></p></div></div><p>Improvements to <strong>AutoML</strong> also adds low-code functionality to finetuning LLMs. Any tweaks customers make to their LLMs will be owned by the enterprise, with data kept in-house and not shared with any third parties. This model can also be sahred within an organization with Model Serving, Unity Catalog and MLflow integrations.</p><p>Finally, Databricks has launched several open source models through its marketplace including MosaicML’s MPT-7B and Hugging Face’s Falcon-7B, as well as Stable Diffusion.</p><p>On the LLMOps front, <strong>MLflow AI Gateway</strong> lets customers centrally manage credentials for software as a <a href="https://www.itpro.com/cloud/software-as-a-service-saas/362655/what-is-saas">service (SaaS)</a> models or APIs, and provide access-controlled querying. This also allows for prediction caching to track repeat prompts and rate limiting to reduce costs.</p><p><strong>MLflow Prompt Tools</strong>, meanwhile, is a no-code feature that allows users to compare different outputs from various models based on prompts that are automatically tracked within MLflow.</p><h2 id="new-governance-features-in-lakehouse">New governance features in Lakehouse</h2><p>Also new in Lakehouse is <strong>LakeHouse Federation</strong> capabilities, which has broken down silos between data systems that previously existed on the platform. As a result, businesses are now able to discover, query and govern data across all their systems without the need to copy or move data. </p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/policy-legislation/data-governance/369834/building-a-data-governance-strategy">Building a data governance strategy in 2023</a></p></div></div><p>Query federation comprises catalog and querying functionality to consolidate and map out data asses from different platforms beyond the confines of Databricks, including MySQL, PostgreSQL, Redshift, Snowflake, Azure AQL Database, Azure Synapse, BigQuery, and others. Users can secure, audit and access data from one interface rather than having to move the data between different platforms.</p><p>Additional governance tools in Unity Catalog also lets customers apply access policies on any data asset that’s registered, including tables, rows, columns and tags. This comes ahead of future plans to define data access policies and push them out to other data warehouses.</p><p>The Lakehouse Federation&apos;s capabilities also include giving users a common approach to discovering and exploring both <a href="https://www.itpro.com/big-data-analytics/34532/structured-vs-unstructured-data-management">structured and unstructured data</a> with a single query. Domain-owned data sources for <a href="https://www.itpro.com/business-strategy/28163/what-is-big-data-analytics">analytics</a> and AI use cases are also rapidly exposed, offering faster access to data. Finally, the changes make it possible for users to apply a single permission model for the data estate.</p>
                                                            </article>
                            ]]>
                        </content:encoded>
                                                </item>
                                <item>
                                                            <title><![CDATA[ What's new in Delta Lake 3.0? ]]></title>
                                                                                                <dc:content><![CDATA[ <p>Delta Lake is an open source framework designed by Databricks that allows businesses to manage and analyze the data they generate in real-time. </p><p>Taking the form of a transactional storage layer, it provides a foundation for storing data and tables on Data Lakehouses – which Databricks refers to as an open data management architecture. This allows for data warehousing and <a href="https://www.itpro.com/strategy/28071/what-is-machine-learning"><u>machine learning</u></a> operations to be applied directly onto a data lake.</p><p>It’s the default storage format for all Databricks operations and is an open protocol, with the organization completely open sourcing Delta Lake with the launch of <a href="https://www.itpro.com/development/open-source/368384/databricks-announces-major-contributions-to-flagship-open-source">Delta Lake 2.0</a> in June 2022. </p><p>With Delta Lake 3.0, launched at Databricks Data + AI Summit 2023, Databricks has launched a new unified file format – UniForm – that solves a longstanding compatibility issue in the way businesses store metadata in their data lakes. This launches alongside a bid to remove data silos by addressing connector fragmentation, and performance improvements.</p><h2 class="article-body__section" id="section-what-is-delta-lake"><span>What is Delta Lake?</span></h2><p>Introduced as an open source project in 2019, Delta Lake was created to ensure the data that powers real-time insights that organizations crave is genuinely reliable. It was billed at the time as Databricks’ most important innovation, even including the Apache Spark engine for <a href="https://www.itpro.com/business-operations/31681/what-is-data-processing"><u>data processing</u></a>.</p><div  class="fancy-box"><div class="fancy_box-title">RELATED RESOURCE</div><div class="fancy_box_body"><figure class="van-image-figure "  ><div class='image-full-width-wrapper'><div class='image-widthsetter' ><p class="vanilla-image-block" style="padding-top:56.25%;"><img id="oFGNRqZmBsfWMRvsmW3rdg" name="Modern storage The answer to multi-cloud complexity_listing.jpg" caption="" alt="Whitepaper cover with title below an image of a cityscape with the sunrise in the background" src="https://cdn.mos.cms.futurecdn.net/oFGNRqZmBsfWMRvsmW3rdg-1920-80.jpg" mos="" link="" align="" fullscreen="" width="" height="" attribution="" endorsement="" class="pinterest-pin-exclude" /></p></div></div><figcaption itemprop="caption description" class=""><span class="credit" itemprop="copyrightHolder">(Image credit: Dell)</span></figcaption></figure><p class="fancy-box__body-text"><strong>Modern storage: The answer to multi-cloud complexity</strong></p><p class="fancy-box__body-text"><em>Innovative organizations need innovative storage to manage and leverage their data no matter where it lives</em></p><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/cloud/cloud-hosting/370149/modern-storage-the-answer-to-multi-cloud-complexity"><strong>DOWNLOAD FOR FREE</strong></a></p></div></div><p>The need for Delta Lake has emerged from the way businesses have traditionally managed data in massive and unrefined data lakes. Because these repositories contain both <a href="https://www.itpro.com/big-data-analytics/34532/structured-vs-unstructured-data-management">structured and unstructured data</a>, at different scales, any operations running on top of this data – such as <a href="https://www.itpro.com/business-strategy/data-insights/369981/machine-learning-vs-data-science-whats-the-difference">data analytics or machine learning</a> – might not be optimized.</p><p>Databricks designed Delta Lake to transform these messy data lakes into cleaner ‘Delta Lakes’ with higher-quality data. This ultimately ensures any additional processes or operations performed on top generate much better insights, even if performed in real-time.</p><p>“Nearly every company has a data lake they are trying to gain insights from, but data lakes have proven to lack data reliability,” said Databricks cofounder and CEO, Ali Ghodsi. “Delta Lake has eliminated these challenges for hundreds of enterprises. By making Delta Lake open source, developers will be able to easily build reliable data lakes and turn them into ‘Delta Lakes’.”</p><p>Delta Lake competes with a number of other services on the market that perform similar functions, including Apache HUDI, Azure Data Lake, and Snowflake.</p><h2 class="article-body__section" id="section-how-does-delta-lake-work"><span>How does Delta Lake work?</span></h2><p>Based in the cloud, Delta Lake is a Spark proprietary extension that allows organizations to create Delta tables by default, whether they’re using Apache Spark or <a href="https://www.itpro.com/sql/30242/what-is-sql"><u>SQL</u></a>. The open source platform extends the classic parquet data file format with a file-based transaction log, which allows for additional functions. </p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/data-insights/sql/360642/14-best-sql-courses">The best SQL courses</a></p></div></div><p>Delta Lake features include providing atomicity, consistency, isolation and durability (ACID) transactions on Spark, as well as scalable metadata handling. The former ensures there’s never inconsistent data, while the latter takes advantage of Spark’s distributed processing power to handle masses of metadata at once.</p><p>Organizations can also take advantage of streaming and batch unification – by allowing both types of data to land in the same sink. This comes alongside some quality-of-life features such as rollbacks and historical audit trails, as well as the capability to reproduce machine learning experiments.</p><p>It comes with a variety of open source connectors, including Apache Flink, Presto and Trino, alongside standalone readers/writers that lets clients for <a href="https://www.itpro.com/business-strategy/careers-training/356640/how-to-become-a-python-software-developer"><u>Python</u></a>, Ruby and <a href="https://www.itpro.com/security/ransomware/368476/why-are-ransomware-gangs-pivoting-to-rust"><u>Rust</u></a> write directly into Delta Lake without needing a massive data engine like Apache Spark.</p><p>In 2021, Databricks launched Delta Sharing, a protocol that lets different companies share massive data sets securely and in real time. This aims to break the notion of vendor lock-in, as well as break down data silos due to proprietary data formats and computing resources that are required to read data. Users can access shared data through Pandas, Tableau, Presto, and other platforms without the need to use proprietary systems.</p><h2 class="article-body__section" id="section-what-s-new-in-delta-lake-3-0"><span>What’s new in Delta Lake 3.0?</span></h2><p>With the latest iteration of Delta Lake, Databricks is hoping to end the ‘format wars’ that’s long posed challenges for storing and managing data in data lakes. </p><p>Most data (99%) is stored in data lakes using the parquet format, but organizations must choose from one of several competing standards to store all metadata: Delta Lake, Apache Iceberg, and Apache Hudi. Each format handles metadata differently are fundamentally incompatible with one another. Once enterprises choose a format, they’re stuck using it for everything.</p><p>With Delta Lake 3.0, Databricks has created the <strong>universal format (UniForm)</strong>, which serves as a unification of the three standards. Customers who use a Databricks data lake will find their data stored in the parquet format, as usual, alongside three different versions of the metadata. It means the Delta Lake data can be read as if it’s stored in either Iceberg or Hudi.</p><p>“The metadata is very important, and if you get the metadata wrong, you can’t actually access this stuff,” says Databricks CEO Ali Ghodsi, speaking at Databricks AI + Data Summit 2023. “Since all three projects are open source, we just went and understood exactly how to do it in each of them, and now inside Databricks, when we create data, we create data for all three.”</p><p>Also new in the latest iteration of Delta Lake is the <strong>Delta Kernel</strong>, which addresses connector fragmentation by ensuring connectors are built against a core library that implements Delta specifications. This alleviates the burden of having to update Delta connectors with each new version or protocol change. With one API, developers can keep connectors up-to-date and ensure the latest innovations are pushed out as soon as they’re ready.</p><p>The final new addition is the <strong>Delta Liquid Clustering</strong>, which boosts the performance of reads and writes. With a flexible data layout technique, the Delta Liquid Clustering marks a departure from the decades old hive-style table partitioning system, which uses a fixed layout. It means organizations can take advantage of cost-efficient data clustering as their data lake grows with time.</p> ]]></dc:content>
                                                                                                                                            <link>https://www.itpro.com/business/data-and-insights/what-is-delta-lake</link>
                                                                            <description>
                            <![CDATA[ With Delta Lake version 3.0, Databricks has created a new universal format to resolve compatibility issues ]]>
                                                                                                            </description>
                                                                                                                                <guid isPermaLink="false">dKg35j9vvZYYDvkJoCwii4</guid>
                                                                                                <enclosure url="https://cdn.mos.cms.futurecdn.net/irtxTXY3tMz7nz8XnNTA3Y-1920-80.jpg" type="image/jpeg" length="0"></enclosure>
                                                                        <pubDate>Wed, 28 Jun 2023 13:00:55 +0000</pubDate>                                                                                                                                <updated>Thu, 19 Nov 2026 13:16:04 +0000</updated>
                                                                                                                                            <category><![CDATA[Deep Learning]]></category>
                                                    <category><![CDATA[Data and Insights]]></category>
                                                    <category><![CDATA[Data and Analytics]]></category>
                                                    <category><![CDATA[Big Data]]></category>
                                                    <category><![CDATA[Business]]></category>
                                                                                                <author><![CDATA[ keumars.afifi-sabet@futurenet.com (Keumars Afifi-Sabet) ]]></author>                    <dc:creator><![CDATA[ Keumars Afifi-Sabet ]]></dc:creator>                                                                                    <dc:source><![CDATA[ https://cdn.mos.cms.futurecdn.net/EAvwpZggMZ2K5h8s2pTAEm-320-70.jpg ]]></dc:source>
                                                                <dc:description><![CDATA[ null ]]></dc:description>
                                                                                                                                <cf:isSponsored>false</cf:isSponsored>
                <cf:hasAffiliateLinks>false</cf:hasAffiliateLinks>
                <cf:isPaid>false</cf:isPaid>
                                                                                                                                                                                    <media:content type="image/jpeg" url="https://cdn.mos.cms.futurecdn.net/irtxTXY3tMz7nz8XnNTA3Y-1920-80.jpg">
                                                            <media:credit><![CDATA[Getty Images]]></media:credit>
                                                                                                                                                                                                                                    <media:description><![CDATA[Big data concept stock image]]></media:description>                                                            <media:text><![CDATA[Big data concept stock image]]></media:text>
                                <media:title type="plain"><![CDATA[Big data concept stock image]]></media:title>
                                                    </media:content>
                                                    <media:thumbnail url="https://cdn.mos.cms.futurecdn.net/irtxTXY3tMz7nz8XnNTA3Y-1920-80.jpg" />
                                                                                                                                                                    <content:encoded >
                            <![CDATA[
                            <article>
                                <p>Delta Lake is an open source framework designed by Databricks that allows businesses to manage and analyze the data they generate in real-time. </p><p>Taking the form of a transactional storage layer, it provides a foundation for storing data and tables on Data Lakehouses – which Databricks refers to as an open data management architecture. This allows for data warehousing and <a href="https://www.itpro.com/strategy/28071/what-is-machine-learning"><u>machine learning</u></a> operations to be applied directly onto a data lake.</p><p>It’s the default storage format for all Databricks operations and is an open protocol, with the organization completely open sourcing Delta Lake with the launch of <a href="https://www.itpro.com/development/open-source/368384/databricks-announces-major-contributions-to-flagship-open-source">Delta Lake 2.0</a> in June 2022. </p><p>With Delta Lake 3.0, launched at Databricks Data + AI Summit 2023, Databricks has launched a new unified file format – UniForm – that solves a longstanding compatibility issue in the way businesses store metadata in their data lakes. This launches alongside a bid to remove data silos by addressing connector fragmentation, and performance improvements.</p><h2 class="article-body__section" id="section-what-is-delta-lake"><span>What is Delta Lake?</span></h2><p>Introduced as an open source project in 2019, Delta Lake was created to ensure the data that powers real-time insights that organizations crave is genuinely reliable. It was billed at the time as Databricks’ most important innovation, even including the Apache Spark engine for <a href="https://www.itpro.com/business-operations/31681/what-is-data-processing"><u>data processing</u></a>.</p><div  class="fancy-box"><div class="fancy_box-title">RELATED RESOURCE</div><div class="fancy_box_body"><figure class="van-image-figure "  ><div class='image-full-width-wrapper'><div class='image-widthsetter' ><p class="vanilla-image-block" style="padding-top:56.25%;"><img id="oFGNRqZmBsfWMRvsmW3rdg" name="Modern storage The answer to multi-cloud complexity_listing.jpg" caption="" alt="Whitepaper cover with title below an image of a cityscape with the sunrise in the background" src="https://cdn.mos.cms.futurecdn.net/oFGNRqZmBsfWMRvsmW3rdg-1920-80.jpg" mos="" link="" align="" fullscreen="" width="" height="" attribution="" endorsement="" class="pinterest-pin-exclude" /></p></div></div><figcaption itemprop="caption description" class=""><span class="credit" itemprop="copyrightHolder">(Image credit: Dell)</span></figcaption></figure><p class="fancy-box__body-text"><strong>Modern storage: The answer to multi-cloud complexity</strong></p><p class="fancy-box__body-text"><em>Innovative organizations need innovative storage to manage and leverage their data no matter where it lives</em></p><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/cloud/cloud-hosting/370149/modern-storage-the-answer-to-multi-cloud-complexity"><strong>DOWNLOAD FOR FREE</strong></a></p></div></div><p>The need for Delta Lake has emerged from the way businesses have traditionally managed data in massive and unrefined data lakes. Because these repositories contain both <a href="https://www.itpro.com/big-data-analytics/34532/structured-vs-unstructured-data-management">structured and unstructured data</a>, at different scales, any operations running on top of this data – such as <a href="https://www.itpro.com/business-strategy/data-insights/369981/machine-learning-vs-data-science-whats-the-difference">data analytics or machine learning</a> – might not be optimized.</p><p>Databricks designed Delta Lake to transform these messy data lakes into cleaner ‘Delta Lakes’ with higher-quality data. This ultimately ensures any additional processes or operations performed on top generate much better insights, even if performed in real-time.</p><p>“Nearly every company has a data lake they are trying to gain insights from, but data lakes have proven to lack data reliability,” said Databricks cofounder and CEO, Ali Ghodsi. “Delta Lake has eliminated these challenges for hundreds of enterprises. By making Delta Lake open source, developers will be able to easily build reliable data lakes and turn them into ‘Delta Lakes’.”</p><p>Delta Lake competes with a number of other services on the market that perform similar functions, including Apache HUDI, Azure Data Lake, and Snowflake.</p><h2 class="article-body__section" id="section-how-does-delta-lake-work"><span>How does Delta Lake work?</span></h2><p>Based in the cloud, Delta Lake is a Spark proprietary extension that allows organizations to create Delta tables by default, whether they’re using Apache Spark or <a href="https://www.itpro.com/sql/30242/what-is-sql"><u>SQL</u></a>. The open source platform extends the classic parquet data file format with a file-based transaction log, which allows for additional functions. </p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/data-insights/sql/360642/14-best-sql-courses">The best SQL courses</a></p></div></div><p>Delta Lake features include providing atomicity, consistency, isolation and durability (ACID) transactions on Spark, as well as scalable metadata handling. The former ensures there’s never inconsistent data, while the latter takes advantage of Spark’s distributed processing power to handle masses of metadata at once.</p><p>Organizations can also take advantage of streaming and batch unification – by allowing both types of data to land in the same sink. This comes alongside some quality-of-life features such as rollbacks and historical audit trails, as well as the capability to reproduce machine learning experiments.</p><p>It comes with a variety of open source connectors, including Apache Flink, Presto and Trino, alongside standalone readers/writers that lets clients for <a href="https://www.itpro.com/business-strategy/careers-training/356640/how-to-become-a-python-software-developer"><u>Python</u></a>, Ruby and <a href="https://www.itpro.com/security/ransomware/368476/why-are-ransomware-gangs-pivoting-to-rust"><u>Rust</u></a> write directly into Delta Lake without needing a massive data engine like Apache Spark.</p><p>In 2021, Databricks launched Delta Sharing, a protocol that lets different companies share massive data sets securely and in real time. This aims to break the notion of vendor lock-in, as well as break down data silos due to proprietary data formats and computing resources that are required to read data. Users can access shared data through Pandas, Tableau, Presto, and other platforms without the need to use proprietary systems.</p><h2 class="article-body__section" id="section-what-s-new-in-delta-lake-3-0"><span>What’s new in Delta Lake 3.0?</span></h2><p>With the latest iteration of Delta Lake, Databricks is hoping to end the ‘format wars’ that’s long posed challenges for storing and managing data in data lakes. </p><p>Most data (99%) is stored in data lakes using the parquet format, but organizations must choose from one of several competing standards to store all metadata: Delta Lake, Apache Iceberg, and Apache Hudi. Each format handles metadata differently are fundamentally incompatible with one another. Once enterprises choose a format, they’re stuck using it for everything.</p><p>With Delta Lake 3.0, Databricks has created the <strong>universal format (UniForm)</strong>, which serves as a unification of the three standards. Customers who use a Databricks data lake will find their data stored in the parquet format, as usual, alongside three different versions of the metadata. It means the Delta Lake data can be read as if it’s stored in either Iceberg or Hudi.</p><p>“The metadata is very important, and if you get the metadata wrong, you can’t actually access this stuff,” says Databricks CEO Ali Ghodsi, speaking at Databricks AI + Data Summit 2023. “Since all three projects are open source, we just went and understood exactly how to do it in each of them, and now inside Databricks, when we create data, we create data for all three.”</p><p>Also new in the latest iteration of Delta Lake is the <strong>Delta Kernel</strong>, which addresses connector fragmentation by ensuring connectors are built against a core library that implements Delta specifications. This alleviates the burden of having to update Delta connectors with each new version or protocol change. With one API, developers can keep connectors up-to-date and ensure the latest innovations are pushed out as soon as they’re ready.</p><p>The final new addition is the <strong>Delta Liquid Clustering</strong>, which boosts the performance of reads and writes. With a flexible data layout technique, the Delta Liquid Clustering marks a departure from the decades old hive-style table partitioning system, which uses a fixed layout. It means organizations can take advantage of cost-efficient data clustering as their data lake grows with time.</p>
                                                            </article>
                            ]]>
                        </content:encoded>
                                                </item>
                                <item>
                                                            <title><![CDATA[ Google Cloud aims to remove data limits through BigLake ]]></title>
                                                                                                <dc:content><![CDATA[ <p>Google Cloud has introduced a preview of BigLake today, a data lake storage engine, it said would remove data limits by unifying data lakes and warehouses.</p><p>The tech giant said that managing <a href="https://www.itpro.com/business-strategy/28163/what-is-big-data-analytics" target="_blank" data-original-url="https://www.itpro.com/business-strategy/28163/what-is-big-data-analytics">data</a> across disparate lakes and warehouses create silos and increases risk and cost, especially when data needs to be moved. Through BigLake, it hopes to allow companies to unify their data warehouses and lakes to analyse data without worrying about the underlying storage format or system. This, it said, eliminates the need to duplicate or move data from a source and reduces cost and inefficiencies.</p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/cloud/cloud-computing/367334/google-cloud-unveils-new-chronicle-mssp-program" data-original-url="/cloud/cloud-computing/367334/google-cloud-unveils-new-chronicle-mssp-program">Google Cloud unveils new Chronicle MSSP Program</a> <a data-analytics-id="inline-link" href="https://www.itpro.com/business-strategy/digital-transformation/365392/google-cloud-partners-with-flipkart-for-digital" data-original-url="/business-strategy/digital-transformation/365392/google-cloud-partners-with-flipkart-for-digital">Google Cloud partners with Flipkart for digital transformation</a> <a data-analytics-id="inline-link" href="https://www.itpro.com/cloud/cloud-computing/366980/google-cloud-pricing-increase" data-original-url="/cloud/cloud-computing/366980/google-cloud-pricing-increase">Google Cloud will hike pricing for some storage and data services</a></p></div></div><p>Google Cloud customers will gain access controls through BigLake, with an <a href="https://www.itpro.com/application-programming-interface-api/33557/the-api-economy-what-your-business-needs-to-know" target="_blank" data-original-url="https://www.itpro.com/application-programming-interface-api/33557/the-api-economy-what-your-business-needs-to-know">API</a> interface spanning Google Cloud and open file formats like Parquet, as well as open-source processing engines like Apache Spark. The company said these capabilities extend a decade’s worth of innovations with BigQuery to data lakes on Google Cloud Storage to enable a flexible and cost-effective open lake house <a href="https://www.itpro.com/server-storage/31782/what-is-serverless-architecture" target="_blank" data-original-url="https://www.itpro.com/server-storage/31782/what-is-serverless-architecture">architecture</a>.</p><p>BigLake is set to be the centre of the company’s overall strategy from a storage perspective, revealed Sudhir Hasbe, Google Cloud’s data analytics product manager.</p><div  class="fancy-box"><div class="fancy_box-title">RELATED RESOURCE</div><div class="fancy_box_body"><figure class="van-image-figure "  ><div class='image-full-width-wrapper'><div class='image-widthsetter' ><p class="vanilla-image-block" style="padding-top:56.25%;"><img id="oMkurt6uLjEqZhbFrQbgnP" name="oMkurt6uLjEqZhbFrQbgnP.jpg" caption="" alt="" src="https://cdn.mos.cms.futurecdn.net/oMkurt6uLjEqZhbFrQbgnP-1920-80.jpg" mos="https://cdn.mos.cms.futurecdn.net/oMkurt6uLjEqZhbFrQbgnP.jpg" link="" align="" fullscreen="" width="0" height="0" attribution="" endorsement="" class="pinterest-pin-exclude" /></p></div></div></figure><p class="fancy-box__body-text"><strong>Automating the modern data warehouse</strong></p><p class="fancy-box__body-text">Freedom from constraints on your data</p><p class="fancy-box__body-text">FREE DOWNLOAD</p></div></div><p>“We are going to make sure all of our tools and capabilities work seamlessly with BigLake,” he said. “Similarly, all of our analytics engines, whether it's BigQuery, whether it's our Spark engine, or whether it's our data flow engine, all of these engines will seamlessly integrate out of the box with BigLake.”</p><p>The company also announced the formation of the Data Cloud Alliance, a new initiative to ensure global businesses have more seamless access and insights into the data required for <a href="https://www.itpro.com/strategy/29899/three-reasons-why-digital-transformation-is-essential-for-business-growth" target="_blank" data-original-url="https://www.itpro.com/strategy/29899/three-reasons-why-digital-transformation-is-essential-for-business-growth">digital transformation</a>. </p><p>The alliance comprises Google Cloud, Accenture, Confluent, Databricks, Dataiku, Deloitte, Elastic, Fivetran, MongoDB, Neo4j, Redis, and Starburst. The companies said they are committed to making data more portable and accessible across disparate business systems, platforms, and environments.</p><p>Google Cloud said that the proliferation of data means businesses increasingly need common digital data standards and a commitment to open <a href="https://www.itpro.com/business-operations/31681/what-is-data-processing" target="_blank" data-original-url="https://www.itpro.com/business-operations/31681/what-is-data-processing">data</a>, to effectively use data to digitally transform.</p><p>The Data Cloud Alliance is committed to accelerating the adoption of data analytics, <a href="https://www.itpro.com/strategy/29848/is-artificial-intelligence-safe" target="_blank" data-original-url="https://www.itpro.com/strategy/29848/is-artificial-intelligence-safe">artificial intelligence</a>, and machine learning best practices across industries through common industry data models, open standards, and integrated processes. </p><p>The alliance’s members will work together to help reduce customer challenges and complexity with data governance, data privacy, data loss prevention, and global compliance. They will provide infrastructure, APIs, and integration support to ensure data portability and accessibility between multiple platforms and products across multiple environments.</p><p>Each alliance member is also set to collaborate on new, common industry data models, processes, and platform integrations to increase data portability and reduce the complexity associated with data governance and global compliance.</p> ]]></dc:content>
                                                                                                                                            <link>https://www.itpro.com/data-insights/data-management/367341/google-cloud-aims-to-remove-data-limits-through-biglake</link>
                                                                            <description>
                            <![CDATA[ Tech giant also announced the formation of the Data Cloud Alliance initiative to help global businesses undergo digital transformation ]]>
                                                                                                            </description>
                                                                                                                                <guid isPermaLink="false">t9uvGFjsiEKFryybNZa7mH</guid>
                                                                                                <enclosure url="https://cdn.mos.cms.futurecdn.net/Bde5Kr3zxtpjGC9wFZusa4-1920-80.jpg" type="image/jpeg" length="0"></enclosure>
                                                                        <pubDate>Wed, 06 Apr 2022 10:23:50 +0000</pubDate>                                                                                                                                <updated>Thu, 19 Nov 2026 13:16:04 +0000</updated>
                                                                                                                                            <category><![CDATA[Deep Learning]]></category>
                                                    <category><![CDATA[Big Data]]></category>
                                                    <category><![CDATA[Data and Analytics]]></category>
                                                                                                                    <dc:creator><![CDATA[ Zach Marzouk ]]></dc:creator>                                                                                    <dc:source><![CDATA[ https://cdn.mos.cms.futurecdn.net/GFZtdGsYoXrkh3Jhj4ZKTc-320-70.jpg ]]></dc:source>
                                                                <dc:description><![CDATA[ &lt;p&gt;Zach Marzouk is a former ITPro, CloudPro, and ChannelPro staff writer, covering topics like &lt;a href=&quot;https://www.itpro.co.uk/security/cyber-security/361099/cyber-security-and-insurance-companies-evolving-with-the-threat-of-ransomware&quot;&gt;security&lt;/a&gt;, &lt;a href=&quot;https://www.itpro.co.uk/security/privacy/361191/is-australia-becoming-a-surveillance-state&quot;&gt;privacy&lt;/a&gt;, &lt;a href=&quot;https://www.itpro.co.uk/technology/artificial-intelligence-ai/359765/with-ai-on-the-rise-is-it-time-to-join-a-union&quot;&gt;worker rights&lt;/a&gt;, and &lt;a href=&quot;https://www.itpro.co.uk/business-strategy/startups/361311/why-does-japan-lag-behind-on-startups&quot;&gt;startups&lt;/a&gt;, primarily in the Asia Pacific and the US regions. Zach joined ITPro in 2017 where he was introduced to the world of B2B technology as a junior staff writer, before he returned to Argentina in 2018, working in communications and as a copywriter. In 2021, he made his way back to ITPro as a staff writer during the pandemic, before joining the world of freelance in 2022.&lt;/p&gt;
&lt;p&gt;After studying an undergraduate degree in Arabic and Spanish at the University of Leeds, Zach completed a journalism internship at The Argentina Independent in Buenos Aires where he wrote about the country’s history, politics, and technology.&lt;/p&gt;
&lt;p&gt;Contact him at zach.marzouk@futurenet.com or find Zach’s thoughts (and more) on Twitter &lt;a href=&quot;https://twitter.com/ZachMarzouk&quot;&gt;@ZachMarzouk&lt;/a&gt;&lt;/p&gt; ]]></dc:description>
                                                                                                                                <cf:isSponsored>false</cf:isSponsored>
                <cf:hasAffiliateLinks>false</cf:hasAffiliateLinks>
                <cf:isPaid>false</cf:isPaid>
                                                                                                                                                                                    <media:content type="image/jpeg" url="https://cdn.mos.cms.futurecdn.net/Bde5Kr3zxtpjGC9wFZusa4-1920-80.jpg">
                                                            <media:credit><![CDATA[Getty Images]]></media:credit>
                                                                                                                                                                                                                                    <media:description><![CDATA[A glowing particle and binary wave pattern on dark background]]></media:description>                                                            <media:text><![CDATA[A glowing particle and binary wave pattern on dark background]]></media:text>
                                <media:title type="plain"><![CDATA[A glowing particle and binary wave pattern on dark background]]></media:title>
                                                    </media:content>
                                                    <media:thumbnail url="https://cdn.mos.cms.futurecdn.net/Bde5Kr3zxtpjGC9wFZusa4-1920-80.jpg" />
                                                                                                                                                                    <content:encoded >
                            <![CDATA[
                            <article>
                                <p>Google Cloud has introduced a preview of BigLake today, a data lake storage engine, it said would remove data limits by unifying data lakes and warehouses.</p><p>The tech giant said that managing <a href="https://www.itpro.com/business-strategy/28163/what-is-big-data-analytics" target="_blank" data-original-url="https://www.itpro.com/business-strategy/28163/what-is-big-data-analytics">data</a> across disparate lakes and warehouses create silos and increases risk and cost, especially when data needs to be moved. Through BigLake, it hopes to allow companies to unify their data warehouses and lakes to analyse data without worrying about the underlying storage format or system. This, it said, eliminates the need to duplicate or move data from a source and reduces cost and inefficiencies.</p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/cloud/cloud-computing/367334/google-cloud-unveils-new-chronicle-mssp-program" data-original-url="/cloud/cloud-computing/367334/google-cloud-unveils-new-chronicle-mssp-program">Google Cloud unveils new Chronicle MSSP Program</a> <a data-analytics-id="inline-link" href="https://www.itpro.com/business-strategy/digital-transformation/365392/google-cloud-partners-with-flipkart-for-digital" data-original-url="/business-strategy/digital-transformation/365392/google-cloud-partners-with-flipkart-for-digital">Google Cloud partners with Flipkart for digital transformation</a> <a data-analytics-id="inline-link" href="https://www.itpro.com/cloud/cloud-computing/366980/google-cloud-pricing-increase" data-original-url="/cloud/cloud-computing/366980/google-cloud-pricing-increase">Google Cloud will hike pricing for some storage and data services</a></p></div></div><p>Google Cloud customers will gain access controls through BigLake, with an <a href="https://www.itpro.com/application-programming-interface-api/33557/the-api-economy-what-your-business-needs-to-know" target="_blank" data-original-url="https://www.itpro.com/application-programming-interface-api/33557/the-api-economy-what-your-business-needs-to-know">API</a> interface spanning Google Cloud and open file formats like Parquet, as well as open-source processing engines like Apache Spark. The company said these capabilities extend a decade’s worth of innovations with BigQuery to data lakes on Google Cloud Storage to enable a flexible and cost-effective open lake house <a href="https://www.itpro.com/server-storage/31782/what-is-serverless-architecture" target="_blank" data-original-url="https://www.itpro.com/server-storage/31782/what-is-serverless-architecture">architecture</a>.</p><p>BigLake is set to be the centre of the company’s overall strategy from a storage perspective, revealed Sudhir Hasbe, Google Cloud’s data analytics product manager.</p><div  class="fancy-box"><div class="fancy_box-title">RELATED RESOURCE</div><div class="fancy_box_body"><figure class="van-image-figure "  ><div class='image-full-width-wrapper'><div class='image-widthsetter' ><p class="vanilla-image-block" style="padding-top:56.25%;"><img id="oMkurt6uLjEqZhbFrQbgnP" name="oMkurt6uLjEqZhbFrQbgnP.jpg" caption="" alt="" src="https://cdn.mos.cms.futurecdn.net/oMkurt6uLjEqZhbFrQbgnP-1920-80.jpg" mos="https://cdn.mos.cms.futurecdn.net/oMkurt6uLjEqZhbFrQbgnP.jpg" link="" align="" fullscreen="" width="0" height="0" attribution="" endorsement="" class="pinterest-pin-exclude" /></p></div></div></figure><p class="fancy-box__body-text"><strong>Automating the modern data warehouse</strong></p><p class="fancy-box__body-text">Freedom from constraints on your data</p><p class="fancy-box__body-text">FREE DOWNLOAD</p></div></div><p>“We are going to make sure all of our tools and capabilities work seamlessly with BigLake,” he said. “Similarly, all of our analytics engines, whether it's BigQuery, whether it's our Spark engine, or whether it's our data flow engine, all of these engines will seamlessly integrate out of the box with BigLake.”</p><p>The company also announced the formation of the Data Cloud Alliance, a new initiative to ensure global businesses have more seamless access and insights into the data required for <a href="https://www.itpro.com/strategy/29899/three-reasons-why-digital-transformation-is-essential-for-business-growth" target="_blank" data-original-url="https://www.itpro.com/strategy/29899/three-reasons-why-digital-transformation-is-essential-for-business-growth">digital transformation</a>. </p><p>The alliance comprises Google Cloud, Accenture, Confluent, Databricks, Dataiku, Deloitte, Elastic, Fivetran, MongoDB, Neo4j, Redis, and Starburst. The companies said they are committed to making data more portable and accessible across disparate business systems, platforms, and environments.</p><p>Google Cloud said that the proliferation of data means businesses increasingly need common digital data standards and a commitment to open <a href="https://www.itpro.com/business-operations/31681/what-is-data-processing" target="_blank" data-original-url="https://www.itpro.com/business-operations/31681/what-is-data-processing">data</a>, to effectively use data to digitally transform.</p><p>The Data Cloud Alliance is committed to accelerating the adoption of data analytics, <a href="https://www.itpro.com/strategy/29848/is-artificial-intelligence-safe" target="_blank" data-original-url="https://www.itpro.com/strategy/29848/is-artificial-intelligence-safe">artificial intelligence</a>, and machine learning best practices across industries through common industry data models, open standards, and integrated processes. </p><p>The alliance’s members will work together to help reduce customer challenges and complexity with data governance, data privacy, data loss prevention, and global compliance. They will provide infrastructure, APIs, and integration support to ensure data portability and accessibility between multiple platforms and products across multiple environments.</p><p>Each alliance member is also set to collaborate on new, common industry data models, processes, and platform integrations to increase data portability and reduce the complexity associated with data governance and global compliance.</p>
                                                            </article>
                            ]]>
                        </content:encoded>
                                                </item>
                                <item>
                                                            <title><![CDATA[ Data is key to the next space race – and Toulouse is at the heart of it ]]></title>
                                                                                                <dc:content><![CDATA[ <p>The French city of Toulouse is famously known as ‘La Ville Rose’, the Pink City, due to the sheer volume of red bricks that run through its streets and buildings, creating a breathtaking spectacle. It’s also, however, home to a burgeoning startup scene with a focus that's a little more out of this world.</p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/infrastructure/network-internet/357490/microsoft-and-spacex-collaborate-to-take-the-cloud-into" data-original-url="/infrastructure/network-internet/357490/microsoft-and-spacex-collaborate-to-take-the-cloud-into">Microsoft and SpaceX collaborate to take the cloud into space</a></p></div></div><p>Space technology has almost taken over the fourth largest city in France; many of its news tech businesses are focused on satellite data, which has caught the attention of the French Space Agency (CNES). </p><p>Towards the end of 2021, CNES chose <a href="https://www.itpro.com/cloud/31522/orange-acquires-basefarm-to-boost-european-cloud-services" target="_blank" data-original-url="https://www.itpro.com/cloud/31522/orange-acquires-basefarm-to-boost-european-cloud-services">Orange Business Services (OBS)</a> to lead an industry consortium to design, deploy and maintain a new storage system for its scientific spatial data. This is, in part, to help CNES modernise its <a href="https://www.itpro.com/server-storage/31876/the-history-and-evolution-of-storage" target="_blank" data-original-url="https://www.itpro.com/server-storage/31876/the-history-and-evolution-of-storage">storage infrastructure</a>, but also find ways it can use the plethora of satellite data it receives in better ways. When completed, it’ll resemble <a href="https://www.itpro.com/cloud/private-cloud/360284/the-benefits-of-private-cloud-for-your-data-storage-environment" target="_blank" data-original-url="https://www.itpro.com/cloud/private-cloud/360284/the-benefits-of-private-cloud-for-your-data-storage-environment">a private cloud platform</a>, and will be based in Toulouse.</p><p>“Toulouse is really the capital of space in France,” according to Etienne Bonhomme, France’s director of cloud activities at Orange Business Services. “This is where you've got all the companies in the space market – they are located in Toulouse.” </p><h2 id="oceans-of-data">Oceans of data</h2><div  class="fancy-box"><div class="fancy_box-title">RELATED RESOURCE</div><div class="fancy_box_body"><figure class="van-image-figure "  ><div class='image-full-width-wrapper'><div class='image-widthsetter' ><p class="vanilla-image-block" style="padding-top:56.25%;"><img id="oMkurt6uLjEqZhbFrQbgnP" name="oMkurt6uLjEqZhbFrQbgnP.jpg" caption="" alt="" src="https://cdn.mos.cms.futurecdn.net/oMkurt6uLjEqZhbFrQbgnP-1920-80.jpg" mos="https://cdn.mos.cms.futurecdn.net/oMkurt6uLjEqZhbFrQbgnP.jpg" link="" align="" fullscreen="" width="0" height="0" attribution="" endorsement="" class="pinterest-pin-exclude" /></p></div></div></figure><p class="fancy-box__body-text"><strong>Automating the modern data warehouse</strong></p><p class="fancy-box__body-text">Freedom from constraints on your data</p><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/data-insights/databases/361794/automating-the-modern-data-warehouse" data-original-url="/data-insights/databases/361794/automating-the-modern-data-warehouse">FREE DOWNLOAD</a></p></div></div><p>The topic of space is interesting in itself, Bonhomme explains, but what’s more exciting is the data coming from the many <a href="https://www.itpro.com/business-strategy/it-infrastructure/358952/the-it-pro-podcast-the-tech-keeping-your-telly-on" data-original-url="https://www.itpro.com/business-strategy/it-infrastructure/358952/the-it-pro-podcast-the-tech-keeping-your-telly-on">satellites</a> orbiting Earth. OBS aims to use the data CNES captures for the benefit of society, and to boost the economy, citing examples such as transportation and tourism. The French space sector employs roughly 20,000 to 30,000 people, but their work with data can also help the two million that work on tourism, he explains, touching on how the sector can bed roots into other sections of the economy.</p><p>“This is why CNES has launched this project,” he says. “A private cloud auto-scaling infrastructure, which is being built in one place; that they expect will be a new data zone designed to take advantage of the split between compute and storage. When you’re looking for data, you won’t be slowing down the user with silos. You have, instead, each individual element that can scale when necessary.” </p><p>CNES has specifically drafted in OBS to build a data lake because the volume of data coming from this space is “exploding”, Bonhomme explains. So much so, the firm foresees the volume of data could actually double by 2025 because there will be more powerful satellites relaying even more data to <a href="https://www.cloudpro.co.uk/it-infrastructure/cloud-deployment/7805/aws-ground-stations-link-satellites-to-the-cloud">ground stations</a>, which, in turn, inevitably means <a href="https://www.itpro.com/business-strategy/28163/what-is-big-data-analytics" target="_blank" data-original-url="https://www.itpro.com/business-strategy/28163/what-is-big-data-analytics">more data to analyse</a>.</p><figure class="van-image-figure pull- inline-layout" data-bordeaux-image-check="" ><div class='image-full-width-wrapper'><div class='image-widthsetter' ><p class="vanilla-image-block" style="padding-top:56.25%;"><img id="avCQLxJ98pHZu82zpZJPdd" name="" alt="A cityscape shot of Toulouse showing its red buildings against the blue sky" src="https://cdn.mos.cms.futurecdn.net/avCQLxJ98pHZu82zpZJPdd-1920-80.jpg" mos="https://cdn.mos.cms.futurecdn.net/avCQLxJ98pHZu82zpZJPdd.jpg" align="" fullscreen="" width="" height="" attribution="" endorsement="" class="pull-" /></p></div></div><figcaption itemprop="caption description" class="pull- inline-layout"><span class="credit" itemprop="copyrightHolder">(Image credit: Getty Images)</span></figcaption></figure><p><em><strong>Bonhomme describes Toulouse as the capital of space in France</strong></em></p><p>The reasoning behind a data lake is very much tied to the potential for satellite data because it removes certain barriers for users. A data lake is a type of centralised repository that's different from a data warehouse, in that it doesn’t require a holder to store data with any specific ideas about its intended use – it <a href="https://www.itpro.com/big-data-analytics/34532/structured-vs-unstructured-data-management" target="_blank" data-original-url="https://www.itpro.com/big-data-analytics/34532/structured-vs-unstructured-data-management">accepts both structured and unstructured data</a>. One of the reasons people sometimes have structured and unstructured forms of data is that it often gets stored in different locations – on-premise and the cloud – with the data lake offering a single virtual facility to house it. This is arguably the best place to store data collected from satellites, which have, seemingly, endless possibilities. </p><h2 id="the-next-space-race">The next space race</h2><p>“The purpose of the data lake is actually at the heart of the digital transformation of the French Space Agency,” Bonhomme explains. “Beyond storage, the challenge is to process and transform the information in order to accelerate innovation cycles and tasks, and use data in various data initiatives it carries out.”</p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/data-insights/data-processing/358600/its-time-to-give-edge-computing-some-space" data-original-url="/data-insights/data-processing/358600/its-time-to-give-edge-computing-some-space">It’s time to give edge computing some space</a></p></div></div><p>The <a href="https://www.itpro.com/data-insights/data-processing/358600/its-time-to-give-edge-computing-some-space" target="_blank" data-original-url="https://www.itpro.com/data-insights/data-processing/358600/its-time-to-give-edge-computing-some-space">new data lake</a>, technically, will be capable of hosting 100PB of special data. Just 1PB is the equivalent of 500 billion pages of standard printed text or 11,000 <a href="https://www.itpro.com/business/business-strategy/358658/dont-listen-to-martin-scorsese-the-netflix-algorithm-is-your" target="_blank" data-original-url="https://www.itpro.com/business/business-strategy/358658/dont-listen-to-martin-scorsese-the-netflix-algorithm-is-your">4K movies</a>. Much of the data collected will have an economic value, particularly for geolocation and imagery, but it will also hold information of importance to technologies that monitor climate change. These include systems monitoring rising sea levels, or machines that analyse the ozone layer. Geospatial data is also important for emerging technologies such as driverless cars or for other, more traditional services, like digital maps and even urban planning. </p><iframe frameborder="0" height="200px" width="100%" class="position-center" data-lazy-priority="high" data-lazy-src="https://widget.spreaker.com/player?episode_id=44928385&amp;theme=light&amp;playlist=false&amp;playlist-continuous=false&amp;autoplay=false&amp;live-autoplay=false&amp;chapters-image=true&amp;episode_image_position=right&amp;hide-logo=false&amp;hide-likes=true&amp;hide-comments=true&amp;hide-sharing=true&amp;hide-download=true&amp;color=ffe019" src="https://widget.spreaker.com/player?episode_id=44928385&amp;theme=light&amp;playlist=false&amp;playlist-continuous=false&amp;autoplay=false&amp;live-autoplay=false&amp;chapters-image=true&amp;episode_image_position=right&amp;hide-logo=false&amp;hide-likes=true&amp;hide-comments=true&amp;hide-sharing=true&amp;hide-download=true&amp;color=ffe019"></iframe><p>In supplying the infrastructure for CNES, OBS is helping to facilitate the advancement of scientific understanding, but also aid the emergence of new services and products within the industry. Having such data stored in a single repository, that doesn’t require stipulations to structure it, is also a powerful capability. </p><p>“The dream for every organisation is to be able to transform this data into gold,” Bonhomme adds. “This is also the case in the space market – they want to be able to transform their data into something that's billable.”</p> ]]></dc:content>
                                                                                                                                            <link>https://www.itpro.com/data-insights/big-data-analytics/362201/french-space-agency-orange-toulouse-space-race</link>
                                                                            <description>
                            <![CDATA[ A data lake capable of housing 100PB of data could revolutionise the way France uses geospatial data ]]>
                                                                                                            </description>
                                                                                                                                <guid isPermaLink="false">tCJs8wDHLZcwmZhTeEZubY</guid>
                                                                                                <enclosure url="https://cdn.mos.cms.futurecdn.net/t7NwjABPcLHjJZ4Vxr9bQe-1920-80.jpg" type="image/jpeg" length="0"></enclosure>
                                                                        <pubDate>Wed, 16 Feb 2022 08:00:06 +0000</pubDate>                                                                                                                                <updated>Thu, 19 Nov 2026 15:10:53 +0000</updated>
                                                                                                                                            <category><![CDATA[Deep Learning]]></category>
                                                    <category><![CDATA[Data and Analytics]]></category>
                                                    <category><![CDATA[Big Data]]></category>
                                                                                                <author><![CDATA[ itpro@futurenet.com (Bobby Hellard) ]]></author>                    <dc:creator><![CDATA[ Bobby Hellard ]]></dc:creator>                                                                                    <dc:source><![CDATA[ https://cdn.mos.cms.futurecdn.net/bsR2tHSyVKUoyXZF5pNsDA-320-70.jpg ]]></dc:source>
                                                                <dc:description><![CDATA[ &lt;p&gt;Bobby Hellard&amp;nbsp;is&amp;nbsp;ITPro&#039;s Reviews Editor and has worked on&amp;nbsp;CloudPro and ChannelPro since 2018. In his time at ITPro, Bobby has covered stories for all the major technology companies, such as Apple, Microsoft, Amazon and Facebook, and regularly attends industry-leading events such as AWS Re:Invent and Google Cloud Next.&lt;/p&gt;
&lt;p&gt;Bobby mainly covers hardware reviews, but you will also recognize him as the face of many of our video reviews of laptops and smartphones.&lt;/p&gt;
&lt;p&gt;He has been a journalist for ten years, originally covering sports, before moving into business technology with ITPro. He has bylines in The Independent, Vice and The Business Briefing. Contact him at &lt;a href=&quot;mailto:bobby.hellard@futurenet.com&quot;&gt;bobby.hellard@futurenet.com&lt;/a&gt; or find him on Twitter: &lt;a href=&quot;https://twitter.com/bobbyhellard&quot;&gt;@bobbyhellard&lt;/a&gt;&lt;/p&gt; ]]></dc:description>
                                                                                                                                <cf:isSponsored>false</cf:isSponsored>
                <cf:hasAffiliateLinks>false</cf:hasAffiliateLinks>
                <cf:isPaid>false</cf:isPaid>
                                                                                                                                                                                    <media:content type="image/jpeg" url="https://cdn.mos.cms.futurecdn.net/t7NwjABPcLHjJZ4Vxr9bQe-1920-80.jpg">
                                                            <media:credit><![CDATA[Getty Images]]></media:credit>
                                                                                                                                                                                                                                    <media:description><![CDATA[A sea of observatory antenna at sunset]]></media:description>                                                            <media:text><![CDATA[A sea of observatory antenna at sunset]]></media:text>
                                <media:title type="plain"><![CDATA[A sea of observatory antenna at sunset]]></media:title>
                                                    </media:content>
                                                    <media:thumbnail url="https://cdn.mos.cms.futurecdn.net/t7NwjABPcLHjJZ4Vxr9bQe-1920-80.jpg" />
                                                                                                                                                                    <content:encoded >
                            <![CDATA[
                            <article>
                                <p>The French city of Toulouse is famously known as ‘La Ville Rose’, the Pink City, due to the sheer volume of red bricks that run through its streets and buildings, creating a breathtaking spectacle. It’s also, however, home to a burgeoning startup scene with a focus that's a little more out of this world.</p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/infrastructure/network-internet/357490/microsoft-and-spacex-collaborate-to-take-the-cloud-into" data-original-url="/infrastructure/network-internet/357490/microsoft-and-spacex-collaborate-to-take-the-cloud-into">Microsoft and SpaceX collaborate to take the cloud into space</a></p></div></div><p>Space technology has almost taken over the fourth largest city in France; many of its news tech businesses are focused on satellite data, which has caught the attention of the French Space Agency (CNES). </p><p>Towards the end of 2021, CNES chose <a href="https://www.itpro.com/cloud/31522/orange-acquires-basefarm-to-boost-european-cloud-services" target="_blank" data-original-url="https://www.itpro.com/cloud/31522/orange-acquires-basefarm-to-boost-european-cloud-services">Orange Business Services (OBS)</a> to lead an industry consortium to design, deploy and maintain a new storage system for its scientific spatial data. This is, in part, to help CNES modernise its <a href="https://www.itpro.com/server-storage/31876/the-history-and-evolution-of-storage" target="_blank" data-original-url="https://www.itpro.com/server-storage/31876/the-history-and-evolution-of-storage">storage infrastructure</a>, but also find ways it can use the plethora of satellite data it receives in better ways. When completed, it’ll resemble <a href="https://www.itpro.com/cloud/private-cloud/360284/the-benefits-of-private-cloud-for-your-data-storage-environment" target="_blank" data-original-url="https://www.itpro.com/cloud/private-cloud/360284/the-benefits-of-private-cloud-for-your-data-storage-environment">a private cloud platform</a>, and will be based in Toulouse.</p><p>“Toulouse is really the capital of space in France,” according to Etienne Bonhomme, France’s director of cloud activities at Orange Business Services. “This is where you've got all the companies in the space market – they are located in Toulouse.” </p><h2 id="oceans-of-data">Oceans of data</h2><div  class="fancy-box"><div class="fancy_box-title">RELATED RESOURCE</div><div class="fancy_box_body"><figure class="van-image-figure "  ><div class='image-full-width-wrapper'><div class='image-widthsetter' ><p class="vanilla-image-block" style="padding-top:56.25%;"><img id="oMkurt6uLjEqZhbFrQbgnP" name="oMkurt6uLjEqZhbFrQbgnP.jpg" caption="" alt="" src="https://cdn.mos.cms.futurecdn.net/oMkurt6uLjEqZhbFrQbgnP-1920-80.jpg" mos="https://cdn.mos.cms.futurecdn.net/oMkurt6uLjEqZhbFrQbgnP.jpg" link="" align="" fullscreen="" width="0" height="0" attribution="" endorsement="" class="pinterest-pin-exclude" /></p></div></div></figure><p class="fancy-box__body-text"><strong>Automating the modern data warehouse</strong></p><p class="fancy-box__body-text">Freedom from constraints on your data</p><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/data-insights/databases/361794/automating-the-modern-data-warehouse" data-original-url="/data-insights/databases/361794/automating-the-modern-data-warehouse">FREE DOWNLOAD</a></p></div></div><p>The topic of space is interesting in itself, Bonhomme explains, but what’s more exciting is the data coming from the many <a href="https://www.itpro.com/business-strategy/it-infrastructure/358952/the-it-pro-podcast-the-tech-keeping-your-telly-on" data-original-url="https://www.itpro.com/business-strategy/it-infrastructure/358952/the-it-pro-podcast-the-tech-keeping-your-telly-on">satellites</a> orbiting Earth. OBS aims to use the data CNES captures for the benefit of society, and to boost the economy, citing examples such as transportation and tourism. The French space sector employs roughly 20,000 to 30,000 people, but their work with data can also help the two million that work on tourism, he explains, touching on how the sector can bed roots into other sections of the economy.</p><p>“This is why CNES has launched this project,” he says. “A private cloud auto-scaling infrastructure, which is being built in one place; that they expect will be a new data zone designed to take advantage of the split between compute and storage. When you’re looking for data, you won’t be slowing down the user with silos. You have, instead, each individual element that can scale when necessary.” </p><p>CNES has specifically drafted in OBS to build a data lake because the volume of data coming from this space is “exploding”, Bonhomme explains. So much so, the firm foresees the volume of data could actually double by 2025 because there will be more powerful satellites relaying even more data to <a href="https://www.cloudpro.co.uk/it-infrastructure/cloud-deployment/7805/aws-ground-stations-link-satellites-to-the-cloud">ground stations</a>, which, in turn, inevitably means <a href="https://www.itpro.com/business-strategy/28163/what-is-big-data-analytics" target="_blank" data-original-url="https://www.itpro.com/business-strategy/28163/what-is-big-data-analytics">more data to analyse</a>.</p><figure class="van-image-figure pull- inline-layout" data-bordeaux-image-check="" ><div class='image-full-width-wrapper'><div class='image-widthsetter' ><p class="vanilla-image-block" style="padding-top:56.25%;"><img id="avCQLxJ98pHZu82zpZJPdd" name="" alt="A cityscape shot of Toulouse showing its red buildings against the blue sky" src="https://cdn.mos.cms.futurecdn.net/avCQLxJ98pHZu82zpZJPdd-1920-80.jpg" mos="https://cdn.mos.cms.futurecdn.net/avCQLxJ98pHZu82zpZJPdd.jpg" align="" fullscreen="" width="" height="" attribution="" endorsement="" class="pull-" /></p></div></div><figcaption itemprop="caption description" class="pull- inline-layout"><span class="credit" itemprop="copyrightHolder">(Image credit: Getty Images)</span></figcaption></figure><p><em><strong>Bonhomme describes Toulouse as the capital of space in France</strong></em></p><p>The reasoning behind a data lake is very much tied to the potential for satellite data because it removes certain barriers for users. A data lake is a type of centralised repository that's different from a data warehouse, in that it doesn’t require a holder to store data with any specific ideas about its intended use – it <a href="https://www.itpro.com/big-data-analytics/34532/structured-vs-unstructured-data-management" target="_blank" data-original-url="https://www.itpro.com/big-data-analytics/34532/structured-vs-unstructured-data-management">accepts both structured and unstructured data</a>. One of the reasons people sometimes have structured and unstructured forms of data is that it often gets stored in different locations – on-premise and the cloud – with the data lake offering a single virtual facility to house it. This is arguably the best place to store data collected from satellites, which have, seemingly, endless possibilities. </p><h2 id="the-next-space-race">The next space race</h2><p>“The purpose of the data lake is actually at the heart of the digital transformation of the French Space Agency,” Bonhomme explains. “Beyond storage, the challenge is to process and transform the information in order to accelerate innovation cycles and tasks, and use data in various data initiatives it carries out.”</p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/data-insights/data-processing/358600/its-time-to-give-edge-computing-some-space" data-original-url="/data-insights/data-processing/358600/its-time-to-give-edge-computing-some-space">It’s time to give edge computing some space</a></p></div></div><p>The <a href="https://www.itpro.com/data-insights/data-processing/358600/its-time-to-give-edge-computing-some-space" target="_blank" data-original-url="https://www.itpro.com/data-insights/data-processing/358600/its-time-to-give-edge-computing-some-space">new data lake</a>, technically, will be capable of hosting 100PB of special data. Just 1PB is the equivalent of 500 billion pages of standard printed text or 11,000 <a href="https://www.itpro.com/business/business-strategy/358658/dont-listen-to-martin-scorsese-the-netflix-algorithm-is-your" target="_blank" data-original-url="https://www.itpro.com/business/business-strategy/358658/dont-listen-to-martin-scorsese-the-netflix-algorithm-is-your">4K movies</a>. Much of the data collected will have an economic value, particularly for geolocation and imagery, but it will also hold information of importance to technologies that monitor climate change. These include systems monitoring rising sea levels, or machines that analyse the ozone layer. Geospatial data is also important for emerging technologies such as driverless cars or for other, more traditional services, like digital maps and even urban planning. </p><iframe frameborder="0" height="200px" width="100%" class="position-center" data-lazy-priority="high" data-lazy-src="https://widget.spreaker.com/player?episode_id=44928385&amp;theme=light&amp;playlist=false&amp;playlist-continuous=false&amp;autoplay=false&amp;live-autoplay=false&amp;chapters-image=true&amp;episode_image_position=right&amp;hide-logo=false&amp;hide-likes=true&amp;hide-comments=true&amp;hide-sharing=true&amp;hide-download=true&amp;color=ffe019" src="https://widget.spreaker.com/player?episode_id=44928385&amp;theme=light&amp;playlist=false&amp;playlist-continuous=false&amp;autoplay=false&amp;live-autoplay=false&amp;chapters-image=true&amp;episode_image_position=right&amp;hide-logo=false&amp;hide-likes=true&amp;hide-comments=true&amp;hide-sharing=true&amp;hide-download=true&amp;color=ffe019"></iframe><p>In supplying the infrastructure for CNES, OBS is helping to facilitate the advancement of scientific understanding, but also aid the emergence of new services and products within the industry. Having such data stored in a single repository, that doesn’t require stipulations to structure it, is also a powerful capability. </p><p>“The dream for every organisation is to be able to transform this data into gold,” Bonhomme adds. “This is also the case in the space market – they want to be able to transform their data into something that's billable.”</p>
                                                            </article>
                            ]]>
                        </content:encoded>
                                                </item>
                                <item>
                                                            <title><![CDATA[ AWS launches HealthLake to help health care organizations query medical data ]]></title>
                                                                                                <dc:content><![CDATA[ <p>AWS has made its <a href="https://www.itpro.com/cloud" data-original-url="https://www.itpro.com/cloud">cloud</a>-based health care data lake solution, Amazon HealthLake, generally available to customers, enabling health care organizations to pull medical information from different sources into a single environment and use it to improve patient care.</p><p>With the solution, the tech giant wants to end the enormous spread of medical data across various applications and other systems. </p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/cloud/cloud-security/359945/university-medical-center-mainz-taps-ibm-to-secure-health-care-data" data-original-url="/cloud/cloud-security/359945/university-medical-center-mainz-taps-ibm-to-secure-health-care-data">University Medical Center Mainz taps IBM to secure health care data</a> <a data-analytics-id="inline-link" href="https://www.itpro.com/security/data-breaches/358864/third-party-attacks-expose-12m-health-care-records" data-original-url="/security/data-breaches/358864/third-party-attacks-expose-12m-health-care-records">Third-party attacks expose 12 million health care records</a> <a data-analytics-id="inline-link" href="https://www.itpro.com/security/ransomware/358231/health-care-organizations-under-siege-from-ryuk-ransomware" data-original-url="/security/ransomware/358231/health-care-organizations-under-siege-from-ryuk-ransomware">Health care organizations under siege from Ryuk ransomware</a> <a data-analytics-id="inline-link" href="https://www.itpro.com/cloud/software-as-a-service-saas/357443/vectorcare-expands-offerings-to-meet-surging-mobile-health" data-original-url="/cloud/software-as-a-service-saas/357443/vectorcare-expands-offerings-to-meet-surging-mobile-health">VectorCare expands offerings to meet surging mobile health care demands</a></p></div></div><p>Various patient medical information, such as clinical notes, lab results, and insurance information, is often housed in separate databases. This makes it extremely difficult for health care professionals to get a complete overview of a patient’s information. Moreover, collecting this information takes a lot of time.</p><p>With HealthLake, this could be a thing of the past, as it merges all patient data into a single data repository. In concrete terms, the solution offers a collection of connectors that automatically import records from commonly used health applications. When these records are in the data repository, algorithms based on natural language processing ensure the data is organized. This makes it easier to work with, according to AWS.</p><p>The algorithms scan medical documentation, such as clinical notes, and extract patient information. Then, Amazon HealthLake indexes this information and offers it as search pages in a search engine, adapting the raw data into easily searchable data. The ultimate result is doctors can search patient information faster than manually scrolling through all the information.</p><p>Additionally, Amazon HealthLake makes it possible to move the combined patient data to other systems for further processing. For this purpose, the data is stored in the industry-standard FHIR. The moving process takes place with a separate API that AWS provides in its Amazon product.</p><p>AWS also aims to provide added services to health care customers within the solution. This includes providing artificial intelligence (AI) services to improve patient care. AI can automate tasks, such as making medical diagnoses and planning treatment trajectories. AI can also provide general recommendations.</p><p>The underlying infrastructure of the Amazon HealthLake solution is fully managed by AWS. IT specialists from health care institutions only need to create a new data store. This data store is fully operational within minutes, the tech giant indicates. AWS handles the management, configuration and setup of the underlying infrastructure.</p><div  class="fancy-box"><div class="fancy_box-title">RELATED RESOURCE</div><div class="fancy_box_body"><figure class="van-image-figure "  ><div class='image-full-width-wrapper'><div class='image-widthsetter' ><p class="vanilla-image-block" style="padding-top:56.25%;"><img id="NhvLChm5wHQDKR7tJWkDhD" name="NhvLChm5wHQDKR7tJWkDhD.jpg" caption="" alt="" src="https://cdn.mos.cms.futurecdn.net/NhvLChm5wHQDKR7tJWkDhD-1920-80.jpg" mos="https://cdn.mos.cms.futurecdn.net/NhvLChm5wHQDKR7tJWkDhD.jpg" link="" align="" fullscreen="" width="0" height="0" attribution="" endorsement="" class="pinterest-pin-exclude" /></p></div></div></figure><p class="fancy-box__body-text"><strong>Cloud operational excellence</strong></p><p class="fancy-box__body-text">Everything you need to know about optimising your cloud operations</p><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/cloud/cloud-management/359733/cloud-operational-excellence" data-original-url="/cloud/cloud-management/359733/cloud-operational-excellence">FREE DOWNLOAD</a></p></div></div><p>“More and more of our customers in the healthcare and life sciences space are looking to organize and make sense of their reams of data, but are finding this process challenging and cumbersome,” said Swami Sivasubramanian, vice president of Amazon Machine Learning for AWS. </p><p>“We built Amazon HealthLake to remove this heavy lifting for healthcare organizations so they can transform health data in the cloud in minutes and begin analyzing that information securely at scale. Alongside AWS for Health, we’re excited about how Amazon HealthLake can help medical providers, health insurers, and pharmaceutical companies provide patients and populations with data-driven, personalized, and predictive care.”</p> ]]></dc:content>
                                                                                                                                            <link>https://www.itpro.com/data-insights/big-data/360271/aws-launches-healthlake-to-help-health-care-organizations-query</link>
                                                                            <description>
                            <![CDATA[ Cloud-based services and solutions could personalize patient care and bring new therapies to market ]]>
                                                                                                            </description>
                                                                                                                                <guid isPermaLink="false">oTyY6XHn4BaKdWPV2S3Jtp</guid>
                                                                                                <enclosure url="https://cdn.mos.cms.futurecdn.net/tXqJNLL29ZERv7RyA3pJuB-1920-80.jpg" type="image/jpeg" length="0"></enclosure>
                                                                        <pubDate>Fri, 16 Jul 2021 18:14:01 +0000</pubDate>                                                                                                                                <updated>Thu, 19 Nov 2026 13:16:04 +0000</updated>
                                                                                                                                            <category><![CDATA[Deep Learning]]></category>
                                                    <category><![CDATA[Cloud Hosting]]></category>
                                                    <category><![CDATA[Data and Analytics]]></category>
                                                    <category><![CDATA[Big Data]]></category>
                                                    <category><![CDATA[Cloud]]></category>
                                                                                                <author><![CDATA[ itpro@futurenet.com (Rene Millman) ]]></author>                    <dc:creator><![CDATA[ Rene Millman ]]></dc:creator>                                                                                    <dc:source><![CDATA[ https://cdn.mos.cms.futurecdn.net/vwWuTPNRCuw9vEaWzuXYnR-320-70.png ]]></dc:source>
                                                                <dc:description><![CDATA[ null ]]></dc:description>
                                                                                                                                <cf:isSponsored>false</cf:isSponsored>
                <cf:hasAffiliateLinks>false</cf:hasAffiliateLinks>
                <cf:isPaid>false</cf:isPaid>
                                                                                                                                                                                    <media:content type="image/jpeg" url="https://cdn.mos.cms.futurecdn.net/tXqJNLL29ZERv7RyA3pJuB-1920-80.jpg">
                                                            <media:credit><![CDATA[Shutterstock]]></media:credit>
                                                                                                                                                                                                                                    <media:description><![CDATA[A stethoscope on top of a MacBook keyboard]]></media:description>                                                            <media:text><![CDATA[A stethoscope on top of a MacBook keyboard]]></media:text>
                                <media:title type="plain"><![CDATA[A stethoscope on top of a MacBook keyboard]]></media:title>
                                                    </media:content>
                                                    <media:thumbnail url="https://cdn.mos.cms.futurecdn.net/tXqJNLL29ZERv7RyA3pJuB-1920-80.jpg" />
                                                                                                                                                                    <content:encoded >
                            <![CDATA[
                            <article>
                                <p>AWS has made its <a href="https://www.itpro.com/cloud" data-original-url="https://www.itpro.com/cloud">cloud</a>-based health care data lake solution, Amazon HealthLake, generally available to customers, enabling health care organizations to pull medical information from different sources into a single environment and use it to improve patient care.</p><p>With the solution, the tech giant wants to end the enormous spread of medical data across various applications and other systems. </p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/cloud/cloud-security/359945/university-medical-center-mainz-taps-ibm-to-secure-health-care-data" data-original-url="/cloud/cloud-security/359945/university-medical-center-mainz-taps-ibm-to-secure-health-care-data">University Medical Center Mainz taps IBM to secure health care data</a> <a data-analytics-id="inline-link" href="https://www.itpro.com/security/data-breaches/358864/third-party-attacks-expose-12m-health-care-records" data-original-url="/security/data-breaches/358864/third-party-attacks-expose-12m-health-care-records">Third-party attacks expose 12 million health care records</a> <a data-analytics-id="inline-link" href="https://www.itpro.com/security/ransomware/358231/health-care-organizations-under-siege-from-ryuk-ransomware" data-original-url="/security/ransomware/358231/health-care-organizations-under-siege-from-ryuk-ransomware">Health care organizations under siege from Ryuk ransomware</a> <a data-analytics-id="inline-link" href="https://www.itpro.com/cloud/software-as-a-service-saas/357443/vectorcare-expands-offerings-to-meet-surging-mobile-health" data-original-url="/cloud/software-as-a-service-saas/357443/vectorcare-expands-offerings-to-meet-surging-mobile-health">VectorCare expands offerings to meet surging mobile health care demands</a></p></div></div><p>Various patient medical information, such as clinical notes, lab results, and insurance information, is often housed in separate databases. This makes it extremely difficult for health care professionals to get a complete overview of a patient’s information. Moreover, collecting this information takes a lot of time.</p><p>With HealthLake, this could be a thing of the past, as it merges all patient data into a single data repository. In concrete terms, the solution offers a collection of connectors that automatically import records from commonly used health applications. When these records are in the data repository, algorithms based on natural language processing ensure the data is organized. This makes it easier to work with, according to AWS.</p><p>The algorithms scan medical documentation, such as clinical notes, and extract patient information. Then, Amazon HealthLake indexes this information and offers it as search pages in a search engine, adapting the raw data into easily searchable data. The ultimate result is doctors can search patient information faster than manually scrolling through all the information.</p><p>Additionally, Amazon HealthLake makes it possible to move the combined patient data to other systems for further processing. For this purpose, the data is stored in the industry-standard FHIR. The moving process takes place with a separate API that AWS provides in its Amazon product.</p><p>AWS also aims to provide added services to health care customers within the solution. This includes providing artificial intelligence (AI) services to improve patient care. AI can automate tasks, such as making medical diagnoses and planning treatment trajectories. AI can also provide general recommendations.</p><p>The underlying infrastructure of the Amazon HealthLake solution is fully managed by AWS. IT specialists from health care institutions only need to create a new data store. This data store is fully operational within minutes, the tech giant indicates. AWS handles the management, configuration and setup of the underlying infrastructure.</p><div  class="fancy-box"><div class="fancy_box-title">RELATED RESOURCE</div><div class="fancy_box_body"><figure class="van-image-figure "  ><div class='image-full-width-wrapper'><div class='image-widthsetter' ><p class="vanilla-image-block" style="padding-top:56.25%;"><img id="NhvLChm5wHQDKR7tJWkDhD" name="NhvLChm5wHQDKR7tJWkDhD.jpg" caption="" alt="" src="https://cdn.mos.cms.futurecdn.net/NhvLChm5wHQDKR7tJWkDhD-1920-80.jpg" mos="https://cdn.mos.cms.futurecdn.net/NhvLChm5wHQDKR7tJWkDhD.jpg" link="" align="" fullscreen="" width="0" height="0" attribution="" endorsement="" class="pinterest-pin-exclude" /></p></div></div></figure><p class="fancy-box__body-text"><strong>Cloud operational excellence</strong></p><p class="fancy-box__body-text">Everything you need to know about optimising your cloud operations</p><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/cloud/cloud-management/359733/cloud-operational-excellence" data-original-url="/cloud/cloud-management/359733/cloud-operational-excellence">FREE DOWNLOAD</a></p></div></div><p>“More and more of our customers in the healthcare and life sciences space are looking to organize and make sense of their reams of data, but are finding this process challenging and cumbersome,” said Swami Sivasubramanian, vice president of Amazon Machine Learning for AWS. </p><p>“We built Amazon HealthLake to remove this heavy lifting for healthcare organizations so they can transform health data in the cloud in minutes and begin analyzing that information securely at scale. Alongside AWS for Health, we’re excited about how Amazon HealthLake can help medical providers, health insurers, and pharmaceutical companies provide patients and populations with data-driven, personalized, and predictive care.”</p>
                                                            </article>
                            ]]>
                        </content:encoded>
                                                </item>
                                <item>
                                                            <title><![CDATA[ Amazon launches big data service focused on health care  ]]></title>
                                                                                                <dc:content><![CDATA[ <p>Amazon has launched another business venture, this time pushing into the health care and life sciences sectors. The company announced a preview of <a href="https://aws.amazon.com/healthlake">HealthLake</a>, a big data service housed in its Amazon Web Services (AWS) cloud.</p><p>Announced at Amazon's AWS re:Invent conference, the service will help companies in the health care and pharmaceutical sectors analyze medical data trends.</p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/development/application-programming-interface-api/355165/nylas-announces-free-scheduling-tool-for" data-original-url="/development/application-programming-interface-api/355165/nylas-announces-free-scheduling-tool-for">Nylas announces free scheduling tool for health care providers</a> <a data-analytics-id="inline-link" href="https://www.itpro.com/marketing-comms/communications/355204/how-technology-is-changing-healthcare" data-original-url="/marketing-comms/communications/355204/how-technology-is-changing-healthcare">How technology is changing health care</a> <a data-analytics-id="inline-link" href="https://www.itpro.com/security/cyber-security/355791/cyberpeace-institute-urges-governments-to-work-together-to-stop" data-original-url="/security/cyber-security/355791/cyberpeace-institute-urges-governments-to-work-together-to-stop">Governments urged to work together to stop health care cyber attacks</a> <a data-analytics-id="inline-link" href="https://www.itpro.com/marketing-comms/communications/355164/chatbunkers-free-offering-streamlines-health-care" data-original-url="/marketing-comms/communications/355164/chatbunkers-free-offering-streamlines-health-care">ChatBunker's free offering streamlines health care communication during coronavirus outbreak</a></p></div></div><p>HealthLake is a data lake tailored to health applications. Data lakes store structured data, like financial records and contact information, alongside unstructured data, like Word documents, PDFs, and images, indexing it all so big data applications can mine that information. </p><p>HealthLake consumes information, ranging from diagnoses, medications, doctors' notes, and medical imaging reports to insurance claims. Many of these are in non-standard formats, and some are handwritten, but Amazon uses <a href="https://www.itpro.com/strategy/28071/what-is-machine-learning" data-original-url="https://www.itpro.com/strategy/28071/what-is-machine-learning">machine learning</a> to mine the text.</p><p>The service interprets handwritten text and makes it readable, along with digital documents. It then uses natural language processing to identify and index distinct pieces of health care information, tagging it with metadata to make it searchable. This includes chronological metadata, enabling data scientists to create a timeline of medical events.</p><p>Amazon wants health care companies to use this information for <a href="https://www.itpro.com/tag/big-data-analytics" data-original-url="https://www.itpro.com/big-data-analytics">big data analytics</a>. That includes making predictions about individual patients, such as when a person with diabetes is likely to develop high blood pressure based on parameters, including their age, gender, and lifestyle. It will also help analyze larger trend data across populations for applications ranging from health insurance to epidemiology.</p><p>The system stores all this information in the Fast Healthcare Interoperability Resources (FHIR) format from medical data standards nonprofit organization Health Level Seven. It enables health providers to share data, Amazon explained.</p><p>The system supports data privacy and security standards created under the 1996 Health Insurance Portability and Accountability Act (HIPAA), The Department of Health and Human Services (HHS) imposes penalties of up to almost $60,000 per violation with an annual cap of $1,786,000.</p><p>Insecure medical document storage has been a thorny issue in the US. In September 2019, investigative nonprofit journalism team organization ProPublica <a href="https://www.propublica.org/article/millions-of-americans-medical-images-and-data-are-available-on-the-internet">uncovered</a> medical images and health records for over five million US patients publicly exposed on 187 servers.</p><p>This is Amazon's second major health-care-related move in a few weeks. In mid-November, it announced <a href="https://press.aboutamazon.com/news-releases/news-release-details/introducing-amazon-pharmacy-prescription-medications-delivered">Amazon Pharmacy</a>, an online store enabling people to buy prescription medicines via the Amazon App using their health insurance. </p><p>In 2019, it launched its <a href="https://amazon.care">Amazon Care</a> health insurance initiative, targeting its own employees. This service offers telemedicine services to participants using a mobile app.</p> ]]></dc:content>
                                                                                                                                            <link>https://www.itpro.com/cloud/amazon-web-services-aws/358081/amazon-launches-big-data-service-focused-on-health-care</link>
                                                                            <description>
                            <![CDATA[ Data lake promises to keep health care organizations' information secure ]]>
                                                                                                            </description>
                                                                                                                                <guid isPermaLink="false">iRivr2Hxy1G5WjyXDaDdJC</guid>
                                                                                                <enclosure url="https://cdn.mos.cms.futurecdn.net/WZSr7JRK9hx9UDwaUQEjBR-1920-80.jpg" type="image/jpeg" length="0"></enclosure>
                                                                        <pubDate>Thu, 10 Dec 2020 11:56:28 +0000</pubDate>                                                                                                                                <updated>Thu, 19 Nov 2026 15:10:52 +0000</updated>
                                                                                                                                            <category><![CDATA[Deep Learning]]></category>
                                                    <category><![CDATA[Data and Analytics]]></category>
                                                    <category><![CDATA[Big Data]]></category>
                                                                                                                    <dc:creator><![CDATA[ Danny Bradbury ]]></dc:creator>                                                                                    <dc:source><![CDATA[ https://cdn.mos.cms.futurecdn.net/HdKafzrfv3Z6M5axRN8fhh-320-70.jpeg ]]></dc:source>
                                                                <dc:description><![CDATA[ null ]]></dc:description>
                                                                                                                                <cf:isSponsored>false</cf:isSponsored>
                <cf:hasAffiliateLinks>false</cf:hasAffiliateLinks>
                <cf:isPaid>false</cf:isPaid>
                                                                                                                                                                                    <media:content type="image/jpeg" url="https://cdn.mos.cms.futurecdn.net/WZSr7JRK9hx9UDwaUQEjBR-1920-80.jpg">
                                                            <media:credit><![CDATA[Shutterstock]]></media:credit>
                                                                                                                                                                                                                                    <media:description><![CDATA[Doctor behind cloud- and IT-related emblems ]]></media:description>                                                            <media:text><![CDATA[Doctor behind cloud- and IT-related emblems ]]></media:text>
                                <media:title type="plain"><![CDATA[Doctor behind cloud- and IT-related emblems ]]></media:title>
                                                    </media:content>
                                                    <media:thumbnail url="https://cdn.mos.cms.futurecdn.net/WZSr7JRK9hx9UDwaUQEjBR-1920-80.jpg" />
                                                                                                                                                                    <content:encoded >
                            <![CDATA[
                            <article>
                                <p>Amazon has launched another business venture, this time pushing into the health care and life sciences sectors. The company announced a preview of <a href="https://aws.amazon.com/healthlake">HealthLake</a>, a big data service housed in its Amazon Web Services (AWS) cloud.</p><p>Announced at Amazon's AWS re:Invent conference, the service will help companies in the health care and pharmaceutical sectors analyze medical data trends.</p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/development/application-programming-interface-api/355165/nylas-announces-free-scheduling-tool-for" data-original-url="/development/application-programming-interface-api/355165/nylas-announces-free-scheduling-tool-for">Nylas announces free scheduling tool for health care providers</a> <a data-analytics-id="inline-link" href="https://www.itpro.com/marketing-comms/communications/355204/how-technology-is-changing-healthcare" data-original-url="/marketing-comms/communications/355204/how-technology-is-changing-healthcare">How technology is changing health care</a> <a data-analytics-id="inline-link" href="https://www.itpro.com/security/cyber-security/355791/cyberpeace-institute-urges-governments-to-work-together-to-stop" data-original-url="/security/cyber-security/355791/cyberpeace-institute-urges-governments-to-work-together-to-stop">Governments urged to work together to stop health care cyber attacks</a> <a data-analytics-id="inline-link" href="https://www.itpro.com/marketing-comms/communications/355164/chatbunkers-free-offering-streamlines-health-care" data-original-url="/marketing-comms/communications/355164/chatbunkers-free-offering-streamlines-health-care">ChatBunker's free offering streamlines health care communication during coronavirus outbreak</a></p></div></div><p>HealthLake is a data lake tailored to health applications. Data lakes store structured data, like financial records and contact information, alongside unstructured data, like Word documents, PDFs, and images, indexing it all so big data applications can mine that information. </p><p>HealthLake consumes information, ranging from diagnoses, medications, doctors' notes, and medical imaging reports to insurance claims. Many of these are in non-standard formats, and some are handwritten, but Amazon uses <a href="https://www.itpro.com/strategy/28071/what-is-machine-learning" data-original-url="https://www.itpro.com/strategy/28071/what-is-machine-learning">machine learning</a> to mine the text.</p><p>The service interprets handwritten text and makes it readable, along with digital documents. It then uses natural language processing to identify and index distinct pieces of health care information, tagging it with metadata to make it searchable. This includes chronological metadata, enabling data scientists to create a timeline of medical events.</p><p>Amazon wants health care companies to use this information for <a href="https://www.itpro.com/tag/big-data-analytics" data-original-url="https://www.itpro.com/big-data-analytics">big data analytics</a>. That includes making predictions about individual patients, such as when a person with diabetes is likely to develop high blood pressure based on parameters, including their age, gender, and lifestyle. It will also help analyze larger trend data across populations for applications ranging from health insurance to epidemiology.</p><p>The system stores all this information in the Fast Healthcare Interoperability Resources (FHIR) format from medical data standards nonprofit organization Health Level Seven. It enables health providers to share data, Amazon explained.</p><p>The system supports data privacy and security standards created under the 1996 Health Insurance Portability and Accountability Act (HIPAA), The Department of Health and Human Services (HHS) imposes penalties of up to almost $60,000 per violation with an annual cap of $1,786,000.</p><p>Insecure medical document storage has been a thorny issue in the US. In September 2019, investigative nonprofit journalism team organization ProPublica <a href="https://www.propublica.org/article/millions-of-americans-medical-images-and-data-are-available-on-the-internet">uncovered</a> medical images and health records for over five million US patients publicly exposed on 187 servers.</p><p>This is Amazon's second major health-care-related move in a few weeks. In mid-November, it announced <a href="https://press.aboutamazon.com/news-releases/news-release-details/introducing-amazon-pharmacy-prescription-medications-delivered">Amazon Pharmacy</a>, an online store enabling people to buy prescription medicines via the Amazon App using their health insurance. </p><p>In 2019, it launched its <a href="https://amazon.care">Amazon Care</a> health insurance initiative, targeting its own employees. This service offers telemedicine services to participants using a mobile app.</p>
                                                            </article>
                            ]]>
                        </content:encoded>
                                                </item>
                                <item>
                                                            <title><![CDATA[ Home Office seeks AWS and Windows experts for criminal data lake ]]></title>
                                                                                                <dc:content><![CDATA[ <p>The Home Office is aiming to have its long-awaited criminal data lake up and running by 2023 after putting out a public tender for relevant experts. </p><p>The Law Enforcement Data (LED) service is on track to be completed by 2023, according to the Home Office, which will see the upgrade of outdated police resources into one searchable service.</p><p>The idea was first put forward in 2014 as the <a href="https://www.gov.uk/government/publications/law-enforcement-data-service-privacy-impact-assessment">National Law Enforcement Data Programme (NLEDP)</a>, but the scheme would eventually fall behind schedule and over budget. However, the Home Office is now seeking to attract contractors with a £3 million public tender through the <a href="https://www.digitalmarketplace.service.gov.uk/digital-outcomes-and-specialists/opportunities/11985">Police and Public Protection Technology directorate</a>, in an effort to revive the proposals.</p><p>The scheme is hoping to encourage applications from those with knowledge of AWS security services such as S3, EC2, DMS Databases, Cloudtrail, Cloudwatch, and also expertise of Windows, <a href="https://www.itpro.com/cloud/33999/ibm-doubles-down-on-red-hat-independence" data-original-url="https://www.itpro.com/cloud/33999/ibm-doubles-down-on-red-hat-independence">Red Hat</a> and <a href="https://www.itpro.com/security/25332/splunk-wants-you-to-monitor-your-employees-behaviour" data-original-url="https://www.itpro.com/security/25332/splunk-wants-you-to-monitor-your-employees-behaviour">Splunk</a>. The application window is open until 17 March.</p><p>The LED service will replace the Police National Computer (PNC) and the Police National Database (PND). Due to the age of these systems, the technology that underpins them is increasingly difficult and expensive to maintain - PNC was first launched in 1974 and is largely run on the same technology.</p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/databases/33927/mod-unlawfully-searches-police-biometric-data" data-original-url="/databases/33927/mod-unlawfully-searches-police-biometric-data">MoD 'unlawfully' searches police biometric data</a> <a data-analytics-id="inline-link" href="https://www.itpro.com/digital-transformation/34223/manchester-police-forced-to-release-suspects-after-botched-it-upgrade" data-original-url="/digital-transformation/34223/manchester-police-forced-to-release-suspects-after-botched-it-upgrade">Manchester police forced to release suspects after botched IT upgrade</a> <a data-analytics-id="inline-link" href="https://www.itpro.com/digital-transformation/34772/capita-partners-with-what3words-to-improve-999-emergency-locating" data-original-url="/digital-transformation/34772/capita-partners-with-what3words-to-improve-999-emergency-locating">Capita partners with What3Words to improve 999 emergency locating</a></p></div></div><p>PND was a service introduced as a recommendation following the Bichard enquiry into the Soham Murders in 2002. The report recommended a national system for sharing police intelligence to ensure better protection for the public, but, despite launching in 2009, that too is in need of upgrading.</p><p>"To tackle modern crime it is vital that the police have access to fast and accurate data and intelligence that can be shared between forces," a Home Office spokesperson said. "The Police National Computer and Police National Database will be replaced by the new Law Enforcement Data Service (LEDS), saving £216 million over ten years when it is rolled out nationally."</p><p>The Police and Public Protection Technology directorate contains a number of delivery programmes, including the Law Enforcement Cloud Platform, which will be hosted on a public cloud. The programme is still in the design and planning phase and the Home Office were unable to confirm when this will be live.</p> ]]></dc:content>
                                                                                                                                            <link>https://www.itpro.com/business-strategy/digital-transformation/354942/home-office-seeks-aws-and-windows-experts-for</link>
                                                                            <description>
                            <![CDATA[ UK government is hoping to replace legacy systems first installed in 1974 ]]>
                                                                                                            </description>
                                                                                                                                <guid isPermaLink="false">7fB5sNm9zGXSaKfoXVYeiV</guid>
                                                                                                <enclosure url="https://cdn.mos.cms.futurecdn.net/HbtGReLqfVifH5HPEUFMB4-1920-80.jpg" type="image/jpeg" length="0"></enclosure>
                                                                        <pubDate>Mon, 09 Mar 2020 11:18:33 +0000</pubDate>                                                                                                                                <updated>Thu, 19 Nov 2026 13:16:04 +0000</updated>
                                                                                                                                            <category><![CDATA[Deep Learning]]></category>
                                                    <category><![CDATA[Digital Transformation]]></category>
                                                    <category><![CDATA[Data and Analytics]]></category>
                                                    <category><![CDATA[Big Data]]></category>
                                                    <category><![CDATA[IT Leadership]]></category>
                                                                                                <author><![CDATA[ itpro@futurenet.com (Bobby Hellard) ]]></author>                    <dc:creator><![CDATA[ Bobby Hellard ]]></dc:creator>                                                                                    <dc:source><![CDATA[ https://cdn.mos.cms.futurecdn.net/bsR2tHSyVKUoyXZF5pNsDA-320-70.jpg ]]></dc:source>
                                                                <dc:description><![CDATA[ &lt;p&gt;Bobby Hellard&amp;nbsp;is&amp;nbsp;ITPro&#039;s Reviews Editor and has worked on&amp;nbsp;CloudPro and ChannelPro since 2018. In his time at ITPro, Bobby has covered stories for all the major technology companies, such as Apple, Microsoft, Amazon and Facebook, and regularly attends industry-leading events such as AWS Re:Invent and Google Cloud Next.&lt;/p&gt;
&lt;p&gt;Bobby mainly covers hardware reviews, but you will also recognize him as the face of many of our video reviews of laptops and smartphones.&lt;/p&gt;
&lt;p&gt;He has been a journalist for ten years, originally covering sports, before moving into business technology with ITPro. He has bylines in The Independent, Vice and The Business Briefing. Contact him at &lt;a href=&quot;mailto:bobby.hellard@futurenet.com&quot;&gt;bobby.hellard@futurenet.com&lt;/a&gt; or find him on Twitter: &lt;a href=&quot;https://twitter.com/bobbyhellard&quot;&gt;@bobbyhellard&lt;/a&gt;&lt;/p&gt; ]]></dc:description>
                                                                                                                                <cf:isSponsored>false</cf:isSponsored>
                <cf:hasAffiliateLinks>false</cf:hasAffiliateLinks>
                <cf:isPaid>false</cf:isPaid>
                                                                                                                                                                                    <media:content type="image/jpeg" url="https://cdn.mos.cms.futurecdn.net/HbtGReLqfVifH5HPEUFMB4-1920-80.jpg">
                                                            <media:credit><![CDATA[null]]></media:credit>
                                                                                                                                                                        <media:description><![CDATA[Police on the street]]></media:description>                                                            <media:text><![CDATA[Police bill parliament]]></media:text>
                                <media:title type="plain"><![CDATA[Police bill parliament]]></media:title>
                                                    </media:content>
                                                    <media:thumbnail url="https://cdn.mos.cms.futurecdn.net/HbtGReLqfVifH5HPEUFMB4-1920-80.jpg" />
                                                                                                                                                                    <content:encoded >
                            <![CDATA[
                            <article>
                                <p>The Home Office is aiming to have its long-awaited criminal data lake up and running by 2023 after putting out a public tender for relevant experts. </p><p>The Law Enforcement Data (LED) service is on track to be completed by 2023, according to the Home Office, which will see the upgrade of outdated police resources into one searchable service.</p><p>The idea was first put forward in 2014 as the <a href="https://www.gov.uk/government/publications/law-enforcement-data-service-privacy-impact-assessment">National Law Enforcement Data Programme (NLEDP)</a>, but the scheme would eventually fall behind schedule and over budget. However, the Home Office is now seeking to attract contractors with a £3 million public tender through the <a href="https://www.digitalmarketplace.service.gov.uk/digital-outcomes-and-specialists/opportunities/11985">Police and Public Protection Technology directorate</a>, in an effort to revive the proposals.</p><p>The scheme is hoping to encourage applications from those with knowledge of AWS security services such as S3, EC2, DMS Databases, Cloudtrail, Cloudwatch, and also expertise of Windows, <a href="https://www.itpro.com/cloud/33999/ibm-doubles-down-on-red-hat-independence" data-original-url="https://www.itpro.com/cloud/33999/ibm-doubles-down-on-red-hat-independence">Red Hat</a> and <a href="https://www.itpro.com/security/25332/splunk-wants-you-to-monitor-your-employees-behaviour" data-original-url="https://www.itpro.com/security/25332/splunk-wants-you-to-monitor-your-employees-behaviour">Splunk</a>. The application window is open until 17 March.</p><p>The LED service will replace the Police National Computer (PNC) and the Police National Database (PND). Due to the age of these systems, the technology that underpins them is increasingly difficult and expensive to maintain - PNC was first launched in 1974 and is largely run on the same technology.</p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/databases/33927/mod-unlawfully-searches-police-biometric-data" data-original-url="/databases/33927/mod-unlawfully-searches-police-biometric-data">MoD 'unlawfully' searches police biometric data</a> <a data-analytics-id="inline-link" href="https://www.itpro.com/digital-transformation/34223/manchester-police-forced-to-release-suspects-after-botched-it-upgrade" data-original-url="/digital-transformation/34223/manchester-police-forced-to-release-suspects-after-botched-it-upgrade">Manchester police forced to release suspects after botched IT upgrade</a> <a data-analytics-id="inline-link" href="https://www.itpro.com/digital-transformation/34772/capita-partners-with-what3words-to-improve-999-emergency-locating" data-original-url="/digital-transformation/34772/capita-partners-with-what3words-to-improve-999-emergency-locating">Capita partners with What3Words to improve 999 emergency locating</a></p></div></div><p>PND was a service introduced as a recommendation following the Bichard enquiry into the Soham Murders in 2002. The report recommended a national system for sharing police intelligence to ensure better protection for the public, but, despite launching in 2009, that too is in need of upgrading.</p><p>"To tackle modern crime it is vital that the police have access to fast and accurate data and intelligence that can be shared between forces," a Home Office spokesperson said. "The Police National Computer and Police National Database will be replaced by the new Law Enforcement Data Service (LEDS), saving £216 million over ten years when it is rolled out nationally."</p><p>The Police and Public Protection Technology directorate contains a number of delivery programmes, including the Law Enforcement Cloud Platform, which will be hosted on a public cloud. The programme is still in the design and planning phase and the Home Office were unable to confirm when this will be live.</p>
                                                            </article>
                            ]]>
                        </content:encoded>
                                                </item>
                                <item>
                                                            <title><![CDATA[ Google DeepMind has learned to read the London Underground ]]></title>
                                                                                                <dc:content><![CDATA[ <p>Learning to read the London Underground map is a rite of passage for any new Londoner, but DeepMind -- Google's deep learning AI with an interest in healthcare and card games has joined the ranks of those that know the fastest way to get from Acton Town to Wapping.</p><p>That may not sound hugely impressive, but the manner in which it has learned to read the Tube map is very interesting for the future of artificial intelligence, as it used basic reasoning and memory to conquer the commute. In other words, it was more human than your average Tube map app.</p><p>"I think this can be described as rational reasoning," Herbert Jaegar, a computer scientist from the University of Bremen, told The Guardian. "They [the tasks] involve planning and structuring information into chunks and re-combining them." Combining deep learning with an external memory means that DeepMind could take what it learned from the London Underground, and apply it to navigating other similar transport networks around the world.</p><p>This is different from things that have gone before. As Alex Graves, a research scientist at DeepMind, told Wired: "You can't give normal neural networks a piece of information and let them keep it indefinitely in their internal state -- at some point it will be overwritten and they will essentially forget it." This neural network, however, could keep the memory forever.</p><p>The same strategy was used on two other tasks both of which, again, seem trivial to humans. DeepMind was given simple extracts of stories, such as "John is in the playground. John picked up the football." From there the AI would be asked where the football was, and provided the correct answer to these kind of puzzles 96% of the time. Graves concedes that while these puzzles "look so trivial to a human that they don't seem like questions at all," it's the methodology that is interesting. Traditional computers, he says, "do really badly at this."</p><p>In another puzzle, <a href="https://www.youtube.com/watch?v=B9U8sI7TcMY&amp;feature=youtu.be" target="_blank">explained in this video</a>, DeepMind was able to establish familial relationships by reading a family tree.</p><p>"Taken together, our results demonstrate that [differentiable neural computers] have the capacity to solve complex, structured tasks that are inaccessible to neural networks without external read-write memory," the authors concluded in their paper. "Like a conventional computer, it can use its memory to represent and manipulate complex data structures, but, like a neural network, it can learn to do so from data."</p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/strategy/27223/google-deepmind-gets-closer-to-sounding-human" data-original-url="/strategy/27223/google-deepmind-gets-closer-to-sounding-human">Google DeepMind gets closer to sounding human</a></p></div></div> ]]></dc:content>
                                                                                                                                            <link>https://www.itpro.com/networking/27404/google-deepmind-just-learned-to-read-the-london-underground-map-through-memory-and</link>
                                                                            <description>
                            <![CDATA[ While it sounds basic, the manner in which the AI is learning is a big breakthrough ]]>
                                                                                                            </description>
                                                                                                                                <guid isPermaLink="false">fUjGuJK7jmRfkXookEYZN9</guid>
                                                                                                <enclosure url="https://cdn.mos.cms.futurecdn.net/p7TrXEEdr7gFgnfsSWwsUn-1920-80.jpg" type="image/jpeg" length="0"></enclosure>
                                                                        <pubDate>Thu, 13 Oct 2016 12:31:00 +0000</pubDate>                                                                                                                                <updated>Thu, 19 Nov 2026 13:16:04 +0000</updated>
                                                                                                                                            <category><![CDATA[Deep Learning]]></category>
                                                    <category><![CDATA[Artificial Intelligence]]></category>
                                                    <category><![CDATA[Data and Analytics]]></category>
                                                    <category><![CDATA[Big Data]]></category>
                                                                                                <author><![CDATA[ itpro@futurenet.com (Alan Martin) ]]></author>                    <dc:creator><![CDATA[ Alan Martin ]]></dc:creator>                                                                                    <dc:source><![CDATA[ https://cdn.mos.cms.futurecdn.net/5bf2e2KF2ydUWCdv7bxuYS-320-70.jpg ]]></dc:source>
                                                                <dc:description><![CDATA[ null ]]></dc:description>
                                                                                                                                <cf:isSponsored>false</cf:isSponsored>
                <cf:hasAffiliateLinks>false</cf:hasAffiliateLinks>
                <cf:isPaid>false</cf:isPaid>
                                                                                                                                                                                    <media:content type="image/jpeg" url="https://cdn.mos.cms.futurecdn.net/p7TrXEEdr7gFgnfsSWwsUn-1920-80.jpg">
                                                            <media:credit><![CDATA[null]]></media:credit>
                                                                                                                                                                                                                                                                                                                                                    </media:content>
                                                    <media:thumbnail url="https://cdn.mos.cms.futurecdn.net/p7TrXEEdr7gFgnfsSWwsUn-1920-80.jpg" />
                                                                                                                                                                    <content:encoded >
                            <![CDATA[
                            <article>
                                <p>Learning to read the London Underground map is a rite of passage for any new Londoner, but DeepMind -- Google's deep learning AI with an interest in healthcare and card games has joined the ranks of those that know the fastest way to get from Acton Town to Wapping.</p><p>That may not sound hugely impressive, but the manner in which it has learned to read the Tube map is very interesting for the future of artificial intelligence, as it used basic reasoning and memory to conquer the commute. In other words, it was more human than your average Tube map app.</p><p>"I think this can be described as rational reasoning," Herbert Jaegar, a computer scientist from the University of Bremen, told The Guardian. "They [the tasks] involve planning and structuring information into chunks and re-combining them." Combining deep learning with an external memory means that DeepMind could take what it learned from the London Underground, and apply it to navigating other similar transport networks around the world.</p><p>This is different from things that have gone before. As Alex Graves, a research scientist at DeepMind, told Wired: "You can't give normal neural networks a piece of information and let them keep it indefinitely in their internal state -- at some point it will be overwritten and they will essentially forget it." This neural network, however, could keep the memory forever.</p><p>The same strategy was used on two other tasks both of which, again, seem trivial to humans. DeepMind was given simple extracts of stories, such as "John is in the playground. John picked up the football." From there the AI would be asked where the football was, and provided the correct answer to these kind of puzzles 96% of the time. Graves concedes that while these puzzles "look so trivial to a human that they don't seem like questions at all," it's the methodology that is interesting. Traditional computers, he says, "do really badly at this."</p><p>In another puzzle, <a href="https://www.youtube.com/watch?v=B9U8sI7TcMY&amp;feature=youtu.be" target="_blank">explained in this video</a>, DeepMind was able to establish familial relationships by reading a family tree.</p><p>"Taken together, our results demonstrate that [differentiable neural computers] have the capacity to solve complex, structured tasks that are inaccessible to neural networks without external read-write memory," the authors concluded in their paper. "Like a conventional computer, it can use its memory to represent and manipulate complex data structures, but, like a neural network, it can learn to do so from data."</p><div  class="fancy-box"><div class="fancy_box-title"></div><div class="fancy_box_body"><p class="fancy-box__body-text"><a data-analytics-id="inline-link" href="https://www.itpro.com/strategy/27223/google-deepmind-gets-closer-to-sounding-human" data-original-url="/strategy/27223/google-deepmind-gets-closer-to-sounding-human">Google DeepMind gets closer to sounding human</a></p></div></div>
                                                            </article>
                            ]]>
                        </content:encoded>
                                                </item>
            </channel>
</rss>