Hadoop is one of the most popular open source cloud platforms used by developers, data scientists, and engineers to process enormous amounts of data, including analysing, sorting and storing information.

The platform is used to deliver key understandings into a company's health, whether financial or operational, allowing stakeholders to make decisions based upon data.

One of the main advantages of using Hadoop over other cloud platforms is that everything happens in real-time, enabling faster reaction times and meaning businesses can always keep one step ahead of competitors.

Debuting more than 10 years ago, Hadoop has undergone many changes, essentially evolving to take account of shifting demands from businesses, including the capacity to process much larger datasets.

For example, the first iteration was able to sort 1.8 TB on 188 nodes in 47.9 hours - revolutionary for its time. However, this has been increased considerably in a decade and much greater datasets can now be handled in less time thanks to contributions from the likes of Facebook, LinkedIn, eBay and IBM.

Hadoop was first developed by Doug Cutting and Mike Carafella, two software engineers that wanted to improve web indexing. It was built upon Google's File System paper and was created as the Apache Nutch project, after tons of experimentation by Cutting and the subsequent creation of Apache Lucene, previous to his tie-up with Carafella.

When Cutting started working for Yahoo in 2006, the platform's development accelerated thanks to a dedicated team to work on it. Fast forward to 2008 and it became clear that Yahoo was no longer the place for Hadoop to flourish.

At this point, Rob Beardon and Eric Badleschieler who had been heavily involved in the development of Hadoop made the decision to split Hortonworks, which was currently responsible for the evolution of the platform, from Yahoo. They created a team of developers that could focus primarily on its future.

Google hasn't completely abandoned Hadoop, but the evolution reigns are now managed by Hortonworks' founder Owen O'Malley.

Hadoop's main role is to store, manage and analyse vast amounts of data using commoditised hardware. What makes it so effective is the way in which it structures the data, making it extremely efficient to sort it autonomously.

Big data has huge potential, but only if you work out how to harness it. That means you must think about how you want to use it, and what answers you want from the data. That will determine your approach to data analytics, and help you reach the right insights that give your products and services an edge over the competition.

How does Hadoop work?

Four key elements comprise Hadoop; the Distributed File System, MapReduce, Hadoop Common and YARN. These 'modules' are each responsible for distinct tasks within the software.

The Distributed File System (DFS) provides the storage component; splitting files into fragments that it then distributes over many servers to improve high bandwidth streaming. This approach also makes it simple to increase storage capacity, compute power and even the bandwidth itself - all you need to do is install more commodity hardware.

Next, we come to the MapReduce module, the piece that performs the analytics on the datasets. It works by taking a large volume of data and crunching it into smaller datasets - perhaps the most crucial part of Hadoop's software.

MapReduce does two things - first, its map function crawls through the data, and slices it into key-value pairs. Reduce then collates the results of the mapping. We can see how this works on word-counting tasks.

Let's say we performed MapReduce on the 'she sells sea shells' tongue-twister, first of all the mapping part of the software would break the text down and count each word, which would look like this:

(she, 1)(sells, 1)(sea, 1)(shells, 1)(on, 1)(the, 1)(sea, 1)(shore, 1)(the, 1)(sea, 1)(shells, 1)(that, 1)(she, 1)(sells, 1)(are, 1)(sea, 1)(shells, 1)(i'm, 1)(sure, 1)

Once this operation has completed, the reduce function then groups the results together, returning an aggregate of the various values. In this case, that would be:

(she, 2)(sells, 2)(sea, 4)(shells, 3)(on, 1)(the, 2)(shore, 1)(that, 1)(are, 1)(i'm, 1)(sure, 1)

While this is an exceedingly simplistic example, it does illustrate the vital role MapReduce plays in breaking down huge datasets into more manageable formats.

The other modules that make up Hadoop are Hadoop Common, which refers to the utilities, libraries, packages and other miscellaneous files that the framework requires to function, and YARN (short for Yet Another Resource Negotiator), a cluster management tool that offers a central platform for research management across Hadoop infrastructure.

Why is Hadoop so popular?

While there are alternatives to Hadoop, it's unquestionably the most popular Big Data processing framework in the enterprise. There are a few very good reasons for this. The first is that it's fantastically economical because a Hadoop cluster can be built using commodity hardware, the setup and operational costs are extremely low compared with other technologies. It's open source, too, which means that there are no inherent licensing costs.

It's also highly scalable, which has been a boon to businesses that are looking to quickly spin up their Big Data operations. Hadoop also offers very high bandwidth across the cluster, allowing large workloads to be completed much more quickly and with fewer bottlenecks.

What are the main problems with Hadoop?

The biggest problem facing Hadoop is that it's still incredibly complex. While other elements of the enterprise IT stack have benefitted from increasing simplification and user-friendliness, Hadoop is still one of the more labyrinthine and confusing technologies in common use.

This makes it difficult for businesses to find people who can start working on Hadoop deployments or to up-skill existing staff in its implementation. It can also make it incredibly hard to sell the board on its benefits, as non-IT folks often have a hard time getting their head around what it actually does.

Big data represents a shift in the enterprise technology landscape. As the volume and variety of data continues, companies are increasingly turning to Hadoop, NoSQL and other tools to tackle information issues not readily addressable with older data technologies.

This whitepaper outlines the challenges many companies face with big data, and sets out a blueprint for succeeding with these scenarios:

• Data warehouse optimisation
• Streamlined data refinery
• Monetising data
• Getting a 360-degree customer view

Download it now to learn more.

Over the last five years, there have been few more disruptive forces in information technology than big data, and at the centre of this trend is the Hadoop ecosystem.

While a whole ecosystem of tools has sprung up around Hadoop to handle and analyse data, many of them are specialised to just one part of a larger process.

This whitepaper shows how taking an end-to-end view of the analytic data pipeline can fulfil Hadoop’s true value, as well as promoting project success and enhanced IT collaboration within the business.

Are you from the Hortonworks side or the Cloudera side? It's a question I found myself asking a lot at this year's DataWorks Summit, the first major event since the two companies completed their $5.2 billion merger just months ago. Naturally, a marriage of this scale throws up a tidal wave of questions. Unfortunately, there were no answers to be found in Barcelona.

It's difficult to put my finger on the mood in the air, but it was closest to uncertainty. Given that DataWorks Summit has conventionally been a Hortonworks event, having the 'new' Cloudera spearhead it was jarring. Not just for the press, but the comms team. The reason? The May-time Washington DataWorks Summit, as well as Cloudera's two Strata Data Conferences had already been planned and organised months before the merger was tied up. So the company has to effectively go through the motions with its 2019 events.

But it was especially confusing given the Hortonworks branding appears to have been discarded entirely. Instead, the two companies, now operating under the Cloudera umbrella, have undergone a complete image refresh, with a newly-designed logo and several buzzy slogans to boot.

A new image is always something to get excited about. But the fact Cloudera was handing out metal pins emblazoned with the company's old logo summed up the feeling quite effectively. Its Twitter page, too, is still displaying the company's old logo at the time of writing.

Meanwhile, the event was fronted by the firm's chief marketing officer (CMO) Mick Hollison. This underpinned the company's almost singular focus on 'image' this week, which on one level made sense. Earnings day the week before made for grim reading. Revenue grew, sure. But so did expenditure, by quite a lot. This doubled losses to more than $85 million. Yet Cloudera is setting itself a target of becoming a billion dollar company before the end of the year, and reinforced its ambitions to target only the largest companies.

But it didn't seem appropriate that a significant portion of the top brass was left at home. Anybody who could give serious answers about Cloudera's financial performance, or specific details about the merger, was not available to chat. Then it hit me during the main keynote when it became clear CMO Hollison would be the only Cloudera voice addressing the press, analysts and delegates that morning. Chiefly, at Cloudera's first major public event since the merger, it begged the question: Where was the CEO?

It's not fair to say that everybody with prominence was left at home. Hilary Mason, Cloudera's resident data scientist and the lead on its research division, dazzled on the evolving nature of AI. Meanwhile, there were some interesting insights to gain on data warehousing, open source, and GDPR. The thematic substance of DataWorks Summit 2019 was actually quite positive despite the company's considered efforts to push its new marketing slogans, namely 'from the edge to AI' and 'the enterprise data cloud'.

But the merger, undoubtedly, was at the forefront of everyone's minds, with many questions lingering. Though now that it has mostly been completed, it was interesting to hear discussions with Hortonworks were actually underway for three-and-a-half years before the two firms tied the knot.

Yet we still don't fully know what its flagship service, named the Cloudera Data Platform (CDP), will look like. We do, however, know it's a mash-up of Hortonworks and Cloudera's legacy systems, Cloudera Distribution Including Apache Hadoop (CDH) and Hortonworks Data Platform (HDP).

Neither do we know when this will launch, with Cloudera officially saying it will come within the next two quarters. But one customer, Swiss insurance firm Zurich, told IT Pro it was coming in June. Meanwhile, while customers are allowed to keep these legacy platforms until around 2022, for Zurich, currently in the process of migrating from HDP 2.0 to 3.0, does this then mean a second big transition in quick succession? The aim is, of course, to transition all customers to CDP eventually.

The future is uncertain. So much so that nobody really knows if the DataWorks Summits held in 2019 will be the last ever. Nevertheless, this presented a fantastic opportunity for Cloudera to address the world post-merger, and take on its major challenges head-on.

But this was an opportunity missed. The fact its most senior staff were left at home spoke volumes, even though the substance of the conference was for the most part engaging. It became clear over the course of the event that there hasn't been, and probably won't be, a honeymoon period for the 'new' Cloudera as it begins to find its feet in a turbulent market.

The 'new' Cloudera has committed to becoming a fully open-source company, having followed an open-core model prior to its $5.2 billion merger with former rival Hortonworks.

All 32 of the current open source projects found between both Hortonworks and Cloudera's legacy platforms will remain available as cloud-based services on its new jointly-developed Cloudera Data Platform (CDP).

There were fears Cloudera's influence could undermine the "100% open source" principles that underpinned Hortonworks, given the former had previously been just an 'open-core' company. This amounted to a business model in which limited versions of Cloudera projects were offered in line with open source principles, with additional features available at a cost.

Cloudera first made reassurances over its commitment to open source on a conference call with journalists last week. This call was made to explain the firm's dismal Q4 2018 financial results which saw the company's net losses double post-merger to $85.5m.

The commitment, which Cloudera elaborated at the company's DataWorks Summit 2019 hosted in Barcelona this week, has also coincided with a complete rebranding of the company logo, and further elaboration over its vision for an 'enterprise data cloud'.

This, according to the firm's chief marketing officer Mick Hollison, includes multi-faceted data analytics and support for every conceivable cloud model from multiple public clouds to hybrid cloud to containers like Kubernetes.

It would also be underpinned with a common compliance and data governance regime, and would retain a commitment to "100% open source", with Hollison insisting several times to journalists at a press briefing the term "isn't just marketing fluff".

Cloudera's vice president for product management Fred Koopmans told journalists at the same press briefing that both company's existing customers valued the principles of 'openness' - which starts with open APIs.

"They don't view that there is one vendor that's going to serve all of their needs today and in the future," Koopmans said. "Therefore it's critical for them to have open APIs so they can bring in other software development companies that can extend it and enhance the platform.

"What open source provides them is no dead-ends; if they're trying to develop something, and there's a particular feature they need. They always have the option of going and adding a feature with their own development team. So this is a huge driver for a lot of our larger customers in particular."

Cloudera also used the DataWorks Summit to outline its intentions to exclusively chase the biggest enterprise customers, insisting the firm is only interested in tackling big data problems for large companies.

CDP, the embodiment of the new vision, is due to make its way to customers as only a public cloud platform later this year, with a private cloud iteration to follow in late 2019 or early 2020. The platform is a mashing-together of Cloudera's Cloudera Distribution including Apache Hadoop (CDH) and Hortonworks' Hortonworks Data Platform (HDP).

Cloud computing and big data systems have without doubt been among the most disruptive technology trends of the past decade. These developments have irrevocably changed the way that technology companies operate, and how value is delivered to stakeholders.

With cloud computing, firms have been able to optimise IT operations through significant reduction of the need to invest in on-premise hardware and software, as well as provide businesses with more strategy options than ever before. Digital transformation and the shift to the cloud continues to show momentum, with the public cloud services market growing 29% from the previous year in 2021 according to IDC.

Simultaneously, big data technologies have provided enterprises with an unprecedented avenue of data value from assets. In the past, high-volume, diverse data proved troublesome for enterprises that were used to working with traditional database technology. But great technical developments allowed businesses to reduce the overhead required to transfer raw data into sorted information within a data store, as well as drastically increasing the efficiency, accessibility and speed of mass data processing.

These innovations have allowed for actionable analysis on a range of once-challenging data sources, including web logs, documents and machine sensors. Even so-called ‘dark data' has been revitalised through these technologies.

Big Data systems help organisations solve the trickiest problems, but they generally require a significant up-front and ongoing IT investment, in addition to dealing with the sheer amount of data in more ambitious projects.

Therefore, it makes sense that enterprises are turning to cloud providers with expertise in managing and maintaining scalable, and flexible, computing and storage infrastructure. While on-premise data systems are far from defunct, organisations are starting to effectively push the limits of big data analytics at scale by tapping into big data systems hosted on cloud infrastructure.

While many of these early cloud projects involve high volumes of structured data, there are 3 key components that are already enabling extraction of value from massive, diverse data sets on cloud infrastructure:

Cloud Analytical Databases

These cloud-based services, such as Amazon RedShift, are elastic data warehouses optimised for analytics with existing Business Intelligence tools. This type of analytical database includes management as well as solutions monitoring for the provider.

Hosted Hadoop Services

Hadoop clusters are often hosted in the cloud, which negates the need for on-premise infrastructure. Some Hadoop cloud offerings come with additional managed services, like job troubleshooting, software installation, testing and other such services.

Data Integration and Analytics

Databases built for high-performance analytics can make use of cloud techniques like compression, column-based storage and high-speed inserts of structured data, perfect for sophisticated queries and truly insightful business analytics.

Early adopters are already demonstrating how the cloud can improve upon the value proposition of big data, delivering elastic and cost-effective solutions for analysing data at unprecedented scale.

This article was first published on 12/09/17, and has since been updated

Oracle is adding new analytics capabilities to Hadoop and NoSQL databases, as it aims to help people struggling to make sense of huge datasets.

Its latest set of tools, labelled Oracle Big Data Spatial and Graph, targets a wider variety of datasets than the company's older solutions in this space, while cutting back on any need to move data around.

Instead, users can process data inside Hadoop while running the tool, saving time and money on analytics compared to moving the data into the cloud or another system to analyse it.

The spatial part of Oracle's tool helps examine data based on a variety of factors.

These include the proximity of the data and where it is in terms of geographical boundaries.

The spatial capabilities also allow users to find geospatial patterns like connections between different organisations, or the proximity of different clients to each other.

For example, it can look at datasets like Twitter feeds that include a zip code or street address, and add or update city, state, and country information.

It can also filter or group results based on spatial relationships, claimed Oracle, like finding how many customers are in each sales territory.

These results can be visualized on a map using its HTML5-based web mapping tool.

Meanwhile, the graph part of the tool enables the parallel in-memory analytics run in Hadoop.

Steve Pierce, CEO of Oracle partner Think Huddle, said: "With the introduction of Oracle Big Data Spatial and Graph, Hadoop users will be able to enrich data based on location and use this to harmonise data for further correlation, categorisation and analysis.

"For traditional geospatial workloads, it will provide value-added spatial processing and allow us to support customers with large vector and raster data sets on Hadoop systems."

Almost two thirds of IT and business leaders think the move to Hadoop is being held back by the skills gap, a report by Gartner has revealed.

Although the tools to help IT department get to grips with the framework are getting better, they are suited to highly skilled workers rather than improving the skills of existing employees.

An additional barrier to adoption is uncovering value from the framework, with 49 per cent of the 125 people surveyed saying it was holding them back.

Nick Heudecker, research director at Gartner said: "Despite considerable hype and reported successes for early adopters, 54 per cent of survey respondents report no plans to invest at this time, while only 18 per cent have plans to invest in Hadoop over the next two years.

"Furthermore, the early adopters don't appear to be championing for substantial Hadoop adoption over the next 24 months; in fact, there are fewer who plan to begin in the next two years than already have."

Other findings revealed in the organisations that have deployed Hadoop, 70 per cent have between one and 20 users accessing services, while four per cent of organisations don't have a single user accessing the framework.

"With such large incidence of organisations with no plans or already on their Hadoop journey, future demand for Hadoop looks fairly anemic over at least the next 24 months. Moreover, the lack of near-term plans for Hadoop adoption suggest that, despite continuing enthusiasm for the big data phenomenon, demand for Hadoop specifically is not accelerating," said Merv Adrian, research vice president at Gartner.

"The best hope for revenue growth for providers would appear to be in moving to larger deployments within their existing customer base."

Hitachi Data Systems (HDS) has expanded its range of software-defined storage so customers can pick and choose their preferred capacity.

Announced at HDS Connect 2015 in Las Vegas, Virtual Storage Platform (VSP) models now include the G200, G400, G600 and an upcoming G800 in addition to its G1000 product, offering users storage virtualisation and data migration functionality no matter what capacity they require.

"Customers now have the ability to choose systems based on the necessary capacity, performance and price required to meet their business goals, not because of functional difference," HDS said.

Running on all the VSP models is Hitachi's Storage Virtualization Operating System (SVOS), which helps customers consolidate and simplify user environments and perform easier migrations.

Converged platform

HDS has also grown its Unified Compute Platform (UCP) portfolio, adding a hyper-converged product in UCP 1000, which supports VMware's Evo:Rail appliance, a combined compute, networking, and storage solution competing with Oracle's more expensive converged systems.

Like the 1000, the newly-announced UCP 2000 uses new rack servers and is aimed at SMB office users, according to HDS.

The company's UCP 6000 features Hitachi CB 2500 blade servers, and the expanded range was welcomed by retailer Spar.

"The Hitachi UCP technologies will help us stay ahead in the markets, fine tune what we need to supply to best meet customer needs, and support a most efficient business model," said Andreas Kranabitl, managing director for SPAR's information and communication services.

"We at SPAR are now wisely prepared to meet and proactively address any changes and growth."

Data lake

Yesterday, upcoming HDS acquisition Pentaho spoke about its ability to make data lakes "easy and safe to swim" in - a timely announcement with HDS now revealing a new Hyper Scale-Out Platform (HSP).

Capable of swallowing massive amounts of different types of data across a distributed, clustered architecture, the firm claims Hitachi file system technology enables this data lake to grow elastically with the help of virtualisation.

HSP is being positioned as an ideal Hadoop platform, allowing users to analyse data there and then rather than moving it out of Hadoop in order to do so.

IT-as-a-service management

Lastly, HDS has bolstered its IT management suite to help out beleaguered IT departments.

It reckons these solutions will lead to greater automation, cutting costs and complexity along the way.

Its Automation Director works with Hitachi Command Suite to let customers create service templates for simple, application-specific provisioning of storage resources to databases, applications and VDI environments.

Once created, end users can also try these templates out in a self-service capacity.

Meanwhile, Infrastructure Director is a management app that hooks into the VSP systems' APIs to automatically control environments that don't need too much manual tweaking.

Instance Director is designed to simplify data protection by automating backup snapshots and data cloning, in addition to live backup and archiving under one platform.

"The fundamentals of IT are changing," Sean Moser, senior VP of HDS product management, said. "Customers are looking for ways to be more agile and flexible to gain a competitive advantage.

"Hitachi is delivering a new generation of software-defined technologies built to support the changing needs of customer application environments to deliver on those goals and connect what works today with what's next."

Storage giant EMC has announced a raft of enhancements for its ViPR storage management platform, which it claims will cut big data processing times by up to 80 per cent.

The product, which was announced by EMC at its annual customer and partner conference last May and went on general release in September 2013, is a core component of the vendor's software-defined datacentre strategy.

It provides users with a self-service portal where they can view available storage resources, allowing them to provision the applications and services they need.

It also allows systems admins to pool resources from a variety of different storage arrays, including those made by rival vendors, and boasts support for standard APIs from the likes of Amazon Web Services, OpenStack and Atmos.

The vendor has now added to the product's functionality by enhancing its object storage capabilities through the inclusion of support for the Hadoop Distributed File System (HDFS) in its ViPR 1.1 release.

This means IT managers can turn their existing storage infrastructure into a big data repository, allowing Hadoop analytics to be applied to any data that's under the control of ViPR.

Speaking to IT Pro, Chris Ratcliffe, vice president of marketing within EMC's advanced software division, said the inclusion of Hadoop support will have a huge impact on big data processing times.

This is because, rather than having to move their data somewhere else for processing, the HDFS enhancements allow the analysis to be carried out where the data already resides.

"For customers it's all about time to results with big data and this can reduce that by 70-to-80 per cent," he said.

Since ViPR made its debut, Ratcliffe said the product has been warmly received, and has even won over users that are new to EMC.

To ensure this momentum continues, the company has also confirmed the 1.0 version of the software will be available from today as a free download.

As well as the ViPR enhancements, the company has also overhauled its Storage Resource Management (SRM) product suite to make it easier for admins to manage multi-vendor storage environments.

In other EMC-related news, the vendor has confirmed that 1,000 jobs at the firm will be axed by the end of the current financial quarter.

The announcement follows the publication of EMC's latest financial results, which saw the firm post lower-than-expected revenue figures for the previous financial year.

HP has unveiled its Haven platform which aims to help enterprises analyse the petabytes of structured and unstructured data they generate.

Haven utlises the open source Hadoop software together with technologies HP acquired from companies including Autonomy, Vertica and ArcSight.

The platform aims to help enterprises sift through and analyse information and also help businesses to identify the data they do not need. HP claims that on average businesses house 16.6PB of data on the back of generating between 100,000 and 100 million business events per second.

Colin Mahony, VP and GM of HP Vertica, told attendees that Haven includes 700 connectors, so it is able to analyse data from sources including social media, video, audio, email texts documents and images.

"We believe [Haven] is the only comprehensive, open, scalable and secure big data platform on the market," he said at HP Discover 2013.

HP will provide analytics applications for use within Haven, but the firm noted that the platform supports multi-vendor tools, giving customers a flexibility and avoiding vendor lock-in.

"We have taken various components and made it easy to use the right tool for the right job and that's what everybody is looking to do when it comes to big data," Mahony said.

One of the most interesting tools offered up by HP is the Autonomy Legacy Data Cleanup ControlPoint 4.

Robert Youngjohns, GM of HP Autonomy, claimed that many businesses are holding onto data in a "digital landfills". This costs businesses money and impacts on efficiency.

"Many companies just [take a view] if in doubt keep it [data]," he said.

"We've built a legacy data cleanup product, so you create rules within that data policy manager and the analytics engine goes away and starts analysing all you legacy data. It begins to advise you on what can be deleted and what should be kept."

Youngjohns claimed that the analytics tool can help provide huge cost savings in some cases reducing up 50 to 70 per cent of legacy data stored.

Other benefits of deleting unwanted data include improved risk mitigation and increased efficiency.

"Clearly, if you only have the information you need, it makes you more efficient," he added.

The latest version of Microsoft's data platform, SQL Server, brings enhanced scalability and flexibility, the company claimed today as it released the software to manufacturing.

SQL Server 2012 will also help customers manage and analyse more data than ever before, according to Microsoft.

"Data is being generated faster than ever before, and organisations need a way to process and analyse all that data," said Ted Kummert, corporate vice president at Microsoft.

"Whatever the type or size of data, SQL Server 2012 delivers the platform and familiar tools to manage data, generate actionable insights and help drive business impact."

With SQL Server 2012 and integrated business intelligence tools, we're processing massive volumes of data queries in near-real time.

Those tools include PowerPivot for Excel 2010 and Power View, and Microsoft said that by the end of June they will also include an Apache Hadoop-based service for Windows Azure.

Early users of SQL Server 2012 said that the software has lived up to its promise.

"Our business depends on delivering customers fast, detailed insight into hundreds of terabytes of social-network data," said David Mariani, vice president of engineering at influence measurement specialist Klout.

"With SQL Server 2012 and integrated business intelligence tools, we're processing massive volumes of data queries in near-real time."

"SQL Server 2012 Enterprise with AlwaysOn gives us exactly the performance we need," added Thomas Pullen, database administrator at BETONSOFT.

"We can sustain 1,500 game rounds per second that gives our players a much faster, better gaming experience."

Microsoft said that next month it would introduce new data warehousing solutions that build on SQL Server 2012.

These include a major software update and new half-rack form factors for Microsoft Parallel Data Warehouse appliances, as well as availability of SQL Server Fast Track Data Warehouse reference architectures for SQL Server 2012.

Partners have already begun announcing support for SQL Server 2012. One of the first was EMC, which said it had optimised its storage hardware and software to work with the new Microsoft platform.

It claimed that combining SQL Server 2012 with its unified storage, storage tiering and flash caching capabilities could quadruple database performance and yield 80 per cent faster server provisioning.

HP may have focused its attentions on hardware this week, but the founder of Autonomy claimed today it was the nature of information that held the most importance.

Mike Lynch, founder of Autonomy and vice president of information management at HP, took to the stage at his new parent company's global partner conference to impart his philosophy to the 3,000 partners gathered.

The IT industry is carrying a hammer and everything it sees it tries to make a nail.

"It is no longer about the data but about the meaning of that data," he said. "There is a fundamental revolution going on in information and the industry is now about the I' not the T' in IT."

He claimed the traditional approach of relational databases just didn't fit the information companies are collecting today, from social networking through to video conferencing.

"The IT industry is carrying a hammer and everything it sees it tries to make a nail," said Lynch. "That hammer is the relational database... anything you put around it like Hadoop or MapReduce doesn't understand the information now being collected... you need technology that understands."

Lynch said the relational database was a tool from the past when people began using computers. Then the machines didn't understand human behaviour so were forced to deal with very strict equations. Now, the world has changed and data tools must move with it.

"Customers do not live their lives in columns, they ring you up," he said. "Understanding this can change your business.

"There is an explosion of human friendly information, such as social media, phone calls... this data is growing much faster than structured information, three times as fast."

Autonomy already has products to deal with this data, known as IDOL, but Lynch revealed more products would be coming this year to integrate the software into HP hardware products.

"What you are going to be seeing from us is we will work very closely with hardware for new products," he said. "It's a very exciting time."

Lynch concluded by summing up his beliefs around information.

"It is not about data, it is about meaning, it is not about machines, it is about people and it is not about the status quo, it is about a new opportunity."

Dell and Cloudera are collaborating to provide complete Apache Hadoop infrastructure solutions for businesses.

The pair are attempting to produce a "single source" for deployment and management of Hadoop-based cloud architecture.

The new offering combines purpose-built hardware in the shape of Dell servers and networking components, together with Cloudera's Apache Hadoop platform. The package is aimed at enterprises with a need to process large-scale datasets and claims to be more cost effective than using proprietary solutions.

Dell and Cloudera said the new solutions will be useful to a range of commercial enterprises, including financial services, energy, utility and telecom companies, research institutions, retail businesses and internet media groups.

The main goal is to reduce complexity for anyone wishing to use Hadoop-based data analytics and management tools. Training, technology support and professional services will also be available.

The package combines Cloudera's Hadoop platform with Dell's Crowbar software and Dell PowerEdge C2100 servers, along with a PowerConnect 6248 48-port Gigabit Ethernet Layer 3 switch.

Dell said additional server models will be added as options, while joint service and support, as well as a deployment guide, are included as standard.

Dell's Crowbar software is a particular lynchpin of the package, as it manages the Apache Hadoop deployment from first boot to primary Hadoop configuration. Dell said the upshot of this is businesses will be able to deploy a complete Hadoop environment in hours rather than days.

Its use doesn't end there though, as post-deployment functionality means Crowbar can help maintain, expand and architect analytics. Such features include BIOS configuration, network discovery, status monitoring, performance data gathering and alerting.

The new setup will be made available from Dell within the next month.

Cloudera is pushing for more enterprise-level adoption of Hadoop with its latest release, Cloudera Enterprise 3.5.

The company claims the new offering will enable businesses to manage Hadoop clusters in a much simpler way, thanks to en

Hadoop enables multiple applications to be run across commodity software thanks to its Map/Reduce functionality, splitting both applications and data into smaller chunks which can be stored anywhere across a cluster.

"We have seen extensive adoption of Apache Hadoop across our client base, but the management challenges of running the Hadoop stack in production can be prohibitive," said Matt Dailey, Hadoop applications developer for the High Performance Computing Centre at SRA International.

"This latest release of Cloudera's Management Suite goes a long way to giving enterprises the confidence to operate a Hadoop system in production."

Headline features include the new Activity Monitor a real time overview of the Hadoop infrastructure at granular levels and an improved Resource Manager and Authorisation Manager for scrutinising usage and general administration.

However, it is the Service and Configuration Manager (SCM) Cloudera is really shouting about. Within Cloudera Enterprise 3.5, this feature speeds up Hadoop deployment and configuration and is capable of automating a number of processes, including security.

An additional tool called SCM Express has also been unveiled as part of the launch. This aids deployment, configuration and management of all the Apache Haoop services on offer, and results in set up times that are minutes rather than weeks, according to Cloudera. It is also offered as a free download.

"You should be able to easily manage your systems while your usage of Hadoop evolves and grows. Cloudera helps enterprises plan for the inevitable changes," said company chief executive Mike Olson.

"With the release of Cloudera Enterprise 3.5, we are not only introducing new levels of transparency and automation, but we're injecting best practices Cloudera has learned over many years of helping enterprise customers build and manage Apache Hadoop-based systems."

Both Enterprise 3.5 and SCM Express are available immediately.

Open source may not often be associated with large-scale corporations, but EMC is looking to get more involved in this area.

At its annual EMC World Conference, the company unveiled new analytics software and an appliance using Hadoop, an Apache development.

This move is headed up by EMC's data computing division, made up of Greenplum employees an open source data warehousing firm EMC acquired in July last year.

But this is just the start of an on-going relationship between EMC and the open source community, according to the company's chief executive (CEO) Joe Tucci.

"We are very committed to open source," he told IT PRO.

Despite Tucci claiming the commitment from the top down, he admitted he was keen to put the projects into the hands of experienced enthusiasts, hence falling under Greenplum's jurisdiction.

"That is why we chose to put [Hadoop] over with them as they have experience," he said.

"It is a way to make sure I have that embracing of open source from a place in EMC that is very committed to it and it is also commitment from me going down that we are going to reach out and do more with open source."

EMC is making a number of product announcements over the next few days of its conference. For all the news, check out our round-up here.

EMC is extending its play into open source with the introduction of Hadoop analytics tools for the enterprise.

Previously a platform only utilised by large internet companies like Facebook and Twitter, EMC hopes to bring the capabilities over to the enterprise market for analysing the "deluge of data" they currently face.

Releasing the two software options, along with an appliance, via its data computing division, the new products build on EMC's Greenplum acquisition from July last year, bringing the two analytics technologies together.

"The Apache Hadoop technology platform is becoming a really big fundamental solution to solve these unprecedented data needs," said Scott Yara, co-founder of Greenplum and vice president of products in EMC's data computing division.

"We are here to build a big data analytics stack [and] EMC Greenplum can be [a company's] single source provider."

The first offering is the Greenplum HD Community Edition. The software only-release claims to be 100 per cent open source and is virtual machine-ready.

The second release is the Greenplum HD Enterprise Edition software package. In addition to including the analytics Hadoop offers to the open source community, it adds extra features optimised for business use, such as disaster recovery, replication and management tools.

Finally, EMC is offering an appliance with the Hadoop software and Greenplum database built in. The Greenplum HD Computing Appliance contains commodity servers with Intel processors and SATA drives with a JBot configuration.

Despite EMC's push into cloud computing, the company has yet to make an offering for the enterprise Hadoop offering as a service, but Luke Lonergan, co-founder of Greenplum and chief technology officer (CTO) of the data computing division, revealed work was underway.

"The market is not quite there yet but [we are] absolutely investing in that area [of analytics as a service]," he said.

The new offerings are already proving popular with partners, however, with the likes of Informatica, Pentaho and Datameer on board and singing its praises.

"We owe Greenplum for making the market interesting again. For so long Oracle said you could do everything with Oracle," said James Markarian, Informatica's chief technology officer (CTO). "[Now] we are seeing all this information being generated [and] the market has grown."

He added: "Without the power of Hadoop, you have no way to gather that information."

Not all Greenplum partners are benefiting from the new offerings though. During a Q&A at EMC World 2011, Lonergan admitted the joint work of his division and Cloudera, announced in September last year, was dead in the water.

"There has been a change and we are now a competitor with Cloudera," he revealed.

He tried to play the move down saying it was "better for the community [when] we can all work together," and claimed EMC was working with a number of rivals and start-ups to build the sector.

The two software launches will be available by the end of the current quarter, with the appliance following in the third quarter of this year.