Digital transformation is at the centre of operational efficiency and innovation across businesses, but without a clear roadmap leaders struggle to find long-term benefits in optimisation initiatives such as automation.

This whitepaper shares how IBM's Process Mining offers unique differentiators and capabilities, and explains how it lowers risk, improves decision-making and maximises the returns of automation.

Download now to read client success stories and how to get started.

Provided by

Intelligent automation firm Laiye has forecasted the end of subscription models in the IT industry within the next five years, citing the inadequate extent to which such models can serve growing automation needs.

Laiye's chief strategy and ecosystem officer Braham Shnider said 'rip and replace’ requirements currently imposed in the software as a service (SaaS) market, in which companies have to remove current systems in order to update to new ones, are an ineffective model for automation.

This is because of the way that Laiye argued end-to-end automation will have to work, as a combination of automated 'digital workers' - programmed bots performing specific manual tasks - with humans working alongside them.

In such a system, digital workers could be loaned for specific tasks like a rental car, rather than paid for as part of a wider systems package in a costly, yearly subscription.

Instead of subscription models, the firm is proposing a more ‘open’ automation ecosystem, replacing digital transformation and vendor lock-in with ‘digital transition’, to address the skill shortage gaps experienced by each company.

In the coming years, the company believes digital workers could be paid for by firms through a consumption model, with a set amount given to vendors per process carried out, akin to virtual freelancers.

“For example, we can provide an invoice-processing digital worker, and we can charge by the number of invoices that digital worker processed,” said Yichuan Hu, co-founder and CTO at Laiye, to IT Pro.

“It's no longer 'you need to pay ten thousand for this digital worker, for a year'. You just take, say, 10 cents for every invoice this digital worker processes. So that will actually be more attractive and more elastic for customers.

“There are three stages, and at the third stage — that is ten years from now — our product will become ‘work as a service’.”

Launching the Work Execution System

Laiye’s claims were made at an event held in London marking the launch of its new Work Execution System (WES), an automation system which provides partners with digital workers powered by machine learning (ML) and artificial intelligence (AI).

WES digital workers learn rules, taught by human employees in no-code interactions, which they can then contextually apply to tasks of varying complexity, from menial form processing to more sophisticated handling of forms.

Laiye advertises that WES can be integrated with systems such as UiPath, Slack, Teams, and SalesForce, and Shnider claimed that it “doesn’t care” if its customers choose to go with Laiye robotic process automation (RPA) or a competitor’s.

At the same event, the company highlighted the importance of harnessing the metadata generated by automated systems, in order for organisations to adequately address the ‘work execution gap’.

Trillions of pieces of data, from integers to complex information such as documents and videos, are caught up in automated systems, capable of producing vast amounts of metadata for big data analytics.

With an aim of getting a better view of this data, the firm announced that upon release, users will be given the choice of a free version or paid enterprise version.

In the former, users will be able to leverage the features of this system at no cost, in exchange for sharing their metadata for public use, giving companies insight into non-identifiable industry and use-case benchmarks and feeding this data to better train ML algorithms.

For example, a company could compare their invoicing automation stats to the industry benchmarks, use this information to determine whether more changes need to be made to their system, and in the process teach their system gaps that should be eliminated in future.

Paid enterprise users will generate only private metadata, for internal benchmarking purposes — IT Pro was given the example of the figures from different departments being compared against one another.

Laiye claims that companies that have adopted its automated systems thus far have seen a 60% improvement in productivity, and is willing to offer a full refund if it cannot integrate its WES by a company's deadline.

Another benefit of accurate metadata is explored in an upcoming white paper by Laiye, which proposes a carbon calculator for its customers.

Through the WES, this will determine the sustainability of companies in comparison to the overall market, and offer partners discounts in the form of credit against future purchases in exchange for meeting sustainability goals, which Gartner highlighted as the key strategic technology trend for 2023.

The beta version of Laiye’s WES will be launched on 30 November, with general release slated for 20 January 2023, and any company that buys a current Laiye product before 16 December will be given free access to WEP in 2023.

UltimateSuite offers four stages to its task mining, from data capture to pre-processing and enrichment, through to data analysis output, and they stand out from competitors with automation of much of the manual analysis that the more typical tools offer.

Download this whitepaper to learn more about UtlimateSuite and what this task mining tool can offer your business, as well as how easy it is to set up.

Provided by

The database is the backbone of the enterprise. Data is the currency of modern business, and organisations need to ensure consistent, optimal database performance to keep their business running smoothly.

The consequences of compromised or even suboptimal database performance can be significant; including delays in responses to customer enquiries, sales processes, and transaction processing.

This report looks at how ensuring efficient and effective operations across multiple databases is critical to maintaining business continuity and performance.

Provided by

Colonialism has left a deep and enduring mark on the world. Its roots have anchored themselves across the globe, and its long shadow maintains an unequal balance of power and wealth between the Global North and South.

There are many architects of colonialism, with the East India Company – and its complex history of trade monopolies, war and slavery – among the most notorious. The company established infrastructure, like railroads and ports, in foreign lands to extract raw materials, which, in turn, were used to manufacture goods. Many of these were often even sold back to the countries from which the materials were taken. This method allowed many imperialist entities to substantially expand their wealth at the expense of others.

Although these practices are largely confined to the past, some suggest big tech companies are following in their footsteps. These companies are, too, accused of engaging in trade monopolies, play a key role in the industry of war, and are even swept up in allegations of benefitting from modern slavery prevalent in supply chains. Where goods once traded included cotton, silk or tea, today they’re minerals mined to make electronic components, and information. Although the days of the East India Company are over, is big tech today partaking in a form of modern colonialism by entrenching themselves in the new power networks of technology and data?

Seizing first-mover advantage

Digital colonialism, says Michael Kwet, a visiting fellow at Yale Law School’s Information Society Project, is the use of technology for the political, economic, and social domination of another territory. It’s principally achieved through the ownership and control of the digital ecosystem, comprising software, hardware, network connectivity, data, platforms, and intellectual property (IP). Kwet adds the US is “by far” the leader, with China and Europe vying to close the gap, alongside some Global South-based corporations seeking to impose their influence abroad, in what he terms “South-on-South” colonialism.

Big tech takes advantage of a borderless internet to impose its products and services in poorer countries, maintaining its dominance by retaining control over technology and property. This means the Global North, in effect, controls a potentially highly lucrative part of their respective economies, Kwet says – making inroads before local companies can. Uber, for instance, had set up in Africa before African companies could compete on the same footing. Global South populations, in turn, play the role of passive user, consumer, and producer of low-level goods and services, ranging from mining to sweatshop labour.

Colonialists of the past established trade monopolies, frequently engaged in warfare and built the slave trade industry

To expand its empire across the Global South, Kwet says tech firms employ various tactics. They set the rules, like IP, and seize the first-mover advantage. They also take advantage of networks by blocking interoperability between platforms, which could allow many services to operate instead of just a few.

Instead of colonising the land and establishing infrastructure for extraction, Kwet continues, through digital colonialism, tech giants colonise the technology for data extraction and rent. He adds that networks like Facebook extract data, process it on server farms, and use it to provide services in a scenario in which the South cannot compete. He explains the North also has “heavy machinery” in place, like cloud centres, that are required for data-driven services. Kwet says these aren’t easy to replicate, and are comparable to machinery once used to drill deep beneath the surface to mine minerals like diamonds and gold.

Navigating the choppy waters of colonial overhang

Big tech takes advantage of relationship patterns that were established historically, argues Olufunmilayo Arewa, Shusterman professor of business and transactional law at Temple University. In Africa, she says, European powers once exercised control over social, political, and economic institutions, with countries integrated into the global economic system primarily as a source of raw materials, including people and agricultural products. The colonies, in turn, imported manufactured goods from the powers that controlled them. Arewa believes these patterns are mirrored in the modern era, as African nations still maintain similar economic relationships with external powers – with an added digital dynamic.

“African countries are an important source of raw materials for the industrial and digital economies, including metals such as tantalum, which is important for electronic components,” she says. “Many countries in Africa, however, remain economically marginalised, in important respects, and a capacity gap is evident, particularly in key skills needed in the digital economy.”

She says the introduction of new technologies draws attention to lawmaking within Africa, as many current laws and regulations were put in place before the digital revolution. The poor fit of existing regulations, many of which may predate modern technologies and business practices, is an issue of ongoing contestation globally, with the lack of regulatory oversight giving tech companies a green light to experiment in various territories as they see fit.

Undoing digital colonialism, Arewa suggests, would involve establishing relationships not built on colonial parallels. “This would mean engaging with countries in the Global South as partners, rather than grounds for exploitation,” she says. Several measures can be adopted from a trade perspective, meanwhile, according to Abhijit Das, head of the Centre for World Trade Organisation (WTO) Studies at the Indian Institute of Foreign Trade. He believes there should be no obligations on digital issues in free trade agreements at the WTO, such as forcing countries to liberalise e-commerce, with these agreements now the “most important legal instrument” for exacerbating digital colonisation. Countries should also have the flexibility to impose restrictions on cross-border data flows, and mandate localisation of servers within their territory. This, Abhijit says, will help developing countries create a vibrant domestic digital sector without becoming dependent on imports of digital products.

He adds countries must also be able to tax digital players, and nations must not be “arm-twisted into not imposing taxes on foreign suppliers of digital services” either. This is on top of governments being able to sufficiently regulate the digital sector, in light of provisions in existing trade agreements that “tie the hands of governments”.

Data is the new tea

The continued appropriation of data from the stream of the collective human experience is a phenomenon of historical significance, comparable with the original land grab, says Nick Couldry, professor of media, communications and social theory at LSE. He believes this “data colonialism” exacerbates historic power dynamics with regards to the world’s resources given that data, and the value derived from it, represents a completely new type of asset to “grab and appropriate”.

The continued appropriation of data by companies thriving in Silicon Valley has been likened to the original land grab

Couldry adds historic colonialism evolved over the course of four centuries into complex imperialist political structures and cultures of racism. It’s much too early to say, therefore, if data colonialism will ever evolve in the same way. He says, however, the underlying principles of colonialism, which centred on imposing ‘Western superiority’ to appropriate resources across the world, will continue through the rhetoric of big data. This language of ‘dataism’, he adds, is based around the idea that the maximum amount of data must always be gathered, whatever the cost.

Data colonialism can, in principle, thrive everywhere, so it takes place not only in the Global South but also in the North. Ultimately, though, the key targets for data extraction will always be shaped by historical colonialism, Couldry continues, leaving many African countries vulnerable to the Trojan Horse offerings by companies like Facebook. He uses Facebook Free Basics as an example of a free, limited internet service for developing markets that was found to ignore local sites and prioritise Western ones instead. “Where we think data colonialism is headed, as it grows within the power structures already inherited from capitalism,” Couldry explains, “is to make possible an entirely new type of social and economic order, based on much more intensive governance of people in the interests of value extraction."

The only way to end digital colonialism, Kwet adds, is by reconstructing the digital economy in a way that strips out the system of private property and production for profit, and replaces it with one organised by society to service society on egalitarian terms. This includes free and open source software, a phasing out of IP, stronger privacy laws, and a digital tech deal that would fund the production, maintenance, and access to technology for everyone. This must be formulated collectively by people across the world, he says, fuelled by a movement that intersects with broader political, economic, and social justice ambitions.

Couldry says he’s “optimistic” about resisting data colonialism in the long term, “provided we’re not misled by false solutions”. He says a good starting point would be to imagine a world that refuses to intensify data colonialism and instead relies less on data-extracting platforms for the conduct of our daily lives. For this to be effective, though, it must be through collective means, as there’s no other way to resist something as large as an entire social order except by “imagining and starting to live out a different one”.

In the age of the internet, more text is generated every day than a person could hope to read in a lifetime. An estimated 319.6 billion emails will be sent every day in 2021, according to The Radicati Group. Add to that social media posts, articles, reports, records and the endless other forms of written document that are constantly created, and you’ll begin to understand the incredible amount of text out there.

Even on a smaller scale, an organisation may find that it is generating and capturing more textual information than it can handle, and as documents are digitised and more types of information are captured, this will only increase. This can lead to situations in which you know that valuable insights are contained in your data, but do not have the capacity to extract them.

Text mining helps us make the most of large amounts of unstructured data, extracting insights from textual information that we would never have the time or capacity to do manually.

What is text mining?

Text mining – sometimes known as text analysis or text analytics – is an aspect of data mining that, through machine learning (ML) and natural language processing (NLP), can extract actionable insights from large, unmanageable amounts of data.

Conventional search tools function on rule-based systems that use keywords or phrases which often fall short in analysing for contextual accuracy. Google goes beyond the standard by incorporating synonyms, query intent, page relevance, quality and domain authority, but ultimately the reading and understanding of all the results must still be done manually and is too time-consuming to execute effectively for large amounts of data.

Text mining is orders of magnitude more sophisticated than a conventional search tool. It helps to provide summaries, entity extraction and sentiment analysis, to establish relationships and to structure data that can be integrated into a database or a data warehouse.

A key branch of AI used in text mining is NLP, which allows text mining systems to parse written information like a human reader. These systems are able to “read” documents and to understand the complex concepts, ambiguities and contextual clues they contain so as to locate and analyse information in a way far beyond the standard search function (this is known as natural language understanding), as well as to structure the information and interpret it to produce summaries that allow us to better understand the data as a whole (natural language generation).

If your team doesn’t have the AI/ML skills for text mining internally, companies like Merit can help build bespoke and cost effective solutions without you having to develop them from the ground up. Text mining services can be personalised to different organisations using ontologies, vocabularies and custom dictionaries that incorporate particular language and terminology used in a specific field or industry. This helps the text mining tool properly understand the target text and to provide insights most appropriate for the business.

Actionable results

There are many examples of how text mining can be applied to your operations.

It can be very useful when it comes to assessing customer feedback or Net Promoter Rating (NPR). A large organisation might receive thousands of responses to a CSAT survey. It would be incredibly time consuming to read them all, and splitting the task between several employees might mean that they miss larger patterns in the feedback. Text mining can similarly be applied to social media or online reviews to analyse sentiment related to your business or field. Using these tools, you can identify the trends and pull out insights that allow you to make material improvements to your products and services based on customers’ needs and desires.

For educational institutions, text mining can offer a more robust way of detecting plagiarism. Traditional tools look for similarities between documents, but not all the written sources will be available, so some cases will slip through the net. Using NLP, text mining tools are able to detect when the writing style changes in a document, indicating segments that were not written by the supposed author. In this way, it doesn’t matter whether the original text is immediately available to your plagiarism detection system.

Cyber security is constantly evolving in an uphill battle against the latest threats. One of the big challenges is to identify and eliminate vulnerabilities in the various security systems we rely on to protect our networks. But reports on these vulnerabilities registered in various databases are difficult to keep track of manually. Text mining tools can locate and report on the mentions of these vulnerabilities so that cyber security firms and government agencies can clamp down on security risks as soon as they are identified.

The healthcare industry stands to be a key beneficiary of text mining. It would be impossible to read every scientific paper published every year related to medicine and health, but these tools are ideal for assessing these documents and pulling out insights that could contribute to healthcare breakthroughs. They can also assist in the development and testing of drugs by highlighting potential safety issues and unsafe doses that have been identified by other studies and during the development of other treatments, leading to quicker and safer refinement.

The applications of text mining are almost endless. Wherever you have large amounts of written information, it offers the opportunity of drawing insights you would unlikely come to on your own, ones that can significantly improve your systems, products, services and the very future of your organisation.

Find out how Merit can help support your business with text mining and other services

The proliferation of onboard computing, centralised vehicle networks, mobility-as-a-service apps, navigation tools, intelligent traffic management systems, cloud computing, and more makes vehicles both major data producers and data collectors.

And it’s an imperative of the mobility ecosystem to enable monetisation of this vehicle data. To realise the full potential of innovation services like a car optimising a driver’s daily commute based on congestion and fuel prices, stakeholders in the mobility ecosystem need to test collaborative business models that allow them to make intelligent use of data in a secure manner.

Download this whitepaper to learn about:

• New technologies that help make data monetisation possible
• The benefits of edge computing in this initiative
• A real-world case study
• The three main pillars of vehicle data monetisation

The Information Commissioner’s Office (ICO) has ordered credit rating giant Experian to stop profiting from the secretive enriching and processing of people’s personal data or face a massive GDPR fine.

The UK data regulator has reprimanded the company after discovering a massive data broking operation across the entire credit rating industry, with all three credit reference agencies (CRAs), including Equifax and TransUnion, under fire for illegal practices.

The investigation found the three firms were trading, enriching and enhancing people’s personal data without their knowledge or consent. This resulted in products which were used by third-party commercial organisations to find new customers, identify those who were most likely to be able to afford products, and build individual profiles around people. Such services were also used by political parties and charities.

This “invisible” processing likely affected millions of adults in the UK, and certainly breaches data protection laws, the ICO concluded in its report - the result of a two-year investigation into the sector following an initial complaint in 2018.

The data regulator uncovered “widespread and systemic data protection failings” across the entire CRA sector, which it found is particularly concerning in an industry entirely dependent on personal data.

Of particular concern is the way these companies were using profiling to generate fresh or previously unknown information about individuals, which can be extremely invasive and can also have discriminatory effects.

“The information the CRAs are privileged to hold for statutory credit reference purposes was unlawfully used by them in their capacity as a data broker, with poor regard for what people might want or expect,” said Information Commissioner Elizabeth Denham.

“The data broking sector is a complex ecosystem where information appears to be traded widely, without consideration for transparency, giving millions of adults in the UK little or no choice or control over their personal data. The lack of transparency and lack of lawful bases combined with the intrusive nature of the profiling has resulted in a serious breach of individuals’ information rights.”

After all three companies were confronted, each made improvements to their direct marketing services business. Equifax and TransUnion made these improvements alongside withdrawing some products and services, so the ICO will be taking no further action.

Experian, however, has not gone far enough in making the required changes, so the ICO has issued an enforcement notice giving the company nine months to make the required changes or face a massive fine under GDPR.

The credit rating giant did not accept it was required to make changes set out by the ICO, and as such was not prepared to issue privacy information directly to individuals, or stop using credit reference data for marketing purposes. Should Experian continue to dig its heels in, the company could face a fine of up to £20m or 4% of the organisation’s total annual worldwide turnover.

“While these companies claim that they can process people's data with or without their consent, today's report has made it clear that the consent relied on to pass on data to third parties was often invalid,” said campaigns organisation Privacy International. “Therefore, the ICO's announcement today about three of the most recognisable data brokers in the ecosystem is an important step forward.

“Every country with data protection laws needs to look at this sector. Every regulator needs to ask what it is doing to protect people from their data being opaquely exploited by 'credit reference agencies' like Experian. As the UK regulator notes, people don't even know the names of most of these companies and yet they hold everyone's data. We believe the deck is stacked against people and this can't continue.”

Experian has hit back, with its CEO Brian Cassin disagreeing with the ICO's judgement and outlining his company's intention to appeal the ruling.

"At heart this is about the interpretation of GDPR and we believe the ICO’s view goes beyond the legal requirements," he said. "This interpretation also risks damaging the services that help consumers, thousands of small businesses and charities, particularly as they try to recover from the COVID-19 crisis."

Data is everywhere. Most businesses today are making efforts to capitalise on the vast potential of an ever increasing range of untapped data sources to produce actionable business insights.

But this is about more than getting the right answers; it’s about getting ahead of the pace of decision making. Speed is everything. In an age of connected devices and automation, decisions increasingly need to be made in real time. The result is a data race.

This whitepaper identifies three key trends that will enable teams to push further, faster, in this race. Download it now to learn how to quickly adapt your data strategy to capitalise on the opportunities these trends are creating today.

A US federal court has rejected efforts from LinkedIn to stop a startup from harvesting publically-accessible information off the professional network.

The Circuit Court of Appeals let stand a 2017 preliminary injunction that required LinkedIn to give hiQ Labs access to user accounts that were publicly available.

The decision has set the Microsoft-owned company back in its battle against 'data scraping', which it said violates its users' privacy. However, Circuit Judge Marsha Berzon said hiQ showed it faced irreparable harm and might go out of business without access.

She also added that giving companies such as LinkedIn "free rein" over who can use public user data risked creating "information monopolies" that harm the public interest.

"LinkedIn has no protected property interest in the data contributed by its users, as the users retain ownership over their profiles," Berzon wrote, according to Reuters. "And as to the publicly available profiles, the users quite evidently intend them to be accessed by others."

"Others", in this case, refers to hiQ, which is a San Francisco company that uses this kind of data to provide prediction models to businesses. The firm's algorithms can be used to estimate when employees might leave or workout areas where in-demand skills can be found.

The company was originally in court on August 2017 and was awarded a preliminary injunction that meant LinkedIn had to allow it access. Classified ad website Craigslist supported LinkedIn, warning the injunction could have a "dangerous impact" that could lead to spam and phishing attacks.

However, Berzon said, however, the startup had raised serious questions about LinkedIn's conduct, such as whether it could invoke a federal law targeting computer fraud and abuse to block "free riders" from accessing user data.

"Of course, LinkedIn could satisfy its 'free rider' concern by eliminating the public access option, albeit at a cost to the preferences of many users and, possibly, to its own bottom line," she said.

IT Pro has contacted LinkedIn for comment.

Data. Our modern world could not function without it. It powers everything around us and impacts our daily lives. Data is also essential to make important business decisions.

Such decisions are made based in insights gained from information. These decisions are either taken by humans or as part of an automated process. Information is acquired from several sources, such as from customers, from market data, and then used to figure out the best course of action for marketing, supply chain logistics, or manufacturing processes.

Data makes many modern organisations more competitive and successful. Data also helps a lot in making businesses more adaptable to our ever-changing world.

However, raw data may not be much use to anyone without analysis and filtering, doing this leads to the key insights needed for business. The use of cloud computing has meant that vast realms of data can be taken at scale from a multitude of sources and stored for real-time analysis at any time. But what is more important here is that copious amounts of data need to be instantaneously assessed to find the right information. Something that humans just cannot do by themselves.

What is data mining?

Data mining is defined by scrutinising large amounts of data to discover patterns and irregularities within the datasets. By mining data, you can create an independent forecast of the future of your business and predict scenarios of potential opportunities as well as challenges.

There are many different ways to mine, and a data-swamped enterprise can use this opportunity to expand the business, streamline costs, mitigate risks, and strengthen relationships with clients

Analytics giant SAS believes data mining is vital because it not only allows an organisation to discover the best data for whatever goals it is trying to achieve but it will also convert the most relevant data into meaningful information that has a heap more value.

Data mining allows businesses to sift through all the chaotic and repetitive noise in their data and understand what is relevant, then make good use of that information to assess likely outcomes. The process identifies patterns and insights that can't be found elsewhere, and by using automated processes to find the specific information, it not only speeds up the time it takes to find the data but also increases the reliability of the data.

Once the data is gathered, it can be analysed and modelled to convert it into actionable insights for the business to use.

Big data mining

Mining Big data means analysing large amounts of data (known here as Big data) and turning all of that into information that is meaningful to the business who then in turn makes decisions based on that data.

The methodology is taken as a strategy within the business intelligence function of an organisation. To develop focused insights for it, including data about systems, processes, and everything else that requires reliable data collection over a protracted time.

The nature of Big data means that it takes a lot longer to gather and is frequently stored in an unstructured format - so some organising is essential before it can be completely analysed.

Mining usually comprises searching through a database, sanitising and then extricating that data to then be ordered into a meaningful structure, frequently based on shared features or types, using an algorithm.

As big data mining is fundamentally data mining on a much greater scale, it also necessitates far more computing power to do successfully. In some instances, only specialised kit, such as research computers, are up to the assignment.

However, the central principles of data mining continue to be the same, irrespective of the size of the data set.

Data mining techniques

Among the techniques, parameters and tasks in data mining are:

• Anomaly detection: unusual data records are identified that could be of interest if errors that need more study.
• Dependency modelling: Looking for relationships between variables. For example, a supermarket will collect information about the purchasing habits of their customers. Using association rule learning, the supermarket can work out which products are bought together and use this for marketing.
• Clustering: this searches for structures and groups in data that are similar, without using known data structures.
• Classification: searching for patterns in new data using known structures. For example, when an email client classifies messages as spam or legitimate.
• Regression: searching for functions that model data with the least number of errors.
• Summarisation: creating a compact dataset representation. This includes visualisation and report generation.
• Prediction: Predictive analytics look for patterns in data that can be used to make reasoned forecasts about the future.
• Association: a more straightforward approach to data mining, this technique allows for making simple correlations between two or more sets of data. For example, matching people's buying habits, such as people who buy razors tend to buy shaving foam at the same time, which would allow for the creation of straightforward buying suggestions served to shoppers.
• Decision trees: related to most of the above techniques, the decision tree model can be used as a means by which to select data for analysis or support the use of further data within a data mining structure. A decision tree starts with a question that has two or more outcomes in turn connecting to other questions, eventually leading to an action, say send an alert or trigger an alarm if analysed data leads to particular answers.

Advantages of data mining

There are a few ways in which organisations can benefit from data mining.

• Predicting trends: finding predictive information in large datasets can be automated using data mining. Questions that used to require lots of analysis can now be answered more efficiently straight from the data.
• Decision-making help: as organisations become more data-driven, decision making becomes more complex. By using data mining, organisations can objectively analyse the available data to make decisions.
• Sales forecasting: businesses with repeat customers can keep track of the buying habits of these consumers by using data mining to foresee future purchase patterns so they can offer the best possible customer service. Data mining looks at when their customers have bought something and predicts when they will buy again.
• Detecting faulty equipment: applying data mining techniques to manufacturing processes can help them detect faulty equipment quickly and produce optimum control parameters. Data mining can be used to regulate these parameters to result in fewer errors during manufacturing and better-finished products.
• Better customer loyalty: low prices and good customer service should ensure repeat custom. Businesses can decrease customer churn by using data mining, especially on social media data.
• Discover fresh insights: data mining can help you discover patterns that reinforce your business practices and strategies, but it can also throw up unexpected information about your company, customers, and operations. This can lead to new tactics and approaches that can open up new revenue streams or find faults in your business that you would never have spotted or have thought to look for otherwise.

Disadvantages of data mining

As with anything in life, while there are many benefits associated with using data mining, there are also some few drawbacks too.

• Privacy issues: Businesses collect information about their customers in many ways for understanding their purchasing behaviours trends, but such businesses aren't around forever, they could go bankrupt or be acquired by another company at any time, which would usually lead to the customers' personal information they own being sold to another or leaked.
• Security issues: Security is a big concern for both businesses and their customers, especially due to the vast number of hacking cases where big data of customers have had their private information stolen. This is a possibility everyone needs to be aware of.
• Misuse of information: Information collected through data mining for ethical reasons could be misused, such as being exploited by people or businesses to take benefits of vulnerable people or discriminate against a group of people.
• Not always accurate: Information collected isn't always 100% accurate, and if used for decision-making, could cause dire consequences.

The future of data and data mining

Recently, companies have seriously ramped up their data collection capabilities and it doesn’t seem like this will decrease anytime soon. Some businesses might find that they are drowning in all the data they’ve collected, and this could end up causing them some serious headaches, rather than produce the results they were hoping for.

This is exactly the reason organisations should consider spending some money on improving their data analytics capabilities. Using the right tech, it's possible to analyse real-time data without needing to transfer it to the cloud or even a data centre. Edge computing is integral to this process, and it's estimated that by 2025, 75% of data produced by enterprises will be processed and created outside of the traditional data centre, with big data analytics’s future lying at the edge, according to Gartner research.

Thanks to super-fast transfer speeds, you can process data at the point where it’s collected, especially when you merge edge computing with the benefits of 5G. The Internet of Things (IoT) environment is one area which has profited well from this, especially as it has become more popular during the pandemic. Since lots of people across society had to spend more time working from home, or even in the house itself during their downtime, smart devices appeared as an attractive way to make those routine everyday tasks more efficient. However, it’s worth noting that this trend could backfire on organisations, due to the much greater threat surface created by so many devices operating on the same network.

Machine learning equally promises to influence the future of data analytics, with more businesses deploying AI-based applications with each passing year. This technology has never been more accessible, with many tools just as easily available to small businesses as they are to data scientists. Some of the newest machine learning tools can provide businesses of all sizes with the capabilities to analyse complex datasets and derive useful insights, with the performance of these systems only set to improve.

In the age of rampant digital transformation, not only is data becoming more important, but so is the speed and accuracy of processing this data, and the quality of insights that organisations can derive.

This article was first published on 22/08/2019, and has since been updated.

The popularity of cryptocurrencies over the past decade has almost been as turbulent and volatile as their monetary valuations. Having spent many years in relative obscurity, the popularity of cryptocurrencies, or digital tokens, soared in 2019 and 2020, as did the number of cryptocurrency miners. But what exactly is cryptocurrency mining?

The process for creating 'new' cryptocurrencies varies depending on the blockchain in question, however many follow a process known as 'mining'. The process is a cause of concern for many lawmakers and environment groups, as it requires an enormous amount of energy to pull off.

Cryptocurrency mining: A simple definition

Cryptocurrency mining is the name given to the process of completing 'work' on the behalf of the supporting blockchain – the digital ledger technology that supports each individual cryptocurrency. There is one blockchain for each cryptocurrency, controlling all the trading and activity of that particular coin.

When it comes to major cryptocurrencies like Bitcoin, there are only a finite number of new cryptocurrency tokens, or coins, that can be 'minted', which is by design.

Perhaps a good way to visualise this idea is to imagine a group of people stumbling across a large block of gold. There is only a single block of gold, and it has a set weight. The only way for this group to take a chunk of it for themselves is to chip away until they each get a nugget. This is essentially how cryptocurrency mining works – the trading of energy or computational power for a reward.

However, the term 'mining' is a little misleading, as the process does not resemble any sort of mining you may be familiar with.

How is cryptocurrency mining related to the blockchain?

Digital tokens like Bitcoin can’t be printed like money, and the only way to generate new coins is through the process of mining, although the term 'mining' is perhaps misleading. It stems directly from the specific blockchain that powers each individual cryptocurrency, from Bitcoin to Ether.

Each particular distributed public ledger supports the operation of a corresponding cryptocurrency and records all transactions across the extent of its network. For our guide on how blockchain technology works in more detail, check out our explainer.

The blockchain records every time cryptocurrency is traded, with these records being compiled into an endless line of blocks that are all connected. For transactions to be validated, they will need to be verified by other users on the network. This process is key to the way that cryptocurrencies work and avoids instances whereby individuals try to initiate several transactions using the same units.

Cryptocurrency mining is effectively a process of rewarding network users with Bitcoin for validating these transactions.

How are new coins mined?

When a slew of transactions is assembled into a block, it is then appended to the blockchain. However, in order to be rewarded with Bitcoin, a ‘mining’ user, or ‘miner’, needs to perform two tasks: validate 1MB worth of transactions as well as be the first to find out a unique 64-digit hexadecimal number - also called a hash.

Similarly to the blockchain, the network user, or 'node', also holds a record of every transaction. As it is notified, the transactions are validated with a series of checks in order to make sure they are legitimate. The checks include scanning the transactions for a unique cryptographic signature, which is created at the beginning of the process, and confirming whether it is valid or not.

In order to be within a chance of securing new Bitcoin, every miner seeks to validate 1MB worth of these transactions. If successful, they then also have to solve a numeric problem which is otherwise known as 'proof of work'. Users who are able to triumphantly generate the correct 64-digit hexadecimal number, or 'hash', which is either less than or equal to the target hash associated with the block, are then remunerated with Bitcoin.

Due to the difficulty of the task, the only feasible way to find the right hash is to calculate as many combinations as possible, and then wait until a match is found.

In order to stand a chance in being the first one to guess a hash, a user needs to have a high hash rate, or hash-per-second, and the more powerful setup, the more hashes a user can sift through - such are the high computing costs of mining. In order to visualise it, imagine a competition where contestants have to guess the correct weight of a cake. Participants have an unlimited number of guesses and the first one to submit the correct weight wins. In this competition, the winner is most likely to be the contestant who is capable of making the most guesses at the fastest rate.

Proof-of-work vs. proof-of-stake

Different blockchains operate on different models, and the advantages and disadvantages of each have long been discussed by cryptocurrency enthusiasts. Ethereum recently made the decision to move to a different, more modern and greener type of blockchain, but the complexities involved in doing so have resulted in delays to the switch.

Bitcoin’s blockchain uses a proof-of-work model, which involves computers dedicated to performing calculations on behalf of the blockchain and then earning a Bitcoin reward. This model has been heavily criticised for its energy use, mainly because the system incentivises the formation of large numbers of powerful computers that are able to act as one and effectively muscle other miners out of the way and take the rewards for themselves.

Ethereum’s upcoming proof-of-stake model aims to eliminate both the high energy use and dominance of ‘mining farms’. It assigns computational power used for cryptocurrency mining at random and it promises to drastically reduce energy since mining farms will not be able to secure their previously dominant share of mining. In fact, Ethereum predicts moving to proof-of-stake will cut its energy use by 99%.

The limits of cryptocurrency mining

What this means is that mining for cryptocurrencies is a matter of competing in a highly competitive race against other participants, all hoping to land the winning ticket and grab a payout.

However, that’s not the only challenge you will face if you attempt to mine yourself. The difficulty of calculating each hash also increases artificially in order to maintain a steady flow of newly created blocks. This means that as the block count rises, so too does the processing power required to solve each calculation, and therefore the harder it is to mine as a hobbyist. There’s also the issue of hard limits on total circulation. For example, within the Bitcoin blockchain, there will only ever be 21 million coins created in total, a deliberate design to prevent inflation.

In the early days of each cryptocurrency, it’s perfectly possible for everyday users to participate in mining, but given the issues outlined above, the maturity of cryptocurrencies such as Bitcoin means that it’s no longer possible to use standard PCs. The complication of the calculations involved, coupled with the sheer number of other people on the network, means that Bitcoin mining can now only be done with large scale processing ‘farms’ – multiple specialised GPUs working in tandem on a 24-hour basis.

In fact, these days it’s almost impossible to create a return on investment from mining as the energy costs required to power GPU farms typically outweigh the value of the currency itself. Either that, or you’ll be forced to funnel the majority of what you earn into the running and maintenance of the equipment.

As the Bitcoin hype is more or less fully nestled in the wider public consciousness, organisations have invested increasingly considerable sums into it, effectively industrialising cryptocurrency mining. Large warehouses packed to the brim with floor-to-ceiling racks of expensive graphics cards, working towards the sole aim of mining new units of Bitcoin, Ether, Litecoin, and so on, have become the norm.

The Bitcoin network – to add some context – processes 5.5 quintillion hashes per second, which means that unless you have the equipment capable of processing a massive quantity of calculations in a very short space of time, the chances of you being able to compete with the more industrial operations are minuscule.

For this reason, miners often band together and pool resources to maximise their chances of profiting from the cryptocurrency mining game – creating 'mining pools' – sharing their power, as well as any returns their efforts may generate between them.

The legality of cryptocurrency mining

The legality of cryptocurrency mining differs across countries, and bans are usually imposed for one of two reasons. The first is usually linked to a state’s approach to cryptocurrency overall, meaning that if a government is against cryptocurrency trading, it will likely outlaw mining as well.

An example of this is China, where the State Council Financial Stability and Development Committee banned all cryptocurrency trading and mining in 2021 due to its alleged links to illegal activity. Chinese banks and financial institutions are also prohibited from providing cryptocurrency-related services – a move that sent the price of Bitcoin tumbling by roughly 30% in May 2021. Prior to the crackdown, China was home to roughly 65% of the world's cryptocurrency mining. Today, the country’s share of global Bitcoin mining has fallen to effectively zero, according to the Cambridge Bitcoin Electricity Consumption Index.

The second reason that states ban cryptocurrency mining is linked to its immense energy consumption – Bitcoin consumes roughly 148 Terawatt hours (TWh), which is more than entire countries. For instance, Argentina, the Netherlands, and the United Arab Emirates consume 121 TWh, 109 Twh, and 113 Twh respectively. In January 2022, during the “worst energy crisis in a decade” affecting a number of European countries, Kosovo officials banned cryptocurrency mining in an effort to prevent additional electricity blackouts. Prior to the ban, Kosovo was considered a cryptocurrency mining haven due to its low – and often state-subsidised – electricity bills, with some miners refusing to pay them altogether. However, for the time being, the state hasn’t passed any legislation that opposes cryptocurrency trading itself.

Transport for London will start tracking the movements of passengers on the London Underground from next month, the company has announced.

Beginning on 8 July, TfL will begin using the free Wi-Fi service provided throughout the tube network to gather information on the movements of passengers through the system. Anyone that connects to the Wi-Fi will be tracked as part of the scheme. An early pilot was held for four weeks in 2016 across more than 50 tube stations, during which TfL gathered data on more than 42 million journeys.

The aim of the programme will be to give TfL more granular data on passenger movement within stations, allowing them to inform customers more accurately about the state of the network, taking things like foot congestion and overcrowding into consideration, as well as train schedules.

"The benefits this new depersonalised dataset could unlock across our network--from providing customers with better alerts about overcrowding to helping station staff have a better understanding of the network in near-real time -- are enormous," said TfL's chief data officer Lauren Sager Weinstein.

"By better understanding overall patterns and flows, we can provide better information to our customers and help us plan and operate our transport network more effectively for all."

The data will also be made available via TfL's open API, meaning that app developers will also be able to make use of it. Journey planner apps like Citymapper, for example, could direct customers to pick the less overcrowded of two equally fast routes based on this passenger data.

"TfL is a world leader in the use of data to improve services," said London's chief digital officer Theo Blackwell. "Today nearly half of all Londoners use an app derived from TfL's open data feeds. Greater insight into how people use the Tube helps better transport planning but also enables the development of new apps to give customers further advance information about their planned journey in order to plan their best, and least congested route."

TfL has anticipated the concerns of privacy campaigners around this kind of mass tracking, however, and said that the data-gathering will be secure and privacy-protected. The data will be depersonalised, it said, promising that "individual customer data will never be shared and customers will not be personally identified from the data collected by TfL".

The organisation also noted that it had collaborated with the Information Commissioner's Office on developing the scheme in order to ensure that privacy and transparency concerns were adequately addressed.

Once the system goes into effect, warning notices will be added to TlF's in-station CCTV signage to alert them to the data collection and direct them to where they can find more information.

"While I am excited about the potential of this new dataset, I am equally mindful of the responsibility that comes with it," Weinstein said. "We take our customers' privacy extremely seriously and will not identify individuals from the Wi-Fi data collected. Transparency, privacy and ethics need to be at the forefront of data work in society and we recognise the trust that our customers place in us, and safeguarding our customers' data is absolutely fundamental."

Commvault's veteran CEO has said the firm's recent advancements in areas such as AI are way ahead of the rest of the industry - one year after admitting he "completely screwed up".

Despite a recent shift to HTML5 on the company's front-end user interface, Commvault's continued use of "legacy" Java in its back-end systems has sought criticism from detractors. But CEO Bob Hammer claims the firm's coding and engineering has set a new bar for the industry.

"You have all of the Commvault customers that are maybe sitting on the Java UI, and you have all the competitors in the market talking to us about legacy," Hammer said during a press Q&A at the company's flagship conference Commvault GO.

"And I'm going to be really blunt, we're way ahead of Rubrik, Cohesity and all these guys, technically.

"They're having a real good time talking about Commvault as legacy, but we're way ahead of them on automation, on AI, on managing to the cloud, on dealing with things like cyber, and GDPR, and data analytics.

"We're on another world compared to where they are, and our Command Centre, which now has all this added automation, is well beyond what their next-generation visions look like."

Asked whether there were plans to transition backend infrastructure to HTML5, or further integrate AI, Bunte, who also attended the Q&A, said the company will continue to attack the issue with further integrations in future.

He added Commvault was "feeling really good" about its position with customers, as well as against its competitors.

Hammer added: "What we said on stage on Monday [Tuesday] was not an accident. So basically we're setting the bar, we're saying to anybody in the industry you have to reach that bar.

"The issue some of these competitors are going to have is they're architecturally-bound, it's going to constrain then in solving some of these fundamental problems, and we know that. So it's a new bar, and we've just set it."

Commvault's CEO and COO taking questions from reporters at Commvault GO 2018

The pair's comments came a day after they put Commvault into the same bracket as such industry giants as Netflix, Amazon, and Uber for the company's aspirations to implement sophisticated machine learning and AI, and change users expectations.

"What do these five companies have in common?," Hammer said during the keynote address at Commvault GO, hosted this year in Nashville, Tennessee.

"They changed our expectations for things we do in our everyday lives. And how did they do that? Well, they applied sophisticated technology like machine learning and artificial intelligence to drive better outcomes."

The pair, who delivered a joint-address, used the keynote to highlight a number of key announcements including a new AI-powered user interface, as well as two new hardware products, and partnerships that build on Commvault's new channel-centric strategy.

For instance, the company's Command Centre, set to launch in December, uses AI and machine learning integration to slash backup times, and vastly reduce the number of notifications.

"When we started this journey, just a little while ago," Bunte concluded, "we showed you logos for five companies who are changing your expectations about almost everything you do in your everyday life.

They apply sophisticated ML technology for better outcomes, and they simplify complicated operations, again, to drive better outcome, better user interfaces, automation, etc.

"And like those five companies, Commvault has applied innovative technology to radically change your expectations for the things you do every day."

Hammer's words are a far cry from the tone he set at 2017's Commvault GO, in which he admitted "we completely screwed up," strategically. The firm had missed the boat on the initiative taken by one of its main competitors, Veeam, which introduced a simple application into the market.

A common criticism levelled at Commvault by existing and prospective customers had largely centred on the sheer complexity of the firm's offering.

This, during the last year, sparked a radical change in how its software was packaged, and priced - with simplicity one of the core themes at this year's conference.

A High Court has blocked a mass lawsuit brought against Google claiming the search engine illegally harvested personal data from UK iPhone users.

The tech giant could have faced a huge compensation bill over the collective High Court action, led by former Which? consumer group director Richard Lloyd, which alleged that Google violated section 4 of the Data Protection Act 1998 and unlawfully gathered personal data from iPhones between 2011 and 2012.

However, on Monday Mr Justice Warby said he would not allow the claims to proceed on the grounds that the facts alleged in the claim did not support the contention by Lloyd that he or any other iPhone user had suffered 'damage' and because it would be impossible to accurately calculate the number of iPhone users who might have been affected.

"The privacy and security of our users is extremely important to us. This claim is without merit, and we're pleased the Court has dismissed it," a Google spokesperson said.

Lloyd set up a consumer campaign group called 'Google You Owe Us' as part of the legal action and the website had 20,000 co-claimants before the High Court made its ruling.

"We want to ensure that big companies like Google respect our privacy in the future. Our personal information is valuable and it must be used in a way that is trustworthy and fair."

It is claimed that Google used the Safari workaround to track internet browsing history and then used the data to sell targeted advertising. According to Google You Owe Us the search company made $80 billion in 2016 from targeted advertising alone.

The group say that approximately 5.4 million individuals were affected and the data collected by Google included race, political learnings, sexuality, social class, financial, shopping habits and physical and mental health information. If the claimants are successful it could potentially be the largest ever bill owed to British consumers regarding an illegal use of data.

Although the claims against Google go back to 2011 and 2012, it is only in the last year that this type of personal data harvesting has become mainstream news, following the Cambridge Analytica scandal and the creation of the GDPR. We are now seeing more scrutiny on tech companies with regards to public data - specifically how, where and why they are using it.

Facebook, for example, has repeatedly had to answer questions regarding its model of service, which is sold as 'free' but collects personal data to be sold to advertising companies. Google too has a similar operation with regards to its user's personal info. Earlier in the year, the search engine was accused of illegally tracking the movements of iPhone and Android users, even if their mobiles had locations services switched off.

Instagram has been spotted prototyping a privacy setting that would allow it to share your location history with its parent company Facebook.

The setting records exact GPS coordinates collected by the Instagram app, even then when it is not in being used, and then sends that information to Facebook to target users with ads and relevant content recommendations.

This has raised concerns about how much surveillance Facebook is undertaking and how it is getting its data. This Instagram location sharing tool is actually located in Facebook's privacy and security settings, where it is explained that Instagram and Messenger data can be used to collect and build a history of locations based on your phone's location features.

"Location History is a setting that allows Facebook to build a history of precise locations received through Location Services on your device," Facebook states. "When Location History is on, Facebook will periodically add your current precise location to your Location History even if you leave the app. You can turn off Location History at any time in your Location Settings on the app.

"When Location History is turned off, Facebook will stop adding new information to your Location History which you can view in your Location Settings. Facebook may still receive your most recent precise location so that you can, for example, post content that's tagged with your location."

With the social network's former VP of its news feed, Adam Mosseri now in charge of Instagram, there is a concern that it will try to squeeze more value out of the picture sharing app. Facebook has previous for this having been fined $122 million for breaking its promise to European regulators that it would not commingle WhatsApp and Facebook data.

The social network is already under scrutiny for how it uses personal data and its encroachment into users lives and its involvement with the Cambridge Analytica scandal.

The company still hasn't reckoned with the full extent of the massive hack it detected last week, in which unknown malicious actors exploited three distinct bugs on Facebook's platform that combined to devastating effect.

A host of anti-malware tools developed by cyber security company Trend Micro have been removed from Apple's Mac App Store.

Six apps including Dr. Cleaner and Dr. Antivirus have disappeared from the Mac App Store after experts, including Malwarebytes Labs' head of Mac Thomas Reed, spotted that user data was needlessly being 'exfiltrated' from these products.

Examining the apps' code, Reed and others also claimed Trend Micro's repertoire of apps was sending data to a server in China based on the fact a domain was registered in the country - a charge Trend Micro flatly denies.

"It's blindingly obvious at this point that the Mac App Store is not the safe haven of reputable software that Apple wants it to be," Reed said.

"I've been saying this for several years now, as we've been detecting junk software in the App Store for almost as long as I've been at Malwarebytes. This is not new information, but these issues reveal a depth to the problem that most people are unaware of."

Apple, having revamped its rules earlier this year to prioritise user privacy, began removing Trend Micro's apps once alerted to the complaints two days ago - with all apps now off the store at the time of writing.

Facebook just last month clashed with Apple over data collection concerns surrounding its controversial Onavo Protect VPN app; voluntarily removing its service from the App Store after talks with the iPhone manufacturer.

Trend Micro denied all claims it was stealing user data and sending them to an unidentified server in China, branding them "absolutely false".

In a post responding to the controversy, the Japanese firm said it completed an investigation into the six apps removed from the Apple Mac Store and concluded they "collected and uploaded a small snapshot of the browser history on a one-time basis".

In a further update, Trend Micro confirmed it had removed the data collection features across the consumer products in question, and permanently dumped legacy logs stored on a US-based AWS server.

Finally, the company identified what it claimed to be a "core issue which is humbly the result of common code libraries", learning the data collection functionality was designed the same across all of its apps regardless of whether this was necessary for the app to work.

"The potential collection and use of browser history data was explicitly disclosed in the applicable EULAs and data collection disclosures accepted by users for each product at installation," the company said.

"The browser history data was uploaded to a U.S.-based server hosted by AWS and managed/controlled by Trend Micro.

"We apologise to our community for concern they might have felt and can reassure all that their data is safe and at no point was compromised."

IT Pro has approached Apple for comment but it had not responded at the time of publication.

The personal information of more than four million Facebook users was stored in an easily-accessible online database for the last four years, it has been revealed.

The database, which was created by researchers from the University of Cambridge and features highly sensitive information, was left exposed with minimal protections.

Facebook users' data was compiled through a Facebook app called myPersonality - a personality quiz similar to the ThisIsYourDigitalLife app that Alexandr Kogan used to collect the data that would later be passed to Cambridge Analytica.

Included in the database were details including age, gender, location and relationship status for 4.3 million people, as well as 22 million status updates from 150,000 users, according to the New Scientist, which broke the news following an investigation it conducted.

The purpose of the quiz was to gather information on users' 'Big Five' personality traits - five broad factors that are used in some branches of psychology to analyse an individual's personality. The database included the 'Big Five' scores of 3.1 million Facebook users.

While their datasets were supposedly anonymised, with users' names stripped out prior to publication, the rest of their details were still linked, making it comparatively easy to determine the users' identity, according to the publication.

The men behind the project are Michal Kosinski and David Stillwell, two researchers from Cambridge University's Psychometrics Centre - although the university said that Stillwell created the app prior to his employment.

The data was made available to other researchers who wanted to use it in their own studies, with the proviso that they use it in such a manner that the data could not be traced back to the individual participants of the quiz, while researchers from private companies could use it for non-commercial purposes provided they agreed to be bound by strict data protection policies.

Researchers from Google, Microsoft and Yahoo registered to use the data, as well as academics from around 150 universities, totalling almost 300 people. However, one university lecturer apparently gave students the credentials to access the database as part of a course on analysing Facebook data.

The login details were then uploaded to GitHub, where they were easily accessible via a simple Google search, allowing anyone to access the database. The GitHub page remained up for around four years.

Facebook suspended the myPersonality quiz last month over concerns that the language describing how data was shared violates its terms of service, and has launched an investigation to determine if a violation has taken place. The Information Commissioner's Office is also examining the case, although an official investigation has not been announced.

Stillwell told the New Scientist that over the course of the project's lifespan, it has only experienced one data breach, saying "we believe that academic research benefits from properly controlled sharing of anonymised data among the research community".

The news comes after the Cambridge Analytica scandal, in which the US analytics firm allegedly siphoned millions of users' data from an app created by Cambridge professor Aleksandr Kogan to try to influence voters in the 2016 US presidential election.

It has raised questions around what third-parties do with Facebook users' data, and the lack of controls Facebook had in place to monitor their usage of such information. Last week Facebook revealed it has suspended 200 apps amid an investigation into whether third parties misused users' information.

Cambridge Analytica, the political consultancy firm at the heart of the Facebook data sharing scandal, is shutting down after months of allegations surrounding the exploitation of data.

The Information Commissioner's Office (ICO) said it would be examining closely the details of the announcement, with a spokesperson adding it will "continue its civil and criminal investigations and will seek to pursue individuals and directors, as appropriate and necessary even where companies may no longer be operating".

"We will also monitor closely any successor companies using our powers to audit and inspect, to ensure the public is safeguarded," it added.

Cambridge Analytica, which specialised in data analysis and strategic communication, blasted "a siege of media coverage" that drove away its customers and suppliers, in a statement saying it would commence insolvency proceedings in the UK and parallel bankruptcy proceedings in the US.

"Over the past several months, Cambridge Analytica has been the subject of numerous unfounded accusations" the company said, adding despite trying to "correct the record" it has has been "vilified for activities that are not only legal, but also widely accepted as a standard component of online advertising in both the political and commercial arenas".

But there are concerns Cambridge Analytica has simply used its closure as a rebranding exercise, with a new company, Emerdata Ltd, listed as using the same offices as SLC Elections; also hiring Alexander Taylor, previously the acting CEO of Cambridge Analytica, as director on 28 March.

"Cambridge Analytica and SCL group cannot be allowed to delete their data history by closing," said Damian Collins MP, chair of Parliament's Digital Culture Media and Sport committee investigating the phenomenon of fake news, adding: "The investigations into their work are vital."

The announcement was made only hours after cybersecurity expert Chris Vickery gave evidence before the DCMS committee detailing the role Cambridge Analytica allegedly had in supporting the pro-Brexit 2018 EU referendum campaign, alongsideCanadian data firm Aggregate IQ (AIQ); Vickery also noted that "beyond a shadow of a doubt" there had been collusion between pro-Brexit campaigns during the referendum.

"With all the evidence I've seen I don't think there's any ability for a reasonable, rational person to deny there was some level of coordination or collaboration going on," Vickery told MPs. Different campaign groups are forbidden from coordinating in the UK, under electoral law, unless they declare their spending jointly.

The pro-Brexit campaign groups, AIQ and Cambridge Analytica have consistently denied any wrongdoing. And while Vickery also outlined how Cambridge Analytica used psychological insights derived from Facebook data in the Trump campaign, the company had consistently denied it had breached any data use and privacy laws.

Last month the ICO revealed the scope of its investigation into how data is used in political campaigns spanned 30 companies, including Facebook, AIQ, and Camgridge Analytica.

Information commissioner Elizabeth Denham said at the time: "The ICO is looking at how data was collected from a third party app on Facebook and shared with Cambridge Analytica. We are also conducting a broader investigation into how social media platforms were used in political campaigning."

She added: "This is an important time for privacy rights. Transparency and accountability must be considered, otherwise it will be impossible to rebuild trust in the way that personal information is obtained, used and shared online."

Meanwhile Mark Zuckerberg has been issued a formal invitation by the DCMS committee to give evidence or face being issued a summons, after Mike Schroepfer, Facebook's CTO, who Zuckerberg sent in his place, failed to answer a series of questions to the satisfaction of the committee.

The scandal engulfing Facebook and Cambridge Analytica has raised overwhelming personal data and privacy concerns, especially in light of the upcoming GDPR enforcement with Facebook determined to appease lawmakers - for instance rolling out a wave of new privacy toolsahead of the 25May deadline.

Big data poses big challenges. But if businesses can tackle these head on, there are huge opportunities awaiting them.

The first time we heard the phrase "big data," we were wondering whether storage companies had gone into children's programming, describing in some Jackonory dialect what they did for a living. Perhaps they were trying to make the concept simple yet appealing to the kids in our schools, we pondered.

At a second glance though, it pithily sums up the deluge of data hitting us, both at home and in the office.

Different companies have different definitions of course, but it all comes down to more unstructured data than ever before piling through our networks, with the likes of rich media taking up even more space in areas like social networking.

Never before has the business seen so much customer content, pictures, videos, databases, file systems and even more types of information that, besides making you dizzy, has to be stored.

Big data offers a massive competitive advantage for the organisations that embrace it.

Big data sounds like a headache to most companies and, in most cases, is becoming a serious migraine. The provisioning of storage to handle what one needs to keep is a challenge in itself, but to keep up with the ever-growing numbers feels like a continual race.

Black sand

Where big data really comes into play though is not how you store it but how you use it to your advantage.

"You might want to think of big data as being black sand," said Colin Mahony, vice president of products and business development at HP. "You just need a way to analyse all the black sand (the massive amounts of raw data), in real-time, to find the gold (the nuggets of useful information) when and where you need it."

This is the key. Having all this data, be it on the performance of your staff or the preferences of your customers, gives you more knowledge than ever before. Bring in the old adage of "knowledge is power" and it seems big data is a big chance to know your market and your own company better than ever before, opening up new opportunities to grow.

"Big data offers a massive competitive advantage for the organisations that embrace [it] and its opportunities," added Mahony. "The companies that thrive in today's information economy will be the ones that can find the commonalities and the outliers in the data to let them see usage trends as well as opportunities for improvements."

It all sounds great and doors for companies will be opening in the future for all to take advantage of big data. But it's not that straightforward.

Big data is not something you should begin to think about soon, big data is here now. This is the conclusion not just of vendors trying to sell you the technology to deal with it, but of analysts looking at what businesses need to do to address this immediate plethora of data.

"Businesses are now working in an incredibly different environment compared with 10-15 years ago," said Carl Streatfield, presales manager for business intelligence and In-Memory at SAP.

"In 2005, mankind created 150 exabytes of information. In 2011, 1,200 exabytes will be created. This is the new world of big data.'"

That is a lot of data. They are numbers which many could not have imagined when they started working in IT, but companies must put the shock and awe to oneside and work out the best way of dealing with this data in a competitive way.

"Opportunities to improve the bottom line exist in a flood of information; however, gaining insight from data becomes challenging as it grows extremely large," said Brian Hopkins, senior analyst at Forrester, in his Big opportunities in big data' report.

"Emerging technology applies the power of distributed, virtual computing to the problem of large data, providing new tools and techniques that shift the way businesses use information to compete."

The cloud is one method a lot of vendors are choosing to put their weight behind to deal with the data surge hitting their customers.

EMC is one key example, building its annual conference around the "journey to the private cloud" and "big data" for the past three years, along with its annual sponsored survey conducted by IDC to come up with new and more terrifying measures for data size.

The thinking behind using cloud would be to remove these huge volumes from your own network, allowing for an optimisation of your own data centre without clogging it with reams of unstructured data.

However, as we have already pointed out, this data could be turned into amazing IP for companies, discovering what your customers really want and using the information to provide it. This is where firms get very nervous about moving to the cloud for security reasons and ensuring they have control over their property.

That being said, Hopkins does believe it is one of the most viable options.

"Forrester finds the elastic, scalable nature of the cloud to be a primary driver of big data implementation," he said. "Virtual versions of commodity infrastructure such as servers and storage blocks are now commonplace."

"Further, virtual networking and intelligent workload management tools enable elastic scaling of this infrastructure on demand, creating an ideal environment for cheaply sorting and processing large data sets across distributed systems."

Don't wait till it's too late

There are many options for companies both big and small to address the problem and opportunity that this data explosion brings. But they key message we have discovered is don't wait; big data needs to be tackled now before it floods your data centre and used to its full advantage.

"Enterprises of all sizes can take advantage of the information they gather, and use this increased knowledge into their networks, services, and customer behaviours to reduce costs and enhance customer satisfaction," concluded Mahony.

"In an era where customers have all the power, the ability to define new services based on actual usage patterns and behaviours is a powerful competitive advantage."

Kognitio has hailed the death of physical cube databases with the launch of Pablo, designed to enable "train of thought" analysis.

The MDX connector lets users create and query virtual cubes "on the fly," without any of the lag and complexity that comes with physical cubes, the business intelligence firm claimed.

Pablo, named after the cubist master Picasso, can leverage any MDX-compatible tool, including Microsoft Excel, to help the end user carry out in-depth analysis.

This will no doubt please those unwilling to splash out on expensive front-end applications.

Pablo will only work, however, on top of Kognitio's WX2 analytical database and will come as an addition to the offering from 1 June.

Because Pablo helps create in-memory images of a database, this makes the process of analysing information considerably quicker than using OLAP [online analytical processing] cubes, which can take hours to build, according to John Coppin, product director at Kognitio.

"It has an immense number of implications if you can do things in that time," Coppin told IT PRO.

"You can move analysis much closer to real time. You can change things around as much as you like because you don't have to spend time rebuilding things."

Coppin said businesses want the ability to be flexible and remain fluid in how they mine data.

Pablo's introduction will mean Kognitio "is no longer niche" as it should open up a wider market for the firm, he claimed.

"Just because we don't have a separate OLAP technology it doesn't mean we're not actually competing against OLAP as a technology," he added.

"It does away with the need for businesses to do a whole layer of stuff."

Businesses should be increasing their investment in data management, in order to create consistent and high-quality master data sets, according to industry analysts.

Gartner, the IT research firm, believes that spending on master data management (MDM) grew 14 per cent to US$1.5bn this year, despite the tough economy.

Investing in data management will become more important still as the economy recovers, the analysts suggest. Customers will demand better service from businesses, and companies will need to improve data quality for compliance, reporting and risk management.

The firm expects spending on master data management to continue to expand, with a compound annual growth rate of 18 per cent between now and 2014. Growth is being driven both by offerings from specialist vendors and the large, enterprise software companies, along with increasing demands for data management from specialist business functions such as human resources and the supply chain.

But a growing market for MDM software will not, in itself, solve all business issues around data quality, Gartner cautioned. Businesses may lack the skills to deploy in-house MDM tools, especially for niche applications. Companies should look instead to use cloud-based data management services, in order to bridge the skills gap. Subscription or cloud-based models will also be more acceptable where data management projects are being carried out for just one area of the business.

And data management also needs to be tackled as a business project; if it is handled by IT alone, companies will not see its full value. "The business needs to take responsibility and be accountable for master data management," warned Gartner research vice president, John Radcliffe. "With MDM, CIOs can create a unified view of existing data leading to greater enterprise agility and ultimately improved profitability."

Previous industry research has shown that most business data is still not good enough to drive quality decision making.

• Gartner is holding a summit on master data management in London in February 2011.

British fashion retailer Jaeger has used a new data-mining tool to achieve a return on investment (ROI) in just three months.

The company installed the IDM Software tool to identify potential fraudulent staff activity and internal procedural issues in its stores by merging and analysing electronic point-of-sale (EPOS) transaction information alongside other, more complex data streams.

Steve Hearn, Jaeger head of safety and security, said the tool had already triggered interesting and unexpected lines of enquiry that would not have been visible before.

"LossManager has been up and running for three months now and, not only have we uncovered fraudulent activity, but we have also identified numerous procedural issues that in the current economic climate, make a significant yet wholly unnecessary impact on the bottom line."

Hearn said the Jaeger audit team had obtained business-wide view of data and process risk as well as obviously fraudulent activity.

"The audit team have taken custody of the day-to-day management of data mining and have already driven significant savings and identified huge profit improvement potential well beyond any ROI calculations run at inception," he added.

Jaeger worked with the vendor to fully integrate the new tool with its complex data streams, including EPOS transaction, electronic article surveillance (EAS) and CCTV data.