How we predict and explain future trends and outcomes is not a simple process. Even with the best will – and data scientists – in the world, it won’t and can’t always be 100% correct. In today’s age, this is normally where machine learning or statistics come into the equation.
History has always relied on humans to examine and interpret statistical data when making decisions. Now, though, the emergence of machine learning has put computers at the centre of answering that vital question of “what happens next?”.
Models driven by artificial intelligence (AI) and analysis are becoming the norm. There is still, however, genuine confusion between the benefits of taking a machine learning approach versus a statistical one. These differences even extend to educational pathways and career opportunities, including machine learning courses versus alternatives in statistics.
What is machine learning?
Machine learning uses computers to identify patterns in data, without the computers needing to follow explicit instructions. Instead, they are programmed with initial algorithms and models, and they learn to adapt these based on the data inputted to offer answers. However, this also means there’s less human control over the results.
Dr Michail Basios, co-founder of TurinTech, says: “Machine learning models are built for providing accurate predictions without explicit programming. Machine learning models can provide better predictions, but it’s more difficult to understand and explain them.”
What are statistics?
Statistical models make their predictions from numerical data. This is often gained from questioning people and asking them to make a choice; for example, it’s how political and election polling works, or consumer shopping surveys. Human involvement is usually more involved with the interpretation of this data. “Although some statistical models can make predictions,” Basios explains, “the accuracy of these models is usually not the best as they cannot capture complex relationships between data.”
What are the major differences between machine learning and statistics?
Statistics will be fed into computers for the purpose of machine learning, but there is one key difference between the two methods. This difference doesn’t lie in how the data is analysed, but in the objective or desired outcome, says professor Paul Clough of the University of Sheffield and head of AI and data science at Peak Indicators.
Clough gives the example of measuring the amount of chicken feed a hen gets versus the number of eggs it lays. “A statistical analysis would aim to explain how the amount of feed affects the number of eggs produced,” he says. “A predictive [machine learning] analysis would use the data to predict how many eggs the farmer will get next week.”
When would you use machine learning over statistics?
Clough’s egg analogy is useful in exploring this further. “The machine learning model doesn’t tell you anything about running a more efficient farm,” he suggests, “whereas the statistical model would be unwieldy if you owned tens of thousands of chickens.”
When looking at huge data sets, machine learning can be more optimal. Many concede statistics are unable to offer a deeper analysis on the relationships and correlations between the data when the levels of data are high.
They also cannot be relied on for causation, probability, and certainty because the danger is they might be misinterpreted or intentionally misused to back up a particular argument. As the famous saying goes: “There are lies, damn lies, and there are statistics.” Human involvement presents a weakness in statistics.
Basios, however, explains you can apply statistical modelling when you understand “specific interaction effects between variables” and “have prior knowledge about their relationships”. Machine learning, meanwhile, can be used when aiming for “high predictive accuracy”.
It tends to be ‘deployed’ more at the source of the data; as this is gathered and grows, algorithms automatically begin to provide intelligence. In manufacturing, for instance, it can predict when machines will need maintenance, reducing disruption. It can also analyse one option when compared to another – predicting which outcome would be better.
When would you choose statistics over machine learning?
MLOps and trustworthy AI for data leaders
A data fabric approach to MLOps and trustworthy AI
Statistics are still useful in understanding particular problem points that require deeper thought, as well as to produce metrics, tables and KPIs, says Graham Upton, chief architect of intelligent industry at Capgemini. Indeed, according to Eleanor Watson, IEEE AI ethics engineer at Singularity University, statistics is a human-focused method and easily explainable, while machine learning is far less explainable.
Although machine learning may be “more powerful” in circumstances with complex patterns – for instance, those with too many variables for a human to manage – one shouldn’t need to rely on the most powerful tool to solve a problem. “Companies often apply the jackhammer of AI to surgically split a peanut of a problem when good old-fashioned data science would be far cheaper and quicker,” she warns. “Data science holds up the modern economy far more than machine learning, but it remains an unsung hero.”
Is there a role for both machine learning and statistics?
Governments, businesses, and organisations are increasingly becoming reliant on machine learning as well as traditional statistics to help them make better and faster decisions. Each method, though, has its own individual drawbacks to consider.
“Statistical models have limited predictive accuracy since, sometimes, the underlying assumptions of the model are far too strict to represent reality,” Basios says. “Today’s businesses are adopting hybrid methods combining characteristics of statistical modelling and machine learning to understand in-depth how the underlying models work as well as to generate accurate predictions.”
What does the future hold for machine learning and statistics?
Decision intelligence (DI) is the space to understand in the future, highlights Dr Lorien Pratt, chief scientist at Quantellia. She explains it bridges both technology and the everyday decisions we all make hundreds or thousands of times to construct our world.
“There is a certain class of problem – ranging anywhere from day-to-day business strategy to the realities of climate change – that involves intangibles, complex systems dynamics, and human-machine interaction,” she says. “Machine learning, as a ‘supercharged’ form of AI, can cut through the noise and make a real difference.”
Pratt continues: “Rather than starting by thinking about the data we have, the key is to begin by asking what outcomes we want and what actions can be taken to achieve those outcomes, and then to fit the data into this action-to-outcome pathway.”
