Chief data officers (CDOs) might not be the most important executives in the C-suite right now, but they’ll rank among the most influential figures in no time.

That’s according to Deloitte's Chief Data Office (CDO) report, which found 54% of CDOs believe they have less influence on the business than their C-suite colleagues, but 44% believe that CDOs will become of equal influence by the turn of the decade.

"While over half of CDOs currently feel less influential than other C-suite leaders, there's a powerful shift on the horizon," said Andy Whitton, data, privacy and analytics partner at Deloitte.

The survey found that 57% of CDOs now report to a CIO or COO, up from 39% last year. Beyond the C-suite, nearly two-thirds of CDOs predict their teams will add employees over the next year, growing by an average 27%.

Budgets are also on the rise, with 43% saying they've seen a boost over the last year and 56% reporting an increase in data spend.

"Organizations are clearly doubling down on data, rapidly increasing their data maturity and literacy to gain crucial competitive advantage," said Whitton.

"This is positioning the CDO role as a pivotal and increasingly influential force across the C-suite, helping to drive sustained growth."

Chief data officers are riding the AI wave

The rise in influence for CDOs is perhaps no surprise given the sharp enterprise focus on AI and the key role they play in driving adoption of the technology.

Half of CDOs polled said they were accountable or responsible for AI or generative AI in their organizations, for example. Indeed, three-quarters of those surveyed revealed they have AI deployments or experiments operational right now.

Data governance and AI were by far the top priorities for CDOs, though their importance differed by sector. AI was deemed the top priority for CDOs in financial services, highlighted by two-thirds of respondents in that industry.

Meanwhile, 50% of corporate CDOs and 70% of public sector CDOs said data governance was their key focus.

Whitton said this highlights a “nuanced approach” taken by CDOs, with those operating across different sectors tailoring their priorities based on individual needs.

"For example, AI and GenAI are top of the agenda for 66% of financial services CDOs, whereas corporates are homing in on data products, and the public sector is rightly prioritizing data governance,” he said.

Those CDOs who named AI as their core priority also tended to predict their organization will realize benefits more quickly than those who aren't prioritizing the technology – though perhaps the eye on AI payoffs like productivity gains are why they're choosing to focus on the technology in the first place.

It’s not all plain sailing for CDOs, however. Long-running finance-related issues were identified as key barriers to progress.

Nearly half of the CDOs said budget and resource limitations were the biggest hurdle to AI adoption, for example, followed by skills gap and ethical and regulatory concerns.

A separate report from earlier this year found that two-thirds of CDOs believed data quality was the biggest challenge to AI adoption, hampering them from maximizing the use of the technology.

MORE FROM ITPRO

• Why diversity is key to a successful AI strategy
• Data governance in the spotlight as UK firms ramp up AI adoption
• A quarter of firms still don’t have a formal data strategy

Snowflake - initially called Snowflake Computing - was a 2012 brainchild of former Oracle data architects Benoit Dageville and Thierry Cruanes, with database scientist Marcin Zukowski, in San Mateo, California.

Today the company is focused above all on its AI data cloud concept, Snowflake is known for its engineering chops in cloud data warehousing, separating data processing from data storage. While it has offices around the world, it does not operate with a traditional headquarters.

Its 2026 Q1 ended 30 April netted $1 billion in revenue, following a full-year 2025 totaling $3.5 billion. However, despite revenues 30% up from 2024, Snowflake reported a $1.5bn loss.

Snowflake’s CEO is Sridhar Ramaswamy, with Chris Niederman joining from AWS as alliances and channels leader in July. In 2025, the company boasts 8500 staff across 40 offices and roughly 12,000 partners, including phData, Qlik, Capgemini, Dataiku, Alteryx, and Blue Yonder. The partner program recently underwent some changes, which received positive feedback from the channel community.

Key offerings

The serverless, managed Snowflake Platform targets built-in governance, finops, and observability to help enterprises benefit from their data under an AI Data Cloud umbrella. It's all about having a data platform as a self-managed service, according to the vendor, which is largely end-user hardware and software-free. Snowflake handles the maintenance, management, upgrades, and tuning.

• Capabilities incorporate data integration and collaboration functionalities, data source connectivity, and developer tooling.
• Highlighted are Snowflake ML for model development and ML ops, Snowpark libraries and code execution, and Cortex AI for accessing large language models (LLMs).

Recent news

• Snowflake in June 2025 announced plans to acquire open-source Postgres provider Crunchy Data. The aim? Developing secure agentic AI faster.
• Fawad Qureshi, Snowflake global industry field CTO, told ITPro in March that businesses need AI to convert data from byproduct into "digital fuel", requiring complete lineage traceability and audit of entire processes.
• CEO Sridhar Ramaswamy, opening Snowflake's World Tour 2024, reiterated Snowflake and partners as poised to usher in "the era of enterprise AI. where AI is easy, efficient, and trusted".

Snowflake for partners

End-users harness Snowflake to build, use, and share data, apps, and AI; many partners offer professional, managed, or implementation services. Within the AI Data Cloud concept, Snowflake has remodeled its partner approach and is gearing up for more updates in its bid to hit $10bn by 2030.

Dan Waters, Snowflake's regional vice president for partners, notes that partners are key to its global ambitions across a complex ecosystem, especially as further advancements such as agentic Snowflake intelligence roll out.

"Focusing on that democratized data and use of Snowflake as a data platform, the great thing is, partners are part of that with us. We work with partners to develop market solutions and their capability," Waters says. "So we train them, give them early access to that innovation. With product release, we already have a community already set up to speak to clients on how they take this on."

As new-minted channels chief Niederman said in July, it's all about accelerating joint successes.

"Throughout my career, I've seen how a thriving partner ecosystem and partner-led growth directly impact customer success. Snowflake's partners are at the forefront of innovation, extending the power of the AI Data Cloud and enabling our joint customers to unlock maximum value from their data and drive business outcomes," he says.

The overarching Snowflake Partner Network (SPN) sits up top. "We have partners that we work with to sell, and work with on delivery and strategic advisory. We have partners that build applications, manage native-connected applications, and partners that are strategic from a technology and platform perspective, like, for instance, Workday, Microsoft, AWS," confirms Waters.

Partner programs, tiers, and partner types

• SPN partner types include Select, Premier, and Elite tiers:
  • AI Data Cloud Services partners - delivering and managing Snowflake platform services.
  • AI Data Cloud Product partners - building products and taking them to market.
  • Data Cloud partners such as strategic AWS, Microsoft, and Google partnerships or ISVs.
• Certifications, renewable every two years:
  • SnowPro Associate introductory certification.
  • SnowPro Core technical certification - demonstrates the ability to implement Snowflake.
  • Three SnowPro Specialty certifications - Snowpark, Native Apps, and Gen AI.
  • Five role-based SnowPro Advanced certifications - Architect, Data Engineer, Data Scientist, Administrator, and Data Analyst.

Certification exam guides are on the same page. Click here to create an account and schedule exams. Further resources are here, with a report on certification value here.

Further benefits, rewards, and incentives

Training via Snowflake University - Get access via the Snowflake Community. The array of resources includes:

• Five SPN Learn partner-only courses - Snowflake Beginner Essentials, Mastering Marketing with Snowflake, SPN Gen AI Foundation, Snowflake Partner Technical Foundations (PTF), and Data Engineering Foundation Learning Track.
• Other learning tracks with:
  • Hands-on badges - data warehousing workshop, collaboration, marketplace, and cost estimation workshop.
  • Level up: 1st concept series - first concepts, Snowflake key concepts, and architecture.
  • Level up: Performance series - Query history and caching, Context in Snowflake.
  • Level up: Snowpark and Python Essentials - getting started with Snowpark for Python, Setting up the Snowpark environment for Snowpark Python.
  • SnowPro Core Study (for SnowPro Core certification) - Fundamentals, SnowPro Core Certification prep course.
• Additional training on demand - 63 tailored courses for partners, including advanced data engineering and data lakes or agentic AI and marketing. View all courses here.
• CoSell Referrals incentives program, with ReSell launching in September. Registered partners can earn 5% gross annual contract value (GACV) for referrals, Select, Elite, and Premier 10%. ACV tiering credits are available for expanded customer capacity contracts.
• Deal registration.
• Services registration (starts in August).
• Services funds program (SFP) for registered service partners' pre-sales proofs-of-concept (POCs), migrations in implementation, or growth renewals.Variable up to 3-5%.
• Partner migration services.
• Systems integrator (SI) partner services.
• Practice Development Program funding for Elite partners.
• ISV funding program for Data Cloud partners.
• Snowflake forum.
• Data sharing rebate program - 10% to 50% for consumption of shared data.
• Developer community.
• Partner, apps and solutions finder.
• Powered by Snowflake program for applications built on Snowflake.
• Resource library[https://www.snowflake.com/en/resources/] - blogs, studies, ebooks, and guides.
• QuickStarts guides.
• Live demos with Q&A.
• Snowflake Marketplace - 3000+ searchable listings.

Sign up and get on board

To join the program, sign up to SPN with this Enroll button. "We are looking for domain expertise, and data specialists, but also partners that have reach and broad segment or vertical expertise that are looking to drive innovation for their customers," says Waters. "We want to bring in partners that are focused and clear on what they want to work with us on and how this will drive value to our joint customers." Create a user account here.

Email partner support at partnersupport@snowflake.com or contact sales here.

Snowflake is on LinkedIn, YouTube, Instagram, Threads, and X. There's also a LinkedIn SPN group for partners.

Only a third of businesses are making meaningful progress in AI adoption, and chief data officers (CDOs) are pinning the blame on poor data quality.

More than two-thirds (68%) of CDOs said data quality was their top challenge, according to the Ataccama Data Trust Report 2025. Meanwhile, four-in-ten struggle to maintain consistent data quality, directly hindering AI outcomes.

The report noted that trust is a critical factor when leveraging data in daily operations. Without it, organizations face inefficiencies, poor decision-making, and the risk of compliance failures, ultimately limiting their ability to achieve business objectives.

However, the report found most organizations still struggle with siloed systems.

"Untrusted data erodes every decision it informs. Without real insights into data quality, businesses risk cascading failures, from unreliable AI outputs to stalled growth," said Krishna Cheriath, chief digital Officer at Thermo Fisher Scientific.

"Trust must permeate every layer — data, models, and decisions."

Bad data leads to bad insights, the report noted, affecting decision-making, slowing down operations, and wasting valuable resources. Similarly, it jeopardizes compliance, leaving organizations vulnerable to regulatory and financial risks, and diminishes ROI.

Knowledge gaps are hampering digital progress

Knowledge gaps around data trust and governance are slowing progress, the report found, with a lack of unified standards leading to inconsistency. Without guidelines for data formats, definition, and validation, CDOs find it hard to establish a centralized system of control.

A third of organizations experience processing delays because there are too many barriers that stand in the way of integration.

"Fragmented systems bleed efficiency and inflate costs," said Andrew Foster, chief data officer at M&T Bank.

The report also found legacy systems are still a major barrier to innovation, with CDOs finding that outdated systems are ill-equipped to handle increasing data volumes.

Many systems are designed to provide period data updates, for example, rather than continuous, real-time streams. It’s this area in particular that’s causing serious trouble for CDOs ramping up AI adoption, with only a third of enterprises reporting meaningful progress on this front.

The report called for new national standards for data quality in the UK, and suggested that the proposed National Data Library - a core goal within the UK’s AI Action Plan - could play a key role in bolstering national data governance benchmarks and best practices.

These standards would ensure clear compliance guidelines while supporting the UK’s pro-innovation regulatory goals.

"The report makes one thing clear: enterprise AI initiatives rely on a foundation of trusted data," said Jay Limburn, chief product officer at Ataccama.

"Without addressing systemic data quality challenges, organisations risk stalling progress. The UK’s approach to AI regulation shows how aligning data trust principles with national standards and infrastructure modernization can deliver tangible results."

IBM has announced an extension of its partnership with The All England Club to generative AI-based services at the 2023 Wimbledon tournament. 

A raft of new technologies will be used to create richer spectator experiences, including AI-based commentary for match highlights and analysis of player statistics throughout the tournament. 

The announcement builds on a longstanding relationship between the two organizations, which IBM said will greatly enhance experiences for fans. 

“We are constantly innovating with our partners at IBM to provide Wimbledon fans, wherever they are in the world, with an insightful and engaging digital experience of The Championships,” said Usama Al-Qassab, marketing and commercial director at The All England Club. 

“This year, we’re introducing new features for our digital platforms that use the latest AI technology from IBM to help fans gain even more insight into the singles draw and access commentary on a wider variety of matches through our match highlights videos”.

AI Draw Analysis tools

New generative AI features set to be rolled out at Wimbledon this year include the IBM AI Draw analysis platform, which IBM described as a “first for tennis”.

This will use AI to produce statistical analyses of player performances that will then inform predictions such as a given player’s most favorable path to the final, for example.

Player draw favorability will be defined by ratings based on a range of factors, including matchups against future opponents and how a player’s position in the draw compares to competitors, IBM said. 

“This new insight will help tennis fans to uncover anomalies and potential surprises in the singles draw, which would not be apparent by looking only at the players’ ranking, inspiring more debate and engagement in the fan community.” 

AI commentary features

Powered by IBM watsonx, this year’s Wimbledon championships will also be augmented with AI commentary capabilities that will identify key moments in a match and provide automatically generated audio commentary on them.

IBM said the tool has been designed specifically to give fans a “more insightful experience when catching up on key moments from matches”. 

“Its introduction this year is a step towards making commentary available in an exciting way for matches outside of Wimbledon’s Show Courts, which already have live human commentary.”

Development of the new feature was led by staff at IBM iX, the experience design partner within IBM Consulting, with staff working alongside the All England Club to use foundation models from watsonx.

These foundation models were used to train the AI in the “unique language of tennis” and applied to provide narration of match highlights with “varied sentence structure and vocabulary to make the clips informative and engaging”.

Jonathan Adashek, senior vice president for marketing and communications at IBM, said the introduction of the new features will bring “new layers of insight and engagement” to the 2023 championships, while marking a significant step in leveraging generative AI technologies for fan experiences. 

“We’ve seen first-hand how these technologies have the power to help major sporting events like Wimbledon to grow their audiences through outstanding digital experiences,” he said. 

“The AI and data platform that IBM is using to create unique fan experiences for Wimbledon is the same technology that we’re using to drive business transformation with clients across all sectors and industries.”

IBM’s long-term Wimbledon partnership

Fans are no strangers to AI-powered features and services at the tournament. IBM has been a long-term partner of The All England Club, and to date has launched a range of tools and features for match-goers, the firm said. 

This includes the IBM Power Index Leaderboard, personalized highlight reels and recommendations, and the IBM Match Insights platform, which offers fans detailed insights on player performances throughout the competition. 

These tools draw upon over 100,000 data points from “every shot played across the tournament” and are analyzed by Watson’s AI technology, the firm said. 

“These digital features are designed to make it easier for fans to understand which players to follow, how they compare to their opponents, and who’s likely to win.”

Less than half of data and analytics (D&A) leaders believe their team is effectively delivering value to their organisation.

A survey of more than 500 D&A leaders globally found that just 44% reported their teams are unlocking tangible long-term benefits for their firms despite rising budgets and growing expertise.

D&A budgets have risen steadily as organisations seek to deliver greater value from data, streamline operations, and enhance operational agility. The mean D&A budget now stands at $5.41 million (£4.4 million), the research from Gartner found, while 44% of D&A teams also increased in size over the last year.

However, while rising budgets reflect increasing confidence in the data function, this is placing additional pressure to deliver impactful results, resulting in a negative impact on both teams and leaders, said Donna Medeiros, senior director analyst at Gartner.

“The demands being placed upon D&A, as well as increased investment, reflect a growing confidence in CDAO’s abilities and recognition of the data office as an indispensable business function,” she said.

“However, this leads to more work as pressure grows for D&A to achieve tangible business results.”

Gartner’s study found that D&A functions are receiving increased investment across a range of areas. Nearly two-thirds (65%) of leaders reported increased investment in data management while 63% and 60% reported investment in data governance and advanced analytics capabilities respectively.

Growing responsibilities present challenges

Chief data and analytics officers (CDAOs) now find themselves tasked with an expanding range of responsibilities within their organisation, Gartner found.

60% said they are now responsible for defining and implementing data strategies, with 59% also taking responsibility for oversight of said strategies.

Management of data-driven culture change was also a key responsibility highlighted by 54% of respondents.

This widening array of responsibilities means that some CDAOs are struggling to meet increasingly complex leadership demands, however.

Leaders that committed time to their own professional development were found to be far more effective than those who did not, highlighting the critical need for continual learning and evolution to accommodate the changing demands of the role.

Nearly half (43%) of “top-performing” D&A leaders reported increased effectiveness in both their own ability and the broader team by committing time to professional development, compared with just 19% of “low performers”.

“Successful CDAOs must be elite leaders,” said Alan Duncan, distinguished VP analyst at Gartner. “Top-performing CDAOs invest in their success by developing skills to thrive in ambiguous circumstances, articulate compelling value stories and identify D&A products, and services that can drive business impact.”

Human-related challenges to success

Difficulty meeting the complexity of demands placed on D&A teams is being exacerbated by an acute shortage of talent, Gartner noted.

More than one-third (39%) said that a lack of available talent had become a major impediment to success in the data function.

In fact, the top six key roadblocks for D&A leaders were all human-related challenges, including poor data literacy across the broader organisation and culture change acceptance.

Organisations make key business decisions on the basis of what they know, and what they know is derived from analysing their data. To enable the best decisions, data has to be accurate and reliable. But there are still organisations out there with data silos, which by their very nature can be anything but accurate and reliable. These silos often contain duplicate, out-of-date, or otherwise dangerous data.

Added to this, maintaining two or more data sets on the same topic uses time and technology resources an organisation could more fruitfully channel into other work. It’s time to purge the data silos and stop them from ever coming back. Doing so, however, involves hunting for silos across the breadth of an organisation and implementing measures that address the cultural deficiencies that let silos flourish in the first place, all of which is easier said than done.

What damage can data silos do?

There’s widespread agreement that data silos are a bad thing. “There can be no debate,” insists Dave Williams, consulting associate director for leading data consultancy Dufrain, Dave Williams. “Data silos are the scourge of the modern company.”

Chief among the problems they raise is the lack of centralised control, he tells IT Pro. Without strong governance to ensure the data an organisation holds is accurate and up to date, data silos are simply repositories that can lead to ill-informed decision-making.

“Silos can deeply impact the decision intelligence of any organisation,” agrees Janet Bastiman, chief data scientist at Napier and chair (data science and artificial intelligence (AI)) at the Royal Statistical Society. “Without a single holistic view, decisions may be taken without full information,” she elaborates. “More importantly, updates may not filter to all silos, meaning that duplicate data can be in different states.”

How to identify data silos

If you think your organisation has no data silos, are you absolutely certain? Have you checked everywhere? Well, you might be surprised, Williams suggests. “Data silos are typically created in business areas when a centralised data platform is either too well governed to accept disparate business data, or where an IT team is too disinterested to incorporate data together because the corporate value is considered low.”

There can be other factors at play too, such as business units thinking they have some kind of advantage if they maintain their own data. This can be born from a sort of psychology of competitiveness that has its roots well outside the technology, in the overall way business units view each other – as competitors rather than collaborators.

Whatever the reason for the existence of silos, finding them can be tricky. Just asking around isn’t always going to be enough. Some tech-based sleuthing might be able to identify large files such as spreadsheets, the owners of which can be quizzed. But if this approach is taken, Bastiman offers some sage advice. “The individual who does this must have clear authority from the executive team to have access to sensitive silos,” she says. This is especially important if there’s already a competitive, perhaps antagonistic, vibe between business units.

Indeed, former chief data officer (CDO) for Network Rail, as well as CEO and co-founder of Carruthers and Jackson, Caroline Carruthers, tells IT Pro technology isn’t the ideal tool for hunting down data silos. told IT Pro that technology isn’t the ideal tool for hunting down data silos. ”I’d argue hunting data silos doesn’t really need any sort of technological method,” she says. “Tech can do whatever you want it to if you put your mind to it, but data silos themselves are fundamentally people problems.”

Preventing data silos is better than ‘curing’ them

Once data silos have been identified, the next steps are to remove them and prevent them from appearing again. This too, for Carruthers, is a people thing, not a technology thing, and the responsibility often lies with the CDO. “A CDO’s role here is to understand why data silos exist in the first place, as they’ll have happened in the first place because some employees thought they were needed,” she explains. “Fundamentally, you can’t wipe out a silo without putting something better in its place, and you can’t put something better in its place without first understanding the need.”

Williams is also clear that workplace culture really matters, and has a key role to play in preventing data silos from reappearing once they’ve been eliminated. “Once all data silos are identified and broken down, companies must rethink their cultural and technological processes to protect themselves from data silos popping up in the future,” he tells IT Pro. He proposes the promotion of the development of “collegiate ways of working to ensure everyone is working from the same one true source of data”. Once such a regime is in place, he continues, “culture and technology must work together in lockstep to affect long-term change”.

Developing and maintaining a collegiate, collaborative approach to work, using cloud-based data systems, might take time. But there seems to be general agreement that data silos arise as a result of certain ways of working and are prevented by the adoption of others. In the end, then, data silos are a technological solution to what is very much a human situation. Change the situation – eliminate the data silos – and make better business decisions, too.

This article originally appeared in issue 34 of IT Pro 20/20, available here. To sign up to receive each new issue in your inbox, click here

Data now sits and the heart of enterprise strategy and senior stakeholders recognise its significance. The most forward-looking businesses are appointing a chief data officer (CDO) to ensure their organisations take advantage of the information they collect. The role of a CDO, however, has changed over time and will continue to evolve as data becomes more important to businesses in the coming months.

Such is pivotal the role of information in modern business that consultancy firm PwC reports that two-thirds of companies have seen a growth in references to data in their annual reports since 2017. Data frequency at these firms has increased by 78% in the last five years.

While most organisations know the power of data, not all executive teams support that appreciation with the appointment of a CDO. Recruiter Nash Squared reports that two-thirds of digital leaders see big data being as key to creating a competitive advantage during 2023, but only a fifth feel their organisations are effective at using data.

In fact, PwC found – despite growing recognition of the role played by data in an effective business strategy – less than a quarter of companies have a CDO in place at a senior executive level. Even when a data chief is in situ, these C-level appointments are concentrated in a few sectors and regions.

It’s hard to find a good CDO these days

Nash Squared CEO Bev White believes one explanation for this paucity of data leadership is that good CDOs are tough to find. “It's a talent-short market – and what we're seeing is a lot of people moving around roles because they're getting poached repeatedly,” she says.

White says the best data chiefs are likely to have a solid background in large-scale analytics but are also great communicators: “CDOs have got to be able to talk with people who are in decision-making roles and convince them that an investment is a good one because the business will then capture a slice of the market it can't currently reach.”

The requirement to demonstrate the value of emerging technologies – in areas such as analytics and machine learning – means data leaders must be proactive and persuasive. White says CDOs holding these capabilities will be very marketable through 2023 and onwards.

“They've got to be able to form a strategy and have a strategic mind,” she says. “And they need to be able to create a really strong data environment, where people understand the value of bringing data together into a place that they can use to make good, solid decisions and create new offers for their markets.”

One data chief who’s already demonstrating these strengths is David Charnley, head of transformation and strategy at NatWest Group. He says data leadership was historically focused on data governance and information management, with responsibilities that stretched from data quality and lineage to controlling metadata repositories.

While those responsibilities still hold true in some organisations, Charnley believes the CDO role is evolving – and the result is a broad spectrum of data chief roles. At NatWest, data leadership involves looking across analytics, digital decision-making processes and data science capabilities for customer-facing businesses, but also dealing with data management concerns and the technology that underpins the work that data teams do for the business.

“I think that combination of all of those disciplines together, unified with the technology, is really, really powerful,” says Charnley. “Because it then means the CDO is very much in control of the data strategy and its implementation for the bank.”

Data platforms are now seen as an investment

Research suggests the most pioneering businesses are now aware of the role a talented data chief can play. The boards and CEOs of these organisations are appointing senior executives who are accountable for data success. PwC says three out of five CDOs are now members of the C-Suite with oversight of enterprise-wide data.

That’s a trend that’s likely to continue, says Alex Read, senior manager of data platforms at EDF UK, who suggests next-generation data leaders will have to do more than simply extract value from the technologies they buy. He says successful data chiefs will focus on how their organisations can work effectively to create value.

“Historically, there was too much focus on the data platform – just enabling the business-as-usual elements, such as reporting at a point in time, and enabling regulatory requirements, which is all of the standard stuff you'd expect the business to need,” he says. “Now, data platforms are seen as an investment. And I think what you need to do as a data leader is to communicate to your key stakeholders what that investment means.”

At EDF UK, Read is creating an integrated data stack that runs on the Snowflake platform. By using tactical solutions – such as the Snowpark development framework, AWS SageMaker for machine learning, Collibra’s data catalogue, AirFlow for orchestration, and Matillion for data extraction and transformation – Read focuses on how this combination of tools can support innovative, information-rich services for EDF UK and its customers.

“I think that's a massive step up,” he says. “This work is about more than just platform consolidation and the reduction of legacy platform costs. It's about, ‘OK, great, once we've done that foundational work, what's the value-add in having this platform, such as data science and other downstream analytical processes?’ These are lenses that a data leader should be looking through as part of that journey.”

Transmitting strong data skills across the enterprise

Data chiefs can’t just focus on developing internal capabilities, though. As well as ensuring their team has the techniques to make the most of information, CDOs need to find ways to turn the exploitation of data into an enterprise-wide activity. For organisations to create useful services, it’s important that all kinds of professionals hold strong data skills.

That’s something that resonates with Marc Jennings, CIO of analytics and artificial intelligence (AI) at holiday firm TUI, who says machine learning and data science skills must be embedded across the business. While the governance of the technology platform rests under the CDO’s wing, the significance of insight means data savviness is now everyone’s responsibility. Smart CDOs will ensure this gestation of skills takes place through 2023.

“For me, that's absolutely crucial,” he says. “It's not just my space, it's not just my team’s. We want to democratise data. We want to make great decisions available everywhere. Everyone needs to have informed evidence and facts and be able to build capabilities, give insights, interpret information and make the right decisions. Responsibility for data has got to be held everywhere across the organisation.”

The British Computer Society has revealed that interest in computing degrees is growing at a greater rate than any other university course, and has credited this rise to the popularity of AI and data science.

92,980 18-year-olds applied to start computing degrees this year in the UK, a 9.6% rise and demonstration of the sector’s continued appeal.

BCS identified leading-edge technologies such as artificial intelligence and data analytics as an especially attractive element that the sector can offer to prospective students.

“Young people are seeing and using real-world generative AI tools, like Dall-E for pictures, so it’s no surprise that career aspirations and education and skills choices lead them this way,” a spokesperson from BCS told IT Pro.

Although the 9.6% rise is more than the Universities and Colleges Admissions Service (UCAS) measured in other courses, it is a smaller increase for the subject than in previous years. In February 2022, BCS reported a 13% rise in applications.

This is likely a reflection of the smaller overall number of applicants this year. 597,000 students applied by UCAS’ January deadline as compared to 611,000 in 2022.

Amidst the broadening of disciplines within computing, there has also been a narrowing of the gender gap. The number of women applying for computing degrees rose 18% in 2022, a larger shift in gender makeup than seen in other degrees.

Despite this positive shift for gender diversity at a degree level, men applying for these courses still outnumber women 3.8 to 1, a slight reduction from the 4.2 to 1 ratio recorded in 2022.

“Young people – and an increasing proportion of young women – see that a computing degree is a passport to change the world,” said Julia Adamson, MD for education and public benefit at BCS.

“AI and machine learning are transforming how kids complete homework and how job applicants write the covering letters. It’s no wonder so many people see their futures in technology. The more diverse range of people we have working in computing the fairer and more inclusive the results will be for all of us.”

BCS also identified the use of data science and tech like AI and machine learning in the fight against climate change as another draw for young applicants.

This aligns with new insights into the need for Gen-Z staff to be given clear purpose in their roles through schemes such as those focused on sustainability.

The UK government recorded a near 50% rise in the number of students accepted to computer science courses between 2011 and 2020, and noted that STEM graduates are more likely to snag a higher starting wage in their first post-university job.

High wages could also drive students toward the computer sciences, and BCS listed the “range of career options” available to computing graduates as a deciding factor.

Salaries in roles such as software engineering top $1m in Silicon Valley, and the highest-paying tech jobs often go to those with an academic background in the sector.

In 2022, computer science became the UK’s fastest-growing degree subject, and in the three years up to the data being published the number of students taking computing at A-level had risen 41%.

More UK computing graduates are needed to fill skills gaps across the sector, with a recent Gartner report revealing that one in four UK businesses is hiring overseas to tackle the talent shortage.

The UK continues to lag behind most of Europe when it comes to tech skills, and some in the sector have called for more investment in domestic tech literacy to bring the dearth to an end.

With coding bootcamps touted as an alternative to degrees, not all those learning programming in the sector do so at university.

Stack Overflow’s 2022 Developer Survey noted that while 62% of respondents were more likely to have learned to code from traditional education, 71% cited online sources such as video tutorials, blogs, and forums as better resources for learning.

Also shown in the report was the large number of young people learning to code, with 56% of respondents learning to do so being aged 18-24 and 18% being under 18.

Data scientists and machine learning engineers are in hot demand as more companies look to leverage real-time data and uncover insights and trends that they can then use to gain an advantage in their industry.

The terms ‘data science’ and ‘machine learning’ may often be used interchangeably by those who aren’t experts in either field, but they are two completely separate disciplines.

In short, data science is used to turn large amounts of data into useful insights. Machine learning, on the other hand, falls under the umbrella category of AI and leans on the insights derived through data science to create projections or otherwise improve a system’s performance.

Machine learning vs data science: What are the similarities?

Machine learning and data science have the shared purpose of lifting insight from data. A data scientist will deploy statistical modeling and data vizualisation on this quest, generating structured findings from unstructured data.

Both may use algorithms, but machine learning relies entirely on algorithms to function. Modern data science projects often involve machine learning to speed up manual processes. 

However, despite the similarities and overlaps between these two fields, each has distinct characteristics and uses.

Machine learning vs data science: What are the differences?

While data science brings order to disordered data, or pulls order from it, machine learning aims to learn from data to inform future actions or predictions.

A core part of data science is data cleaning, in which erroneous data, anomalies, and non-defined values are corrected or deleted.

Machine learning relies on some level of data science, as its algorithms cannot properly use data to learn and make decisions unless it has been cleaned and proven to be accurate and reliable.

Which skills are required for data science and machine learning?

Data science requires a wide range of skills, including domain knowledge, programming, statistics, and data visualization.

A career in machine learning requires experience in computer science and mathematics. In particular, skills such as linear algebra, calculus, and probability theory are handy to be able to implement the algorithms machines will need to make predictions.

Both disciplines can come in use in a wide range of roles across various industries, including AI engineer, business intelligence analyst, data analyst and research scientist.

How are machine learning and data science used across different industries?

In technology, data science and machine learning are used by engineers to build intelligent systems and improve user experience (UX). For example, Google uses machine learning to improve its search results, and Amazon uses it to personalize product recommendations.

Financial institutions, on the other hand, would use data science and machine learning to detect fraudulent transactions, predict stock prices, and identify potential investment opportunities.

The technologies both have a clear role in healthcare. For example, machine learning algorithms can be used to identify patterns in medical images and make accurate diagnoses, while data science can be used to identify inefficiencies in medical processes and improve the management of a hospital environment.

Similarly, in retail, companies analyze customer data and make personalized product recommendations, while also working to optimize the supply chain and logistics. In transport, meanwhile, machine learning is used to optimize traffic flow, prejudice maintenance, and demand for a specific route.

Finally, data science and machine learning are widely used in research fields such as physics, genomics, biology, and environmental science to analyze and understand complex data sets.

What are some of the most common applications of data science?

There’s an array of use cases for strong data science skills across various organizations. Among these include the following.

1. Predictive maintenance for manufacturers

Data collated from sensors on a factory floor can be used to predict when a piece of machinery might break down and allow engineers to carry out maintenance before a fault occurs. Coupled with automation, this is the basis for smart manufacturing and smart ports.

2. Fraud detection

By analyzing transaction data, data scientists can build models to identify unusual behavior that may indicate fraudulent activity and help with fraud detection and prevention.

3. Market segmentation

Data scientists can segment a market and identify customer groups with similar characteristics based on enterprise data. This allows businesses to target their marketing efforts effectively.

4. Sentiment analysis

Analyzing text data by carrying out text mining can help a data scientist understand customers' perceptions of a particular product, service, and brand and help businesses make decisions accordingly.

What are the most common applications of machine learning?

Machine learning is used across the entire economy, with varying sizes and types of organizations able to leverage the approach to iteratively improve and automate business aims.

1. Forming the basis for more complex AI systems

Machine learning is used as the building blocks for widespread AI approaches such as generative AI, in which it is used to map the contextual web that connects text, images, video, and audio. At a basic level, this forms the basis for large language models (LLM) to accept user input and produce relevant outputs.

2. Image and speech recognition

Machine learning algorithms can be used to analyze images, such as for facial recognition in security systems or for computer vision systems that can help improve accessibility. 

Visual data is analyzed based on known visual markers, so that a system may recognize audio patterns such as speech or discern visually between objects and people.

3. Recommendation systems

Machine learning algorithms can be used to analyze user data and make personalized recommendations for products or content.

This has become a core part of the user experience across many online platforms, with most shopping and streaming websites using machine learning to make tailored recommendations based on user data.

4. Handling language

Machine learning models can be used for tasks like natural language processing (NLP), language translation, text summarization, sentiment analysis, and much more.

5. Autonomous vehicles

Machine learning can be used to iteratively train self-navigating vehicles, such as autonomous robots on factory floors or self-driving cars. In practice, machine learning is applied to sensor data, to make low-latency decisions at the edge to ensure vehicles stay within markers and don’t endanger people.

Everyday developers and data scientists are leveraging the power of machine learning from AWS to solve the world’s biggest challenges. From connecting patients and caregivers to supporting sustainability, machine learning models are at the core of this evolution.

This eBook explores how this transformative technology is pioneering impactful solutions.

Download now to read how innovators are tapping into the best practices, in-depth expertise, and powerful machine learning solutions to launch new initiatives that are improving lives and protecting our planet right now.

Provided by

Snowflake has revealed new enhancements designed to make it easier for developers to build apps on its data cloud platform while reducing reliance on third-party tools.

Following its acquisition of Streamlit in March 2022, an open source app framework in the Python language, the company is now integrating the framework with its own technology. It said this will help developers bring their data and machine learning (ML) models to life as secure and interactive applications on Snowflake.

The cloud data company hopes the Streamlit integration will help developers build powerful applications without the complexity of building and deploying web applications. The integration will help developers create applications with Python using their data in Snowflake, and then deploy and run these applications on Snowflake’s platform before sharing them with business teams.

“As we continue to disrupt application development, we’re giving builders the data access and tools they need to accelerate their pace of innovation securely under Snowflake’s one unified platform,” said Torsten Grabs, director of Product Management at Snowflake.

“Snowflake’s advancements provide developers with the capabilities to build powerful applications, pipelines, and models with the utmost confidence, and eliminate complexity so they can drive value across their organisations with the Data Cloud.”

To further reduce the need for developers to turn to third-party tools, Snowflake is also providing access to Python and its open source libraries to all users and teams, with the general availability of Snowpark for Python. This means developers will be able to use Python to query and process data queries in a data pipeline while remaining on the company’s platform.

Snowflake said that since its public preview announcement in June 2022, Snowpark for Python has seen six times growth in adoption, with hundreds of customers building their data using the platform, including EDF, Sophos, and Northern Trust.

Through Snowpark, developers also get a streamlined architecture that natively supports users’ programming languages of choice, including Java, Scala, SQL, and Python. Snowflake is also releasing Snowpark-optimised warehouses, which are currently in public preview in AWS, so Python developers can run large scale ML training and other memory-intensive procedures directly in Snowflake. They can also use Python Worksheets, which is in a private preview, to develop applications, data pipelines, and ML models inside Snowflake.

Lastly, Snowflake is also making it easier to work with streaming data within a single platform and further eliminate silos for customers. Users are able to boost productivity by onboarding faster with Schema Interface, and execute data pipelines with Serverless Tasks on Snowflake’s platform.

Data is among the most valuable assets many organisations have, and if you have the right training there's a world of high paying, challenging work waiting for you – two of the most prominent are data scientist and data analyst.

And while the two have some similarities, the common pathways into careers in each area can be quite different, and candidates have a wealth of choices around types of study. The responsibilities and workloads of each are different, as are the further career prospects.

Two of the most popular career paths in data are those of the data scientist and data analyst. While similar in nature, on the face of it, they have varying routes of entry – including data science courses, data science bootcamps, as well as university degrees. They also have differences in terms of roles and responsibilities, as well as varying career prospects.

What does a data scientist do?

Online information systems are coming into being and growing so fast, the need to analyse data is exploding. In 2023, we're set to produce 120 zettabytes of data, up from 97 zettabytes last year and set to be outstripped next year with 147 zettabytes, according to Statista.

But companies and organisations that collect that data have to store and action it in particular ways for it to have any value. Enter the data scientist, who sets up frameworks for others to synthesise and extract insight from the data that's been gathered.

A data scientist combines computer science and mathematics to do this, with a particular focus on statistical tests and distributions. The companies that collect data need to ask it questions, and the data scientist figures out how best to frame the question and translate the answer. Like any business asset, data can be used to provide a financial or social advantage, and the data scientist helps to reach that goal.

Data science is the art of collecting, collating, processing, analysing and interpreting data in both structured and unstructured environments, creating frameworks that standardise it for further interrogation. Their arsenal includes machine learning or AI, data mining, statistical algorithms and more to 'smooth' data into a comprehensible form.

What does a data analyst do?

A data analyst takes a dataset from multiple sources, cleans it, and reorganises it for analysis. The tools of the trade include automated systems for unearthing unstructured data as well as stripping out or fixing corrupted or garbled files.

Whether it's designing and building an analytics system or delivering results from the analysis performed, a data analyst can enter the process anywhere along the workflow (or perform it entirely), including training other staff in conducting analyses that relate to their own business units.

A data analyst also needs to be a presenter, putting reports and summaries of results and interpretations into formats executives or other stakeholders can digest.

What skills do data scientists and data analysts need?

Data scientist skills

As with other scientific fields, a qualified data scientist is usually at least a holder of a Master's degree or PhD, and typically an expert in both technical and non technical principles.

The non-technical skills include critical thinking – clients and employers will have specific questions the data needs to answer, and the data scientist must structure the workflow so the available information can do so in direct relation to the industry or company concerned.

Data scientists also need to be competent communicators, able to put arcane concepts into relatable language for other business units, and the ability to think on your feet to solve problems as they come up is critical.

When it comes to the more technical aspects of the job description, the data scientist needs to know how to prepare datasets for the best analysis tools and methods. They need to know how to write effective code to do so, applying maths, statistics, and machine learning on a daily basis.

It also calls for a thorough understanding of database programming languages like Python, R Programming, SQL, Scala, MongoDB, MySQL, and more.

Data analyst skills

The responsibilities of a data analyst have some overlap with those of a data scientist. The data analyst similarly needs to be proficient in maths, which is applied more to the modelling and analysing of data once it has been prepared for treatment (possibly by a data scientist).

Every data analyst needs to have a deep knowledge of the various techniques needed to clean data, so analysis produces the best possible results, calling for high attention to detail.

When a project is complete, they then need similar interpersonal skills as a data scientist in order to communicate and present work to colleagues, particularly the ability to create visual or graphical representations of findings.

Formats and frameworks data analysts need to know include statistical programming languages like R or Python, data management, machine learning/AI, and systems like SQL.

Data science vs data analysis: Training pathways and career prospects

It's easy for outsiders to confuse the two professions because both are concerned with properly preparing and interrogating data for organisational insight. But there are quite stark differences.

Firstly, the data scientist is a more senior position, with more generalised responsibility in applying programming knowledge to create data models used for multiple purposes. A data analyst, by contrast, designs examinations of the data according to the established aims of other business units.

A career in data science

Becoming a data scientist will usually start with a degree in a computing, statistical, or scientific field, which will prepare you for the generalised knowledge of data science.

Depending on your area of focus, industry or the employer(s) you want to target, data analysis certifications like the CAP or vendor-specific study (Cloudera Data Platform Generalist Certification, Microsoft Azure Data Scientist Associate, SAS Certified Big Data Professional, etc) can be a good next step to specialise further.

According to Payscale.com, the salary range of a data analyst in the UK in 2023 is around £22K-£44k, with an average of £29,699. For data scientists, the range is £28k-£62k, with an average salary of £41,483, although seniority or further specialisation can put you over the £100k mark.

A career in data analysis

To many, the best way into data analysis is to get an entry-level job in the  field, giving you invaluable exposure to datasets and the tools you'll apply to them from real-world business processes.

You don't strictly need qualifications in maths or statistics to find work as a data analyst, but it certainly won't hurt to have them. Once you have a foot in the door, further training to boost your knowledge is common, and that can only expand your career advancement possibilities.

That might mean anything from the entry-level Associate Certified Analytics Professional (ACAP) to the Certified Analytics Professional (CAP), a vendor and technology-agnostic qualification that's more specialised and suits data analysts further into their careers.

A record number of people will begin computer science degrees this year, with more women taking computing than any other university subject group.

Analysis of this year’s exam results in the UK found that the number of students placed on undergraduate degrees in computing across the country has increased by 7% when compared to last year, with just over 24,900 students due to start courses – the largest percentage increase of any subject.

The figures, compiled by BCS, The Chartered Institute for IT, also revealed that computer science degrees have seen a 20% increase in accepted applications from women since 2019, which is also a higher increase than any other subject group listed by UCAS.

When it comes to A level entries, Computing has also shot up 41% across England, Northern Ireland and Wales in the last three years – the highest increase of any subject, BCS said.

“From autonomous vehicles, to media and communications, to banking and health care, we are in a digital society and computing skills are amongst the highest in demand,” commented Julia Adamson, Director of Education at BCS.

“So, it’s exciting that more young people have opted for advanced studies in computing at university - and particularly important to report a continued growth in the number of young women represented and achieving the highest grades.”

However, BCS has cautioned that the overall number of women that had accepted applications for computing, around 4,830, remains low when compared to men (20,080). This means that male students continue to outnumber women by 4.2:1 this year – only a slight improvement from 4.7:1 in 2019.

Despite this, female students continue to outperform their male counterparts at A level, with 38.8% achieving an A* or A, compared with 34.4% of men. That’s a considerable rise on both counts, with the comparative figures from 2019 showing that 18.7% of women and 17.4% of men hit those same grades that year.

Charlene Hunter MBE, CEO of Coding Black Females and member of the BCS board, commended the figures but warned against any potential complacency.

“It’s great to see young women choosing to take computer science in record numbers, but we can’t be complacent at all,” she said.

“In a world where big challenges like climate change and cyber security require highly skilled technologists, we need ever greater numbers of people from a diverse range of backgrounds to see computer science as an ethical, aspirational career choice.”

Data science platform provider Anaconda has announced the acquisition of PythonAnywhere, a cloud-based Python development and hosting environment.

PythonAnywhere’s platform is designed to remove the burden of infrastructure management and helps Python developers to create web applications in the cloud for collaboration and sharing.

Anaconda said the added expertise will allow it to better support its community of over 30 million people, as well as ensure that Python developers have access to a cloud-based environment that offers notebooks, tools, and a simple way to collaborate with their team.

In an announcement, Anaconda CEO and co-founder Peter Wang noted that non-programmers are the fastest growing group of Python users today.

“For Python to maintain this growth and remain the most widely used data science programming language in the world, it’s imperative to increase accessibility and remove barriers to collaboration,” he said. “With the PythonAnywhere acquisition, Anaconda will extend its services to all Python developers while building on capabilities for data scientists, engineers, data science enthusiasts, and students.

“This is an exciting milestone for our company, and we’re thrilled about the opportunity to strengthen the breadth of our Python offerings.”

Founded in 2012, UK-based PythonAnywhere has since accumulated 400,000 users across 100 countries and hosts over 50,000 websites.

The firm’s acquisition follows hot on the heels of Anaconda’s release of PyScript, an open source framework running Python applications within the HTML environment. The provider said that both the PythonAnywhere acquisition and the development of PyScript are central to its focus on “democratising Python and data science.”

“Anaconda’s commitment to advancing and supporting the Python community aligns with what has made PythonAnywhere successful,” said Giles Thomas, founder of PythonAnywhere. “We’ve experienced incredible growth as a company, creating a tight-knit community by simplifying a complicated process.

“By joining the Anaconda team, we can merge our cloud execution platform with Anaconda's expertise and experience in numerical computing—we’re looking forward to the opportunities and innovations we’ll accomplish together."

Global interviewing giant Karat has announced a strategic investment from tennis icon Serena Williams to significantly scale its Brilliant Black Minds program, an initiative designed to improve access and inclusion across the tech industry.

The program is now open to all current and aspiring Black software engineers, with Serena serving as Karat’s “Champion of Brilliance”. In doing so, she is supporting the firm’s call on the industry to add more than 100, 000 new Black engineers to tech over the next decade.

Brilliant Black Minds was launched in response to the multiple barriers to entry jobs that Black software engineers face in the tech industry, aiming to close the Interview Access Gap and tackle this disproportionate issue.

A microcosm of a wider societal problem, these issues range from structural inequities that delay early exposure to computer science to the limited information about how the industry hires, fewer connections in their professional networks, as well as the smaller number of opportunities to practice technical interviews.

In fact, in research conducted by Karat and Howard University, only 50% of the Black engineers surveyed said they had experienced a technical interview before they look for a job. However, the same data shows that confidence levels increase with more practice – with 79% of respondents with three practice interviews saying they were more likely to succeed.

With this investment from Serena Williams, the program is now open to all current and aspiring Black software engineers in the US who are looking to make progress on their technical interviewing skills.

As Champion of Brilliance, the tennis superstar will teach the importance of practice and building a championship mindset to help participants land secure their dream job in tech.

“The technology industry is focused on solving some of the world’s biggest challenges. My focus is ensuring the solutions to those challenges are developed by all of us,” explained Serena Williams. “There has never been a shortage of brilliance in Black America; only limits to the access and opportunities extended to our community.

“That is why I am proud to team up with companies like Karat who are taking actionable steps to bring more diversity and equity to the industry, as well as call on others to be part of the change.”

Brilliant Black Minds leverages Karat’s Interviewing Cloud platform to deliver free interview practice, feedback and coaching to help aspiring engineers prepare to successfully enter the tech industry. It’s a human-plus-tech solution that offers predictive and fair live technical interviews conducted by a global network of interview engineers.

Karat says the expertise and data it has generated from hundreds of thousands of interviews have enabled it to create more candidate-centric experiences – particularly for engineers who have been previously overlooked by traditional hiring practices.

“The tech industry is the biggest opportunity generator of our lifetime. And yet, those opportunities have not been distributed equally. People who look like me have been mostly kept out of the highest levels of tech,” commented Anthony Mays, Senior Advisor to the Brilliant Black Minds program and former software engineer at Google.

“With Brilliant Black Minds, Black software engineers have a safe and collaborative space to learn how to crack the code on how the industry hires. All it takes is just one opportunity to change a person’s life, and the lives of those who come after.”

IBM researchers have unveiled a novel method of training machine learning (ML) models that minimises the amount of personal data required and preserves high levels of accuracy.

The research is thought to be a boon to businesses that need to stay compliant with data protection and data privacy laws such as the General Data Protection Regulation (GDPR) and the California Privacy Rights Act (CPRA).

In both GDPR and CPRA, 'data minimisation' is a core component of the legislation but it's been difficult for companies to determine what the minimal amount of personal data should be when training ML models.

It's especially difficult when the goal of training ML models is usually to achieve the highest degree of accuracy in predictions or classifications, regardless of the amount of data used.

The findings from the study, thought to be a world-first development in the field of machine learning, showed that fewer data could be used in training datasets by undergoing a process of generalisation while preserving the same level of accuracy compared to larger ones.

At no point did researchers see a drop in prediction accuracy below 33% even when the entire dataset was generalised, preserving none of the original data. In some cases, the researchers were able to achieve 100% accuracy even with some generalisation.

In addition to adhering to the data minimisation principle of major data protection laws, researchers suggest that smaller data requirements could also lead to reduced costs in areas like data storage and management fees.

Data generalisation process

Businesses can become more compliant with data laws by removing or generalising some of the input features of runtime data, IBM researchers showed.

Generalisation involves taking a feature value and breaking it down into specific values and generalised values. For a numerical feature 'age', the specific values of which could be 37 or 39, a possible generalised value range could be 36-40.

A categorical feature of 'marital status' could have the specific values 'married, 'never married', and 'divorced'. A generalisation of these could be 'never married' and 'divorced' which eliminates one value, decreasing specificity, but still provides a degree of accuracy as 'divorced' implies that an individual has, at one point, been married.

The numerical features are less specific, adding three additional values, while the categorical feature is less detailed. The quality of these generalisations is then analysed using a metric. IBM chose to use the NCP metric over others in consideration as it lent itself best to the purposes of data privacy.

Researchers then selected a dataset and trained one or more target models on it to create a baseline. Generalisation was then applied, the accuracy was calculated and re-calculated (see diagram above) until the final generalisation was ready to be compared to the baseline.

The accuracy of the target model is calculated using decision trees (see above) which are gradually trimmed from the bottom upwards, taking note of any significant decreases in accuracy.

If accuracy is maintained or meets the acceptable threshold after generalised data is applied, the researchers then work to improve the generalisation by gradually trimming the decision tree from the bottom upwards, increasing the generalised range of a given feature, until the final optimised generalisation is made.

The Senate Armed Services Committee is reviewing a bill that would require the military to measure its use of artificial intelligence (AI).

The Department of Defense Artificial Intelligence Metrics Act, introduced on September 30 by Senator Mike Rounds (R-S.D.), would force the military to introduce key performance indicators measuring how effectively it used AI in operations.

At the time of this writing, the Congress.gov website had not received the text for bill S.2094, but the filing describes it as "A bill to require the Secretary of Defense to establish performance objectives and accompanying metrics for the incorporation of artificial intelligence and digital readiness into Department of Defense platforms, processes, and operations, and for other purposes."

NextGov, which has seen the five-page bill, says it would require government officials to identify potential technology applications for AI and create metrics and performance objectives to evaluate their deployment. Part of the evaluation includes identifying skills gaps in AI-related areas, including software development and data science. Internal DoD departments would then need to create recruiting drives to help build up capable talent.

Rounds reportedly said the bill would help meet the National Security Commission on AI’s (NSCAI) recommendations. The Commission released a report in March warning the US was unprepared to defend itself against AI threats. It said China was a primary threat to the US due to its advanced AI research and prolific AI deployment. It recommended $40 billion in federal government investments in AI technology.

The DoD has made some moves to integrate AI into its operations. In June 2018, it created a Joint Artificial Intelligence Center (JAIC) under the DoD CIO's office to coordinate AI efforts agency-wide.

The DoD launched an AI and data accelerator initiative to create proof-of-concept systems for data-driven warfare this June. The initiative tested algorithms in war scenarios to identify potential adoption hurdles, officials said.

DataCamp has begun to offer free data literacy courses to all Degreed users.

DataCamp and Degreed develop comprehensive learning programs tailored to enterprises’ needs. Extending their partnership, DataCamp is offering free access to three interactive online courses: Data Science for Everyone, Machine Learning for Everyone, and Data Engineering for Everyone.

The courses require no prior coding experience, making them an ideal choice for organizations undergoing or planning an enterprise-wide digital transformation. Essentially, the courses are suitable for all skill levels and roles.

Degreed clients with Edit Settings permission can access DataCamp’s free courses through the Provider's tab or by contacting their Client Experience partner. There’s also the option to upgrade for full DataCamp access.

The “Data Science for Everyone” course includes 15 videos and 48 exercises on data science, data collection and storage, time series analysis, and A/B testing.

“Machine Learning for Everyone” lets users discover intelligent technologies behind self-driving cars and always-on customer experience tools.

Lastly, “Data Engineering for Everyone” explains data engineers’ core responsibilities via Spotflix, a fictional music-streaming firm. Each course takes two hours to complete.

DataCamp CEO and co-founder Jonathan Cornelissen said, "There is incredible power in data—but only if you know what to do with it. All departments and roles must have the skills to analyze that data to extract meaningful insights. Through our renewed partnership with Degreed, we are proud to provide their clients with free access to the powerful data skills they need to make better decisions, better serve their customers, and drive business growth."

"Data is core to every organization and that means everyone must have a baseline understanding of data. We're excited to offer all Degreed clients the DataCamp training, to empower their people ready for the data-driven future and ensure everyone knows how to use data effectively in their work," commented Rob Wellington, director of experience partnerships at Degreed.

A new design of the £50 note featuring famed British scientist Alan Turing has been revealed by the Bank of England (BoE).

The note will be in circulation from 23 June 2021, which coincides with what would have been Turing's 109th birthday.

Turing is widely regarded as one of the world's most influential scientists, known for his codebreaking work at Bletchley Park during World War Two. His Enigma machine would ultimately prove successful in cracking the codes used by the German military, which historians believe shortened the war and saved countless lives as a result.

He was also a leading mathematician, developmental biologist, and pioneer in the field of computer science.

However, Turing's life is also marked by tragedy, as he was later convicted of gross indecency in 1952 after he was discovered having an affair with a 19-year-old man. The resulting persecution, and the forced use of hormonal treatment as an alternative to prison, eventually led to Turing taking his own life in 1954.

By placing him on its new polymer £50 banknote, the BoE said it was celebrating his achievements and "the values he symbolises".

"There's something of the character of a nation in its money, and we are right to consider and celebrate the people on our banknotes. So I'm delighted that our new £50 features one of Britain's most important scientists, Alan Turing," said Andrew Bailey, the governor of the BoE.

Fittingly, the polymer note contains 'advanced' security technology in the form of two-colour foil, which the BoE suggests makes it difficult to counterfeit. There is also a holographic image that changes between the words "fifty" and "pounds" when it is tilted from side to side.

The director of GCHQ, Jeremy Fleming, called the note a "landmark moment" in the UK's history as it also confirms Turing's status as one of the most iconic LGBT+ figures in the world.

"Turing was embraced for his brilliance and persecuted for being gay," Fleming said. "His legacy is a reminder of the value of embracing all aspects of diversity, but also the work we still need to do to become truly inclusive."

The £50 note will also include nods to different stages of Turing's life, including a photo taken of Turing during 1951, his signature taken from a visitor's book from Max Newman's House, and ticker tape displaying Turing's birth date (23 June, 1912) in binary.

Also included is a quote given to the Times Newspaper during an interview in 1949: "This is only a foretaste of what is to come, and only the shadow of what is going to be."

A team of researchers led by Microsoft have withdrawn a research paper into quantum computing that was published in 2018.

The scientists had published a paper in the journal Nature claiming to have found evidence of a subatomic particle that could help develop quantum computers. The paper focuses on creating qubits, a unit of information having both the values of one or zero, using the properties of Majorana particles. These were named after the Italian physicist Ettore Majorana in the 1930s.

In the paper’s retraction, it notes that there were “several inconsistencies... between the raw measurement data that was made available to them and the figures that were published in the paper.”

The scientists established that the data in two of the figures had been “unnecessarily corrected for charge jumps (corrections that were not mentioned explicitly in the paper)” and also that “one of the figure axes was mislabelled.”

The retraction goes on to say that the data was replotted over the full parameter range and concludes with this statement: “We can therefore no longer claim the observation of a quantized Majorana conductance, and wish to retract this Letter.”

Zulfi Alam, vice president at Microsoft, stated in a post on LinkedIn: “We remain confident in our topological approach to scaled quantum computing, as we nurture a culture of continuous improvement to innovate across every component of the Quantum Machine.”

Quantum computing, once it has been developed fully, will purportedly provide a range of benefits for businesses including improved security, better logistics, better fraud detection and more.

Microsoft isn’t the only major tech company foraging into this emerging technology, IBM hit a quantum computing milestone last August and Amazon’s AWS launched a quantum cloud service too.

Microsoft has announced a new platform designed to make it easy to build and operate artificial intelligence-powered technology for use in low-power edge devices, such as cameras and audio equipment.

The Azure Percept Development Kit (DK), which is available in public preview from today, promises to provide a single, end-to-end system that enables customers without coding knowledge to develop an AI product from the ground up.

The hope is that this new platform will help create a Microsoft-powered ecosystem of edge devices designed for low-power implementations, in essence replicating its success with the Windows operating system in the PC market.

The platform, announced at Microsoft Ignite, will run alongside Azure Percept Vision and Azure Percept Audio, two bolt-on services that can connect to Azure cloud services such as Azure AI, Azure Machine Learning, Azure Live Video Analytics, and Microsoft’s various IoT services.

Early concepts suggest the platform is initially aimed at use-cases involving retail and warehousing, where customers can take advantage of services like object detection, shelf analytics, anomaly detection and keyword spotting, among others.

Microsoft explained that the DK “significantly” lowers the bar for what is required to build edge technology, particularly as most implementations require some degree of engineering and data science expertise to make them a success.

“With Azure Percept, we broke that barrier,” said Moe Tanabian, Microsoft vice president and general manager of the Azure edge and devices group. “For many use cases, we significantly lowered the technical bar needed to develop edge AI-based solutions, and citizen developers can build these without needing deep embedded engineering or data science skills.”

Customers signing up to the platform will also be provided with a range of edge-enabled hardware that allows for processes like speech and image recognition to take place without requiring a connection to the cloud. Initially, this will be built by Microsoft, however, the company also confirmed that third-party manufacturers will be able to build equipment that’s certified to run on the Azure Percept platform.

“We’ve started with the two most common AI workloads, vision and voice, sight and sound, and we’ve given out that blueprint so that manufacturers can take the basics of what we’ve started,” said Roanne Sones, corporate vice president of Microsoft’s edge and platform group. “But they can envision it in any kind of responsible form factor to cover a pattern of the world.”

Microsoft’s own hardware also uses the industry-standard 80/20 T-slot framing architecture, which it claims will make it easier for customers to run pilots of their ideas with existing edge housing and infrastructure.

Elevators that are able to respond to custom voice commands, cameras that notify managers when shelves have low stock, and video streams that monitor for availability in car parks are just a few examples of how the technology could be deployed, Microsoft explained.

Azure Percept Studio, another bolt-on service, will provide step by step guides taking customers through the entire lifecycle of an edge tool, from design to implementation. Perhaps most importantly, customers using Percept Studio will also have access to AI models created by the open source community.

Artificial intelligence (AI) should be treated with the same scrutiny as medication and warn users about potential “undesirable side effects”, according to an AI ethics expert.

Maria Luciana Axente, PwC UK’s Responsible AI and AI for Good lead, made the statement during a discussion on the second day of the newly-launched AI Festival, adding that data scientists should respond to an ethical code just like medical practitioners.

When quizzed by BT's head of AI and Data Science Research, Detlef Nauck, about whether she thinks AI systems should be regulated like medicines that come “with a description of undesirable side effects”, Axente replied: “Of course, no doubt about it.”

"We have to move in that direction,” she said. “Because we started using AI in so many different domains of life that have a significant impact on people's lives.”

However, Axente added that, although she has heard arguments for creating an agency that would regulate the use of AI in a similar way to how the US Food and Drug Administration aims to protect public health, she believes that it might be worth focusing on existing institutions.

“Let's see what we can be doing on this part of the pond, and how can we leverage some of the institutions we already have, rather than creating new ones – but they will have to go in that direction, especially for high-risk application [of AI],” she said.

She referenced the European Commission’s plans to revise EU laws on the use of AI, with new regulations “set to come in April”.

“I think [they] will allow us to at least separate the use cases that need more attention and more governance from the ones that we can be a little bit more relaxed,” she said.

Axente’s comments come as Facebook’s head of hardware confirmed that the tech giant is considering using facial recognition technology for its upcoming smart glasses, which are set to be released later this year.

Facial recognition has been the subject of a petition launched last week by a coalition of privacy advocates, who are pressuring EU regulators to take advantage of its upcoming revision of AI laws and ban the use of biometric mass surveillance tools.

Investing in fleets of data scientists and expecting them to do all the heavy lifting in artificial intelligence (AI) is one of the most common mistakes businesses make, according to former IBM executive Phil Westcott.

Speaking at the virtual AI Festival, Westcott, who formerly lead the IBM Watson platform in Europe and is now co-CEO of AI consultancy firm Filament, told attendees that he had witnessed “a lot of overspending and wasted resources” during his career.

“A couple of the mistakes I've seen is to go out and buy a huge fleet of data scientists and pop them in in a large room and expect that the business outcome will flow magically,” he said.

Another “cautionary tale”, according to Westcott, is “going too far on the operational steps before you've proved the hypothesis within the state of the art of applied AI technology”.

Instead of wasting time and resources on these mistakes, businesses should invest in creating “more agile, cross-functional teams with the engineering skills and business analysts”, he said.

Businesses looking to build their AI capability on a budget must follow five key steps, which include setting a strategy, building AI assets, developing a culture of agility and collaboration, investing in skilled team members, and partnering with academic and industrial organisations.

“In your key strategic areas, where you want to make your own AI assets and build your capability, that's where you will need the help of consultants (...) to bring the benefits of the full ecosystem and bring their experience from other industries and other clients,” he said, before recommending potential partners such as the Google Cloud, IBM Watson, and the Alan Turing Institute.

Companies seeking to develop their businesses' AI capabilities will also benefit from turning their areas of expertise into a competitive advantage and focusing their AI programme on departments such as customer service, research, or HR recruiting.

“Because if they've already got a differentiated human capability there, they have an opportunity to turn that into a differentiating AI capability by effectively bottling that expertise into machine learning models that can be scaled, and can further augment that human capability,” he added.

Westcott was speaking at a keynote address during the first day of the newly-launched AI Festival. The virtual event, which is backed by BT, aims to explore the influence of AI technology on businesses, skills, and employment, with today’s keynotes focusing on issues such as the future of AI skills, ethical AI, and digital-based approaches to health.

Artificial Intelligence (AI) has become a key component of companies’ bids to modernise in recent years.

Organisations have been scrambling to get an advantage over their competitors and see how greater volumes of data insights can give them an edge.

However, introducing AI without having a full understanding of where it will fit in your strategy can mean your investment won’t generate a return. But one thing’s for certain, that once it’s implemented, you need to run with it, as Gartner predicts that by 2024, 75% of organisations will initiate some form of machine learning (ML).

However, data scientists shouldn’t be extracting deep insights from company data unless they’re aligned to a business goal or solving a business problem. This can then indicate how AI and machine learning can be attributed to success.

Sivan Metzger, managing director, MLOps & Governance at DataRobot, has said that we’re now at a point where businesses will finally start to see the benefit of their AI investments, as pressure from CFOs continues to mount.

He goes on to say that data scientists should also begin to see results that can help them to adapt their work to give their company a better advantage.

Previously, this had been an issue, and Metzger credits these issues to a disconnect between the data team, IT operations and stakeholders on the business side (i.e. the potential consumers of data science insights). Data science and IT operations teams have very different considerations and goals – and machine learning is very different from running software. This disconnect is known as the ‘production gap’, and can prevent AI solutions from being properly executed.

Machine Learning Operations (MLOps) is a combination of processes, best practices and underpinning technologies which seeks to bridge this gap by increasing collaboration and communication between data scientists and operations staff – and ultimately ensuring that AI is properly deployed and can begin to deliver the ROI promised.

To learn more about how MLOps can improve your returns on AI, watch IT Pro and DataRobot’s webinar ‘The Last Mile to AI ROI’, in which Metzger and data scientist Rajiv Shah discuss topics including:

• How to eliminate AI-related risks by adopting MLOps best practices
• The inherent challenges of production model deployment and how to overcome them
• Model-monitoring best practices
• Production lifecycle management and why it matters

Looking for the best data science bootcamps out there can be a tricky process, especially when your future and career prospects depend on it. 

Across the global workforce, many are opting to switch jobs and pursue a career in the burgeoning tech industry, and much of the current wave of interest can be attributed to the generative AI boom.

Recent research from BCS, the Chartered Institute for IT, showed AI interest is fueling a surge in applications for digital apprenticeships across the UK, for example.

There are concerns that some jobs will disappear in the age of AI, however. But while some traditional data science roles will inevitably disappear in years to come, new ones will likely emerge and open up opportunities.

Going forward, data scientists who are AI savvy will be among the most in-demand workers across a number of sectors, from finance to healthcare and manufacturing.

So, if you’re looking to switch careers and become a data scientist, we’ve picked out what we think are some of the best data science bootcamps in both the US and UK to give you a sense of what to expect.

Our list of the best data science bootcamps has been populated based on a combination of factors, including community reviews, the convenience of each course, and whether they cater for absolute beginners or a current professional looking to up-skill or retrain. Our goal is to provide a broad range of options to give you a good sense the market today.

Best data science bootcamps in the US

WeCloudData

Price: $8,300 to $12,500

Course link: WeCloudData

Kicking off our list of the best data science bootcamps is a course offered by data and AI training academy WeCloudData, which hosts courses in Toronto, Vancouver, and both the US east and west coast. 

The package includes 14 weeks of study, followed by six months of project experience and building alongside six months of mentorship.

Lessons cover the basics of SQL, where students will learn to write SQL queries and use SQL and Python to solve data problems. This will equip them with tools needed to “be ready for the new economy”.

Courses typically run from September to January, and January to May. These currently cost between $8,300 and $12,500.

WeCloudData stands out for a couple of reasons. Not only does it have a very high candidate success rate (95%), putting more than 2,000 students into roles, it also operates a Women in Tech scholarship. 

This is aimed at boosting gender diversity and addressing underrepresentation; it can provide financial assistance to those recently laid off and looking to continue their professional development in the tech sector.

Flatiron School

Price: $16,900 – with a $99 deposit

Course link: Flatiron data science bootcamp

Students on this course offered by New York-based educational organisation Flatiron School will start by turning messy, real-world datasets into analytical insights and interactive data visualizations using Python and SQL.

The next phase will involve using scientific computing tools, such as NumPy and SciPy, to produce high-quality data reports worthy of being published. The learning, which covers AI and machine learning, culminates in a final project.

Those who decide to study full-time will benefit from group work with the help of course instructors as well as lab time. Those studying part-time can access pre-recorded lessons, joining live lectures remotely whenever possible, and will be added to a Slack group where they can converse with classmates and course instructors.

Courses usually start at the end of October, and cost $16,900 with a $99 deposit. There are also scholarships available.

A reason to consider Flatiron School: Awarded best coding bootcamp for 2023 by Course Report, where it has an average rating of 4.5/5.

All students will also receive one-on-one career coaching for 180 days following graduation. Past graduates have gone on to be hired by the likes of Facebook, IBM, Shopify and Spotify.

LunarTech

Price: Between $79.97 and $149.97 per month

Course link: LunarTech data science bootcamp

Launched in August, this online bootcamp from Austin, Texas-based LunarTech wants to make AI and machine learning accessible to everyone regardless of their technical expertise. Lessons covered are designed to help students grasp the fundamentals of statistics and machine learning and help them master A/B testing.

“We aspire to arm our students with the knowledge and skills needed to ace even the most demanding tech interviews, including those at FAANG [Facebook, Apple, Amazon, Netflix, Google] companies,” according to LunarTech’s LinkedIn.

The bootcamp costs between $79.97 and $149.97 per month – the top package offers students priority email support as well as free one-on-one career coaching sessions.

Those interested but unsure of whether to apply can book a 15-minute chat with one of the company’s advisors.

Springboard

Price: $9,900 upfront or $1,890 per month

Course link: Springboard data science bootcamp

Online learning platform Springboard’s program is self-paced, but it’s recommended that students complete it in six months. This requires devoting 15 to 20 hours per week, learning from more than 500 hours of articles, videos, hands-on projects and coursework.

“Our flexible, human-guided curriculum featuring advanced specialisation means you can learn when you want with support as you need it,” states the Springboard website.

Springboard was awarded best online bootcamp for 2023 by Course Report, where it has an average rating of 4.6/5.

Another bonus is that although the course costs $9,900 upfront or $1,890 per month, if you fail to be offered a data science job within six months of graduating, then you’ll get your money back.

Best data science bootcamps in the UK

BrainStation

Price: £875 per month for 12 months / £10,500 one-off payment

Course link: BrainStation data science bootcamp

This London-based course from BrainStation kickstarts students’ data science journey by helping them hone critical data skills and learning to work with database management software MySQL.

Another unit covers AI and sees students explore a range of tools that can be used in building machine learning models. They also gain hands-on experience applying machine learning concepts.

Based in Shoreditch’s tech campus, students are in and around tech professionals and have the opportunity to collaborate with designers, web developers, software engineers and digital marketers. Students get to apply what they’ve learnt to a real-world business challenge posed to them by industry partners including Adidas, Google and Mastercard.

Courses typically run November to February, and January to April in both Shoreditch and at sites in New York, Miami, Toronto, and Vancouver.

The course has a healthy rating of 4.7/5 on Course Report and 4.8/5 on SwitchUp, which reviews and ranks bootcamps.

Cambridge Spark

Price: Free (subsidised by the UK government) 

Course link: Cambridge Spark data skills bootcamps

Backed by the Department of Education and local authorities, Cambridge Spark’s crash courses are ideal for anyone who can’t afford to spend hundreds or thousands of pounds on a bootcamp.

“These government organisations have a vested interest in upskilling residents to address the local data skills gap and bolster economic development in their communities,” states the Cambridge Spark website.

There are currently two 10-week courses available to enrol on: essential data skills and data analytics. The latter provides the opportunity to learn how to process data using programming languages like Python and Pandas.

Not only does the course open doors for people otherwise unable to access tuition, Cambridge Spark also holds an average rating of 4.7/5 on SwitchUp.

If there was an upside to the coronavirus pandemic it was that cloud computing had the chance to shine. Whether it was the mass adoption of video conferencing services such as Zoom or more dependence on automated software, the cloud played a big part in 2020 and, as such, was one of the few sectors that continued hiring.

If you already have IT experience and technical skills - or a desire to get some - a career in cloud computing can be very rewarding. What's more, it's an industry that is constantly evolving making it almost a future-proof career move.

The industry has a diverse range of job titles, including some very niche and experimental fields, and can accommodate all manner of skill sets. New recruits can work on internet connections, servers and storage, analytics, software and much more. It really is a case of using one's imagination, digital transformation isn't necessarily a one-time affair and the use case for cloud expand and grow constantly.

Global spend on cloud tech could reach around $150 billion over the next three years, according to Gartner, just to give you some insight into the potential for job creation within the sector.

Required skills for cloud computing jobs

There are a number of skill sets that can lead to a job within cloud computing. The more experience you have, the more likely you are to land a role, but just like in certain IT roles, lots of the learning can be done online. Developing expertise and or working knowledge in these areas can increase your chances:

• Agile
• Amazon Web Services (AWS)
• Ansible
• Azure
• Chef
• Docker
• Java
• Puppet
• Python
• VMware

14 cloud computing jobs

1. Back-End Developer

Median salary: $101,619

As a back-end web developer, you’ll be responsible for server-side web application logic and integrating front-end web developers’ work.

You’ll use programming languages to create the back-end code that makes the applications run as intended. You’ll also write web services and APIs that front-end developers and mobile application developers will use in their roles.

2. Cloud Engineer

Median salary: $121,992

A cloud engineer is responsible for technological duties related to cloud computing, including design, planning, management, maintenance and support. They also assess a business’ existing infrastructure and look for solutions to move different cloud functions.

3. Data Engineer

Median salary: $102,864

A data engineer is a type of software engineer. They are responsible for installing and compiling database systems, writing complex queries, scaling to multiple machines and implementing disaster-recovery systems. They also look for trends in data sets and create algorithms to help the enterprise use the raw data.

4. Data Scientist

Median salary: $113,309

As a data scientist, you’re responsible for extracting meaning from and interpreting data. This involves using statistics and machine learning tools and methods. You would also collect and clean data.

5. Development Operations Engineer (DevOps)

Median salary: $99,604

DevOps engineers oversee code releases and typically work with developers and IT staff. They typically migrate from being developers who want to get involved in deployment and network operations or sysadmins who are interested in scripting and coding.

Working in DevOps enables them to work on planning tests and deployment.

6. Front-End Developer

Median salary: $76,929

As a front-end developer, you would work on packaging the back end’s utility so users can interact with it effectively. Front-end developers convert website design files into HTML, JavaScript (JS) or CSS code, which are the front-end building blocks.

7. Full-Stack Developer

Median salary: $105,813

Full-stack developers are essentially engineers who work with databases, servers, systems engineering and clients. They work with the front and back ends of a website or application. They help clients during the planning phase of projects and develop web stacks, mobile stacks or native application stacks.

8. Java Developer

Median salary: $79,137

Java developers design, develop and manage Java-based applications. Java has many applications in different organizations, so a Java developer can work on a wide range of projects. You could work on a specific application or several applications at the same time.

9. .NET Developer

Median salary: $68,524

Working as a .NET developer typically involves designing and developing software applications that would meet a specific organization’s needs. They provide support and continuous development, as well as determine and analyze the software’s prerequisites.

10. Software Architect

Median salary: $113,757

Software architects function like a software developer. They work with software developers to design and execute solutions for businesses and clients. Software architects are also responsible for making executive software design decisions and wear several hats, such as communicator, designer and developer.

11. Software Engineer

Median salary: $92,046

A software engineer focuses on developing computer software and systems. They employ engineering principles and programming languages to operate network control systems, create software products and develop computer games.

Software engineers also work with clients to analyze their needs and design, test and develop software to meet those needs.

12. System Engineer

Median salary: $77,768

System engineers are responsible for monitoring and managing the infrastructure and installed systems. They also install, configure, test and maintain operating systems, application software, and system management tools to ensure systems are properly maintained and infrastructure is available at all times.

13. Systems Administrator

Median salary: $64,892

A systems administrator manages the network’s day-to-day operations. They’re responsible for organizing, installing and supporting all the computer systems, including local area networks (LANs), wide area networks (WANs), network segments, intranets and data communication systems.

14. Other roles

Organizations will also employ people who work directly with the cloud. Some people might work directly with cloud applications and build infrastructure for cloud providers, including:

• Computer network architects
• Computer systems analysts

Other positions might be responsible for working with the cloud in different ways, such as executing tasks or deploying critical applications. Some of these roles include:

• Computer programmers
• Information security analysts
• Software developers
• Network and computer systems administrators

Where to find cloud computing jobs

If you already work in IT and want to shift into a cloud computing job, you could leverage your professional or personal network to get a job in this field.

Your current workplace might need cloud computing professionals. With more companies moving their data and services to the cloud, you might be able to make a move through someone in your network.

There are many places to find cloud computing jobs, including large job boards like Indeed, Glassdoor, and LinkedIn. There are also IT- and cloud-specifc job boards worth checking out:

• Cloudy Jobs caters specifically to the cloud computing industry. This cloud computing job platform provides job opportunities for professionals in this niche.
• Code4Lib lists cloud computing jobs within the cultural heritage sector (e.g., libraries, archives, museums).
• Dice provides a searchable database of tech and cloud computing jobs, as well as information on salaries, skills, and career resources.
• Cloud Academy provides links to job boards and resources that cater to professionals in this niche.

Online training resources:

Due to the rapid adoption of cloud computing, a number of the biggest providers have made various education resources available for budding cloud enthusiasts. While not all will guarantee qualifications or job placements, they can boost technical knowledge and discovery. Which is always good to have.

Amazon offers a range of free courses for certifications in areas like machine learning and database management. Microsoft offers free entry level lessons for its Azure cloud and Google also offers free basic courses for its cloud computing software.

Given the sheer variety of fields they are involved in, data scientists are always in high demand. If you're looking for something with variety and plenty of opportunities, a career in data science is a solid option.

The continually increasing amount of data being generated offers new business opportunities. Emerging data sources could offer market insights and aid in the development of new tools. However, these insights are dependent upon the data being appropriately collected and analyzed.

This is where data scientists excel – they are able to effectively curate and study the data, before presenting their conclusions in an actionable format for executives who do not have a technical background.

Data scientists were once based in or around central hubs of the technology industry, such as Silicon Valley. However, there has been a gradual shift away from this due to the increased accessibility of data analysis tools and the popularity of remote working following the Covid pandemic. The US remains the country with the most data scientists, due to a large data science infrastructure, and has the highest number of job openings for data scientists. Other countries with a high demand for data scientists include Canada, India, and Germany.

What does a data scientist do?

A data scientist is someone who collects, analyzes and interprets data to enable the decision-making of an organization. They may utilize advanced analytics techniques, such as machine learning and predictive modelling, along with the application of scientific principles.

Data scientists often work with large data sets to identify patterns, test hypotheses and make inferences.  They analyze data sets, such as customer and market trends, financial risks, cybersecurity threats, stock trades, equipment maintenance needs or medical conditions.

Data scientists are typically responsible for extracting the required raw information from data.  Once analyzed, it could then be used for predicting emergent behavior, identifying new market trends or for detecting fraudulent activities. They can offer valuable insight for healthcare providers, academic institutions and government agencies, amongst many others.

Quick facts about data science jobs

• The average US salary for data scientist in 2023 is approximately $144,050.
  
  
• The US Bureau of Labor Statistics expects 11.5 million new jobs for data scientists by 2026.
  
  
• Requires an undergraduate degree in data science or a related field. A Master's degree or PhD is the best way to stand out.
  
  
• 59% of data scientist jobs will be in finance, IT, insurance and professional services (IBM).
  
  
• Requires a strong understanding of statistics and coding languages like Python, SQL, and C.

What career opportunities are available to data scientists?

Businesses are always seeking new ways to derive insight and to provide actionable outcomes for increasing revenue and profitability. As such, the field of data science is expanding rapidly.

Someone who works in data science is not just a data scientist. It incorporates a wide variety of roles, including the following:

• Business intelligence specialists produce insights that guide decision-making.
  
• Data analysts explore complex data sets to identify new patterns.
  
• Data architects plan and manage big data databases.
  
• Data engineers prepare data for analytical or operational uses.
  
• Data scientists collect, analyze and interpret data to drive decision-making.

Despite the highly technical nature of the role, data scientists are often needed in a wide variety of industries, and not just in the technology sector. The core business sectors employing data scientists are healthcare and finance, but data scientists can be found in most fields.

Do I need a degree to be a data scientist?

A degree in a computer science or mathematics (with statistics) is the minimum requirement to become a data scientist, but a Master’s degree or higher is desirable.

Going through a master’s degree program in particular will provide a more complete understanding of data science, which will include predictive analytics, statistical modeling, data mining, data visualization, and enterprise analytics.

For those who have already started a career in data science, then professional certifications, such as Certified Analytics Professional and Open Certified Data Scientist, demonstrate a commitment to continuing professional development in this field.

What skills do I need to become a data scientist?

In order to become a data scientist, you will have to start by gaining the right skills through a mix of on-the-job experience and education. An effective data scientist also needs to have a balance of the technically-focused core skills and the soft skills of collaborating and communicating.

Data science core skills

• Computer science: You'll have to put a wide variety of principles into action related to database systems, software engineering, artificial intelligence, numerical analysis and human/computer interaction
• Programming: This involves developing computer programs and analyzing big sets of data to search for solutions to complicated problems. Get ready to write code in a number of programming languages (e.g. Java, Python, SQL, R)
• Statistical analysis: Fully analyze data to discover patterns and carry out pattern and anomaly detection
• Machine learning: using statistical models and algorithms, "teach" computers to learn from data
• Data storytelling: Utilise data to illustrate, explain and develop actionable insights for non-technical audiences

Data science soft skills

• Analytical thinking: Find analytical solutions for theoretical business issues
• Interpersonal skills: Work and collaborate with all types of people throughout the organization
• Critical thinking: Analyze facts in an objective manner to reach a conclusion
• Inquisitiveness: Discover hidden patterns and solutions within the data
• Business intuition: Work alongside stakeholders to fully understand their problems

How do I become a data scientist?

When first deciding that you want to be a data scientist, it's helpful to consider which sector you wish to work in. Having a specific sector in mind enables applicants to focus their attention on the most suitable job adverts, making it more likely that your application will stand out.

Networking within the tech industry, both online and in person, is always useful for gaining information on potential job openings. This could be through chatting with people on LinkedIn and data science forums, or meeting them in person at tech conferences. Ask them about their career journey and what they feel got them hired in the first place.

We've summarized some specific tips for applying for data science roles below. These assume you are already covering the basics of applying for a job, including creating an engaging and relevant cover letter, and matching your experiences and skills to the job advert.

Tips for applying for data scientist jobs

• Highlight any real-world experience that relates to the discipline. Applying your data science to skills to things you have done in the past demonstrates an understanding of the skills required
  
  
• Consider voluntary positions at non-profit or charity groups to help build your experience. Not only can these opportunities be a great way of building and testing your skills, they are also perfect for demonstrating your passion for the industry. You will also likely make useful contacts for your future career
  
  
• Another way to keep your skills fresh is to apply them to something you're passionate about. Studying data relating to your favorite sports team, for example, is a good way of acquiring experience working with real-world data
  
  
• Keep up to date with the latest developments from across the data science industry, as well as the wider technology landscape. Be prepared to chat about emerging topics at seminars and conferences, and use social media to discuss ideas with your peers
  
  
• Keep your job searches specific to your skills and desired industries. The sheer number and variety of roles being advertised across the industry and across different geographies can be overwhelming. Your best chance of securing a role is to target the jobs you feel match your experience and goals

The data sciences is a rapidly growing field, and careers in this area offer high wages and a solid future. It’s a great time to launch your data scientist career.

Do your homework to determine what types of jobs are available in your area, who’s hiring, and what the role requires. You can consider freelancing as a data scientist or look for jobs in other parts of the country. Research, analyze the data, and develop a plan, and you will soon be launching your career as a data scientist.

It's hard to believe that the deep, rich analytics that NFL fans enjoy as part of sports broadcasts are largely thanks to one man - and it's even more surprising to learn that man is an English superfan from Luton. In this Q&A, we'll talk to Neil Hornsby, founder of Pro Football Focus, about how his hobby of tracking statistics turned into a multi-million dollar business that now powers a huge amount of the NFL's data analysis efforts.

Justin Cupler

Managing Editor, ITPro.com

Justin is an 11-year veteran in the world of journalism in a wide range of verticals, including automotive, personal finance, landscaping, technology and more. He’s also spent years behind the microphone and in front of the camera. He’s appeared on local and national networks, including Cheddar, NBC, CBS and Fox. Cupler’s a professionally trained broadcaster from Connecticut School of Broadcasting and an ACES-Poynter-certified editor. His favorite pastime is spotting grammatical errors in national ads and billboards.

Neil Hornsby

Founder & CEO, Pro Football Focus

Neil Hornsby founded PFF in 2006 as nothing more than a hobby. Somehow it took on a life of its own and in 2014, with 13 NFL teams as customers and 8 full-time staff, the then UK based company was purchased by Cris Collinsworth and moved to the US. With Hornsby as Chief Operating Officer, PFF has grown to become the preeminent provider of data and analytics to all 32 NFL teams, 73 NCAA FBS teams as well as virtually every major media outlet covering football. It now has just under 100 full-time employees covering four countries and nine time zones.

No matter if you're a youngster looking to break into the industry or a seasoned working professional who's considering retraining, a tech degree is a worthwhile consideration for a wide range of reasons.

First, a computer science or IT degree is one of the best ways to get into one of the few industries that's largely unaffected by the COVID-19 outbreak and the financial crisis it created. Although there were fewer job postings last year, the impact wasn't nearly as severe as it was in other segments, like the hospitality industry.

Believe it or not, many areas have already seen tech job listings recover to pre-pandemic levels. What's more, cloud computing, information security, and big data jobs are predicted to increase by more than 530,000 over the next decade, according to the US Bureau of Labor Statistics.

Second, people working in the tech space often earn significantly more than others. In fact, the nationwide median tech salary was at least 218% higher than the average local salary. It’s worth noting that a post-graduation job isn’t always guaranteed, that’s why a university's job placement rate is so important when choosing a college. It's also a good use of your time to contact someone who recently graduated from the degree program you’re considering. This could give you some genuine advice based on their own experience.

While there are plenty of tech jobs that don’t necessarily require a degree, it can certainly help, especially if you earned it from a prestigious school such as Stanford or MIT. These stalwarts aren't the only tech colleges you should consider. There are plenty of other options with higher acceptance rates and cheaper tuition fees that may be just as beneficial to your skill set and future career.

We’ve performed a coast-to-coast search to find the best tech colleges in the US. You can find these great tech schools in our alphabetized listing of the best tech colleges below.

Carnegie Mellon University

Location: Pittsburgh, Pennsylvania

Related department or school: School of Computer Science

Overview: College ranking site Niche.com ranks CMU as its No. 1 college for information technology in the U.S. Its graduate school programs are especially renowned. U.S. News & World Report also ranks CMU as having the best computer science graduate school and the best IT management graduate program.

Degrees offered: CMU offers B.S. degrees in artificial intelligence, computer engineering, computer science, human-computer interaction, information systems, machine learning and robotics. Bachelor’s degree specializations include animation/special effects, business/enterprise systems, game design and quantitative analysis. CMU’s M.S. degrees include computer science, information technology and software engineering. Finally, if you want a Ph.D., CMU offers one in computer science.

Acceptance rate: 15.4% (2020)

Annual undergraduate tuition: $57,560

Total enrollment: 14,799 (fall 2019)

Job placement rate: 60.9% employed, 27.2% continuing education (2019)

Cornell University

Location: Ithaca, New York

Related department or school: Department of Computer Science, Department of Information Science

Overview: Cornell Computing and Information Science (Cornell CIS) is the parent unit of three interconnected departments: computer science, information science, and statistics and data science. Situated in Gates Hall on the university’s Ithaca, New York campus, the unit’s current research includes a wide range of topics, from tech basics such as natural language processing and data science to more current issues like human-robot interaction or technology and equity.

Degrees offered: Cornell offers three separate IT bachelor’s degree programs, which are taught at three different colleges. Applicants can choose between a B.A. in information science at the College of Arts and Sciences, a B.S. in information science at the College of Agriculture and Life Sciences, or a B.S. in information systems and technology at the College of Engineering. Cornell University also offers a Master’s and PhD in information science.

Acceptance rate: 10.9% (2020)

Annual tuition: in-state: $39,244 (state contract colleges) out-of-state: $58,586 (state contract colleges) endowed colleges: $58,586

Total enrollment: 23,620 (fall 2020)

Job placement rate: 66%

Illinois Institute of Technology

Location: Chicago, Illinois

Related department or school: Department of Computer Science

Overview: Illinois Tech is a private, not-for-profit, technology-focused research university with software-engineering-focused computer science programs. That, along with thorough computer science theory, provides solid fundamentals for long-term career success. Illinois Tech conducts research in artificial intelligence, computer networking, cybersecurity, data science, parallel and distributed systems, software engineering, and theory of computation.

Degrees offered: Bachelor’s, master’s and Ph.D. degrees are available in career-oriented specializations, including: cyber, system and network security, and forensics; data management and analytics; digital systems technology and embedded systems; IT entrepreneurship and management; management information systems; networking, information and communications technology; software development; system administration; systems analysis; and web design and application development.

Acceptance rate: 60.2% (2020)

Annual tuition: $47,670 (fall 2020)

Total enrollment: 6,342 (fall 2020)

Job placement rate: 69.2% employed, 14.6% continuing education (2020)

Lehigh University

Location: Bethlehem, Pennsylvania

Related department or school: Computer Science & Engineering Department

Overview: Lehigh’s tech majors provide a strong foundation in computer science and engineering, from the hardware/software interface through systems software, programming languages, software engineering and the mathematical foundations of computing. Electives include artificial intelligence, computer networking, parallel and distributed computing, security, robotics, bioinformatics, data mining, web and mobile application development, and databases.

Degrees offered: Lehigh University offers a variety of computer science and information technology degree programs, including B.A. and B.S. degrees in computer science, and a B.S. debris in computer science and business. The latter is accredited in computer science and business and teaches students the skills necessary to manage and innovate in the world of IT in a corporate setting.

Acceptance rate: 25% (2020)

Annual tuition: $54,790 (fall 2020)

Total enrollment: 7,067 (fall 2020)

Job placement rate: 70% employed, 23% continuing education

Massachusetts Institute of Technology

Location: Cambridge, Massachusetts

Related department or school: Electrical Engineering & Computer Science Department

Overview: World-renowned, MIT consistently tops the U.S. News & World Report and other college rankings. It’s also widely recognized for its world-class faculty and research. From robots that perform with professional dance troupes to medical electronic devices that harvest energy from differences in body temperature, MIT’s work improves the quality of life for people worldwide.

Degrees offered: Undergraduates can enroll in electrical science and engineering; electrical engineering and computer science; computer science and engineering; computer science and molecular biology; or computer science, economics and data science. MIT also offers M.S. and Ph.D. degrees.

Acceptance rate: 6.7% (2020)

Annual tuition: $53,790 (2019)

Total enrollment: 11,254 (2020-21)

Job placement rate: 53% employed, 41% continuing education

Michigan Technological University

Location: Houghton, Michigan

Related department or school: Department of Computer Science

Overview: A medium-sized public university, Michigan Tech offers degrees in computer science, software engineering and cybersecurity. Its Department of Computer Science’s expertise includes software engineering, human-computer interaction, virtual environments, data analytics and visualization. It also has core strengths in computing systems, systems software and parallel computing. Students interested in gaming should consider Michigan Tech’s concentration in game development.

Degrees offered: Michigan Tech offers B.S. degrees in computer science, cybersecurity or software engineering, M.S. degrees in Computer Science or Cybersecurity and a Ph.D. in Computational Science and Engineering.

Acceptance rate: 74.3% (2020)

Annual tuition: In-state $16,130, out-of-state $36,432 (fall 2020)

Total enrollment: 7,041 (2020)

Job placement rate: 93% (2020)

New Jersey Institute of Technology

Location: Newark, New Jersey

Related department or school: College of Computing

Overview: Located 20 minutes from Manhattan by train, NJIT bills itself as one of the most computing-intensive campuses in the country. The university’s multidisciplinary curriculum and computing-intensive approach to education provide technological proficiency, business know-how and leadership skills. NJIT grads frequently land jobs with major companies like IBM, Mercedes-Benz, as well as COVID-19 vaccine developer Pfizer.

Degrees offered: NJIT offers B.S. degrees in bioinformatics, business and information systems, computer science, computing and business, human-computer interaction, information systems, information technology, and web and information systems.

Acceptance rate: 72.8% (2020)

Annual tuition: In-state $17,674, out-of-state $33,386 (2019-20)

Total enrollment: >11,400 (2020)

Job placement rate: 82% (2019)

Rensselaer Polytechnic Institute

Location: Troy, New York

Related department or school: Department of Computer Science

Overview: Founded in 1824, it bills itself as the oldest technological university in the English-speaking world. Rensselaer Polytechnic Institute operates an on‑campus business incubator and a 1,250-acre technology Park. Its Department of Computer Science is aggressively growing with a focus on artificial intelligence, machine learning and cybersecurity. Its students have distinguished themselves by founding companies while still in school and landing high-income jobs at a broad range of companies.

Degrees offered: B.S., M.S. and Ph.D. in Computer Science

Acceptance rate: 47.4% (2020)

Annual tuition: $55,600 (2020-21)

Total enrollment: 7,569 (2020-21)

Job placement rate: 84% (2017)

Rochester Institute of Technology

Location: Rochester, New York

Related department or school: Department of Computer Science

Overview: The computer science program provides students a deep foundation in theory and modern software and hardware concepts, as well as numerous programming languages and paradigms. Students also get paid to work full-time in technological corporate settings.

Degrees offered: RIT offers a wide range of tech degrees, starting with an associate’s degree in applied computer technology. You can also get a B.S. degree in computing exploration, computing and information technologies, computer science, computing security, game design and development, human-centered computing, new media interactive development, software engineering, and web and mobile computing. There are various M.S. and Ph.D. programs as well.

Acceptance rate: 70.7% (2020)

Annual tuition: $50,564 (fall 2020)

Total enrollment: 18,668 (fall 2020)

Job placement rate: 73.4% employed, 19.3% continuing education (2018)

Stanford University

Location: Stanford, California

Related department or school: Computer Science Department

Overview: Hands down, Stanford is one of the best schools in the country — it’s also very hard to get into. The main educational goal of the Computer Science Department is to prepare students for research and teaching careers at universities or in industry. Strong research groups exist in the realms of artificial intelligence, robotics, scientific computing and systems. Students commonly work with researchers at nearby institutions.

Degrees offered: The Computer Science Department offers B.S., M.S. and Ph.D. degrees in computer science. The B.S. degree requires students to choose an education track, such as artificial intelligence, biocomputation, graphics or human-computer interaction.

Acceptance rate: 4.3% (2020)

Annual tuition: $55,473 (2019-20)

Total enrollment: 16,385 (2019-20)

Job placement rate: 94% (2018)

Stevens Institute of Technology

Location: Hoboken, New Jersey

Related department or school: Department of Computer Science

Overview: Located across the river from New York City, Stevens is known for its prestigious cybersecurity education program. Its Department of Computer Science includes labs and research centers funded by grants from the National Science Foundation and other sources. Students get the opportunity to apply for patents, become entrepreneurs, and pitch their engineering projects and business ideas to investors.

Degrees offered: Stevens offers B.S., M.S. and Ph.D. degrees in computer science; B.S. It also offers an M.S. in cybersecurity and a Ph.D. in data science.

Acceptance rate: 40% (fall 2020)

Annual tuition: $55,952 (2020-21)

Total enrollment: 7,283 (fall 2019)

Job placement rate: 96% (2019)

University of California

The University of California is tricky, as many lists of the best tech colleges in the U.S. include the University of California, but each lists a different campus.

What’s called “the University of California” is actually a sprawling public university system — the most comprehensive in the world. It comprises 10 campuses and numerous research centers scattered across California. It has nearly 23,000 faculty members and more than 285,000 students.

Tuition: In-state $14,000, out-of-state $43,800 (2019-20)

Each campus handles admissions separately. Here are the campuses that typically get high marks for their tech education programs:

University of California-Berkeley

Related department or school: Department of Electrical Engineering and Computer Sciences

Overview: This campus and Stanford University are the major universities closest to Silicon Valley, so they’re linked to many of the biggest names and companies in tech. To say it has close ties to the industry is an understatement. Representatives from Apple and Google drop by for speaking appearances and career fairs.

Degrees offered: B.S. or B.A. in computer science. Various M.S. and Ph.D. degrees.

Acceptance rate: 16.3% (2020)

Job placement rate: 61% (2018)

University of California-San Diego

Related department or school: Computer Science and Engineering Department

Overview: This is known as an affordable option because of the number of available scholarships and fellowships. UCSD’s tech grads fuel the internet, wireless communications, biotech and computer industries. Many students work with UCSD’s research centers such as the California Institute of Telecommunications and Information Technology, the San Diego Supercomputer Center and others.

Degrees offered: UCSD offers B.S. degrees in computer science, computer engineering and computer science with a specialization in bioinformatics. There are also M.S. and Ph.D. degrees in computer science and computer engineering.

Acceptance rate: 31.5% (2020)

Job placement rate: 95% of respondents received their first full-time job offer within three months of graduating (2019)

University of California-Irvine

Related department or school: School of Information & Computer Sciences

Overview: A broad foundation in IT disciplines combined with cutting-edge research in strategic areas.

Degrees offered: UC Irvine offers B.S. degrees in computer science, data science, informatics, software engineering, computer game science, business information management, and computer science and engineering. There are also M.S. and Ph.D. degrees in computer science, informatics, networked systems, software engineering and statistics.

Acceptance rate: 26.5% (2020)

Job placement rate: 89.47% employed (2019)

Although you might not need a university degree to get ahead in the tech industry, evidence suggests that job applicants with a degree in computing or IT have a much broader choice of employment opportunities.

In fact, research shows that demand for Computer Systems graduates has risen by 512% since the start of the pandemic, while Information Systems graduates saw an even higher increase in relevant jobs ads at 626%.

A university degree doesn’t only allow applicants to find the tech job of their dreams – it also helps secure a higher salary in a sector where pay varies significantly between roles. It can also be seen as a lifetime investment. according to Danny Aldridge, co-founder of recruitment website Check-a-Salary, who says that “studying a STEM subject is likely to lead to higher levels of pay throughout a career”.

Like many significant investments, a university degree is a costly expenditure. With most courses at English universities starting at a hefty £9,000 a year, it’s no surprise that not everyone wants to dedicate three years of their life – and £27,000 – for a job they haven’t even started yet. After all, many young people are now more openly considering apprenticeships as an alternative to a university degree, while others are opting for intense, specialised online courses – especially in areas such as coding or UX design.

However, if you’re still convinced about investing in a prestigious BSc (or equivalent), you should ensure that you choose the right degree for your plans. Here, we list some of the best tech majors you should consider to futureproof your career.

Bachelor’s in computer science

As you would probably imagine, a degree in computer science focuses on a number of subjects such as mathematics, programming, and computer theory. The courses that are available in this degree might include algorithms, computation, programming principles, and data structures.

Luckily for computer science majors, the options are nearly limitless. This is because a degree in this field can take you to a number of different careers, ranging from a network architect to a computer programmer.

Moreover, a degree in computer science enables you to look further than the IT industry. Your skills will be sought after in government organizations, the healthcare industry, and many more.

Career paths for computer science graduates:

Bachelor’s in information technology

If you graduate from college with a degree in information technology then there are many opportunities that await you.

With a degree in this field, kick off your IT career by maintaining and setting up networks, a key skill that lots of organizations across all industries require. An information technology bachelor's degree can also lead you to work as a database administrator, programmer or computer systems analyst.

During an information technology degree, your coursework is bound to cover all aspects of information systems and information technology. Classes such as database concepts, Windows server networking, and business systems development will help you build a solid foundation in IT fundamentals.

Career paths for information technology graduates:

• Database administrators often have a degree in information technology. In this occupation, you can expect to earn a median annual salary of $93,750. The BLS projects job opportunities for database administrators to grow 9% by 2028.
• Computer support specialists engage in troubleshooting and problem solving while earning a median annual salary of $54,760. Job opportunities are on the rise too, as the BLS projects them to grow 10% by 2028.
• Information technology managers supervise IT teams. While it may take time to land a management position, IT managers earn an average of $85,253 per year, according to Glassdoor. This type of position enables you to work in multiple industries too, further expanding your opportunities.

Bachelor’s in data science

Data scientists are certainly in demand - even Glassdoor ranks this job as the third-best career in the U.S. The BLS also has plenty of good things to say, especially as it predicts a 27.9% rise in employment in the sector by 2026.

To explore this satisfying and fast-growing career path, it might be a good idea to consider a bachelor's degree in data science.

In this field, data scientists are needed in virtually every industry, meaning there are plenty of opportunities outside of tech. However, if working in the technology sector is essential for you, then you don't need to worry as there are lots of technology companies recruiting data scientists too.

Career paths for data science graduates:

• Data scientists earn an average salary of $113,309 per year, per Glassdoor. Companies such as Facebook, Microsoft and IBM are always hunting for experienced data scientists.
• Data architects bring in an average of $108,278 per year and can work in many industries. This flexibility makes this a great career choice if you want an IT job but aren’t interested in working for a tech company.
• Data engineers are responsible for building and maintaining data pipelines and earn an average annual salary of $102,864.

Choose a degree program based on your career interests

The majors listed above are just some of the many IT-focused degrees offered at colleges and universities in the US. There’s a number of other degrees and certifications to help you in your IT career too.

Before jumping into a career in IT, determine what type of tech job you’d like to pursue. Knowing your desired career path will put you in a better position to choose a degree program to match your career aspirations.

Once you've decided on a degree program, select an accredited school that offers the degree program you want.

In the end, there are various ways to break into the IT industry and pursuing a degree is just one.

Over the last few years, data has become one of the most valuable commodities for any business, driving decisions, powering new models and producing insights that can increase a company’s revenue or save it millions of pounds. Yet data alone can’t deliver real-world value – it’s only when it’s applied to projects and systems that these insights emerge. It’s data scientists that put in the research, thinking, planning and hard graft that transform data into tangible results.

To do so, data scientists harness their technical and analytical expertise, planning and developing data-driven projects that meet existing or future business needs. They help define any data architectures, models and operations research techniques involved in making those projects work effectively, and train their machine learning systems to spot patterns, forecast probable scenarios and predict results. They also work closely with teams across the organisation to spot new opportunities for data projects and ensure that the resulting insights become integrated into processes and workflows.

A data-driven business

At Tesco, data-driven projects impact every area of the business, helping it serve customers better. These projects influence everything from how much of a product to stock in a specific store to how online grocery orders are packed onto different vans. Data scientists will be involved in automated processes that recommend products to an online customer, based on sales data and their website interactions, but also in the way products are placed around the floorspace of a Tesco store. “We’re kind of a centralised team that works across different bits of the business,” says Stuart Barrow, head of data science at Tesco, “which means we try to improve how Tesco operates in different ways across different aspects.”

To do so, Barrow’s team works with a variety of rich, anonymised data sources, including sales data, data on purchasing behaviour, data from Tesco’s Clubcard scheme and data relating to the products Tesco sells. The data sources change according to the area of the business and the challenge, so that an online project might focus more on customer interactions with the websites and previous purchase history, while a project aimed at optimising supply chains will focus on historical sales data and how products flow in and out of Tesco stores.

Data scientists aren’t usually responsible for filtering and preparing data, or for coding, integrating and deploying the platforms that do the work, though they may be involved and work alongside development teams. Instead, their core responsibilities focus more on the data and on how it’s processed through analytics and machine learning.

“Often our work comes down to trying to predict or forecast something,” Barrow explains, “which means we will use that data in order to train machine learning models. A simple example might be forecasting sales in a store over the next couple of weeks. We’ll use historic sales data and other sources of information that we have in order to forecast the amount of products that will be sold, and that can then be used to inform how many products we bring into the store, and all the things that are interlinked with that on the supply chain side.”

Yet the impact of the data science team percolates into so many other areas. For instance, when a customer starts an order, a predictive tool goes to work to figure out how much the order might weigh and a scheduling optimisation tool decides which van it should go out on, even if the order might not be finished on that day. Meanwhile, the colleagues who picks the products for that order will be following a data-driven route around the store, defined by modelling its layout, the products stocked and the shopping behaviour of online customers.

Here small changes build up to have a real impact. When assembling an order, about 50% of a store staff member’s time is spent moving between each of the customer’s chosen items, so providing the optimal route saves them time and helps them not only ensure that the online customer gets the right products but also frees them up to help customers shopping in the store. Similarly, optimising the route of delivery vehicles saves time and fuel costs, while – at scale – helping reduce Tesco’s environmental impact.

Requirements for the role

Becoming a data scientist requires a high level of technical and mathematical expertise. At Tesco, the team looks for a good background in mathematics, physics or related subjects, plus a specialism in an area like machine learning or operational research. Experience of coding is a must-have, along with experience in open-source big data technologies such as Apache Hadoop or Spark. Beyond this, though, data scientists need flexibility and the ability to collaborate. “We can never deliver anything of real value on our own,” says Barrow, “it’s always as part of a bigger project, so we’ll often work closely within that team.” Project managers, data analysts and wider stakeholders within the business, including those within marketing and retail teams, will all be involved, and data scientists need to understand their specific wants and needs.

In turn, this enables data scientists to see new opportunities, or how solving one problem might be the first step in transforming something wider. “When working with my colleagues, it’s also occasionally an education piece when you do spot an opportunity they haven’t seen,” adds Ben White, a data scientist for Tesco. “Because we’re the data scientists, we know how the models operate and where they’re going to work well, and for which problems.”

Data science is a fast-moving field, which can make keeping up a challenge. “We need to make sure that we’re not missing opportunities to provide further value by making sure that we’re up to date with new techniques,” says White. Yet it’s this that keeps data science exciting, particularly in a company where you can work across so many areas and explore specific problems in real depth. What’s more, Tesco encourages its data scientists to spend Friday afternoons researching and training, to keep the team on the cutting edge. With a team of data scientists from different disciplines and backgrounds, there’s always opportunity to learn, upskill and share ideas. This can all be done remotely and around flexible working.

Working as a data scientist for a company of Tesco’s scale can be daunting. “It’s sometimes challenging to align the different parts and it’s a very big business,” Stuart Barrow notes. Yet it’s also hugely rewarding. “The most satisfying thing is when you get something out of the door and deliver value,” he says, particularly when you can measure the performance and see the tangible benefit to millions of customers. “When it works, the benefits are massive.”

Learn more about career opportunities at Tesco Technology

CEO Andy Jassy announced a barrage of new machine learning capabilities for AWS SageMaker during his re:Invent keynote on Tuesday.

SageMaker is Amazon's big machine learning hub that aims to remove most of the heavy lifting for developers and let them use ML more expansively. Launched in 2017, there have been numerous features and capabilities introduced over the years, with more than 50 added to it in 2019 alone.

Of the SageMaker announcements made at the company's annual conference in Las Vegas, the biggest was AWS SageMaker Studio, an IDE that allows developers and data scientists to build, code, develop, train and tune machine learning workflows all in a single interface. Within it information can be viewed, stored, collected and used to collaborate with others through the studio.

In addition to SageMaker Studio, the company announced a further five new capabilities: Notebooks, Experiment Management, Autopilot, Debugger and Model Monitor.

The first of these is described as a 'one-click' notebook with elastic compute.

"In the past, Notebooks is frequently where data scientists would work and it was associated with a single EC2 instance," explained Larry Pizette, the global head of ML solutions Lab. "If a developer or data scientist wanted to switch capabilities, so they wanted more compute capacity, for instance, they had to shut that down and instantiate a whole new notebook.

"This can now be done dynamically, in just seconds, so they can get more compute or GPU capability for doing training or inference, so its a huge improvement over what was done before."

All of the updates to SageMaker have a specific purpose to simplify the machine learning workflows, like Experiment Management, which enables developers to visualise and compare ML model iterations, training parameters, and outcomes.

Autopilot lets developers submit simple data in CSV files and have ML models automatically generated. SageMaker Debugger provides real-time monitoring for ML models to improve predictive accuracy, reduce training times.

And finally, Amazon SageMaker Model Monitor detects concept drift to discover when the performance of a model running in production begins to deviate from the original trained model.

"We recognised that models get used over time and there can be changes to the underlying assumptions that the models were built with - such as housing prices which inflate," said Pizette. "If interest rates change it will affect the prediction of whether a person will by a home or not."

"When the model is initially built to keep statistics, it will notice what we call 'Concept Drift' if that concept drift is happening, and the model gets out of sync with the current conditions, it will identify where that's happening and provide the developer or data scientist with the information to help them retrain and retool that model."

The company also announced a ML service to help write code - AWS Code Guru. This is an automated tool that's been trained on several decades of code reviews at Amazon, according to the company. If it discovers an issue, it will add human-readable comments to pull requests that identify lines of code with a specific issue and recommended remediation, including example code and links to relevant documentation.

AI, machine learning and data science have great potential value in a variety of healthcare use cases, none more so than anomaly detection. From readmission prediction to fraud, accurate, fast anomaly detection is necessary in order to provide a high standard of care and protect the finite resources available to do so.

Establishing a firm understanding of what data science, machine learning, and AI technologies can bring to anomaly detection use cases in healthcare today is a first step to making headway in these areas.

This whitepaper explores fraud detection as a use case for anomaly detection systems in healthcare, identifying what anomaly and fraud detection are, how they work, and when healthcare organisations can apply them.

Businesses at the top of their respective industries all have one thing in common: the knowledge that data, both structured and unstructured, forms the heart of their organization and is key to their success.

A comprehensive and nuanced data strategy is therefore vital.

Creating one, however, is not an easy feat. It isn’t just a matter of dealing with all the sources of data; it’s also having expertise in the tools required to manipulate and process that data and understand what needs to be structured and where a rigid model is inappropriate.

Do you have people that are experts in Excel, SQL, NoSQL, PostgreSQL and Oracle? Does your business have a suitable data lake so all corners of the company can benefit from the data itself and generate their own specific insights?

If you answered ‘no’ to any of these, then your data management strategy might not be up to scratch. How you feed that data into your business, or data lake, is up to you. But understanding how to store that data - either in a structured or unstructured manner - is crucial in making that next big step in your business’ journey to the top.

What is structured data and how is it managed?

When it comes to databases, structured data is often what first comes to mind.

Structured data - also known as relational data - is stored in tables with columns and rows. The structure is described by a schema, with relationships defined between the tables. For example, one table might contain a list of customers, another table might contain a list of telephone numbers with one row in the customer table potentially having many connected rows in the telephone numbers table.

Structured data is, by its very nature, already “managed”. The definition of the tables and columns means that data must be kept in an orderly fashion. Constraints can be added to the columns to, for example, ensure that only numbers are entered for telephone numbers and that no telephone number can be added without an associated customer.

The structured approach to data storage has both benefits and drawbacks. Users are able to understand and access data easily, with a wide variety of tools for querying and analysis available. However, it can also be regarded as highly inflexible when the data needing to be stored does not fit the structured data model.

It's also worth noting that a relational database management system (RDBMS) is often embedded in products that also offer far more bells and whistles than just managing data and making it available to queries.

For example, Salesforce, the cloud-based customer relationship management (CRM) platform, manages the structured data put into it, but also offers tools like chat, access to the Force.com development platform, analytics, and so on.

What is unstructured data and how is it managed?

Unstructured data is anything that can't be organised into a structured database. Common examples are free-flowing text-based interactions, such as email conversations or chat logs, word processing documents, slideshow presentations, image libraries, or videos.

Estimates vary for how much unstructured data lies in business. One recent projection estimated that 80% of global data would be unstructured in the near future. 

It contains a wealth of corporate information but, by its nature, was difficult to access until modern big data analytics and AI have become a reality.

Together with structured data, it's also one of the three Vs of Big Data variety (the other two being velocity and volume).

There are many benefits to using unstructured data. It is rapid to accumulate and there is no need for time-consuming parsing since there is no predefined structure. It can also be stored in its native format and lends itself well to corporate data lakes.

However, the benefits can also be pitfalls; specialized skills and tools are required to analyze unstructured data and users can be left alienated when faced with data that does not adhere to a uniform format.

Common tools for dealing with unstructured data (also referred to as NoSQL) include MongoDB and DynamoDB.

Unstructured data management (UDM) is essential for successfully making use of all this data. Rather than there being a handful of tools to point to for UDM, there are instead some basic tenets to be followed, which we outline below.

Indexing

This term, sometimes known as "discovering" as well as other related terms, means compiling your data to really see what's there, how frequently it is accessed, for how long it has existed, and more. 

The objective of indexing is to find out whether this information will potentially bring future value to the organisation and see if it is worth putting in an UDM system and archiving it.

This, however, can be a long process, and can take many weeks to sift and scan all the data. Be ready to dedicate a lot of effort and time to this process in the initial stage. This is also the section where you should add metatags so that the data is easy to search later on in the process.

Storage and availability

Now that the data has been organised, it now requires storing in a suitable location with the correct attributes that make it automatically and easily accessible.

There are a number of storage locations to choose from, which includes general cloud storage like Microsoft Azure or AWS S3 or on-premises data lakes. When the information resides here, it is able to be stored in its "natural" state, which means there is no need to store it in a database format, but also allows it to be available for automated querying through APIs.

When thinking about which type of storage to utilise, it's worth considering how frequently the data that is being stored is accessed. For example, if it's relatively frequent, it might need to be put in "cold" storage, which is usually much cheaper than if it is kept in storage that makes the information accessible at all times. However, in this "cold" storage it will be slower to access initially when you do need to sift through it and query it.

Semi-structured data

Semi-structured data can be regarded as a “halfway house” between structured and unstructured data. While it does not have the predefined model of structured data, it is easier to store and work with than unstructured data.

Examples of semi-structured include JSON or XML. Although the columns and definitions of traditional structured data are not usually present, the data does contain tags and markers to separate elements, making for simpler querying. These tags and markers are referred to as “metadata” and mean data can be better catalogued.

Vector databases

Also found in the category of unstructured data  - although increasingly meriting a section of its own - and becoming progressively more popular in the era of generative AI, are vector databases.

As with unstructured data, pretty much anything can be dropped into the database. However, an embedding function encodes the data into vectors to capture its meaning and context. The result is something that can be searched based on similarity or vector distance.

An example would be searching for an image based on content or style. Or a large language model (LLM) generating more relevant and coherent text.

Examples of vector databases include Pinecone and Weaviate.

It is admittedly quite late in the game to consider the advantages of AI given how commonly used it already is. Nevertheless, it's an important technology to ponder and debate as it will inevitably affect all life on the planet.  

From minuscule algorithms that offer a little convenience for everyday life to international defense systems, AI is already impacting us, often in ways we cannot see. What you see on your social media feeds, how we predict the weather, and how we manage our staff members and workflows, are all now common use cases of AI.   

While we have a thriving AI industry and seemingly a new use case popping up daily, we still have only scratched the surface of what this technology has to offer. Before adopting it, however, it's always worth considering the advantages and disadvantages of AI.

What are the advantages AI?

The advantages of AI are currently being talked about, to death, by every tech company that has a product to ship. You'll have noticed 'AI' being tagged on just about everything. It's in our laptops, our collaboration software, and even our kettles. But what are the actual benefits of artificial intelligence? 

Improved efficiency

This is arguably where the business interest will be, particularly when it comes to their IT stacks and their workforce. Things working more efficiently will lead to improved performance and cost savings. Automation and generative AI tools are coming out in mass and getting both more affordable and more advanced as we speak. 

A key element here is data; Copilots, such as Microsoft's, run on web and user data, analyzing how, when, and why we use our devices and software and offer us shortcuts and simple directions. For a more business example, SAP's Joule Copilot takes in SAP customer data – essentially pure b2b information – to improve how its customers use ERP and financial platforms. That process basically explains most AI efficiencies, data feed into an algorithm to find better, faster, and simpler ways of working. 

Reducing human error

Another efficiency is the reduction of errors – human ones. We are all prone to the odd laps of concentration and we also simply get things wrong. AI-based machines carrying out specific tasks don't – if they do, you'll likely find that's also some kind of human error.  

This is why Amazon and Ocado use robots in fulfillment centers, and why financial systems, such as those that evaluate credit, are based on algorithms. The reasoning here is volume; you can do more if it's automated, but you're also not increasing the risk of error. 

Advancing AI

In some cases, by using AI you are, by default, also improving AI. Maybe only by a little bit, but it is still learning and developing with the data you create. When your business uses a Copilot, said Copilot will have been designed to learn from its users. It will essentially tailor its service towards the users. So, for example, SAP's Joule will help you navigate Concur when filling out business expense applications and gradually figure out your preferences and those of your company.

If we take that example and apply it to a more grander use case, such as quantum computing, or medical diagnostics, AI will only get faster and more efficient as we apply it. This will be dependent on privacy policies, regulations, and the types of systems being used; AI will largely improve through training and fine-tuning. But it is data (our actions) that will help it to advance.

What are the disadvantages of AI?

It’s natural to be fearful of powerful technology. Recent history with data scandals, malware, and social media has made that clear. AI is no different and many of the concerns held by onlookers are, in some areas, justified. But that doesn’t mean great work isn’t being done to mitigate the drawbacks.

In 2016, an industry-wide organisation including five Silicon Valley giants was formed, known as the Partnership on Artificial Intelligence to Benefit People and Society. This body works to promote the fair and ethical development of artificial intelligence technologies that have the potential to bring as much disruption as it will benefit.

Decision-making AI in the workplace

The speed and efficiency of certain AI applications make them appealing to executives looking to find more value across their organizations. 

IBM's Watson has been used to decide if employees are worthy of a pay rise, a bonus, or a promotion by looking at the experience and past projects of employees to indicate the future qualities and skills individuals could bring to the company. 

Decision-making software used in this way has caused some concern. The Trades Union Congress, the federation that represents the majority of trade unions in the UK, called for legislative changes last year to safeguard employees against this kind of technology. It is also recommended that employers consult trade unions before deploying such systems.

"Our prediction is that left unchecked, the use of AI to manage people will also lead to work becoming an increasingly lonely and isolating experience, where the joy of human connection is lost," TUC general secretary Frances O'Grady said.

Job losses

The potential for human job losses is widely regarded as the number one downside to AI, the implementation of which could set in motion a wave of lay-offs as employees struggle to outperform machines.

However, while this scary scenario is often presented as just over the horizon, AI is expected to create more jobs than it takes. The World Economic Forum’s (WEF) 2020 Future of Jobs Report predicts that by 2025, automation will have affected 85 million jobs around the world but 97 million jobs will be created in industries such as artificial intelligence.

“No matter what prediction you believe about jobs and skills, what is bound to be true is heightened intensity and higher frequency of career transitions, especially for those already most vulnerable and marginalized,” stated FutureFit AI CEO Hamoon Ekhtiari.

The report also stated that, although the pandemic has accelerated the automation of many repetitive and dangerous tasks, “around 40% of workers will require reskilling of six months or less”. One area of skills worth developing in time for the AI-based future is data, but soft skills shouldn’t be ignored either. John Whittingdale OBE, former minister of state for media and data, described soft skills as “hugely important”, adding that, “without them, there is the potential for data to be misread or miscommunicated, which can have significant implications for businesses and the decisions they make”.

Human error

Although AI can virtually remove human error from processes, its code is still subject to bias and prejudice. Being largely algorithm-based, the technology can knowingly or unknowingly be coded to discriminate against minorities or fail to cater to groups that its programmers failed to consider.

If security measures are not followed carefully, hackers can exploit AI seeking to collect public data. For example, Microsoft's ill-fated chatbot Tay Tweets had to be taken down after only 16 hours as it had started to tweet racist and inflammatory content driven by input from other Twitter users.

Importantly, Tay Tweets was purposefully fed hateful content in an effort by Twitter and 4chan users to break it. But other examples of AI going astray have come despite the best efforts by its developers.

For example, in 2018 Amazon decided to retire a recruitment algorithm after it was discovered to discriminate against non-male candidates. The AI system was intended to provide hiring recommendations and had been fed ten years of application data to help train its decision-making. However, as the majority of submissions have been handed in by men, the conclusion the AI came to was that men were preferred candidates.

Responsible use of AI

There is a great deal to be positive about when it comes to AI. Any emerging technology that has the power to disrupt the existing structures of individuals and organisations must be assessed for its potential risks.

But being mindful of the downsides does not mean becoming blinkered to the benefits. Indeed, decision-makers have been warned against doing just this, or else risk losing out on the clear improvements that careful use of AI can bring.

"Look at how you are using technology today during critical interactions with customers – business moments – and consider how the value of those moments could be increased. Then apply AI to those points for additional business value," said Whit Andrews, distinguished vice president analyst at Gartner.

"AI projects face unique obstacles due to their scope and popularity, misperceptions about their value, the nature of the data they touch, and cultural concerns. To surmount these hurdles, CIOs should set realistic expectations, identify suitable use cases and create new organisational structures."

Gartner advises that business and IT leaders should endeavour to cut the AI hype away from reality by carefully considering and weighing up the opportunities vs risks. Obsessively focusing on automation, rather than the bigger picture, will only obscure the wider benefits, the analyst firm warns.

In July 2022, the UK government and Alan Turing Institute jointly announced the establishment of the Defence Centre for AI Research (DCAR). Its goal is to develop areas of AI research that are currently proving challenging to implement, such as training without the need for large data sets, AI ethics, and war gaming.

"Everything we love about civilisation is a product of intelligence," said Max Tegmark, president of the Future of Life Institute.

"Amplifying our human intelligence with artificial intelligence has the potential of helping civilisation flourish like never before as long as we manage to keep the technology beneficial."

Organisations can also check if their use of AI systems breaches data protection laws using a risk assessment toolkit launched by The Information Commissioner's Office (ICO). The AI and Data Protection Risk Assessment Toolkit, available in beta, draws upon the regulator's previously published guidance on AI, as well as other publications provided by the Alan Turing Institute.

It contains risk statements that organisations can use while processing personal data to understand the implications this can have for the rights of individuals. Based on an auditing framework developed by the ICO’s internal assurance and investigation teams, the toolkit also provides suggestions for best practices that companies can put in place to manage and mitigate risks.

Over the past year or so, a maxim has crept into the IT industry's common lexicon: "Data is the new oil". In other words, it's a precious resource that has the potential to be extremely useful and incredibly valuable.

With this in mind, many organisations are now seeking to transform themselves into "data-driven businesses", making use of tools like data analytics and machine learning to extract value from the vast reams of information they generate on a daily basis.

This task isn't necessarily as simple as it sounds, though. Becoming a data-centric organisation can be a long and challenging process, with many potential hazards to navigate. In this month's IT Pro Panel discussion, we spoke to some of our expert panellists who have been through this journey to get their insights on how best to navigate the road to data nirvana.

Driving business outcomes

Speaking to our panellists, it quickly becomes clear that the wave of interest around data is more than just empty hype. It can have a real, tangible impact on multiple areas throughout a company, as Ritam Gandhi, founder of digital design firm Studio Graphene, explains.

"What I have realised is that 'data' touches nearly every part of our business," he says "We use data gathered on user engagement across websites we build to inform decisions taken by our digital marketing team. We use data collected as a result of user testing to inform UX decisions taken by our design team. We use historical data collected on a variety of tasks by our project management team to predict how long a particular project is going to take to complete."

This use of user behaviour data gathered from within applications and websites to improve and optimise them is one of the most universal examples of data analysis in action, but it can also enable business-critical functions such as security.

"As a business with a large online presence, William Hill uses data much like any online retailer," says William Hill's group CISO Killian Faughnan. "With a heavy focus on improving customer experience and service. From my own perspective in security, and perhaps where I can speak more authoritatively on the subject, we use it primarily for the automation of our protection platforms and security controls."

"We pull in data from every layer of our infrastructure so that we can kick off predefined playbooks to reduce the impact to our customers of directed attacks," he explains. "Without access to the vast quantity of data available through modern information systems we would simply be unable to protect the business in the way we do today."

In some cases, data analysis can help augment and support gut feeling, providing a solid foundation for business decisions when personal instinct cannot solely be relied upon. This can come in particularly handy if the business finds itself in a "brain drain" situation where it lacks in-house expertise in certain areas - with Addison Lee being a prime example.

"Addison Lee spent 38 years under private family ownership, where they knew the business and market inside out," chief data officer Graeme McDermott reveals. "However, when it was sold to a private equity firm and the many family members moved on, they needed to revert to data and insight to make decisions - not least as their market was disrupted by the arrival of Uber. Now they use data and tech to the extreme."

"Data is vital in the humanitarian sector," adds Oxfam International CIO Michael Duggan. "When a disaster happens, it's essential to get timely data in order to support critical decisions on what type of aid is needed, how to get it there and making sure it's used in the most effective way possible."

Big data, little data

One of the first hurdles for any business to overcome when planning a data strategy is deciding what data they need to collect.

"I think the starting point is for organisations to identify what they are trying to achieve from 'data'," Studio Graphene's Gandhi says. "They then need to identify what data sources they have access to and lastly what treatment needs to be applied to the data set in order to make it valuable to them."

For McDermott, the obvious starting point is data on your clients or customers, and on any transnational business you're doing.

"Clearly, as a data exploitation person at heart, I want it all. But the reality is not all of it adds value or richness," he says. "I suppose the crucial point is to evaluate systems you implement to support your product and ask 'where is the data, what does it collect and how easy is it for me to get to?'"

However, another important consideration raised by Duggan is that of safety and compliance, particularly in a post-GDPR world. Organisations, he says, don't just need to be legally compliant, but must also have a set of ethical guidelines about acceptable data use - something that he thinks will become a critical factor in organisational governance.

"Many data projects use a lot of personally identifiable information," he points out. "If you don't ensure you understand your data risks adequately you might set yourself up for a lot of pain later.

"Most of the data we collect is about people, many of whom are vulnerable. An expression I heard last year was that 'If data is the new oil, data protection is the new climate change'. "As a result we are putting a lot of energy into making sure the way we collect data is safe, and the way we store and share it is safe."

Ensuring the accuracy of the data they're analysing is also a key concern for all our panellists. As a CISO, the margin for error in Faughnan's data analysis is close to zero, so he goes to great pains to make sure it's as accurate as possible.

"I've not found a silver bullet that's for sure. From my own experience I've found I need to categorise data sources as either 'reliable' or 'unreliable' and act accordingly," he says. "Once you start watering down the quality, you begin to add in uncertainty to the decision tree, which, for us, means human intervention is required."

He does, however, accept that a certain amount of flexibility is required when it comes to the quality of data being processed in non-business critical analysis, arguing that "realistically, outside of mission-critical circumstances, unreliable data is the norm and businesses need to be cognisant and accepting of this."

"What I tend to say is that 'Big Data' is informed by 'Little Data'," Duggan argues, "which can give us an insight and potentially a way of understanding if our analysis of Big Data is right. Given the velocity of many of our operations, it's unlikely our Big Data set is as good as we would like it to be, so it's important to have strong check on making decisions. The best thing being people on the ground who can assess."

"Michael's point is so true," McDermott agrees. "A few years ago I saw a presenter talk about the 'data underdogs'. His summary was 'don't forget about the smaller, less important data items as a way of explaining what's really going on'."

A question of hype

This is one of the potential pitfalls organisations can fall into when looking at data, but it's far from the only one. The most common issue identified by our panellists as a possible threat to the success of a data programme is a poorly-thought out or unrealistic strategy. As Duggan points out, a data-driven organisation is in many ways a different beast from a more traditional company.

"A key question is if your business model is designed for data-centric decision making," he says. "A lot of organisations have a very linear value chain - a bit like Porter's value chain - which incrementally uses Big Data to make decisions but not in a transformational way. The models we see companies like Uber (and Addison Lee) shifting to are platform models where Big Data is used in real-time to influence things like capacity and pricing. It's a fundamentally different way of architecting your organisation."

In McDermott's view, it's imperative to strike the right balance between getting started with a proof-of-concept quickly, and adjusting it for long-term viability once the project has been approved. He points out that while delivering rapid time to value is important, if you don't take the time to reassess the technical architecture every so often, you can end up with an unstable end result.

"No one remembers that it was only a proof-of-concept until five years later, when the systems dies and someone asks why it was up on bricks. Trust me, it happened to me several times - with inherited systems, I might add," he explains. "If everyone's not on the same page about the final picture you'll have a crazy outcome. However, you can't be planned to the nth degree before starting."

For Faughnan, it's important to maintain a sense of perspective when looking at data projects, as many decision-makers tend to get carried away by the potential of data analysis without being realistic about what can actually be achieved.

"While we should definitely think strategically about where we want to get to, it's important to be pragmatic about how realistic it is to attain this nirvana, under what conditions, and in what timeframe. Give a glimpse of the blue sky distant future, but pitch a pragmatic solution that gets you some of the way there," he advises. "Be adventurous, but not irresponsible, in your approach."

In fact, all of our panellists agreed that although data analysis can have huge benefits, it has also been over-hyped to a certain degree.

"I think part of the challenge is that drawing inferences from a dataset isn't always as straightforward as one thinks," says Gandhi. "Organisations often jump to incorrect conclusions because of the way they've interpreted the data they have or often because they do not have a large enough data set to draw an accurate conclusion."

"I think there has to be a sense of realism here," Duggan muses. "Many might think that data is 'cheap' or easy but good, clean data is actually expensive to acquire ... this is why many data projects stall when they get out of pilot as organisations struggle to scale."

"Like every new innovation we overestimate the initial impact, but underestimate the longer term change it will make at industry level. I suspect Graeme is at the heart of this, as he's competing with an organisation built around data: Uber."

Fundamentally, however, data - and the insights that it gives access to - remains an integral part of all of the panellists' businesses. While it can be challenging to effectively exploit, it can deliver real value to organisations, assuming they approach it in the right way.

If you're a senior IT decision-maker and you'd like to apply to be part of the IT Pro Panel, please email panel@itpro.co.uk.

Scientists from Brown University have discovered a way to store data on metabolic molecules, matter smaller than a piece of DNA, by manipulating molecules to mimic the 1s and 0s used in binary code.

Storing data in biological matter isn't exactly new, as scientists have been working with data storage on DNA for a few years now, but this research shows it's capable on a much smaller scale.

Being able to store data on ultra-small pieces of matter presents a wealth of benefits to the IT industry. For starters, it would make hacking into the data much more difficult, meaning it could be ideal for the most sensitive of information that might be more vulnerable in the cloud.

It's also thought that storing data in this way could present benefits for those in harsher climates and environments where traditional computer hardware can be known to suffer. Specifically, with DNA, data could theoretically be preserved for hundreds of years if kept in the right conditions.

The researchers proved the possibility to store data in molecules by using solutions containing sugars, amino acids and other small molecules - known as mixtures of common metabolites used by living creatures to digest food.

Using these solutions, liquid handling robots placed 1,024 dots on a plate and within each dot, the six metabolites were either present or absent. Whether the metabolites were present or not was translated into 1s and 0s binary information, enough to encode a 6,142-pixel (thumbnail sized) image.

Researchers were able to extract the data with around 99% accuracy but they admit DNA is still superior for encoding larger datasets and the small molecule method is comparably much slower than traditional computers but it does have some advantages over DNA.

"Compared to DNA, our metabolite data has low latency, in that we can write and read data sets quickly from start to finish," said Jacob Rosenstein, lead researcher to New Scientist. "These ideas are ready to be used in research labs, but we would need to speed things up and shrink the size of the analysis hardware before it would be practical outside the lab."

We could expect commercial DNA storage devices to be on shelves in the next few years, according to claims made by Catalog, a startup based out of Harvard Life Lab. The company said devices capable of storing up to 1TB of data in a DNA pellet will be completed by the end of 2019 with a view to them being on shelves by 2021.

The company recently announced that it successfully extracted all of Wikipedia, totaling 16GB, on a single DNA pellet.

Somewhat famously, a Ph.D. student cracked a three-year-old bitcoin challenge with three days to spare last year by sequencing DNA to decode files revealing the wallet address for one bitcoin as the prize. It was a prize started by Nick Goldman in 2015 after addressing the World Economic Forum about the possibility of DNA data storage.

As the technology is still in its infancy, the real-world use cases are still unknown but open to speculation. Although DNA and metabolite storage present benefits, such as not being vulnerable to hacking, physical attacks will still be possible which means it won't be a security silver bullet.

Transport for London will start tracking the movements of passengers on the London Underground from next month, the company has announced.

Beginning on 8 July, TfL will begin using the free Wi-Fi service provided throughout the tube network to gather information on the movements of passengers through the system. Anyone that connects to the Wi-Fi will be tracked as part of the scheme. An early pilot was held for four weeks in 2016 across more than 50 tube stations, during which TfL gathered data on more than 42 million journeys.

The aim of the programme will be to give TfL more granular data on passenger movement within stations, allowing them to inform customers more accurately about the state of the network, taking things like foot congestion and overcrowding into consideration, as well as train schedules.

"The benefits this new depersonalised dataset could unlock across our network--from providing customers with better alerts about overcrowding to helping station staff have a better understanding of the network in near-real time -- are enormous," said TfL's chief data officer Lauren Sager Weinstein.

"By better understanding overall patterns and flows, we can provide better information to our customers and help us plan and operate our transport network more effectively for all."

The data will also be made available via TfL's open API, meaning that app developers will also be able to make use of it. Journey planner apps like Citymapper, for example, could direct customers to pick the less overcrowded of two equally fast routes based on this passenger data.

"TfL is a world leader in the use of data to improve services," said London's chief digital officer Theo Blackwell. "Today nearly half of all Londoners use an app derived from TfL's open data feeds. Greater insight into how people use the Tube helps better transport planning but also enables the development of new apps to give customers further advance information about their planned journey in order to plan their best, and least congested route."

TfL has anticipated the concerns of privacy campaigners around this kind of mass tracking, however, and said that the data-gathering will be secure and privacy-protected. The data will be depersonalised, it said, promising that "individual customer data will never be shared and customers will not be personally identified from the data collected by TfL".

The organisation also noted that it had collaborated with the Information Commissioner's Office on developing the scheme in order to ensure that privacy and transparency concerns were adequately addressed.

Once the system goes into effect, warning notices will be added to TlF's in-station CCTV signage to alert them to the data collection and direct them to where they can find more information.

"While I am excited about the potential of this new dataset, I am equally mindful of the responsibility that comes with it," Weinstein said. "We take our customers' privacy extremely seriously and will not identify individuals from the Wi-Fi data collected. Transparency, privacy and ethics need to be at the forefront of data work in society and we recognise the trust that our customers place in us, and safeguarding our customers' data is absolutely fundamental."

Students at a university in Munich have developed an algorithm that predicts who is most likely to die in the TV series Game of Thrones.

With the show's final season underway, a team at the Technical University of Munich (TUM) has a used basic data science and machine learning to work out which characters are most likely to die.

Their prediction for season eight (potential spoiler alert) is that Ser Bronn of the Blackwater is most likely to kick the medieval bucket with a 93.5% chance of dying. To get this statistic, the team at TUM used approaches similar to those used in medicine and life insurance calculations.

They mined statistical information about how long people lived, along with biographical data that might correlate to when they die. A real-life example might include information such as whether someone is a smoker or how frequently they exercise. But in the world of Game of Thrones, the more relevant information is what house a particular character belongs to, whether they're married, and who their allies are.

"In this project, we wanted to find features that are common to all already dead characters and use these features to predict the percentage likelihood of death (PLOD) for yet alive characters, aka to answer the question - who is likely to die next?" the team said in a blog post.

TUM's data scientists combed through the lives of hundreds of characters via the help of fan-maintained Wikis. And, along with collecting in-universe data like their gender and location, they also included information such as whether someone is a major or minor character and how often they're cited in fan Wikipedias.

By using this data, the team were able to reveal some basic truths about mortality in the Game of Thrones universe, such as the fact that being male is more dangerous than being female - men have a 22% death rate, compared to 11% for women. Certain houses are more long-lived than others, reflecting their ascendancy in Westeros. Being a Baratheon, for example, makes you 5% more likely to die than the average character, while being a Lannister makes you 45% more likely to survive.

To turn all this into predictions for individual characters, the team analysed this data using two separate models. Firstly with a fairly straightforward statistical approach known as Bayesian inference and the second relied on techniques involving machine learning and neural networks.

The shows cut-throat approach to characters has seen it become one of the most popular TV shows around and an even more popular death pool, where fans predict when characters will die. Using the TUM team's algorithm would be cheating though.

Nvidia and SAS have teamed up to connect businesses with AI technologies through Nvidia GPUs and CUDA-X AI acceleration libraries, combined with SAS's machine learning and data streaming technologies.

For example, SAS's Viya software can work with Nvidia's GPUs to help businesses identify objects and classify images, run speech-to-text applications and detect sentiment in communications, whether written or voice.

The tie-up is focused on the healthcare, life science, manufacturing and financial service sectors, helping those industries to connect better with their customers and ensure they deliver fault-free products to end-users.

"Analytics is the core of AI," said Laurie Miles, director of analytics at SAS UK and Ireland. "SAS is constantly looking at ways to improve its customers' analytics capabilities for successful AI deployments.

"This partnership with Nvidia combines our software expertise with their powerful GPUs, creating world-class engines for rapid, accurate AI operations. AI is opening up huge possibilities for organisations of all varieties, and SAS is at the forefront of innovation in this important field."

The partnership will also allow the firms to jointly develop new applications for their hardware and software that could be applied by tech companies in future. One example is equipping a drone with AI technology to monitor infrastructure and predict when machinery will need to go in to be serviced.

Because the analytics engine and the GPU are both implanted in the drone, the data can be analysed wherever the drone is, rather than sending the data back to another device to be analysed.

"Our collaboration with SAS will help enterprise customers extract the true value of AI for their company," said Ian Buck, vice president and general manager of Accelerated Computing at Nvidia. "With Nvidia technology, businesses will be able to accelerate their entire data science workflow to innovate, add new services and increase profitability."

SAS isn't the only company that Nvidia has partnered with in this area; the company has also announced this week that it has teamed up with major hardware vendors including Dell, HP and Lenovo to create purpose-built, high-powered workstations for data scientists also using Quadro RTX GPUs and CUDA-X AI libraries.

Here's a conundrum: why is it that computer science graduates find it so hard to get a job after university, when there's such a desperate shortage of people with IT skills?

It's one of the questions that the Institute of Coding was set up to tackle and one that featured heavily at the first annual conference of the IoC, held in Manchester last week. Lack of diversity among IT staff and students were also centre stage, as well as what the field of data science might look like in the years to come.

Established last year off the back of government funding, the Institute of Coding initiative was built to bridge the gap between academia and the highly lucrative digital industry. In a video address at the start of the conference, the digital minister, Margot James, highlighted that the UK's tech industry is worth 116.5 billion a year, and is enjoying growth twice that of other industries.

The demand for skilled talent is there, but one of the common complaints that's heard from business is that it's hard to recruit people with the right skills. There is still much debate about whether UK schools and universities are doing enough to produce the talent needed, particularly when compared to schools internationally.

An interesting approach comes from a school called 42, a French non-profit and tuition-free institution that specialises in computer-programming. Based in Paris, 42 was set up to address the mismatch between France's position as the 6th largest economy in the world but only the 25th biggest digital economy. Olivier Crouzet, Dean of Studies at 42, said that education had been disconnected from society as the skills that were being taught were associated with the old industrial revolution.

The 42 approach is notably different from standard teaching: there are no tutors, as lessons are based heavily on project work and, most importantly, on a co-operative approach. "There is a view that collaboration is cheating," said Crouzet, but teamwork has proved essential. Thanks to funding from billionaire Xavier Niel, the school is able to operate tuition-free and is open 24/7, although is considered the extreme of what's possible given enough funding.

To improve the number of people with the requisite skills, however, there's a need to broaden the base of recruits, hence the emphasis on improving diversity. While much of the discussion centred around the need to bring in more women, IoC director Rachid Hourizi spoke of the "diversity of diversity", that gender was just one element in this debate.

For example, Mark Smith, of Ada College, explained how his institution had been established in Tottenham Hale, one of the poorest areas of London, as a means of providing a digital education to children of low-income families.

But there's more to the subject than gender and income. IoC co-chair Jacqueline De Rojas explained that one of the areas that needs to be explored is neurodiversity.

"We need to tackle diversity in all its forms," De Rojas told IT Pro. "In cybersecurity, for example, people with Aspergers or on the autism spectrum can spot trends that we mere mortals can't spot. We'd love people on the spectrum to come forward. But there's a stigma attached and we must find a way to embrace those people."

But turning to the theme of women in IT, De Rojas said that the gender split came early. She highlighted a study by the Girl Guiding Association which found that by the age of 11, only 11% of girls had been given the opportunity to study a tech-related subject, compared to 33% of boys.

"There is a bias in the teaching," she argued. "We need to address this."

She added that there was also a need to educate everyone in digital skills at an early age. She explained that much of the debate is about GCSEs but "it's too late by then".

"By the age of 12, most children have already made up their minds about what they're interested in. We now have to think about how we relook and reshape digital education, aiming at 4-year-olds - or even younger."

One of the areas where there's much debate is in the field of data science, and the conference touched on the idea of preparing the ground to find data scientists of the future. There's common agreement that there's a need for more data scientists, but the issue is clouded by the lack of a firm definition of what the job title actually refers to. As the Open University's Patricia Charlton explained, AI is part of the data science make-up, but the terms aren't interchangeable and are often confused in discussions about machine learning. She added that there was work being carried out to develop a standard definition of data science.

Beyond promises and well-meaning gestures, what stood out most about the IoC's annual technology conference was the lack of industry representation. The Institute of Coding was set up to ensure that the industry gets the digital professionals that it needs, but there was a complete absence of major tech companies or anyone from user organisations. The debates were, on the whole, conducted by pedagogues for an audience of pedagogues. There was plenty of effort involved in analysing the issues, including why there are so few women in IT, but it was sorely in need of input from the other side of the fence.

Twitter is possibly one of the hardest social media nuts to crack. Although it promises a lot for business users, and specifically those that want to use its leadership and influence capabilities, it can be very time consuming to manage, with a focus on interaction and responding, rather than simply posting.

One of Twitter's differentiators compared to other social media tools is that it generates so much more conversation, with people posting quickly and often very direct replies.

Twitter is a great customer service platform, too. Because it's so open, it can expose a company's flaws and potentially damage reputations. That said, it's a great way to demonstrate crisis management, with informative and accurate replies crucial to winning back some of that positive energy.

It's also a good place to connect with influencers, find out about the latest leaks and news and converse with people in real time. Instagram and Facebook just aren't designed to be used in the same way.

Twitter is growing in importance for politicians too. Donald Trump, for example, has pushed the platform forward, using it as a platform to voice his opinions, connect with voters and vie for support.

Twitter is first and foremost a marketing tool and, as such, it should be analysed and activity scrutinised in exactly the same way as your website is to make sure you're making the most of it.

Twitter Analytics is an extremely handy tool to help you understand how your followers are engaging, helping you to decide which posts people want more of and which you should perhaps scrap. You'll be able to see which accounts following you are most active, which click through to view more on your website and, ultimately, who is most likely to convert to become a loyal customer.

Although there are lots of tools to decipher how people view your Twitter account, the in-built Twitter Analytics are the first place to check out, before you start investing in extras. Here's how to use them.

How to see how many people have viewed a tweet

Many Twitter users will be au fait with the service's basic analytics package. These are the reply, like and retweet buttons that sit underneath every tweet. These will give you a broad understanding of the people who have interested in a particular tweet. But to get under the skin of how others respond to your glowing tweets, you need to go deeper and use Twitter's full-on toolset of goodies.

To get more data on how well your tweet is doing, click on the bar graph icon beneath the tweet to see its activity. This works both on the mobile app and Twitter's desktop site. This displays the basic analytics data for that tweet, including the number of times people have seen it, how much they've interacted with it by liking, retweeting or replying to the tweet, and how many times they've clicked on an image or link within it.

If you need a little more in-depth data as well as wanting to track a tweet's activity across a greater time frame, you can use the analytics tools integrated into Twitter's desktop website. There are aimed mainly at organisations that want to use the service for adverts and campaigns, but these tools can also help in tracking how well non-promoted tweets are performing as well.

How to access Twitter analytics

To access Twitter analytics, go to analytics.twitter.com, or click your profile picture in the top-right corner of the Twitter website to bring up the menu and click on 'analytics'. The homepage of the Twitter analytics site will show you a breakdown of all the key data from your Twitter account, with performance metrics (including number of tweets, mentions, followers and profile visits) from the last 28 days. It also compares those metrics to the previous 28 days to give you a sense of how you're doing.

Scroll down, and you'll find a monthly summary, with the metrics from each calendar month in chronological order, along with a number of highlights such as top-performing tweets or notable new followers.

Navigate to the 'Tweets' tab, and you'll find more detailed information. Here, you can examine the analytics data for tweets published within a specific period, as well as sorting tweets based on their number of impressions and engagements. You can also sort them based on the engagement rate, which is the percentage of people who saw the tweet and then interacted with it in some way, such as retweeting it or clicking on a picture.

The 'Audiences' tab offers a wealth of interesting information about who it is who's actually seeing your tweets. This has an almost frightening level of data; it includes all the usual data you'd expect such as age, gender and location breakdowns, but it also tells you which topics your audience is interested in. You can even access data about how many of your followers are homeowners, their household income brackets and even their net worth! Bear in mind, however, that this data is obtained from Twitter's partners, and applies only to your US-based followers.

If you want to tie your tweeting into what's happening in the world, you can use the 'Events' page, which basically functions as an expanded version of Twitter's trending topics. It'll give you an overview of current and upcoming events - like trade shows or public holidays - telling you which kinds of people tweet about it the most. For example, the upcoming CES expo in Las Vegas is most often tweeted about by men aged 18 to 24 (surprise, surprise).

If you're aiming to create a sponsored promotional campaign, Twitter can facilitate that in various ways, such as offering conversion tracking which allows you to track visitors to your website on Twitter, in order to target them with more relevant ads.

There are various third-party tools which you can explore in order to expand your data-gathering and analysis efforts, but by mastering Twitter's built-in analytics tools, you'll be able to spruce up your social strategy until you're sharing like the pros.

UK Parliament has seized a cache of Facebook documents which allegedly contain important information relating to the Cambridge Analytica data sharing scandal.

The House of Commons's Serjeant at Arms intercepted the documents from the founder of a US software company, Six4Three, while staying at a London hotel. The serjeant-at-arms gave the founder a two-hour deadline to hand over the documents in question and when he refused, he was reportedly escorted to Parliament and told he risked fines and imprisonment if he failed to comply.

The documents, which allegedly contain confidential correspondence between senior executives and Mark Zuckerberg, are said to hold illuminating information about the decisions Facebook made which led to now-defunct political strategist firm Cambridge Analytica receiving a mass of Facebook user data.

The documents were seized using rarely-used parliamentary powers and are the subject of an ongoing trial in California. The documents were with Six4Three's founder because the software firm took legal action against Facebook after investing $250,000 into an app.

Six4Three allege the cache of documents not only hold key information but shows that Facebook actively exploited the loophole in its privacy policy, the one used by Cambridge Analytica.

"We are in uncharted territory," said Damian Collins, the chair of the culture, media and sport committee to The Observer. "This is an unprecedented move but it's an unprecedented situation. We've failed to get answers from Facebook and we believe the documents contain information of very high public interest.

"We have very serious questions for Facebook. It misled us about Russian involvement on the platform. And it has not answered our questions about who knew what, when with regards to the Cambridge Analytica scandal," he said.

The seizure comes in the wake of a feud between the UK and Facebook after Mark Zuckerberg has repeatedly refused summons issued by MPs to explain Facebook's conduct around the scandal.

Cambridge Analytica was a British data analytics firm which helped political campaign target voters online. The scandal is that they used a loophole in Facebook's privacy policy to take user data and use it for purpose of voter modelling, feeding the company's ability to offer political strategy services to clients.

The mass collection of data came from 270,000 facebook profiles who completed an interactive quiz on the site and due to how access to Facebook data works, a number of their friends' data was also taken, leaving the final number of people to have their data stolen at 50 million.

It's been a busy year for Facebook's PR team who have had to deal with and deflect numerous questions about the company's conduct. Last month, it was fined 500,000 by the Information Commissioner's Office under the Data Protection Act 1998.

The string of negative press attention have seen it lose more than $100 billion in value since the start of the Cambridge Analytica scandal. More recently, it has been questioned regarding allegations that they hired a PR firm to make allegations about the financier George Soros.

US Congress has requested that ex-Cambridge Analytica employee Matt Oczkowski answer questions regarding the use of data by his new company Data Propria.

Oczkowski has been asked to explain whether the data being used by Data Propria was supplied by researcher Aleksandr Kogan, who collected data from millions of Facebook users in 2016, according to a letter sent by Democrats on the House Energy and Commerce Committee last week.

"Given news reports indicating that Data Propria is being led by former Cambridge Analytica employees, including yourself, we believe the American people must be assured that Data Propria is not using consumer data wrongfully obtained by Cambridge Analytica or engaging in other inappropriate practices," the letter said.

Not only does Congress want Oczkowski to spill the beans on whether the data it's using as collected via Facebook, Democrats including Frank Pallone, Jr., Mike Doyle and Jan Schakowsky want to understand exactly how Data Propria is getting its data from Facebook, data brokers or other means.

It also requested that Data Propria advise Congress on whether or not it's telling consumers how their data is being used and if it's using the data for consulting purposes. For companies in the US, there's no legal requirement to disclose this information if customers request it, unlike the stringent rules imposed on those in the EU by GDPR.

"You have acknowledged in interviews with press that the work of Data Propria will be very similar to Cambridge Analytica," the group continued in their letter. "The admitted overlap between the personnel and work of Cambridge Analytica and Data Propria raises serious concerns about Data Propria's practices regarding the collection and use of Americans' personal information."

05/06/2018: SCL Elections ignores watchdog order to hand over user's data

Cambridge Analytica has refused to comply with a deadline to hand over the data it holds on a US citizen, throwing a regulatory battle between itself and the Information Commissioner's Office (ICO) into "uncharted waters".

The ICO had given the now-defunct political consultancy firm until Monday to comply with a legal notice after Professor David Carroll filed a Subject Access Request (SAR) under the Data Protection Act 1998 to discover what information the company held on him.

But the organisation's data controller, SCL Elections, failed to respond to the ICO's enforcement order - with the UK data regulator now considering prosecution.

"We are looking at the apparent failure to comply with our Enforcement Notice by SCL Elections Limited and its directors," an ICO spokesperson told IT Pro. "A decision on options for next steps, including any decision on prosecution, will now be taken in line with the prosecutors code."

Despite being a US citizen, Professor Carroll launched a landmark test case in 2017 under UK data protection law, after learning that Cambridge Analytica had processed his personal data in the UK as part of its alleged profiling of US voters for work on the Trump presidential election campaign. His case, separately filed with the High Court, was backed by the ICO, with its outcome bearing implications for millions of Americans whose data had also been harvested by the company.

However, Cambridge Analytica did not recognise his right to submit a SAR as a non-UK citizen, and refused to provide the full extent of the data it held. Consequently the ICO last month issued a legal notice for SCL Elections to comply in full - or face prosecution.

"My story is a peculiar example of how the typically American deference to the tech companies and a reluctance to consider comprehensive privacy rules contrasts with strong data protection regimes, enforced by a muscular regulator equipped with the necessary tools," said Carroll, giving evidence before the European Union's (EU) Civil Liberties, Justice and Home Affairs (LIBE) committee yesterday.

He revealed that Cambridge Analytica administrators had not yet provided the ICO with the login credentials to recover his data from its servers, which the regulator seized earlier this year as it investigates the use of data in political campaigns.

He later tweeted: "The company did not respond to the @ICOnews Enforcement Order due today. We are now in uncharted waters."

Information commissioner Elizabeth Denham, also giving evidence before the LIBE committee, outlined the scale of the ICO's probe into the misuse of data in political campaigning, branding it the "largest investigation ever undertaken by a data protection authority".

Involving 40 full-time ICO staff as well as a further 20 external legal and forensic IT recovery experts, the enquiry is examining more than 30 separate organisations, and the actions of around a dozen individuals.

"We are investigating social media platforms, data brokers, analytics firms, political parties and campaign groups and academic institutions," she told the committee, adding: "We are looking at both regulatory and criminal breaches. We are working with other regulators, EU Data protection authorities and law enforcement in the UK and abroad."

Denham added the ICO has seized dozens of servers containing hundreds of terabytes of data from several searches, and has conducted dozens of interviews.

"We are looking at the complete range of sanctions at our disposal at this time including our new powers under the new UK Data Protection Act for no-notice inspections, quicker warrants, to compel delivery of evidence and to seal digital evidence where it cannot be immediately recovered," she continued.

Cambridge Analytica, which specialised in data analysis and strategic communication, shut down in early May, blaming "a siege of media coverage" that drove away its customers and suppliers, as it commenced parallel bankruptcy proceedings in the US.

"Over the past several months, Cambridge Analytica has been the subject of numerous unfounded accusations," the company said at the time, adding that despite trying to "correct the record" it has has been "vilified for activities that are not only legal, but also widely accepted as a standard component of online advertising in both the political and commercial arenas".

The ICO has made clear that companies shutting themselves down must still comply with subject access requests like Professor Carroll's. The regulator will publish a report outlining the wider implications of its ongoing investigations before the end of June, which will include a series of recommendations on regulatory gaps in areas such as data protection.

Picture: Shutterstock

08/05/2018: ICO orders firm to hand over data to US citizen

The UK's data protection watchdog has served a legal notice to Cambridge Analytica, ordering it to give a US academic all of the personal data it holds on him.

Professor David Carroll, who is based at the Parson School of Design in New York, had filed a Subject Access Request (SAR) under the terms of the Data Protection Act 1998 to Cambridge Analytica after the scandal broke that the data modelling company had allegedly amassed 240 million Americans' Facebook profile data.

However, he only received information that showed how the company had scored him on certain political categories, such as gun control and national security, leading him he to believe the information was incomplete and launch legal action.

Cambridge Analytica's data controller is UK-based company SCL Elections, and Professor Carroll took his case to England's High Court, and also filed a complaint with the UK Information Commissioner's Office (ICO).

The ICO said that not only does it believe Cambridge Analytica held back information, but pointed out that the firm did not offer an adequate explanation of where it had obtained Professor Carroll's data, or how it would be used.

However, in a statement, information commissioner Elizabeth Denham, said that SCL Elections was unwilling to cooperate with its investigation over Professor Carroll's personal data.

"The company has consistently refused to cooperate with our investigation into this case and has refused to answer our specific enquiries in relation to the complainant's personal data - what they had, where they got it from and on what legal basis they held it," she said.

"The right to request personal data that an organisation holds about you is a cornerstone right in data protection law and it is important that Professor Carroll, and other members of the public, understand what personal data Cambridge Analytica held and how they analysed it."

The regulator's decision means potentially millions of US Facebook users whose data was harvested by Cambridge Analytica could make similar demands of the organisation.

Both Cambridge Analytica and SCL Elections have filed for insolvency after the Facebook data harvesting scandal and have questioned the ICO's jurisdiction given the data in question belongs to an American citizen, but the ICO doesn't agree and has warned the companies could face criminal charges for withholding the information.

"We are aware of recent media reports concerning Cambridge Analytica's future but whether or not the people behind the company decide to fold their operation, a continued refusal to engage with the ICO will potentially breach an Enforcement Notice and that then becomes a criminal matter," Denham added.

While Cambridge Analytica is closing, some of the people involved in the firm have started another venture called Emerdata.

Picture: Shutterstock

19/04/2018: Alexander Nix may face formal summons to Parliament

Alexander Nix, the suspended CEO of Cambridge Analytica, may face an official summons to appear before the Digital, Cultural, Media and Sport Committee after refusing to attend voluntarily.

The committee wanted him to address inconsistencies in testimony he gave in February stating that Cambridge Analytica did not have or work with Facebook users' data. Shortly afterwards, The Observer revealed that Cambridge Analytica allegedly had access to tens of millions of Facebook profiles.

Nix's lawyers said on Tuesday, however, that Nix was "not able to give evidence tomorrow as a consequence of him having been served with an information notice and being subject of a criminal investigation by the Information Commissioner's Office".

In response, committee chairman Damian Collins said he may order Nix to give more evidence to the committee.

"It is certainly the intention of the committee to take this further and consider issuing a summons for Mr. Nix to appear on a named day at some point in the near future," said Collins, who warned Nix last month on the severity of giving false statements to a select committee.

Though Nix did not appear, Cambridge Analytica whistleblower Brittainy Kaiser delivered written testimony and documents to the committee on Tuesday.

Kaiser wrote that Arron Banks of Leave.EU collected the personal data of people he sold car insurance to through his company GoSkippy, and that this data may have been used to target possible Brexit supporters.

"I do not believe Banks's written evidence to this inquiry is a full account of what happened," said Kaiser, submitting proposals for projects under Banks, along with supporting emails.

Cambridge Analytica also used quizzes to mine data on users, including a "sex compass" quiz, according to Kaiser's testimony.

"I do not know the specifics of these surveys or how the data was acquired or processed," said Kaiser.

She started the #OwnYourData campaign to push Mark Zuckerberg to change Facebook's terms of service so users get more rights over their own data. The campaign, which has 147,000 signatures, asks Zuckerberg to respond by the end of April. Facebook today introduced changes to its terms and conditions that bring them in line with the EU's impending GDPR data protection legislation.

12/04/2018: Zuckerberg claims ignorance about 'shadow profiles'

Facebook founder Mark Zuckerberg was grilled by members of a US congressional committee over the social network's use of so-called "Shadow Profiles".

Shadow Profiles are said to be data collected by Facebook about non-users of the network, such as email addresses, names tagged in photos, or details in a smartphone contact list. This data is hidden from users but used to recommend friends and new connections as well as social data analysis. This data is not normally disclosed to account owners but is used in Facebook's "People You May Know" feature.

The existence of shadow profiles came to light in 2013 when a bug in Facebook showed not just contact details of a user's friend's in a download file but also their friend's shadow contact information.

At the hearing, of the US House Energy and Commerce Committee, New Mexico Representative Ben Lujan asked Zuckerberg about these profiles.

Zuckerberg replied that "in general we collect data on people who have not signed up for Facebook for security purposes to prevent the kind of scraping you were just referring to."

Zuckerberg denied he was familiar with the term "Shadow Profile". Lujan then asked how many data points on average Facebook collects on its users. Zuckerberg replied that he did not know.

Lujan then asked how many data points Facebook collected on non-users. Again, Zuckerberg said he did not know. Lujan then asked how someone who does not have a Facebook account opt out of Facebook's involuntary data collection?

The Facebook CEO responded that "anyone e can turn off and opt out of any data collection for ads, whether they use our services or not." He then suggested that Facebook still needed to collect non-user data for security reasons.

"In order to prevent people from scraping public information [...] we need to know when someone is repeatedly trying to access our services," he told the committee.

Lujan pointed out to Zuckerberg that non-users wanting to request what data it has on them need to go a Facebook page and sign up for an account to access their data.

11/04/2018: Zuckerberg deflects US regulations in first Senate hearing

Facebook CEO Mark Zuckerberg emerged relatively unscathed from the first of two US Senate hearings, managing to deflect attempts to enforce tougher regulations on the social media platform.

During a grilling by a joint committee of 44 US Senators, the 33-year-old internet mogul took full responsibility company failings that led to the improper sharing of data on approximately 87 million Facebook users, however, made no commitments to supporting any specific reactionary legislation drawn up by Congress.

Zuckerberg began with an opening speech that was released ahead of schedule, followed by a series of questions aimed at probing Facebook's business model and the events leading to the exploitation of the platform by Cambridge Analytica.

He admitted that the company had failed to audit the agreement drawn up with Cambridge Analytica, and said that it had become apparent that the firm had misled Facebook to exploit its algorithms.

He also conceded that following the discovery of the data breach in 2015, failing to notify both the Federal Trade Commission and the public immediately was a mistake, though he believed the case to be "closed" after dealing with the issue internally.

On the question of whether Zuckerberg would support the introduction of legislation Zuckerberg said:

"I'll have my team follow up with you so that way we can have this discussion across the different categories where I think this discussion needs to happen."

Shares in the company fell sharply on the news of the Cambridge Analytica scandal, however, investors were clearly happy with Zuckerberg's performance on Tuesday, as shares saw the single biggest daily jump in almost two years, up by 4.5%.

It could be argued that some Senators' questions failed to probe enough into the failings behind the scandal, as many were targeted at Facebook's business model or the way it currently manages data on the platform.

At one point, it appeared Zuckerberg was answering basic questions on how an internet company operates. Senator Orrin Hatch asked: "How do you sustain a business model in which users don't pay for your service?"

"Senator, we run ads," said Zuckerberg.

This was shortly followed by a question by Senator John Kennedy on whether the company would work on providing better ways for users to delete their data, only for Zuckerberg to highlight that such functions already exist.

It wasn't all smooth sailing for the CEO however. At around 2 hours into the hearing, Senator Lindsey Graham from South Carolina launched a barrage of questions on the subject of market monopolies and customer choice, frequently interrupting Zuckerberg before he could finish answering.

On the question of what competitors Facebook has, Zuckerberg attempted to offer up rival tech companies such as Google, Apple and Twitter, only for the Senator to dismiss them as he believes Facebook offers a unique service.

Graham: "I'm talking about is there real competition you face. Because car companies face a lot of competition. If they make a defective car, it gets out in the world, people stop buying that car, they buy another one. Is there an alternative to Facebook in the private sector?"

Zuckerberg: "Yes Senator, the average American uses 8 different apps...to communicate with their friends and stay in touch with people, ranging from text to email."

Graham: "OK, which is the same service that you provide."

Zuckerberg: "Well, we provide a number of different services."

Graham: "Is Twitter the same as what you do?"

Zuckerberg: "It overlaps with a portion of what we do."

Graham: "You don't think you have a monopoly?"

Zuckerberg responded with a pause, before adding that "it certainly doesn't feel like that to me!", prompting laughter from the chamber.

While this moment was perhaps the toughest for Zuckerberg, other Senators were clearly displeased with his answers.

Writing on Twitter after the hearing, Senator Kamala Harris said: "Mark Zuckerberg's failure to answer several critical questions during his appearance before the Senate today leaves me concerned about how much Facebook values trust and transparency."

Tuesday's hearing was the first of two for Zuckerberg, who is also due to appear before the House Energy and Commerce Committee on Wednesday.

Speaking on Tuesday, Senator John Thune said a separate hearing would be held specifically on Cambridge Analytica, and other companies that may have been involved with the improper sharing of data on Facebook.

10/04/2018: Zuckerberg apologises for data scandal ahead of Congress hearing

Facebook CEO Mark Zuckerberg has apologised for failings in his company's data protection policies that led to 87 million users having their information improperly shared to third-parties, according to documents released yesterday ahead of his Congressional hearing on Tuesday.

Zuckerberg, who is due to appear before Congress to answer questions relating to improper data sharing deals on the social media site, also took responsibility for failing to stop the spread of fake news to influence public opinion.

"Facebook is an idealistic and optimistic company," said Zuckerberg, in a transcript compiled by CNBC. "For most of our existence, we focused on all the good that connecting people can bring.

"But it's clear now that we didn't do enough to prevent these tools from being used for harm as well. That goes for fake news, foreign interference in elections, and hate speech, as well as developers and data privacy."

"We didn't take a broad enough view of our responsibility, and that was a big mistake," he added. "It was my mistake, and I'm sorry. I started Facebook, I run it, and I'm responsible for what happens here."

In the documents released ahead of Tuesday's hearing, Zuckerberg said the company was committed to making changes to policy that would "significantly impact our profitability", but these would take some time to implement.

The scandal surrounding Facebook and its data policies show no sign of abating. Recent reports from New Zealand suggest that as many as 63,714 users could have had their information exposed to Cambridge Analytica's data harvesting, after only 10 people took part in an online personality quiz.

John Edwards, New Zealand's data protection commissioner, said he was urgently reviewing the case and would seek answers from Facebook on how the data was being used by Cambridge Analytica, according to the Guardian.

Edwards reportedly deleted his Facebook account upon hearing news of the incident and has said that New Zealanders should consider doing the same.

Apple's co-founder, Steve Wozniak, told USA Today that he would also be deleting his Facebook account over concerns that the social media site was becoming increasingly careless with user data.

"Users provide every detail of their life to Facebook and ... Facebook makes a lot of advertising money off this," he said to the newspaper. "The profits are all based on the user's info, but the users get none of the profits back."

Zuckerberg will appear before a joint hearing of the US Senate Judiciary and Commerce committee on Tuesday 10 April at 2:15 pm ET (7.15pm BST), and again before the US House Energy and Commerce Committee on Wednesday at 10 am ET (3 am BST). Details on how to watch Tuesday's hearing can be found here.

06/04/2018: Sheryl Sandberg admits 'mistakes'

Weeks after the Cambridge Analytica scandal first started making headlines, Facebook's chief operating officer (COO), Sheryl Sandberg, has finally broken her silence on the matter.

Speaking to the Financial Times, Sandberg admitted the company had underinvested in safety and security but added that this attitude was changing. Indeed, there has been a flurry of announcements this week from senior figures detailing changes that have been made to the way the company shares data and giving users more granular control over what data Facebook holds on them (read on below).

"We made mistakes and I own them and they are on me," Sandberg told the FT, adding: "There are operational things that we need to change in this company and we are changing them ... We have to learn from our mistakes and we need to take action."

Although Facebook admitted earlier this week that around 87 million of its 2.2 billion users are thought to have had their data scraped by Cambridge Analytica, Sandberg said the company is in the dark as to what exactly was taken, because it can't undertake an internal probe until the one being conducted by the Information Commissioner's Office (ICO) in the UK is completed.

Sandberg also acknowledged that she and CEO Mark Zuckerberg should have spoken out on the matter sooner, but added that they "wanted to make sure [they] knew exactly what happened" before making any public statements.

She also claimed that Russian meddling in the US election in 2016 through disinformation and incendiary posts took the social platform's top executives by surprise.

"Things happened in 2016 which we were too slow to understand. Now we are following and investigating very thoroughly," she said.

Her comments come after Zuckerberg held a press Q&A in which he admitted the company hadn't done enough to ensure third-party apps were using people's data responsibly and in line with Facebook's terms and conditions.

"In retrospect, I think we clearly should have been doing more all along," he said.

Facebook yesterday revealed new measures to bring Facebook's data protection policies in line with the EU's tougher rules coming into force next month, including making the policy easier to understand, and explaining how and why Facebook uses people's information.

05/04/2018: Facebook admits it 'could have done more' to scrutinise third-party apps

Mark Zuckerberg has admitted that Facebook "should have been doing more" to ensure third-party companies were using the social network's data responsibly, as revised figures suggest as many as 87 million users may have been affected by the Cambridge Analytica scandal.

The admission came as Facebook's chief faced a grilling from the press over the company's data protection practices and how much it scrutinised those services accessing social media data.

"In retrospect, I think we clearly should have been doing more all along," said Zuckerberg, speaking at the conference on Wednesday. "I'm not trying to defend this now: I think our view in a number of aspects of our relationship with people is that our job is to give them tools, and that it was largely people's responsibility how they chose to use them.

"I think it was wrong in retrospect to have that limited of a view," he added. "I think we need to take a broader view of our responsibility."

Facebook now believes as many as 87 million users may have had their data improperly shared with Cambridge Analytica as a result of past policies, some 37 million more than previously thought.

"We wanted to take a broad view that is a conservative estimate," said Zuckerberg. "I am quite confident that given our analysis that it is not more than 87 million. It very well could be less, but we wanted to put out the maximum we felt that it could be as that analysis says."

In response to concerns about its data protection policies, Facebook has said it now intends to roll out EU General Data Protection Regulations worldwide, dismissing earlier reports that the measures would be diluted for US users.

Facebook also announced a string of updates to its API that aim to restrict the access third-parties have to user data. This includes a new App Controls link at the top of a user's news feed that gives an overview of the data they're sharing with third-party applications. As part of this addition, Facebook will also be telling users if their data had been improperly shared with Cambridge Analytica.

From now on, if a user allows an app to access data on the events they're attending, that app will no longer have free reign to scrape information from the list of users also interested in the event.

Apps will also no longer be able to use the Pages API to access posts or comments on any page by default, and instead only those "providing useful services to our community" will be able to petition Facebook for approval.

The ways in which Facebook groups work are also changing. Instead of a single member being able to give permission for an app to access group content, any third-parties using the API will need to be approved by Facebook first.

The ability to search for a person using their phone number or email address has also been scrapped, after it was found that the feature was being abused to harvest profile data, and any application seeking to access a user's feed, including likes, posts and photos, will need to first ask Facebook.

Last week it was revealed that Facebook had been storing user call and text histories as part of an opt-in feature on the Android version of Facebook. Although it appears the company will still maintain this practice, Facebook has said it will review the data it collects to ensure no message content is stored, and that it will delete all logs older than one year. Broader data, such as the time of calls, will now no longer be collected.

Zuckerberg revealed that his company had opened an investigation into the data Cambridge Analytica data agreement, as it still doesn't know exactly what information the firm held on its systems. However, this has since been put on hold while the UK's Information Commissioner's Office conducts its own investigation, which includes the examination of files taken during a raid on Cambridge Analytica's London headquarters in March.

Zuckerberg is due to appear before Congress next week to defend company policies, having already refused to testify before a UK parliamentary committee.

26/03/2018: Cambridge Analytica: Watchdog examines 'evidence' after London office raid

The UK's data watchdog raided Cambridge Analytica's London headquarters on Friday evening after finally getting a search warrant approved by a High Court judge.

Twenty officials from the Information Commissioner's Office entered the Oxford Street address at 8pm to look for evidence relating to allegations the analytics firm harvested data from 50 million Facebook users without their consent.

The property search lasted seven hours, and an undisclosed volume of evidence was taken from the property, the ICO said.

"We will now need to assess and consider the evidence before deciding the next steps and coming to any conclusions," an ICO spokesperson said in a statement. "This is one part of a larger investigation by the ICO into the use of personal data and analytics by political campaigns, parties, social media companies and other commercial actors."

The ICO announced it was investigating the allegations against Cambridge Analytica on 17 March, and issued a Demand for Access to records two days later. However, the judge adjourned the case from last Wednesday to last Friday, meaning a warrant wasn't granted until late that evening. Media reports showed pictures of boxes being carried out of Cambridge Analytica's shared offices prior to the warrant hearings.

The search forms part of wider public scrutiny into whether Facebook did enough to secure the data it held on its users, as well as whether data was abused to influence elections.

A former Cambridge Analytica director told the Guardian on Friday that the firm's management deliberately misled the British public about its involvement with a pro-Brexit group, Leave.EU, during the referendum campaign period.

In an attempt to mitigate fallout, Facebook took out full-page adverts in both UK and US newspapers over the weekend, in which CEO Mark Zuckerberg said the incident was a "breach of trust" and said he was "sorry we didn't do more at the time".

His comments were later echoed by COO Sheryl Sandberg, who told CNBC on Friday that the case "was a critical moment for our company". "We are going to do everything we can," added Sandberg. "I would definitely have had Mark and myself out speaking earlier, but we were trying to get to the bottom of this."

Friday's raid kicked off a turbulent weekend for the social media company, which saw scathing opinion polls and fresh revelations of data misuse.

A Reuters poll in the US found that less than half of Americans now trust Facebook to adhere to US privacy laws, while 60% of users in Germany believe social networks are negatively impacting the democratic process, according to national newspaper Bild am Sonntag.

There were also multiple reports of users downloading their account files stored on Facebook's website, only to discover that call data from mobile phones was also being logged, including who they called and for how long, seemingly without their knowledge.

Facebook has since refuted the claim that this was done secretly, pointing to a usage agreement as part of its Messenger platform that allows users to opt-in to call and text history logging.

Zuckerberg has now been called to appear before a UK parliamentary committee to give evidence on the Cambridge Analytica's use of personal data. The committee has demanded a response by the end of today.

22/03/2018: ICO must wait for warrant

UK data watchdog the Information Commissioner's Office (ICO) will not get access to Cambridge Analytica's London offices until Friday at the earliest, after its demand for a warrant to search the premises was adjourned.

In a statement today, an ICO spokesperson said: "A High Court judge has adjourned the ICO's application for a warrant relating to Cambridge Analytica until Friday. The ICO will be in court to continue to pursue the warrant to obtain access to data and information to take forward our investigation."

While IT Pro asked the regulator why it was adjourned, a spokesperson declined to comment further. However, Guardian journalist Carole Cadwalladr reported that the judge granted an extension because Cambridge Analytica's legal counsel was unavailable.

It means the ICO's planned raid of Cambridge Analytica's offices is further delayed, despite reports earlier this week claiming boxes were being removed from the firm's shared premises on Tuesday.

Meanwhile, Facebook CEO Mark Zuckerberg has finally broken his silence over allegations that Cambridge Analytica accessed the data of 50 million Facebook users without their consent, seeking to reassure users that a similar abuse of people's information could never happen again.

The scandal, which broke earlier this week, surrounds the use of a quiz developed for data mining and an analytics firm that harvested the data not just of those who used the app (thisisyourdigitallife), but also of all their Facebook friends.

Several days after the story first broke, Zuckerberg finally issued an official response through Facebook, which includes a timeline of events and an explanation of how an app could have accessed data from individuals who didn't have any direct contact with it.

The Facebook CEO said: "The good news is that the most important actions to prevent this from happening again today we have already taken years ago.

"In 2014, to prevent abusive apps, we announced that we were changing the entire platform to dramatically limit the data apps could access. Most importantly, apps like [thisisyourdigitallife] could no longer ask for data about a person's friends unless their friends had also authorized the app.

"We also required developers to get approval from us before they could request any sensitive data from people. These actions would prevent any app ... from being able to access so much data today."

However, he added that the company is taking further steps to tighten up how it guards user data, including revoking apps' access where users haven't used them for three months, and reducing the data first given to apps on sign in to just the user's name, profile picture and email address.

This is particularly significant as part of the broad range of data collected by Cambridge Analytica included very sensitive data such as sexuality, location, religion and political leanings, and apps could access this data indefinitely.

Zuckerberg also claimed that Cambridge Analytica has once again affirmed that it has deleted all the data linked to this case something it previously said it had done in 2015, but which an Observer article subsequently contradicted with the account of former employee and whistleblower Christopher Wylie.

Cambridge Analytica has voluntarily agreed to a full forensic audit to ensure this is the case, Zuckerberg added.

Meanwhile, Wylie, the ex-director of research at Cambridge Analytica who blew the whistle on this scandal, took to Twitter to confirm he would be accepting invitations to testify in front of several legislative bodies in the US and UK.

20/03/2018: ICO seeks warrant to raid London offices

The UK's data privacy watchdog is seeking a warrant to raid Cambridge Analytica's London headquarters in order to investigate allegations that the company illegally harvested the information of 50 million Facebook profiles.

The Information Commissioner's Office (ICO) confirmed to IT Pro this morning that it is seeking an urgent warrant following the company's failure to comply with its request on 7 March for access to its databases and records.

"Cambridge Analytica has not responded to the commissioner by the deadline provided; therefore, the information commissioner is seeking a warrant to obtain information and access to systems and evidence related to her investigation," an ICO spokesman said.

The news follows an expos by the Observer that claimed Cambridge Analytica had used the Facebook data of more than 50 million people in order to build a powerful predictive analytics algorithm that could accurately predict political preferences, sexuality and other personal traits.

It has also emerged that Facebook was in the process of conducting its own internal investigation into Cambridge Analytica's actions separately to the ICO, which it ceased yesterday at the regulator's behest. Damian Collins, chair of the culture select committee, expressed doubt as to Facebook's motives in conducting its own investigation, telling the BBC that "This is a matter for the authorities".

"The concern would have been, were they removing information or evidence which could have been vital to the investigation? It's right they stood down but it's astonishing they were there in the first place."

Facebook is planning an open meeting today, to let staff grill the company's management on the unfolding incident. According to the Verge, the meeting will be led by Facebook deputy general counsel Paul Grewal, who will give staff background information on the case and field questions from staff.

It will mark the first time that staff have had an official opportunity to ask questions of the company's leadership since the scope of Cambridge Analytica's use of Facebook data was publicly revealed - although Facebook has admitted it was aware of the issue in 2015, a year after it happened.

The meeting is planned to last just half an hour, and an employee told the Verge that the move felt like a temporary solution to buy time until Friday's all-hands meeting where founder and CEO Mark Zuckerberg - who has thus far been silent on the issue - is expected to address employees. Collins has called on Zuckerberg or a senior Facebook executive to give evidence to his Parliamentary committee on the scandal.

To understand what the Cambridge Analytica and Facebook scandal is all about, read our explainer.

Facebook maintains that this is not a data breach in the traditional sense, and is instead a misuse of legitimately-obtained data. Security expert Graham Cluley agreed with the assessment, but said it shows fundamental issues with Facebook's design. "From Facebook's point of view, it's not a traditional data breach," he told IT Pro. "That's because this is how Facebook was designed, and many apps over the years have scooped up users' information and (privacy settings permitting) those of their friends as well."

"Hundreds of millions of times every day Facebook hones the content it displays to you based on what it has determined you are interested in, who you are, and what it thinks will be most effective. So, it's not that different from what Cambridge Analytica does with the same access to the data," he added.

"None of this is news. Facebook has been working this way for years. The only way to reduce your exposure is to refuse to play Facebook's game and not be a member of the site. If you can't bring yourself to leave, at the very least lock down your privacy settings and reduce the level of information that you share."

Cambridge Analytica's CEO, Alexander Nix, was also caught in an undercover sting by Channel 4, which recorded Nix seemingly bragging about using ex-intelligence operatives, micro-targeted propaganda and 'honeypot' traps to influence elections and other political proceedings.

"We're used to operating through different vehicles, in the shadows, and I look forward to building a very long-term and secretive relationship with you," Nix told undercover reporters posing as potential clients in an initial phone call.

"We have two projects at the moment, which involve doing deep, deep depth research on the opposition and providing... really damaging source material, that we can decide how to deploy in the course of the campaign," he said as part of a later meeting recorded by Channel 4.

Cambridge Analytica denied that it uses any of the methods alleged by the investigation, and stated that it was simply 'humouring' the potential client.

19/03/2018: Millions of Facebook profiles 'mined illegally' by Cambridge Analytica

The personal data of 50 million Facebook users was leaked to a data analytics firm and used to help Donald Trump win the 2016 US presidential election, it has been claimed.

Cambridge Analytica used this information to build sophisticated software models in 2014 to target US voters with political advertising, according to a whistleblower who said he helped collect the data.

Facebook, which denied the incident amounted to a data breach, confirmed it knew about it in 2015 but didn't inform affected individuals or data protection regulators.

Whistleblower Christopher Wylie said he worked with a Cambridge University academic to collect user data via a third-party Facebook app called thisisyourdigitallife.

Aleksandr Kogan created the app through his company Global Science Research (GSR) and worked with Cambridge Analytica to pay 270,000 Facebook users to take part in a personality test. They also agreed to have their data, such as their home city and Facebook 'likes', collected for academic use.

But then-Analytica employee Wylie told the Observer, which broke the story, that the app also collected data belonging to Facebook friends of the users who signed up to the quiz, producing a pool of data of tens of millions of people.

"We exploited Facebook to harvest millions of people's profiles," Wylie told the newspaper. "And built models to exploit what we knew about them and target their inner demons. That was the basis the entire company was built on."

Cambridge Analytica claimed it had deleted all the data shared by GSR when it discovered this information was collected in breach of Facebook's terms of service and had no reason to believe the data was obtained illegally.

"In 2014, we contracted a company led by a seemingly reputable academic at an internationally-renowned institution to undertake a large scale research project in the United States," its statement read.

"No data from GSR was used by Cambridge Analytica as part of the services it provided to the Donald Trump 2016 presidential campaign."

On the eve of the story breaking, Facebook said it has banned Cambridge Analytica from its platform.

In explaining its decision, Facebook said that while Kogan obtained people's data legitimately, by then sharing that information with Cambridge Analytica he had "lied" to the company and broken platform rules about not sharing such information with third-parties. Kogan maintains he acted legally and had a "close working relationship" with Facebook, the Guardian reports.

Facebook learned of this violation in 2015 and subsequently removed the app from its platform. It also "demanded certifications" from Kogan, Wylie and Cambridge Analytica that they had deleted the data.

However, while Wylie said he received a letter from Facebook lawyers in August 2016 to this effect, he claimed they never followed up to verify the data was deleted.

"That to me was the most astonishing thing," he said. "They waited two years and did absolutely nothing to check that the data was deleted. All they asked me to do was tick a box on a form and post it back."

The social network also failed to report the leak of data to the affected users or to US state regulators, because it claimed the event was not a data breach.

Facebook said: "Several days ago, we received reports that, contrary to the certifications we were given, not all data was deleted. We are moving aggressively to determine the accuracy of these claims.

"If true, this is another unacceptable violation of trust and the commitments they made. We are suspending SCL/Cambridge Analytica, Wylie and Kogan from Facebook, pending further information."

UK data protection regulator the Information Commissioner's Office is already conducting an investigation into the use of personal data in political campaigns and confirmed it's looking into this incident as part of the investigation.

"Our investigation into the use of personal data for political campaigns, includes the acquisition and use of Facebook data by SCL [Strategic Communication Laboratories, which is affiliated with Cambridge Analytica], Doctor Kogan and Cambridge Analytica," said information commissioner Elizabeth Denham.

"This is a complex and far-reaching investigation for my office and any criminal or civil enforcement actions arising from it will be pursued vigorously."

Damian Collins MP, chair of the Digital, Culture, Media and Sport Committee, accused Cambridge Analytica CEO Alexander Nix of misleading the committee when he told the committee while giving evidence last month that his firm had not received any data from GSR.

Collins said: "It seems clear that he has deliberately mislead (sic) the committee and Parliament by giving false statements. We will be contacting Alexander Nix next week asking him to explain his comments, and answer further questions relating to the links between GSR and Cambridge Analytica, and its associate companies."

The committee will also demand that Facebook CEO Mark Zuckerberg, or someone similarly senior, attends the committee to answer questions about the incident.

Main image credit: Shutterstock

London houses twice the total number of AI companies of Paris and Berlin combined, according to a new report launched on the eve of London Tech Week.

The UK's capital is home to 758 firms for which AI is a central part of their business models, a survey commissioned by the Mayor of London has discovered.

These companies span 30 industries, with focuses on insurance, finance and law, according to AI advice platform CognitionX, which compiled the London: The AI Growth Capital of Europe report. Investment in London's AI businesses grew by 50% last year to surpass 200 million, including a 48 million funding round for smartphone health app Babylon Health and a 27 million Series A round for authentication tool Callsign.

Meanwhile, 32% of the firms having at least one founder from a minority ethnic background, and 25% with at least one female founder.

Speaking at the launch of London Tech Week, Mayor of London Sadiq Khan said: "There are few areas of innovation that have the power to define our future economy and society more than artificial intelligence. As mayor, it is my goal to ensure both that London is at the forefront of developing and capitalising on these new technologies, and that all Londoners can benefit from the opportunities that they create.

"The research describes a city with a rich technology ecosystem, a strong pipeline of AI innovation and an academic and investment base set up for the long term."

Khan hopes to capitalise on London's AI knowledge to turn the capital into a smart city, a goal he announced last year.

Today he outlined a roadmap featuring more than 20 initiatives to achieve that aim, including proposals to bring full fibre broadband to all new homes, working with TfL and councils to roll out 4G more widely across public buildings, and adding smart sensors to public infrastructure like lamp posts, as well as supporting bids to test 5G.

The use of open data at TfL and City Hall to build apps for the public is set to continue, with Khan allocating 365,000 to improve data-sharing among public services.

"We are really looking forward to working with startups and other partners to ensure Londoners have good access to homes and can easily travel around the city," Lilli Matson, director of transport strategy at TfL, said. "This is central to quickly developing innovative solutions to deliver more homes and offer better options to encourage more journeys to be made by walking, cycling and public transport."

The mayor also plans to invest 365,000 to improve data-sharing among public services and proposes to draw up a pan-London cybersecurity strategy to safeguard people's personal information, a plan that would bring together bodies including the London Digital Security Centre and the London Cyber Innovation Centre.

London's chief digital officer, Theo Blackwell, said: "We will strike a new deal with citizens on use of their data so when it is appropriate to share data, it is done in a trusted, safe way and for public benefit."

Khan will also launch the Civic Innovation Challenge, first announced in December, to grant startups 15,000 to test ideas that are designed to tackle social challenges like inequality, climate change and the capital's ageing population.

"As one of the world's leading technology hubs, we need to be bold and think big, to experiment and try things out that have not been done elsewhere," he said. "I see London's future as a global 'test-bed city' for civic innovation, where the best ideas are developed, amplified and scaled."

The personal information of more than four million Facebook users was stored in an easily-accessible online database for the last four years, it has been revealed.

The database, which was created by researchers from the University of Cambridge and features highly sensitive information, was left exposed with minimal protections.

Facebook users' data was compiled through a Facebook app called myPersonality - a personality quiz similar to the ThisIsYourDigitalLife app that Alexandr Kogan used to collect the data that would later be passed to Cambridge Analytica.

Included in the database were details including age, gender, location and relationship status for 4.3 million people, as well as 22 million status updates from 150,000 users, according to the New Scientist, which broke the news following an investigation it conducted.

The purpose of the quiz was to gather information on users' 'Big Five' personality traits - five broad factors that are used in some branches of psychology to analyse an individual's personality. The database included the 'Big Five' scores of 3.1 million Facebook users.

While their datasets were supposedly anonymised, with users' names stripped out prior to publication, the rest of their details were still linked, making it comparatively easy to determine the users' identity, according to the publication.

The men behind the project are Michal Kosinski and David Stillwell, two researchers from Cambridge University's Psychometrics Centre - although the university said that Stillwell created the app prior to his employment.

The data was made available to other researchers who wanted to use it in their own studies, with the proviso that they use it in such a manner that the data could not be traced back to the individual participants of the quiz, while researchers from private companies could use it for non-commercial purposes provided they agreed to be bound by strict data protection policies.

Researchers from Google, Microsoft and Yahoo registered to use the data, as well as academics from around 150 universities, totalling almost 300 people. However, one university lecturer apparently gave students the credentials to access the database as part of a course on analysing Facebook data.

The login details were then uploaded to GitHub, where they were easily accessible via a simple Google search, allowing anyone to access the database. The GitHub page remained up for around four years.

Facebook suspended the myPersonality quiz last month over concerns that the language describing how data was shared violates its terms of service, and has launched an investigation to determine if a violation has taken place. The Information Commissioner's Office is also examining the case, although an official investigation has not been announced.

Stillwell told the New Scientist that over the course of the project's lifespan, it has only experienced one data breach, saying "we believe that academic research benefits from properly controlled sharing of anonymised data among the research community".

The news comes after the Cambridge Analytica scandal, in which the US analytics firm allegedly siphoned millions of users' data from an app created by Cambridge professor Aleksandr Kogan to try to influence voters in the 2016 US presidential election.

It has raised questions around what third-parties do with Facebook users' data, and the lack of controls Facebook had in place to monitor their usage of such information. Last week Facebook revealed it has suspended 200 apps amid an investigation into whether third parties misused users' information.

The world of tech has always been fast moving, but things have accelerated of late and we're heading ever closer to another skills gap.

As new technologies and delivery mediums co-exist at least for now with legacy and old-world processes and techniques, it becomes increasingly hard for IT professionals to keep their skill set relevant to both the demands of today and tomorrow.

Absolutely, some legacy skills will transition into the new world, although while some may be quite different to what we recognise today, but new skills will also be required.

Here's our run down of the future technology skills you should be paying attention to and upskilling in if you haven't started already:

AI

Artificial intelligence (AI) is a the tech industry's current equivalent of marmite. There are many positives to be gained through increased use of AI across the board, whether at work or home, but for every advocate, there are many who fear a sentient takeover or massive job displacement.

As Jane McCallion recently highlightedAI has become a real bone of contention, with industry experts also talking about how we're all doomed as a result.

Research by The University of Oxford suggests that just over one third of jobs will be automated thanks to tech such as AI in the next 20 years. However, that doesn't mean new jobs won't be created. After all, technology can and does go wrong and needs regular health checks. Perhaps we will see a new role emerge, that of the AI doctor, to help balance out role displacement elsewhere?

Whatever your view on AI, it's an issue that's not going away, so you really do need to start getting a better understanding of it and it's implications for you and your business now. CIOs, in particular, need to grasp the balance between the need for human skills and where AI can play a role.

Biometrics

As the security battleground hots up and your data increasingly becomes the next currency of any real value, many are turning to biometrics to beef up their protection.

Biometrics, put simply, uses human elements (such as gestures or body parts), rather than passwords, to gain access to a system or systems. It's not fool-proof, though, and can be bypassed a notable example would be the villains in the film Angels & Demons ripping out a CERN professor's eyeball to gain access to the lab containing antimatter.

That's probably one of the reasons many still distrust biometrics as a credible, regular alternative to passwords and the like.

However, regardless of whether you are for or against biometric usage, it's a good security field to study as, if the world moves further in that direction, it could well ensure you don't get your P45.

Cyber security

Cyber security is absolutely the next battleground. It's a broad umbrella term, so there are many sub areas you can look to up-skill in, ranging from a transfer of existing security practices right the way up to shoring defences for online practices that may not even exist yet.

Data forensics, for example, is becoming a growing and very interesting field of study and demand for such skills is only set to go up rather than down.

Rather than act as a spoiler for another piece that wonderfully details the cyber security skills your business needs, here's a link to that article.

Data science/big data

With all this structured and unstructured data being created as organisations digitally transform themselves and increasingly do business online, someone has to make sense of it.

Obtaining the data is key as is protecting it and then accessing it, but understanding the data and gaining insight to help inform better decisions is a fine art. And it's a fine art that is in danger of being hit by a skills shortage due to lack of supply vs demand.

Given the gap, those with the right skills are likely to be able to soon name their price when it comes to employment.

Machine learning

Machine learning as a concept started off a bit like AI in terms of the level of scepticism out there. However, discussions around the topic have now matured and businesses are starting to realise the benefits on offer.

Far from it being a case of the robots coming, taking over our jobs and killing the human race off, the conversation has now moved to one of real-world application.

But machines break and they need maintenance and fixing. They also need to be told what to do in the first place and work alongside us to deliver increased value, so there's definitely a really valid role humans can still play skills-wise in the new dawn of machine learning.

Software development

Software isn't going anywhere. In fact, as we transition to a future with more apps, devices and machines, the role of the developer becomes ever-more important.

Increasingly, organisations are turning to agile development to get the job done both well and quickly a trend that's set to continue.

The power of what can be achieved by software in the future is limited only by our imagination and the availability of the right software developers, so it's an exciting time to get in on this scene.

Soft skills

While we've focused the list thus far on the tech skills that will keep you in gainful employment, it's important to note that softer skills such as dealing with people, showing empathy and understanding different personality types will become ever more important as automation and robotics pervade the workforce of the future.

We will be updating this article regularly as new trends and research findings emerge so please do check back at a later date for the latest information.