Over the last few years, developments in artificial intelligence (AI) have been able to achieve, and even exceed, human performance on a wide range of tasks. However, hackers – often backed by rogue states and criminal gangs – have also developed ways to leverage and trick AI systems into facilitating cyberattacks. There is no substitute for permanent vigilance when it comes to preventing attacks on your AI systems. “People talk about using AI as a powerful way to ensure cybersecurity, as machine learning tools can continually scan IT systems to detect malware. This is indeed true in general, because automation is needed in cybersecurity. Yet the same tools can also be used by hackers to trick systems,” explained , chief scientist and head of the within the University of Luxembourg’s (SnT).
Understanding AI Attacks
Working to explain AI decisions and counteract malicious processes is a direction that research teams (TruX and ) at SnT have been exploring in recent years. “To conduct smart attacks at scale, you need some kind of intelligence to help,” said Prof. Bissyandé – and cybercriminals have discovered how to pick even the most sophisticated locks. The danger is that system owners and administrators gain a false sense of security when AI tools are deployed, increasing the chances of falling prey to ransomware or data leaks.
The danger is that system owners and administrators gain a false sense of security when AI tools are deployed, increasing the chances of falling prey to ransomware or data leaks.
He highlighted the example of an AI-powered facial recognition software. These systems use models to recognise images – but if a hacker can infer how a model is built, that perpetrator can use the model’s internal logic against itself. This can enable the hacker to disguise themself as an approved user. Furthermore, with respect to privacy, trained AI models can face membership inference attacks. This is where hackers discover which samples have been used to train the model. Another rising concern is deepfakes – images, speech or videos that are artificially generated as events that never occurred, or based on people that do not exist.
Hacking through Artificial Intelligence
Crucially, hackers can detect the responses triggered by machine learning systems, and this can give clues to the operation of the model at the heart of the cybersecurity system. By compiling these clues, internal patterns of the system can be rebuilt, so that inputs can be designed that are able to trick the system, and gain access to data and processes. This is known as an adversarial attack. Machine learning system programmers train the AI with data to get the system used to certain inputs, with a view to triggering actions and outputs in an intelligent, automatic fashion. When the system then receives new information, it will categorise this in a particular way and trigger a predefined response.
A hacker may understand how machine learning works, and can learn about the models by feeding them carefully crafted data inputs. On the basis of responses received, they can build understanding of its architecture. “With this information, you can craft new samples that are completely novel for the AI model, or almost identical to expected inputs, enabling the hacker to pass through,” Prof. Bissyandé explained. ‘Although this sounds relatively complex, he characterises this challenge as actually being very easy to create a sample that bypasses malware detectors. Essentially they are manipulating the features so that the system does not see the malicious nature of the sample.’
Once an attack has been successful, a hacker can seek to extract even more information about the system, steal data, block the system – often with ransomware – or work to shut down or cause permanent damage to the system. There is also the problem that once an attack is detected it undermines trust in how all users are identified and how they can access different data and applications.
Preventing Attacks
“When technicians design their machine learning based cyber protection systems, they can choose to operate as either a ‘black box’ or a ‘white box’,” Prof. Bissyandé explained. “With a white box you know how the AI model – typically statistical – operates, the nature of its algorithm, and how data interacts with it,” he continued. This makes the system easier to crack, but its behaviour is explainable so it can be fixed more readily. However, many choose to operate their systems as closed ‘black boxes’, where the architecture is complex (and leads to higher performance). These can be harder to crack than white boxes, and require greater trial and error, but knowing when a hacker has invaded the system is nearly impossible.
Prof. Bissyandé is concerned that there is a degree of complacency about the effectiveness of AI-based cybersecurity protection systems. “AI is not foolproof because it must constantly adapt to evolving threats, which requires permanent retraining of AI models – and even sometimes simply a better understanding of the nature of malicious intent.”
AI is not foolproof because it must constantly adapt to evolving threats, which requires permanent retraining of AI models – and even sometimes simply a better understanding of the nature of malicious intent.
He added that the development of deepfake technologies is potentially an even bigger problem for the future regarding security. Hence, while progress can be made on the technical side regarding how malware can be detected and neutralised, there can be no let-up in the need for organisations to create a culture geared towards promoting cybersecurity. Education is needed at all levels to understand the dire need for cybersecurity awareness and the limits of tools whether AI-based or not.