In February 2018, a number of seven authoritative institutions (such as the University of Oxford, Cambridge, Electronic Frontier Foundation among others) released a report (https://arxiv.org/pdf/1802.07228.pdf) on Malicious AI that was not really picked up by mainstream media (as of yet).
That is quite a shame, as the study deals with an interesting field of research that has gained little to no attention as of yet: the potential malicious use of artificial intelligence and machine learning. As the researchers put it, “the intersection of AI and malicious intent writ large has not yet been analyzed comprehensively”.
This in stark contrast to the positive effects of ML and AI, which are studied extensively – while the number of helpful applications continues to grow at an unheard-of rate.
What could possibly go wrong?
Developments in ML and AI show that the effort needed to perform cyberattacks could decrease significantly in the near future. Examples given in the study include speech synthesis for impersonation, but CAPTCHA-hacking is probably a bit closer to home. Fully automated services that provide solving of CAPTCHA’s already exist (try a simple Google query on captcha-hacking for some results).
Things get more serious when autonomous systems are involved, particularly in cyber warfare. A rogue legion of drones assisting in a bank robbery – or even performing one – is not something that could be considered as a desirable direction of autonomy.
A third area of risk that is mentioned by the study involves potential impact on society, such as manipulation of data to influence the masses. Think specifically targeted, completely automated news briefs that are presented to anyone fitting a certain profile – without a shady political organization behind it.
Spear phishing
A specific threat that is on the rise is spear phishing, where specific individuals are targeted with very specific information – with the intent to extract for example passwords or credit card details. Running a spear phishing scam is a laborious, manual process. With progress in AI and ML, it is possible to greatly reduce the amount of manual work involved, which potentially not only increases the number of spear phishing attacks, but also makes the application of the threat more accessible, to a larger community of scammers. Spear phishing already is costly: in 2016, a Cloudmark survey showed that the average cost of such an attack is $1.6 Mio per company, whereas of the firms that were hit by the attack 81% suffered negative impact because of it (https://www.infosecurity-magazine.com/news/spear-phishing-incident-average/).
Inherently closed
Let’s face it, data scientists are hard to find – and for the average human being (which definitely includes me) their work is difficult to understand. This, in combination with the ever increasing complexity of the underlying models, prompted a group of MIT researchers to propose a Machine Learning 2.0 (http://news.mit.edu/2018/ml-20-machine-learning-many-data-science-0306), “Machine Learning for Many”. It attempts to automate away the data scientist. This brings machine learning back into a larger domain of experts, but at what cost? Do developments like these introduce an inherent risk of reducing the number of people that understand how models work from few to none?
How soon is soon?
Even though AI and ML progress appears to speed up, this progress should be considered only as an indication of what will be possible in a near – but relatively distant – future. When looking back at the past decade or so, for example advancements in the field of robotics are not that great. Still, proponents of the “Robots are friends” school of thought use this argument – and silly examples such as the Disney Beach Bot (http://www.beachbot.ch/) to downplay progress, which could be countered with examples such as Hiroshi Ishiguro’s Geminoid (http://www.geminoid.jp/projects/kibans/resources.html). The danger of AI and ML lies in the fact that once machines reach the same level of performance as humans, they do not stop but surpass – which is demonstrated in the research paper by current machine accuracy on image recognition: near perfect and better than humans are capable of.
A way forward
With certain “simple” tasks such as image recognition, human capabilities are already surpassed by machines. For many other tasks, however, this threshold has not yet been reached. This is reflected in the research report, which states that its nature is exploratory. This suggests there is ample time to make necessary changes in how malicious use of AI is prevented.
The researchers propose a response consisting of four specific recommendations:
- Let policy makers collaborate with researchers to investigate, prevent and mitigate potential malicious uses of AI and ML;
- Researchers and engineers in the field of AI and ML should take time to focus on possible malicious side-effects;
- Identify and implement best practices;
- Expand the range of stakeholders in this discussion.
These responses present a highly academic point-of-view and appear to be driven by consensus. Increasing the number of parties involved in a discussion does not necessarily mean the response will be better, whereas over-studying a topic is quite possible – possibly even preferable from an academic point of view. Those who are attacked, however, have a different and more urgent need.
Openness
Where (the tone of) the report starts to go sour, however, is when the researchers propose to not publicly share each and every new technology, algorithm and model to make sure malicious actors do not gain access to it because of security reasons, drawing an analogy to the security industry (where it is good practice to not publish a flaw or exploit immediately after discovery). A statement such as “Rather than propose a specific solution, our aim is to foster discussion of whether and when considerations against open sharing might outweigh considerations in favor and what mechanisms might enable this” does not inform us who exactly the data is not shared with. Is it only shared with researchers? Does that imply no new technology is ever maliciously used by researchers?
The statement is, however, in line with a quote from Nick Bostrom, a Swedish philosopher in the field of artificial intelligence and existential risks: “Once a discovery has been published, there is no way of un-publishing it”.
Awareness
Apart from these (minor?) comments, the research report is a highly readable, informative, slightly alarming document that presents quite a lot of examples of (potential) malicious use. Go and read it.