“Imagine a tourist arriving in a foreign city,” the Israeli intel officer tells me. “The first thing they do is open Google Maps and look for a restaurant. Google helps them find a place. Helps them navigate. Helps them get there on time. We do the same.” Well, not exactly. The augmented reality mapping application Lt. Col. “N” is describing is designed to find hidden terrorists, not restaurants. “Mistakes can be fatal,” he tells me, “we need to get the right house on the right street.”
Welcome to the battlefield of the future – artificial intelligence, multi-source data fusion, augmented reality. Everything edge-based and real-time. Except this isn’t really a battlefield, as such. “What happened to us,” the officer tells me, “is that our enemies have adopted a technique to merge into urban areas populated with civilians, we need to unveil the enemy, precisely, to neutralise them and stop the threat.”
So, now you start to get the picture. This is counterterrorism powered by AI, this is about creating an asymmetry, shifting the balance of power. Think Google Street View – except it’s not Google. And an augmented reality overlay that comes from the fusion of multiple sources of highly classified intelligence, not big tech’s cloud servers. And if that isn’t enough, there’s also AI running pattern analytics on prior enemy tactics, techniques and procedures to infer what a hidden enemy is likely to do next, in real time.
This military augmented reality is not unique – such systems are already under development, gaming-style headsets overlaying friendlies and likely combatants, helping targeting and the avoidance of blue on blue. Israel’s new system is different, though. The augmented reality comes from the fusion of multiple intel sources, the intent is not to present ground troops with an advanced gaming-style view of the battlefield, but to use live data to infer where actual targets are hiding.
Nowhere is this more important than in an urban environment. This isn’t just a matter of where the enemy might be hunkered down based on visuals and a map of the terrain. This can fuse data sources to understand the ownership, history and usage of specific buildings, the results of prior surveillance activities. And the millisecond new data is received and processed, the entire map updates for everyone involved.
Picture this Street View lookalike again – no screenshots, I’m afraid, it’s classified. Arrows and graphics explain to a soldier on the ground why the third-floor apartment with the wrought iron balcony is deemed a hostile environment, why anyone exiting the building can be considered a combatant. The intent is to root out threats, but also to keep others safe, to avoid collateral damage. “We need to make sure we only target the aggressor and not any civilians,” LTC “N” tells me.
This convergence between real-life conflict and ever more realistic gaming-style graphics has been developing for years now. During the Iraq conflict, there was that inference that here’s a generation of young soldiers more inured to the horrific visuals of a battlefield than previous generations by the combat games they’ve played. You can add to this the remote piloting of lethal drones on kill missions, all from pods thousands of miles away.
Israel’s idea for this “intelligence saturated combat” has been a decade in the making. The new program sits within Unit 9900, the visual intelligence operation (think of maps, satellite imagery, image analysis) within Aman, the country’s military intelligence directorate, and sister unit to the better known 8200 signals intelligence unit. Unit 9900 generated headlines a few years ago when it was reported that it was recruiting autistic teenagers for their unique analytical skillset.
As LTC “N” describes the work of his team, “the development of 3D mapping that is as realistic as possible,” he continually refers back to the modern-day explorer’s Google Maps view of the world, that feeling of familiarity. Yes, the location might be strange, but the viewpoint is well known, understandable in real-time. “We have to build something with that user experience,” he says, “our soldiers crossing the border for the first time must be familiar with the environment.” It’s hard not to conjure images of gaming graphics as he says this – that level of familiarity.
This “intelligence saturated” viewpoint can be presented to the solider on a smartphone or tablet, all off the shelf and “mostly Android,” or streamed directly into their binoculars or weapons sights. “They don’t know where the intel comes from,” LTC “N” tells me, “but it reaches their sights, their C2 systems in real time.” The officer stresses that all targeting decisions are taken by the soldier on the ground, not by the system itself, this is an aid, not an automated targeting system.
That differentiation is critical. IoBT – the Internet of Battlefield Things – envisages a mix of human operators and autonomous machines, all powered by fast communication networks, algorithms that empower rapid decision-making based on data and inferences. But the final decision sits with a human operator, the world isn’t ready – at least not publicly – to fully pitch man against machine.
The challenges the new unit has overcome, I’m told, include distilling this intel, “terabytes every day,” into what is useful and relevant. That’s the role of the AI, the pattern analytics. The window is short – soldiers are given five to ten seconds to decide on any action they take. They are trained in the field with the technology, their feedback hones the program itself, “what to develop further and what to ditch.”
And the beauty of AI is that the more data you have, the larger and more diverse your datasets, the more powerful your thinking machine becomes. Currently this is early-stage – fast forward a few years, though, and that asymmetry between the haves and the have nots where this level of tech is concerned will be staggering.
But back to the here and now – putting the complexity to one side, this is a 3D, photo-realistic map, “the backbone onto which we build our intel – preliminary and real time – to understand the area and what the enemy is doing in real time.” By mining data from previous combat experiences, the AI “recognizes patterns of enemy behaviour – and can understand where the enemy is and what they’re planning.” This is overlaid with real-time intel, including open-source data on the terrain and the environment.
There has been a lot of talk about the fusion of the cyber and physical domain in the last year, not least from Israel, which became the first country to mount a physical military response to a cyber-attack. A few weeks later, the U.S. did the opposite. This new concept of an “intelligence saturated battlefield” can take the cyber domain and feed it directly to troops on the ground. Those same soldiers are connected with sensors, everything feeding back to the central intelligence system.
LTC “N” often refers to the “disappearing enemy.” He means the urban shadows where combatants and civilians blend together, disasters waiting to happen. Yes, this new style of AR combat is intended to sharpen responses, but also to avoid mistakes. The officer explains that the AR display provides enough information to let soldiers understand why a location has been deemed hostile – but the final targeting decision is theirs, and if they don’t understand they won’t act the right way.
I’m told that this new program within Unit 9900 has become a development hotbed, learning its approach from industry. Inside the “joint lab” you’ll find intel, combat troops, cyber and communications, Israel’s Defense Advanced Research Projects Agency (DARPA) equivalent, defense contractors, even start-ups. When something new is envisaged, it’s prototyped and given to ground troops to field test. Their feedback hones the capability or consigns it to the bin.
The military world has changed, LTC “N” tells me. “We needed to imagine new methods of fighting – as much as possible we use tools created outside the defense industry. We take civilian and open-source as much as possible, we access research from all over the world to help us deliver state of the art products.”
You can think of this mix of real-reality and augmented reality, of converged commercial tech and mil-spec systems, of autonomous machines mixing with human operators as Battlefield 2.0. And while today this might seem to be all about augmented overlays, it’s really about painting an AI-driven picture for the soldier to empower decision making.
Imagine a vast array of connected sensors linked to a thinking machine that can compare what it’s seeing in real time with all the battlefield experience that has gone before. Nothing is a better predictor of the future than the past. Lower level surveillance monitoring decisions made autonomously by an AI module – identifying risks and potential hostiles, saving military time, resources and potentially lives.
I’m told that Israel has accepted that “mil-spec” is not always best – why not plug into the billions of investment dollars piling into mapping and AR and AI, repurposing those capabilities for this? “We keep the user experience as straightforward as possible… Google Maps is a good model – how you see the world as a tourist, when you know what you see and understand where you’re going.”
This new program is now ripe for international collaboration. “Our discussions with various countries fighting terror around the world show they’re facing the same threat, enemies hiding in urban environments. This concept brings together quick intel, enhanced by AI and connected to accurate mapping. That’s its innovation.”
No details on any other countries using the tech, of course, no specifics on intel sources – all highly classified. “I can tell you this is a real-time bridge between intel and soldiers (intel wants to keep its secrets), combat operators want that intel in real time.” Testing of the new capabilities started this year.
You can add this IDF program to the multitude of new AI, IoT and AR systems being procured and developed by military customers world-wide. The concept of real-time dissemination of live intel from multiple sources, right to a soldier’s C2 or weapon’s sights is novel. The challenge is that the soldier must remain the decision maker. The biggest takeaway from any system like this is chilling – If there’s ever any implication that targeting has been automated, that a kill switch or trigger has been handed over to the lightning reflexes of a machine, then the military world will have changed and there will be no going back.
Zak Doffman is Chief Executive Officer of Digital Barriers, a provider of advanced surveillance technologies to the international homeland security and defense markets.
