One year ago, Air Force Lt. Gen. John N.T. “Jack” Shanahan became the first director of a new Pentagon office created to act as a clearinghouse for all of the U.S. military’s work on artificial intelligence. Among a raft of near-term projects the office has taken up is one deploying computer vision technology to track and combat wildfires. 

Taking tools developed for Project Maven, an initiative to analyze and identify objects on the ground from videos shot by aerial drones during the fight against the Islamic State, the Pentagon’s office known as the Joint Artificial Intelligence Center has been working with National Guard units combating wildfires in California and hurricanes elsewhere.

The office began applying computer vision tools to track natural disasters, “realizing that the way things had been done … nothing had changed in 40 years,” Shanahan said in a recent interview with CQ Roll Call. Natural disasters were still being tracked using “acetate and grease pencils in the back of a pickup truck. … No one could track wildfires in real time,” he said.

By flying drones equipped with full-motion video sensors over wildfire zones, the Pentagon was able to assist firefighters with live fire locations using specialized maps sent to hand-held devices. The program eventually will be handed off to National Guard and local firefighting units, Shanahan said.

The firefighting project also helps the Pentagon’s artificial intelligence efforts, Shanahan said, by helping understand how to ingest large quantities of live data from a highly dynamic situation, coordinate response efforts with multiple agencies and deliver results in a timely manner. “If you take wildfires, there’s applicability to what you encounter in combat,” he said.

[jwp-video n=”1″]

One of the key goals of Shanahan’s office is to change the military’s culture from the industrial-age emphasis on hardware and weapons to the digital-age emphasis on data. The United States and China are racing to gain an advantage in artificial intelligence-enabled tools that will allow their militaries to respond to threats at computer speeds rather than human decision-making times.

“This is the world we are in now,” Shanahan said. “Data-centric environment gives us a competitive advantage. How do we adapt faster than our adversary? This is about decision advantage.”

Cloud cover

Recently while visiting a data center operated by an unnamed global company, Shanahan said, he envisioned the large cluster of buildings as the equivalent of a 21st-century aircraft carrier.

But the Pentagon is still stuck with bureaucratic processes that few global corporations face. Unlike the world’s top commercial artificial intelligence companies, the Pentagon doesn’t yet have a single cloud service where all of its data resides. A $10 billion program to build such a cloud service is now stuck in a legal dispute after Amazon last year challenged the Pentagon’s decision to pick Microsoft for the work.

Amazon has said that the Pentagon’s decision in the Joint Enterprise Defense Infrastructure contract, known as JEDI, was tainted by bias after President Donald Trump repeatedly tweeted his opposition to Amazon.

The cloud server is key to the Pentagon’s artificial intelligence efforts. In its absence, Shanahan’s office is making do with smaller “bespoke” solutions but those cannot be long-term answers, he said. The Pentagon needs an enterprise cloud that comes with security and advanced tools, he said.

“There’s a reason that the biggest AI companies in the world are also the biggest cloud companies in the world,” Shanahan said. “It allows us to bring data at scale, allows us to train at scale and allows us to push out things in a continuous fashion.”

The labeling problem

The Pentagon also faces another critical hurdle that commercial companies don’t encounter.

Most artificial intelligence systems today rely on so-called “labeled” data to teach computers to identify objects. For example, medical companies send X-rays, scans and pathology samples to companies in India and China, where outsourced contract workers identify and label lesions and other medical conditions. Those labeled images are then used to train computers to identify unlabeled items in large databases.

The Pentagon cannot outsource such jobs because its data in most cases is highly classified military information that must be handled only by people with security clearances. In the early stages of Project Maven, when objects on the ground photographed by drone videos had to be identified, the Pentagon initially used intelligence analysts to label the objects, but soon it became too tedious a task, Shanahan said.

The slow manual labeling affected the performance of algorithms, which depend on vast libraries of identified objects to find new, unlabeled objects in new video footage.

[Report: Speed up drug development with artificial intelligence]

The Pentagon had to scramble to get contractors with security clearances to label items in video feeds, scaling up the database to about 40 million items and thereby improving the algorithmic performance, Shanahan said.

High-quality labeled data is the holy grail for a fully artificial intelligence-enabled military, Shanahan said. “That may be as high on the list of what gets us to the future of AI-enabled capability faster than anything else.”

Military services have given up on promising artificial intelligence projects in their portfolios because they couldn’t scale up the labeling of objects, Shanahan said.

But it doesn’t mean the Pentagon would have to employ a vast army of contract workers to manually label objects.

Shanahan said he sees promising developments in the field of unsupervised learning, or so-called one-shot learning, in which a computer can identify objects by being exposed to just one identified image or object instead of requiring millions of such objects.

New projects for 2020

For 2020, the Pentagon office is showcasing several projects to demonstrate how artificial intelligence-enabled tools can help the military and make his office the “1-800-AI for the department,” Shanahan said.

The projects include a predictive maintenance program for helicopters, tools to address troops’ health and evaluations of the efficacy of commercial cybersecurity products used by the Defense Department.

Machine learning tools also can help fix outdated Pentagon forms, Shanahan said.

In July last year, Rep. Alcee L. Hastings, D-Fla., alerted the Pentagon to death certificates issued to servicemembers that identified their race as “red” for Native Americans, “negroid” and “yellow Asian.” The forms, presumably designed during the 1940s, continued to be in use and were upsetting families receiving them.

Shanahan said his office was asked, “Can we go through all the [Pentagon’s] forms and find those offensive terms and help remove them?”

A couple of data scientists deploying artificial intelligence-enabled software were able to look through and identify changes to the forms that could “save 10,000 labor hours a year” over manual methods, Shanahan said. The project is continuing to fully implement the fixes.

Despite some early successes demonstrated by the Pentagon office, which was created by an act of Congress, the Joint Artificial Intelligence Center doesn’t have enough resources and authorities to fulfill its mission, according to an assessment by Rand Corp. published in late December.

Shanahan said the report’s critique was accurate and the Pentagon planned to allocate more money for the artificial intelligence center in the 2021 budget proposal. For fiscal 2020, Congress approved $268 million for the office.