“The better the strategy, the more points they score. Based on its previous experience, the algorithm learns over time how it should behave in which situation.” But how can an observer judge the criteria on which the AI’s behaviour ​​is based and how good its decisions are? To assess this, the researchers devised a simple experiment. First, they trained the computer to play Pac-Man, but secretly modified the rules according to which the points were awarded. In one case, for example, the character did not lose any points when it died. The algorithm trained in this way (the researchers also refer to it as an “agent”) was therefore not impressed by any ghosts nearby when making its decisions. For a second agent, they changed the value of the biscuits; a third, on the other hand, played according to the normal rules. "We have now asked test subjects to assess the three agents," Huber explains. “They were not allowed to watch several complete games though, but were only shown brief excerpts.” On this basis, the test subjects were asked to indicate which of the agents they would most likely allow to compete for them in a Pac-Man game. They were also asked to briefly describe the strategy of all three AI processes in their own words. “We wanted to find out whether the test subjects had understood why the algorithm was performing certain actions,” says the computer scientist.

Summary of the "most dramatic" game scenes helps the most

To this end, the participants were divided into four groups. Each was allowed to look at five three-second excerpts from the games of the three agents. For the first group, these short clips were chosen at random. For the second group, a kind of “attention map” was also inserted into the random short clips. It showed which influences in its environment the agent was paying particular attention to at that moment. The third group, on the other hand, saw the most “dramatic” game scenes - those in which the agent's decision had a particularly large impact (for example, could lead to the death of the game character or a particularly high points score). In AI research, this is also referred to as a “strategy summary”. In the fourth group, this summary was supplemented by the attention map. The result of the experiment was clear. "Test subjects who saw the summary were most likely to develop a feeling for the strategy of the respective agent as a result of this," explains Huber. “The attention cards, on the other hand, were of significantly less help to them. Even in combination with the summary, they only gave rise to a small additional benefit.” It was, he said, cognitively very demanding to look at the game excerpts and at the same time to watch out for the information from the attention cards. “We assume that their contribution would be greater if the information were better presented.” The researchers now want to investigate how the attention cards could be optimised in such a way that, together with the strategy summary, they make an AI agent’s decisions even more comprehensible.

Publication

• Tobias Huber, Katharina Weitz, Elisabeth André, Ofra Amir: Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps. Artificial Intelligence, Volume 301, 2021, https://doi.org/10.1016/j.artint.2021.103571.