Plan recognition is the process of "understanding" sensor data (Scripts or sequences of Scenes) by reducing the wealth of sensor information to the Goals and Plans of one or more Agents who cause the changes of the Objects by executing their Plans. Plan recognition together with 3D Reconstruction and the Object Configurator form the Perception Pyramid.
Plan recognition takes as an input a Script and returns a list of Plans (together with a confidence score), potentially explaining the agent's behavior in the script. Any errors in the object classification during 3D Reconstruction will appear here as a script that "doesn't make sense". This will trigger an active investigation of the Script, its objects and the available Plans using the higher reasoning faculties in the Control Room. During this process, new information may be added to the system:
- In case of clearly identified objects and an unknown plan, the plan library (part of the Episodic Memory) will be extended by a "non-understood plan". Repeated occurrences of similar cases may lead to the system to investigate the situation or actually ask the participating agents about the plan ("What are you doing there?"), resulting in a new plan added to the library.
- In the case of objects being recognized with low confidence scores (bad light, scene happens far away, ...), the subsystem will look at plausible Plans matching the overall setting, and revise the object recognition process based on these assumptions ("Is there really a monster in the dark, or is it just a piece of wood?").
Generating and Checking Plan Candidates
We propose a hybrid process for plan recognition including statistical and symbolic elements. The generation of plan candidates is assumed to work statistically consisting of the occurring objects, their attributes and the type of actions. In a second step the plan candidates are checked in detail with the Planner simulating the Plan execution by the observed Agents in order to refine the identified plan or to detect contradictions between the observed and reconstructed plan. The subsystem can take into account the specific goals of the observed Agents using the Social Reasoning subsystem. Unless specific information exists, TinyCog will assume that the Agent's plans are identical with its own plans.
This subsystem is mainly designed to work with optical sensor input. However, the same algorithms also apply to acoustic sensor input, when the noise input is pre-processed to provide information about the 3D distribution of noise source and the underlying objects are identified by the type of noise they create. As an example consider noises coming from a neighbor's kitchen, allowing you to conclude whether the neighbor prepares dinner or a cake. LIDAR data or other sensors could be treated in the same way. Finally, the same scene reconstruction algorithm is applied when listening to the natural language generated by an agent.
Comparison and References
Plan recognition has been used in a variety of scenarios since [Wilensky 1978]. [Carberry 2001] provides an overview of various approaches. Various authors have proposed specific approaches for recognizing HTN plans.
TinyCog currently doesn't implement any plan recognition. We plan to introduce plan recognition on the Hunter Domain in order for hunters to predict the plans of deer ("it's going to graze in position X/Y"). In the next phase, we would use plan recognition for social interaction between hunters.