About AnandaBot

Overall presentation

Recent approaches in conversational AI for social assistive robots primarily focus on a human-robot model, where the robot merely acts as a substitute for a human agent. In AnandaBot we propose to explore a human-(human/robot) approach, where the robot accompanies and complements the human agent.

Much like Ananda for his mentor, Buddha, the sidekick robot will enhance the abilities of its human counterpart and support the actions of its mentor in a triadic relationship with the user to address collaborative tasks. The aims of this approach are to:

  1. enhance human acceptance of a robot serving in a limited role as an assistant;
  2. promote research on the technical requirements for such interactions (in contrast to traditional « autonomous » systems): multi-party management, speakers’ input separation, dual conversation handling, and more.
Example of a disciple helping a psychologist to run tests on a patient

A robot placed in an assistant role will be in a vantage point to observe and learn from its mentor. This « internship » vision of the setup will eventually enable processes for progressively updating the interaction modules.

Application scenario

Within the study framework at AP-HP, the assistant robot is expected to support two types of personnel in a specialised gerontology healthcare service:

  • psychologists — by helping facilitate cognitive assessment tests, positioning professionals as observers while the test occurs between the patient and the support robot;
  • nurse aides — who can use the robot as a source of knowledge in their interactions with patients in the waiting room (appointments, delays, general information), including in a multilingual context.
Waiting room at Broca Hospital

The proposed algorithms will be implemented on a general-purpose robot prototype and experimentally validated in one of the waiting rooms of the Broca Hospital (AP-HP). Usage scenarios will be developed for real-world testing, gradually increasing in complexity.

Ethics, privacy and sovereignty

A primary focus will be on developing local (on-premises) and low-resource solutions providing sovereignty and data-privacy protection for users. Cloud-based solutions for voice treatment will be avoided. Data recordings will include only volunteers who have given informed consent; personal data will be stored on a secure server with restricted access, and the detailed experimental protocol will be submitted for approval to the CNIL and the Ethics Committee associated with the Broca hospital.

Partners

Avignon Université — Laboratoire d’Informatique d’Avignon (Coordinator)
Human-machine vocal interactions, dialogue management, decision-making learning, speech technologies (SpeechBrain).
lia.univ-avignon.fr

LIA

Avignon Université — Laboratoire d’Informatique d’Avignon (Coordinator)
Human-machine vocal interactions, dialogue management, decision-making learning, speech technologies (SpeechBrain).
lia.univ-avignon.fr

INRIA — RobotLearn Team, Inria Grenoble Rhône-Alpes
Audiovisual perception, machine learning for social robotics, continual learning.
team.inria.fr/robotlearn

INRIA

INRIA — RobotLearn Team, Inria Grenoble Rhône-Alpes
Audiovisual perception, machine learning for social robotics, continual learning.
team.inria.fr/robotlearn

CNRS / Université Paris-Saclay — Laboratoire Interdisciplinaire des Sciences du Numérique
Speech emotion recognition, affective computing, ethics of human-machine interaction (AI Chair HUMAAINE).
lisn.upsaclay.fr

LISN

CNRS / Université Paris-Saclay — Laboratoire Interdisciplinaire des Sciences du Numérique
Speech emotion recognition, affective computing, ethics of human-machine interaction (AI Chair HUMAAINE).
lisn.upsaclay.fr

Assistance Publique – Hôpitaux de Paris — Broca Hospital, Lusage Living Lab
Healthcare application in gerontology, experiments with older adults and health professionals, psychological survey.
lusage.org

AP-HP

Assistance Publique – Hôpitaux de Paris — Broca Hospital, Lusage Living Lab
Healthcare application in gerontology, experiments with older adults and health professionals, psychological survey.
lusage.org

Key persons

Fabrice LefèvreLIA, Avignon Université — Professor, PI & Scientific Coordinator, Head of the Vocal Interactions Group, Director of LIA
Yannick EstèveLIA, Avignon Université — Professor, speech recognition & understanding
Mickaël RouvierLIA, Avignon Université — Associate Professor, speech technologies
Xavier Alameda-PinedaINRIA RobotLearn — Research Scientist, team leader, audiovisual perception (site coordinator)
Karteek AlahariINRIA — Senior Research Scientist, continual learning
Laurence DevillersLISN, CNRS/Univ. Paris-Saclay — Professor, affective computing (site coordinator)
Anne-Sophie RigaudAP-HP — Professor, Head of the Geriatric & Research Department, Broca Hospital (site coordinator)
Maribel PinoAP-HP — PhD in Cognitive Ergonomics, Director of Broca hospital’s lab
Sébastien Da CunhaAP-HP — MSc in Experimental Cognitive Psychology, researcher at Broca Living Lab

The permanent staff will be joined by PhD students, post-docs and research engineers recruited by each partner.

Organization

To address the overall objective of AnandaBot — a human-(human/robot) approach where the robot accompanies and complements the human agent — the project will develop an advanced perception-(pro)action processing chain enabling voice exchanges between robots and humans to carry out a triadic collaborative task. The strategy hinges on two subsidiary scientific objectives:

  1. enable autonomous interactions between (humanoid) robotic platforms, mimicking situations of interest, to allow simulation-based training and evaluation;
  2. develop on-site solutions for all modules to enhance users’ privacy.

The work programme is broken down into five work packages:

WP1 — Continual audiovisual robot perception INRIA, LIA

Methods and algorithms to continuously extract human-behaviour-understanding cues from audio and visual data: robustness against real-world perturbations (with quantitative reliability estimation) and continuous adaptation to their variations over time. Tasks: behaviour understanding from visual inputs; continual audiovisual adaptation; adaptation to simulated environments.

WP2 — Speech analysis: recognition, understanding, synthesis and emotion LISN, LIA, INRIA

Methods and algorithms to extract meaning from audio data. Tasks: automatic speech recognition, understanding and synthesis; emotion detection and analysis; adaptation to simulated environments.

WP3 — Engagement strategy for triadic human-(human/robot) interactions LIA, INRIA, LISN, AP-HP

Develop the natural vocal interaction ability of the robot, learned in situ through online training in a continual setup, in both real-world and simulated environments. Tasks: task-oriented and open-world chatbot; engagement strategy; adaptation to simulated environments.

WP4 — Specifications, integration, iterative and final evaluation AP-HP, INRIA, LIA, LISN

Ensure compliance with the Ethical Committees, define the experimental protocol, specify laboratory and real-world protocols for « Wizard of Oz » experiments, integrate WP1–WP3 software modules and conduct the final experiments at Broca Hospital.

WP5 — Dissemination, exploitation and management LIA, all partners

Disseminate the project’s scientific results (journals, conferences, tutorials, workshops), manage the Advisory Board, and handle scientific, administrative and financial management. All modules developed inside the project will be made available through an open-source repository by the end of the project.