AIML Special Presentation: HARNESS: Hierarchical Abstractions and Reasoning for Neuro-Symbolic Robotic Systems—From Perception to Autonomy

Abstract: In this talk, I will present HARNESS: Hierarchical Abstractions and Reasoning for Neuro-Symbolic Robotic Systems—From Perception to Autonomy, showcasing our work under the DARPA Assured Neuro-symbolic Reasoning (ANSR) program. This project focuses on building fully autonomous systems that unite perception, reasoning, and planning through neuro-symbolic frameworks, emphasising explainability, performance robustness and assurance. I will first discuss NEUSIS and NEUSIS++, which were evaluated during the first and second phases of the ANSR program. These systems tackle realistic UAV search missions involving Entities of Interest (EOIs)—like “a red SUV vehicle” or “a pedestrian carrying a blue umbrella”—in complex suburban and urban settings with hazards and keep-out zones (KOZs). Their compositional neuro-symbolic design integrates visual perception, reasoning, and grounding with a probabilistic world model and a hierarchical symbolic planner. Experimental results in AirSim and Unreal Engine show they outperform a state-of-the-art (SOTA) vision-language and symbolic planning baseline in success rate, search efficiency, and 3D localisation. Next, I will introduce our recent compositional neuro-symbolic visual grounding method, NAVER, which merges advances in vision-language models, neural foundation models, probabilistic logic reasoning, and a finite-state automaton with a self-correcting mechanism. NAVER enhances robustness and interpretability and achieves SOTA performance compared to neural and neuro-symbolic baselines. If time allows, I will conclude with Hier-SLAM: Scaling-up Semantics in SLAM with a Hierarchically Categorical Gaussian Splatting, which unifies semantic SLAM under a neuro-symbolic framework—an important step toward integrated perception and mapping in the broader context of HARNESS.

Bio: Dr. Hamid Rezatofighi is a Senior Lecturer in the Faculty of Information Technology at Monash University, Australia. His research spans computer vision, machine learning, and robotics, with notable contributions to advancing robot visual perception and reasoning in dynamic environments. Previously, he was an Endeavour Research Fellow at Stanford's Vision and Learning Lab (SVL) and a Senior Research Fellow at the Australian Institute for Machine Learning (AIML) at the University of Adelaide. Dr. Rezatofighi completed his Ph.D. at the Australian National University in 2015. He has authored over 90 publications, including more than 25 high-quality journal articles and over 50 contributions to premier conferences such as CVPR, ECCV, ICCV, NeurIPS, AAAI, IJCAI, ICRA, and IROS. His work has received over 14,500 citations, with 14 of his papers (four as first author) cited more than 100 times, and two exceeding 1,000 citations. Since 2020, he has served as an area chair for major conferences (CVPR, NeurIPS, ECCV, ICCV, IJCAI, and IROS) and holds positions as a Senior Associate Editor for IEEE Transactions on Image Processing and as an Associate Editor for the Artificial Intelligence Journal. These roles reflect his active engagement in the academic community. Over the past four years, Dr. Rezatofighi has secured more than $17 million in research funding, including significant grants from three DARPA projects (one as Lead PI, two as co-PI) and as lead CI for an ARC Discovery Project. His ongoing work continues to drive innovations in computer vision and robotics.