Focus Sessions

Focus Session #1 – Neuroergonomics

Overview: Learn about the newly emerging field of neuroergonomics and how wearables and other devices can enhance your research through enabling various capabilities such as human-aware autonomy, cognitively aided design, and ultimately cognitive situation management.

---

Focus Session #2 – Semantic Forensics

Overview: Learn about the state of the art of detecting mis- and disinformation through this focus session highlighting the algorithms and techniques developed on the DARPA SemaFor program. This session will include an overview of the DAC model of detection, attribution, and characterization of manipulate media, a view into the interfaces being developed for operational use, and deep dives into newly developed algorithms going up against the latest versions of generative AI techniques.

---

Focus Session #3 – University of Pennsylvania GRASP lab

Overview: The General Robotics, Automation, Sensing and Perception (GRASP) Laboratory is an interdisciplinary academic and research center within the School of Engineering and Applied Sciences at the University of Pennsylvania. Founded in 1979, the GRASP Lab is a premier robotics incubator that fosters collaboration between students, research staff and faculty focusing on fundamental research in vision, perception, control systems, automation, and machine learning. This focus session will highlight relevant and cutting-edge work the GRASP lab has been doing in the field of Situation Management.

Talks

Title: A Picture of the Prediction Space of Deep Networks

Presenter: Pratik Chaudhari

Abstract: Deep networks have many more parameters than the number of training data and can therefore overfit—and yet, they predict remarkably accurately in practice. Training such networks is a high-dimensional, large-scale and non-convex optimization problem and should be prohibitively difficult—and yet, it is quite tractable. This talk aims to illuminate these puzzling contradictions.

We will argue that deep networks generalize well because of a characteristic structure in the space of learnable tasks. The input correlation matrix for typical tasks has a “sloppy” eigenspectrum where, in addition to a few large eigenvalues, there is a large number of small eigenvalues that are distributed uniformly over a very large range. As a consequence, the Hessian and the Fisher Information Matrix of a trained network also have a sloppy eigenspectrum. Using these ideas, we will demonstrate an analytical non-vacuous PAC Bayes generalization bound for general deep networks.

We will next develop information-geometric techniques to analyze the trajectories of the predictions of deep networks during training. By examining the underlying high-dimensional probabilistic models, we will reveal that the training process explores an effectively low dimensional manifold. Networks with a wide range of architectures, sizes, trained using different optimization methods, regularization techniques, data augmentation techniques, and weight initializations lie on the same manifold in the prediction space. We will also show that predictions of networks being trained on different tasks (e.g., different subsets of ImageNet) using different representation learning methods (e.g., supervised, meta-, semi supervised and contrastive learning) also lie on a low-dimensional manifold.

References:

• 1. Does the data induce capacity control in deep learning? Rubing Yang, Jialin Mao, and Pratik Chaudhari. [ICML ’22] https://arxiv.org/abs/2110.14163
• 2. Deep Reference Priors: What is the best way to pretrain a model? Yansong Gao, Rahul Ramesh, and Pratik Chaudhari. [ICML ’22] https://arxiv.org/abs/2202.00187
• 3. A picture of the space of typical learnable tasks. Rahul Ramesh, Jialin Mao, Itay Griniasty, Rubing Yang, Han Kheng Teoh, Mark Transtrum, James P. Sethna, and Pratik Chaudhari [ICML ’23]. https://arxiv.org/abs/2210.17011
• 4. The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold. Jialin Mao, Itay Griniasty, Han Kheng Teoh, Rahul Ramesh, Rubing Yang, Mark K. Transtrum, James P. Sethna, Pratik Chaudhari. 2023. https://arxiv.org/abs/2305.01604

Bio: Pratik Chaudhari is an Assistant Professor in Electrical and Systems Engineering and Computer and Information Science at the University of Pennsylvania. He is a core member of the GRASP Laboratory. From 2018-19, he was a Senior Applied Scientist at Amazon Web Services and a Postdoctoral Scholar in Computing and Mathematical Sciences at Caltech. Pratik received his PhD (2018) in Computer Science from UCLA, and his Master’s (2012) and Engineer’s (2014) degrees in Aeronautics and Astronautics from MIT. He was a part of NuTonomy Inc. (now Hyundai-Aptiv Motional) from 2014-16. He is the recipient of the Amazon Machine Learning Research Award (2020), NSF CAREER award (2022) and the Intel Rising Star Faculty Award (2022).

===============================

Title: Composable Representations for Lifelong Learning in Autonomous Systems

Presenter: Eric Eaton, PhD University of Pennsylvania

Abstract: Lifelong learning is a key characteristic of human intelligence, largely responsible for the variety and complexity of our behavior.  This process allows us to rapidly learn new skills by building upon and continually refining our learned knowledge over a lifetime of experience. Over the past few years, there has been rapid progress toward developing these capabilities, with composable representations showing exceptional promise for enabling lifelong learning. In this talk, I will discuss this progress and its application to autonomous systems, examining how far we have come and the open problems that still remain toward realizing the goal of lifelong machine learning.

==============

Talk title: Attentive Abstractions for Flexible Vision-Based Robot Learners

Author: Dinesh Jayaraman

Abstract: General-purpose robots of the future will need vision and learning, but such vision-based robot learning today is inflexible and inefficient: it needs robot-and-task-specific training experiences, expert-engineered task specifications, and large computational resources. This talk will cover algorithms that dynamically select task-relevant information during sensing, representation, decision making, and learning, enabling flexibilities in pre-training controller modules, layperson-friendly task specification, and efficient resource allocation. I will speak about our work on interactive perception of task rewards for RL, pre-trained object-centric visual representations that track task-directed progress, and task-relevant world model learning for model-based RL.

Bio: Dinesh Jayaraman is an assistant professor at the University of Pennsylvania’s CIS department and GRASP lab. He leads the Perception, Action, and Learning  (Penn PAL) research group, which works at the intersections of computer vision, robotics, and machine learning. Dinesh’s research has received a Best Paper Award at CORL ’22, a Best Paper Runner-Up Award at ICRA ’18, a Best Application Paper Award at ACCV ‘16, an Amazon Research Award ’21, the NSF CAREER award ’23, and been featured on the cover page of Science Robotics and in several press outlets. His webpage is at: https://www.seas.upenn.edu/~dineshj/

---