Thomas G. Dietterich, one of the pioneers of the field of Machine Learning, will join the TAILOR network for a keynote talk at the 3rd TAIILOR conference, 5-6 June 2023 in Siena, Italy.
Competence Models for Machine Learning Systems, by Thomas G. Dietterich
Abstract
Every AI system, and certainly every machine learning system, should have an accurate model of its own competence. This talk will review our recent work on creating competence models for classifiers, computer vision systems, and reinforcement learning. We will discuss both input competence–is the system competent to handle a given query–and output competence–can the system provide calibrated statements about the quality of the answer it has produced?
For input competence, our focus is on detecting queries that fall outside of the training data. We apply anomaly detection algorithms to address this task. For manually-engineered features, these work well, but we have discovered that features learned by deep neural networks are often inadequate for this task. The problem is that features are only learned for “directions of variation” that are present in the training data. An anomaly that varies in some way that was invariant in the training data will not be detected.
For output competence, we have studied calibration (for classifiers) and trajectory-wise prediction intervals (for reinforcement learning). We will report progress on point-wise calibration, which is more challenging that the standard notion of set-wise calibration. Then we will describe our approach to creating trajectory-wise prediction intervals using the tools of conformal
prediction.
All of these results assume stationarity of the distribution that is generating the data. We will conclude by briefly discussing our efforts to extend this work to shifting distributions.
Biography
Dr. Dietterich (AB Oberlin College 1977; MS University of Illinois 1979; PhD Stanford University 1984) is Distinguished Professor Emeritus in the School of Electrical Engineering and Computer Science at Oregon State University. Dietterich is one of the pioneers of the field of Machine Learning and has authored more than 200 refereed publications and two books. His current research topics include robust artificial intelligence, robust human-AI systems, and applications in sustainability.
Dietterich has devoted many years of service to the research community and was recently given the ACML Distinguished Contribution and the AAAI Distinguished Service awards. He is a former President of the Association for the Advancement of Artificial Intelligence and the founding president of the International Machine Learning Society. Other major roles include Executive Editor of the journal Machine Learning, co-founder of the Journal for Machine Learning Research, and program chair of AAAI 1990 and NIPS 2000. He currently serves as lead moderator for the cs.LG category on arXiv.