Cross-Domain MetaDL: Any-Way Any-Shot Learning Competition with Novel Datasets from Pratical Domains

Meta-learning aims to leverage the experience from previous tasks to solve new tasks using only little training data, train faster and/or get better performance. The proposed challenge focuses on “cross-domain meta-learning” for few-shot image classification using a novel “any-way” and “any-shot” setting. This challenge is part of TAILOR WP2 (see more information here).

Goal

The goal is to meta-learn a good model that can quickly learn tasks from a variety of domains, with any number of classes also called “ways” (within the range 2-20) and any number of training examples per class also called “shots” (within the range 1-20). We carve such tasks from various “mother datasets” selected from diverse domains, such as healthcare, ecology, biology, manufacturing, and others. By using mother datasets from these practical domains, we aim to maximize the humanitarian and societal impact.

What

Create a few-shot learning classifier capable of tackling image classification problems from a wide variety of domains (objects, vehicles, plants, animals, OCR, satellite images, microscopy images, etc.), in the range [2-20] classes and [1-20] training examples per class. Meta-Learn the classification algorithm by using meta-training data, not used to carve out tasks on which it will be (meta-)tested, but from similar domains.

Where

The competition is with code submission, fully blind-tested on the CodaLab challenge platform. A single (final) submission will be evaluated during the final phase, using ten datasets previously unused by the meta-learning community. After the competition is over, it will remain active to be used as a long-lasting benchmark resource for research in this field.

How

Join the Cross-Domain MetaDL challenge, https://codalab.lisn.upsaclay.fr/competitions/3627 open until September 1, 2022.

Special 4-hour bootcamp webinar:  https://bit.ly/3ypnGnB Saturday, July 16, 2022 at noon UTC (7 am Bogota, 2 pm Paris)

Competences

The scientific and technical motivations of this challenge include scalability, robustness to domain changes, and generalization ability to tasks (a.k.a. episodes) in different regimes (any-way any-shot).

Why

Get a chance to:

  • get up-to-speed, with our didactic on-line Meta-Learning tutorial (including the method which won our previous MetaDL challenge) and with our white paper describing the challenge and baseline results;
  • have a productive summer and propel your research to the forefront: we’ll invite top ranking participants to co-author a paper on post-challenge analyses, to be published in the NeurIPS proceedings of the competition track;
  • win in any or all 5 challenge leagues: Free-style, Meta-learning, New-in-ML, Women league, Participant of a rarely represented country (see rules);
  • enter a fair competition with code evaluated in controlled conditions on the challenge platform, in 2 phases with different datasets; the code of the participants outperforming the baseline method in the first phase have their code evaluated a single time on fresh data, invisible to the participants, in the final phase.

Timeline

  • Jun 15, 2022: Public phase starts.
  • Jul 1, 2022: Feedback phase starts.
  • Sep 1, 2022: Final phase starts.
  • Oct 1 – 15, 2022: Notification of winners.

Contacts

metalearningchallenge@googlegroups.com

Organizing committee and Acknowlegments

Dustin Carrión (LISN/INRIA/CNRS, Université Paris-Saclay, France)
Ihsan Ullah (LISN/INRIA/CNRS, Université Paris-Saclay, France)
Sergio Escalera (Universitat de Barcelona and Computer Vision Center, Spain, and ChaLearn, USA)
Isabelle Guyon (LISN/INRIA/CNRS, Université Paris-Saclay, France, and ChaLearn, USA)
Felix Mohr (Universidad de La Sabana, Colombia)
Manh Hung Nguyen (ChaLearn, USA)
Joaquin Vanschoren (TU Eindhoven, the Netherlands)

Find more information about this challenge (e.g., Competition Overview, Competition Phases, Prizes) on the dedicated websites:
https://metalearning.chalearn.org
https://codalab.lisn.upsaclay.fr/competitions/3627