Meta Learning from Learning Curves 2

The Meta Learning from Learning Curves challenge is an academic challenge in the 2022 part of the MetaLeran Series of data challenges run by Chalean. a non-for-profit organization lead by Isabelle Guyon (INRIA) in collaboration with TAILOR. The challenge is that of a portfolio of learning algorithms / hyperparameters: it is then possible to run in parallel several of them and to dynamically decide after every evaluation which one to try next, choosing between exploitation (continue with the current best performing) or exploration (try some yet untested algorithm)?

Whereas the first round of this Data Challenge attempted to learn from classical learning curves performance vs learning time (see Figure below, left), the second round used the performance vs dataset size curves (Figure below, right). The question the Data Challenge tries to answer is: learning on a fraction of the whole dataset can be viewed as a proxy for full learning – but how good a proxy is it, and how small can the training dataset be without excessively damaging the performance?

Calendar

  • May 16, 2022: Public phase, on public meta-dataset.
  • May 23, 2022: Development phase, the submissions were meta-trained and meta-tested on 15 hidden datasets. 
  • July 4, 2022: Final/test phase, the last submission of each participant was meta-learned and meta-tested on 15 fresh hidden datasets, never seen before, giving the final ranking of competitors.
  • July 11, 2022: End of competition, start or the Legacy Phase: all test datasets became public, and the Data Challenge became an Open benchmark. 
  • July 15, 2022: winners have been announced at AutoML conference.  

Links