Second TAILOR AutoAI workshop
On June 3rd, 2021, the second TAILOR AutoAI workshop took place, with close to 35 participants. All TAILOR work packages were represented. An important part of the workshop was the two poster sessions with a total of 14 posters. Below, the the programme and links to the posters are shared to give a preview of (some of) the exciting work happening in TAILOR WP7 AutoAI.
Programme
9.00 – 9.20 Workshop introduction 9.20 – 9.40 Poster highlights 9.40 – 10.05 Poster Session 1 10.05 – 10.30 Poster Session 2 10.30 – 10.50 BREAK 10.50 – 11.00 Breakout Session introduction 11.00 – 11.50 Breakout Groups 11.50 – 12.20 Plenary — Breakout Group Summaries 12.20 – 12.30 Outro
Poster Session A
A1 — LSB: Local Self-Balancing MCMC in Discrete Spaces Emanuele Sansone (KU Leuven) [ PDF ]
Abstract: Markov Chain Monte Carlo (MCMC) methods are promising solutions to sample from target distributions in high dimensions. While MCMC methods enjoy nice theoretical properties, like guaranteed convergence and mixing to the true target, in practice their sampling efficiency depends on the choice of the proposal distribution and the target at hand. This work considers using machine learning to adapt the proposal distribution to the target, in order to improve the sampling efficiency in the purely discrete domain. Specifically, (i) it proposes a new parametrization for a family of proposal distributions, called locally balanced proposals, (ii) it defines an objective function based on mutual information and (iii) it devises a learning procedure to adapt the parameters of the proposal to the target, thus achieving fast convergence and fast mixing. We call the resulting sampler as the Locally Self-Balancing Sampler (LSB). We show through experimental analysis on the Ising model and Bayesian networks that LSB is indeed able to improve the efficiency over a state-of-the-art sampler based on locally balanced proposals, thus reducing the number of iterations required to converge, while achieving comparable mixing performance.
A2 — Inductive Programming for Data Wrangling Automation Lidia Contreras-Ochando, Cèsar Ferri, José Hernández-Orallo, Fernando Martínez-Plumed, María José Ramírez-Quintana (VRAIN-DMIP — UPV, València, Spain) [ PNG ]
Abstract: An enormous effort is usually devoted to data wrangling, the repetitive process of cleaning data, such that it is ready for modelling. Here, we present two different methods that try to automate data wrangling. First, we present BK-ADAPT, a system that uses inductive programming (IP) with a dynamic background knowledge (BK) generated by a machine learning meta-model that selects the domain and/or the primitives from several descriptive features of the data wrangling problem. Second, we center on matrix transformations, very common in data science. We introduce AUTOMAT[R]IX, a system that is able to induce R programs from a single (and possibly partial) matrix transformation example. We are able to reduce the search space to find the solution program by with (1) using a typed system that excludes inconsistent combinations of primitives , and (2) using a model to estimate the probability of each sequence of primitives from their frequency of use and a text hint provided by the user. AUTOMAT[R]IX is validated with a set of real programming queries related to matrices from Stack Overflow. Links: https://link.springer.com/chapter/10.1007/978-3-030-46133-1_44 https://link.springer.com/article/10.1007/s10994-021-05950-7
A3 — Metabu: Meta-Learning for Tabular Data Herilalaina Rakotoarison, Louisot Milijaona, Andry Rasoanaivo, Michèle Sebag, Marc Schoenauer (INRIA) [ PNG ]
Abstract: This work focuses on AutoML – the automatic selection of machine learning (ML) algorithms and hyper-parameter configurations – for tabular datasets. The contribution of the paper is twofold. Firstly, the many hand-crafted meta-features proposed to describe tabular datasets and achieve AutoML are reconsidered; optimal transport is leveraged to map these meta-features on an algorithm-dependent representation associated with the configuration space of this algorithm. This new representation effectively supports AutoML tasks for three ML algorithms (Random Forest, Adaboost, SVM) and an ML pipeline (Auto-Sklearn), outperforming state-of-art meta-features on the OpenML CC-18 benchmark. Secondly, the approach enables to estimate the intrinsic dimensionality of the OpenML benchmark suite, and sheds some light on when a given ML algorithm might shine on a dataset.
A4 — Speeding Up Neural Network Verification via Automated Algorithm Configuration Matthias König, Holger Hoos, Jan van Rijn (Leiden University) [ PDF ]
Abstract: Despite their great success in recent years, neural networks have found to be vul- nerable to adversarial attacks. These attacks are often based on slight perturba- tions of given inputs that cause them to be misclassified. Several methods have been proposed to formally prove robustness of a given network against such at- tacks. However, these methods typically give rise to high computational demands, which severely limit their scalability. Recent state-of-the-art approaches state the verification task as a minimisation problem, which is formulated and solved as a mixed integer linear programming (MIP) problem. We extend this approach by applying automated algorithm configuration techniques to the solver and, more specifically, construct a portfolio of MIP solver configurations optimised for the neural network verification task. Thereby, we achieve a substantial reduction in CPU time by a factor of 4.7 when verifying a neural network classifier on the MNIST dataset and, at the same time, reduce the number of timeouts by a factor of 1.42 compared to the state of the art. Link: https://aisecure-workshop.github.io/aml-iclr2021/papers/36.pdf
A5 — autobotlib: A library for automatic text classification Blaž Škrlj, Nada Lavrač (Jožef Stefan Institute) [ PDF ]
Abstract: We present autoBOT, a system for automatic classification of texts. The system evolves the suitable document representation for a given task, and has recently been adapted to perform in language-agnostic settings, targeting less-represented languages. The system has a simple-to-use Python API, and is freely available at: https://github.com/SkBlaz/autoBOT
A6 — AutoML Benchmark Pieter Gijsbers, Erin LeDell, Janek Thomas, Sébastien Poirier, Bernd Bischl, and Joaquin Vanschoren (TU Eindhoven) [ PDF ]
Abstract: In recent years, an active field of research has developed around automated machine learning (AutoML). Unfortunately, comparing different AutoML systems is hard and often done incorrectly. We introduced an open, ongoing, and extensible benchmark framework which follows best practices and avoids common mistakes. The framework is open-source, uses public datasets, and currently has 13 AutoML tools integrated. Links: ArXiv (accepted at the AutoML Workshop @ ICML 2019): https://arxiv.org/abs/1907.00909 Github: https://github.com/openml/automlbenchmark
A7 — Interactive Automatic Algorithm Configuration for Bi-Objective Optimization Juan E. Diaz Leiva and Manuel López-Ibáñez (NEO-UMA — Málaga) [ PDF ]
Abstract: Automatic configuration (AC) methods are increasingly used to tune and design optimisation algorithms for problems with multiple objectives. Most AC methods use unary quality indicators, which assign a single scalar value to an approximation to the Pareto front, to compare the performance of different optimisers. These quality indicators, however, imply preferences beyond Pareto-optimality that may differ from those of the decision maker (DM). Although it is possible to incorporate DM’s preferences into quality indicators, e.g., by means of the weighted hypervolume indicator, expressing preferences in terms of weight function is not always intuitive nor an easy task for a DM, in particular, when comparing the stochastic outcomes of several algorithm configurations. A more visual approach to compare such outcomes is the visualisation of their empirical attainment functions (EAFs) differences. This paper proposes using such visualisations as a way of eliciting information about regions of the objective space that are preferred by the DM. We present a method to convert the information about EAF differences into a weighted hypervolume that will assign higher quality values to approximation fronts that result in EAF differences preferred by the DM. We show that the resulting weighted hypervolume may be used by an AC method to guide the configuration of multi-objective optimisers according to the preferences of the DM. We evaluate the proposed approach on a well-known benchmark problem. Finally, we apply our approach to re-configuring, according to different DM’s preferences, a multi-objective optimiser tackling a real-world production planning problem arising in the manufacturing industry. Link: Juan Esteban Diaz and Manuel López-Ibáñez. Incorporating Decision-Maker’s Preferences into the Automatic Configuration of Bi-Objective Optimisation Algorithms. European Journal of Operational Research, 289(3):1209–1222, 2021. http://doi.org/10.1016/j.ejor.2020.07.059
Poster Session B
B1 — Automating Data Science Luc De Raedt, Adem Kikaj, Mohit Kumar, Gust Verbruggen and Samuel Kolb (KU Leuven) [ PNG ]
Abstract: Our goal in the frame of this project is to automate data science. To do so we combine multiple components in a single interactive framework called VisualSYNTH designed primarily to run in Spreadsheet Environments. VisualSYNTH, a framework that wants to democratize data science by enabling naive end-users to specify the data science tasks that match their needs. In VisualSYNTH, the user and the spreadsheet application interact by highlighting parts of the data using colors. The colors define a partial specification of a data science task (such as data wrangling or clustering), which is then completed and solved automatically using artificial intelligence techniques. The user can interactively refine the specification until she is satisfied with the result.
B2 — Streaming Cascade-based Speech Translation leveraged by a Direct Segmentation Model Javier Iranzo-Sánchez, Javier Jorge, Pau Baquero-Arnal, Joan Albert Silvestre-Cerdà, Adrià Giménez, Jorge Civera, Albert Sanchis and Alfons Juan (VRAIN-MLLP — UPV, València, Spain) [ PDF ]
Abstract: The cascade approach to Speech Translation (ST) is based on a pipeline that concatenates an Automatic Speech Recognition (ASR) system followed by a Machine Translation (MT) system. Nowadays, state-of-the-art ST systems are populated with deep neural networks that are conceived to work in an offline setup in which the audio input to be translated is fully available in advance. However, a streaming setup defines a completely different picture, in which an unbounded audio input gradually becomes available and at the same time the translation needs to be generated under real-time constraints. In this work, we present a state-of-the-art streaming ST system in which neural-based models integrated in the ASR and MT components are carefully adapted in terms of their training and decoding procedures in order to run under a streaming setup. In addition, a direct segmentation model that adapts the continuous ASR output to the capacity of simultaneous MT systems trained at the sentence level is introduced to guarantee low latency while preserving the translation quality of the complete ST system. The resulting ST system is thoroughly evaluated on the real-life streaming Europarl-ST benchmark to gauge the trade-off between quality and latency for each component individually as well as for the complete ST system.
B3 — Value propagation-based spatial interpolation Laurens Arp, Mitra Baratchi, Holger Hoos (Leiden University) [ PDF ]
Abstract: Given the common problem of missing data in real-world applications from various fields, such as remote sensing, ecology and meteorology, the interpolation of missing spatial and spatio-temporal data can be of tremendous value. Existing methods for spatial interpolation, most notably Gaussian processes and spatial autoregressive models, tend to suffer from (a) a trade-off between modelling local or global spatial interaction, (b) the assumption there is only one possible path between two points, and (c) the assumption of homogeneity of intermediate locations between points. Addressing these issues, we propose a value propagation method, inspired by Markov reward processes (MRPs), as a spatial interpolation method, and introduce two variants thereof: (i) a static discount (SD-MRP) and (ii) a data-driven weight prediction (WP-MRP) variant. Both these interpolation variants operate locally, while implicitly accounting for global spatial relationships in the entire system through recursion. We evaluated our proposed methods by comparing the mean absolute errors and running times of interpolated grid cells to those of 7 common baselines. Our analysis involved detailed experiments on two synthetic and two real-world datasets over 44 total experimental conditions. Experimental results show the competitive advantage of MRP interpolation on real-world data, as the average performance of SD-MRP on real-world data under all experimental conditions was ranked significantly higher than that of all other methods, followed by WP-MRP. On synthetic data, we show that WP-MRP can perform better than SD-MRP given sufficiently informative features. We further found that, even in cases where our methods had no significant advantage over baselines numerically, our methods preserved the spatial structure of the target grid better than the baselines. Links: Code: https://github.com/LaurensArp/VPInt Google Scholar (paper under review to appear here): https://scholar.google.com/citations?user=BQPXsloAAAAJ&hl=en
B4 — Dynamic Algorithm Configuration for Pseudo-Boolean Solving André Biedenkapp, Frank Hutter, Daniel Le Berre, Romain Wallon (ALU-FR — Freiburg and CRIL — Université d’Artois) [ PDF ]
Abstract: Solvers are configurable and no configuration outperforms all the others on all problems. Algorithm Configuration (Hutter et al. 2009) allows to select the best configuration for each problem. However such selection is static: it applies during all the solver execution on a given problem. Most efficient solvers currently change the configuration at runtime, to take into account the progress of the search. Dynamic Algorithm Configuration (DAC) (Biedenkapp et al. 2020) allows to change at runtime the current configuration, and to learn when to do so. In this work, we apply DAC to the specific case of Pseudo-Boolean solving with the solver Sat4j (Le Berre and Parrain 2010). Recent work shows that various strategies can be proposed to get the best of the particular case of Pseudo-Boolean constraints (Le Berre and Wallon 2021). Those strategies are complementary. The aim of our work is focussed on a particular strategy: the bumping strategy during conflict analysis. Preliminary results on benchmarks from the same family are promising: the proposed dynamic strategy performs better than the best static strategy on each benchmark.
B5 — OPTION: OPTImization Algorithm Benchmarking ONtology Ana Kostovska, Diederick Vermetten, Carola Doerr, Sašo Dzeroski, Panče Panov, Tome Eftimov (Jožef Stefan Institute) [ JPG ]
Abstract: Many platforms for benchmarking optimization algorithms offer users the possibility of sharing their experimental data with the purpose of promoting reproducible and reusable research. However, different platforms use different data models and formats, which drastically inhibits identification of relevant data sets, their interpretation, and their interoperability. Consequently, a semantically rich, ontology-based, machine-readable data model is highly desired. To this end, developed such an ontology, which we name OPTION (OPTImization algorithm benchmarking ONtology). Our ontology provides the vocabulary needed for semantic annotation of the core entities involved in the benchmarking process, such as algorithms, problems, and evaluation measures. It also provides means for automated data integration, improved interoperability, powerful querying capabilities and reasoning, thereby enriching the value of the benchmark data. The utility of OPTION has been demonstrated by annotating and querying a corpus of benchmark performance data from the BBOB workshop data — a use case which can be easily extended to cover other benchmarking data collections.
B6 — Neural architecture search using a training-free performance metric Andrés Camero, Jamal Toutouh, Hao Wang, Enrique Alba and Thomas Bäck (NEO-UMA — Málaga) [ PDF ]
Abstract: Recurrent neural networks (RNN) are a powerful tool, but they are very sensitive to the quality of their hyperparameter configuration. Therefore, building an appropriate structure before training an RNN is critical. Varied strategies have been proposed to tackle the issue of finding the architecture that provides the best performance. Most of these strategies are still impractical because of the time/resources needed to evaluate the architectures during the search process, mainly due to the intensive training process used. To tackle this issue, we propose the mean absolute error random sampling (MRS), a low computational cost model to evaluate the expected performance of a given architecture based on the distribution of the error of random samples of the weights, i.e., a training-free approach to assess the performance of an architecture. The proposed model can be used by any neural architecture search (NAS) method that needs to evaluate numerous RNN architectures during the search process to reduce the time/resources requirements. The MRS poses a promising alternative to predict the performance of an RNN without training it, thus it is a way to reduce the cost of NAS. References: [1] Camero, A., Toutouh, J. and Alba, E., 2018. Low-cost recurrent neural network expected performance evaluation. arXiv preprint arXiv:1805.07159. [2] Camero, A., Toutouh, J. and Alba, E., 2020. Random error sampling-based recurrent neural network architecture optimization. Engineering Applications of Artificial Intelligence, 96, p.103946. [3] Camero, A., Wang, H., Alba, E. and Bäck, T., 2021. Bayesian neural architecture search using a training-free performance metric. Applied Soft Computing, 106, p.107356.
B7 — Fully automatic design of control software for robot swarms Jonas Kuckling, Antoine Ligot, Mauro Birattari (IRIDIA ULB — Brussels) [ PNG ]