Skip to contents

The AutoTuner wraps a mlr3::Learner and augments it with an automatic tuning process for a given set of hyperparameters. The auto_tuner() function creates an AutoTuner object.

Details

The AutoTuner is a mlr3::Learner which wraps another mlr3::Learner and performs the following steps during $train():

  1. The hyperparameters of the wrapped (inner) learner are trained on the training data via resampling. The tuning can be specified by providing a Tuner, a bbotk::Terminator, a search space as paradox::ParamSet, a mlr3::Resampling and a mlr3::Measure.

  2. The best found hyperparameter configuration is set as hyperparameters for the wrapped (inner) learner stored in at$learner. Access the tuned hyperparameters via at$tuning_result.

  3. A final model is fit on the complete training data using the now parametrized wrapped learner. The respective model is available via field at$learner$model.

During $predict() the AutoTuner just calls the predict method of the wrapped (inner) learner. A set timeout is disabled while fitting the final model.

Resources

Nested Resampling

Nested resampling can be performed by passing an AutoTuner object to mlr3::resample() or mlr3::benchmark(). To access the inner resampling results, set store_tuning_instance = TRUE and execute mlr3::resample() or mlr3::benchmark() with store_models = TRUE (see examples). The mlr3::Resampling passed to the AutoTuner is meant to be the inner resampling, operating on the training set of an arbitrary outer resampling. For this reason it is not feasible to pass an instantiated mlr3::Resampling here.

Super class

mlr3::Learner -> AutoTuner

Public fields

instance_args

(list())
All arguments from construction to create the TuningInstanceSingleCrit.

tuner

(Tuner)
Optimization algorithm.

Active bindings

archive

ArchiveTuning
Archive of the TuningInstanceSingleCrit.

learner

(mlr3::Learner)
Trained learner

tuning_instance

(TuningInstanceSingleCrit)
Internally created tuning instance with all intermediate results.

tuning_result

(data.table::data.table)
Short-cut to result from TuningInstanceSingleCrit.

predict_type

(character(1))
Stores the currently active predict type, e.g. "response". Must be an element of $predict_types.

hash

(character(1))
Hash (unique identifier) for this object.

Methods

Inherited methods


Method new()

Creates a new instance of this R6 class.

Usage

AutoTuner$new(
  learner,
  resampling,
  measure = NULL,
  terminator,
  tuner,
  search_space = NULL,
  store_tuning_instance = TRUE,
  store_benchmark_result = TRUE,
  store_models = FALSE,
  check_values = FALSE,
  callbacks = list()
)

Arguments

learner

(mlr3::Learner)
Learner to tune.

resampling

(mlr3::Resampling)
Resampling that is used to evaluated the performance of the hyperparameter configurations. Uninstantiated resamplings are instantiated during construction so that all configurations are evaluated on the same data splits. Already instantiated resamplings are kept unchanged. Specialized Tuner change the resampling e.g. to evaluate a hyperparameter configuration on different data splits. This field, however, always returns the resampling passed in construction.

measure

(mlr3::Measure)
Measure to optimize. If NULL, default measure is used.

terminator

(Terminator)
Stop criterion of the tuning process.

tuner

(Tuner)
Optimization algorithm.

search_space

(paradox::ParamSet)
Hyperparameter search space. If NULL (default), the search space is constructed from the TuneToken of the learner's parameter set (learner$param_set).

store_tuning_instance

(logical(1))
If TRUE (default), stores the internally created TuningInstanceSingleCrit with all intermediate results in slot $tuning_instance.

store_benchmark_result

(logical(1))
If TRUE (default), store resample result of evaluated hyperparameter configurations in archive as mlr3::BenchmarkResult.

store_models

(logical(1))
If TRUE, fitted models are stored in the benchmark result (archive$benchmark_result). If store_benchmark_result = FALSE, models are only stored temporarily and not accessible after the tuning. This combination is needed for measures that require a model.

check_values

(logical(1))
If TRUE, hyperparameter values are checked before evaluation and performance scores after. If FALSE (default), values are unchecked but computational overhead is reduced.

callbacks

(list of CallbackTuning)
List of callbacks.


Method base_learner()

Extracts the base learner from nested learner objects like GraphLearner in mlr3pipelines. If recursive = 0, the (tuned) learner is returned.

Usage

AutoTuner$base_learner(recursive = Inf)

Arguments

recursive

(integer(1))
Depth of recursion for multiple nested objects.

Returns

Learner.


Method importance()

The importance scores of the final model.

Usage

AutoTuner$importance()

Returns

Named numeric().


Method selected_features()

The selected features of the final model.

Usage

AutoTuner$selected_features()

Returns

character().


Method oob_error()

The out-of-bag error of the final model.

Usage

AutoTuner$oob_error()

Returns

numeric(1).


Method loglik()

The log-likelihood of the final model.

Usage

AutoTuner$loglik()

Returns

logLik. Printer.


Method print()

Usage

AutoTuner$print()

Arguments

...

(ignored).


Method clone()

The objects of this class are cloneable with this method.

Usage

AutoTuner$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

Examples

# Automatic Tuning

# split to train and external set
task = tsk("penguins")
split = partition(task, ratio = 0.8)

# load learner and set search space
learner = lrn("classif.rpart",
  cp = to_tune(1e-04, 1e-1, logscale = TRUE)
)

# create auto tuner
at = auto_tuner(
  method = tnr("random_search"),
  learner = learner,
  resampling = rsmp ("holdout"),
  measure = msr("classif.ce"),
  term_evals = 4)

# tune hyperparameters and fit final model
at$train(task, row_ids = split$train)

# predict with final model
at$predict(task, row_ids = split$test)
#> <PredictionClassif> for 69 observations:
#>     row_ids     truth  response
#>           7    Adelie    Adelie
#>          13    Adelie    Adelie
#>          15    Adelie    Adelie
#> ---                            
#>         338 Chinstrap Chinstrap
#>         340 Chinstrap Chinstrap
#>         341 Chinstrap    Adelie

# show tuning result
at$tuning_result
#>           cp learner_param_vals  x_domain classif.ce
#> 1: -8.927971          <list[2]> <list[1]> 0.06521739

# model slot contains trained learner and tuning instance
at$model
#> $learner
#> <LearnerClassifRpart:classif.rpart>: Classification Tree
#> * Model: rpart
#> * Parameters: xval=0, cp=0.0001326
#> * Packages: mlr3, rpart
#> * Predict Types:  [response], prob
#> * Feature Types: logical, integer, numeric, factor, ordered
#> * Properties: importance, missings, multiclass, selected_features,
#>   twoclass, weights
#> 
#> $tuning_instance
#> <TuningInstanceSingleCrit>
#> * State:  Optimized
#> * Objective: <ObjectiveTuning:classif.rpart_on_penguins>
#> * Search Space:
#>    id    class    lower     upper nlevels
#> 1: cp ParamDbl -9.21034 -2.302585     Inf
#> * Terminator: <TerminatorEvals>
#> * Result:
#>           cp classif.ce
#> 1: -8.927971 0.06521739
#> * Archive:
#>           cp classif.ce
#> 1: -8.927971 0.06521739
#> 2: -8.511987 0.06521739
#> 3: -8.218003 0.06521739
#> 4: -6.080020 0.06521739
#> 

# shortcut trained learner
at$learner
#> <LearnerClassifRpart:classif.rpart>: Classification Tree
#> * Model: rpart
#> * Parameters: xval=0, cp=0.0001326
#> * Packages: mlr3, rpart
#> * Predict Types:  [response], prob
#> * Feature Types: logical, integer, numeric, factor, ordered
#> * Properties: importance, missings, multiclass, selected_features,
#>   twoclass, weights

# shortcut tuning instance
at$tuning_instance
#> <TuningInstanceSingleCrit>
#> * State:  Optimized
#> * Objective: <ObjectiveTuning:classif.rpart_on_penguins>
#> * Search Space:
#>    id    class    lower     upper nlevels
#> 1: cp ParamDbl -9.21034 -2.302585     Inf
#> * Terminator: <TerminatorEvals>
#> * Result:
#>           cp classif.ce
#> 1: -8.927971 0.06521739
#> * Archive:
#>           cp classif.ce
#> 1: -8.927971 0.06521739
#> 2: -8.511987 0.06521739
#> 3: -8.218003 0.06521739
#> 4: -6.080020 0.06521739


# Nested Resampling

at = auto_tuner(
  method = tnr("random_search"),
  learner = learner,
  resampling = rsmp ("holdout"),
  measure = msr("classif.ce"),
  term_evals = 4)

resampling_outer = rsmp("cv", folds = 3)
rr = resample(task, at, resampling_outer, store_models = TRUE)

# retrieve inner tuning results.
extract_inner_tuning_results(rr)
#>    iteration        cp classif.ce learner_param_vals  x_domain  task_id
#> 1:         1 -7.849491 0.05263158          <list[2]> <list[1]> penguins
#> 2:         2 -8.479986 0.03947368          <list[2]> <list[1]> penguins
#> 3:         3 -4.951637 0.07792208          <list[2]> <list[1]> penguins
#>             learner_id resampling_id
#> 1: classif.rpart.tuned            cv
#> 2: classif.rpart.tuned            cv
#> 3: classif.rpart.tuned            cv

# performance scores estimated on the outer resampling
rr$score()
#>                 task  task_id         learner          learner_id
#> 1: <TaskClassif[50]> penguins <AutoTuner[46]> classif.rpart.tuned
#> 2: <TaskClassif[50]> penguins <AutoTuner[46]> classif.rpart.tuned
#> 3: <TaskClassif[50]> penguins <AutoTuner[46]> classif.rpart.tuned
#>            resampling resampling_id iteration              prediction
#> 1: <ResamplingCV[20]>            cv         1 <PredictionClassif[20]>
#> 2: <ResamplingCV[20]>            cv         2 <PredictionClassif[20]>
#> 3: <ResamplingCV[20]>            cv         3 <PredictionClassif[20]>
#>    classif.ce
#> 1: 0.07826087
#> 2: 0.05217391
#> 3: 0.05263158

# unbiased performance of the final model trained on the full data set
rr$aggregate()
#> classif.ce 
#> 0.06102212