Skip to contents

Function for Nested Resampling

Source: R/tune_nested.R
tune_nested.Rd

Function to conduct nested resampling.

Usage

tune_nested(
  tuner,
  task,
  learner,
  inner_resampling,
  outer_resampling,
  measure = NULL,
  term_evals = NULL,
  term_time = NULL,
  terminator = NULL,
  search_space = NULL,
  store_tuning_instance = TRUE,
  store_benchmark_result = TRUE,
  store_models = FALSE,
  check_values = FALSE,
  allow_hotstart = FALSE,
  keep_hotstart_stack = FALSE,
  evaluate_default = FALSE,
  callbacks = list(),
  method
)

Arguments

tuner

(Tuner)
Optimization algorithm.

task

(mlr3::Task)
Task to operate on.

learner

(mlr3::Learner)
Learner to tune.

inner_resampling

(mlr3::Resampling)
Resampling used for the inner loop.

outer_resampling

mlr3::Resampling)
Resampling used for the outer loop.

measure

(mlr3::Measure)
Measure to optimize. If NULL, default measure is used.

term_evals

(integer(1))
Number of allowed evaluations. Ignored if terminator is passed.

term_time

(integer(1))
Maximum allowed time in seconds. Ignored if terminator is passed.

terminator

(Terminator)
Stop criterion of the tuning process.

search_space

(paradox::ParamSet)
Hyperparameter search space. If NULL (default), the search space is constructed from the TuneToken of the learner's parameter set (learner$param_set).

store_tuning_instance

(logical(1))
If TRUE (default), stores the internally created TuningInstanceSingleCrit with all intermediate results in slot $tuning_instance.

store_benchmark_result

(logical(1))
If TRUE (default), store resample result of evaluated hyperparameter configurations in archive as mlr3::BenchmarkResult.

store_models

(logical(1))
If TRUE, fitted models are stored in the benchmark result (archive$benchmark_result). If store_benchmark_result = FALSE, models are only stored temporarily and not accessible after the tuning. This combination is needed for measures that require a model.

check_values

(logical(1))
If TRUE, hyperparameter values are checked before evaluation and performance scores after. If FALSE (default), values are unchecked but computational overhead is reduced.

allow_hotstart

(logical(1))
Allow to hotstart learners with previously fitted models. See also mlr3::HotstartStack. The learner must support hotstarting. Sets store_models = TRUE.

keep_hotstart_stack

(logical(1))
If TRUE, mlr3::HotstartStack is kept in $objective$hotstart_stack after tuning.

evaluate_default

(logical(1))
If TRUE, learner is evaluated with hyperparameters set to their default values at the start of the optimization.

callbacks

(list of CallbackTuning)
List of callbacks.

method

(character(1))
Deprecated. Use tuner instead.

Value

mlr3::ResampleResult

Examples

# Nested resampling on Palmer Penguins data set
rr = tune_nested(
  tuner = tnr("random_search", batch_size = 2),
  task = tsk("penguins"),
  learner = lrn("classif.rpart", cp = to_tune(1e-04, 1e-1, logscale = TRUE)),
  inner_resampling = rsmp ("holdout"),
  outer_resampling = rsmp("cv", folds = 2),
  measure = msr("classif.ce"),
  term_evals = 2)

# Performance scores estimated on the outer resampling
rr$score()
#>     task_id          learner_id resampling_id iteration classif.ce
#>      <char>              <char>        <char>     <int>      <num>
#> 1: penguins classif.rpart.tuned            cv         1 0.11627907
#> 2: penguins classif.rpart.tuned            cv         2 0.08139535
#> Hidden columns: task, learner, resampling, prediction

# Unbiased performance of the final model trained on the full data set
rr$aggregate()
#> classif.ce 
#> 0.09883721