Skip to contents

Extract Inner Tuning Archives

Source: R/extract_inner_tuning_archives.R
extract_inner_tuning_archives.Rd

Extract inner tuning archives of nested resampling. Implemented for mlr3::ResampleResult and mlr3::BenchmarkResult. The function iterates over the AutoTuner objects and binds the tuning archives to a data.table::data.table(). AutoTuner must be initialized with store_tuning_instance = TRUE and mlr3::resample() or mlr3::benchmark() must be called with store_models = TRUE.

Usage

extract_inner_tuning_archives(
  x,
  unnest = "x_domain",
  exclude_columns = "uhash"
)

Arguments

x

(mlr3::ResampleResult | mlr3::BenchmarkResult).

unnest

(character())
Transforms list columns to separate columns. By default, x_domain is unnested. Set to NULL if no column should be unnested.

exclude_columns

(character())
Exclude columns from result table. Set to NULL if no column should be excluded.

Value

data.table::data.table().

Data structure

The returned data table has the following columns:

  • experiment (integer(1))
    Index, giving the according row number in the original benchmark grid.

  • iteration (integer(1))
    Iteration of the outer resampling.

  • One column for each hyperparameter of the search spaces.

  • One column for each performance measure.

  • runtime_learners (numeric(1))
    Sum of training and predict times logged in learners per mlr3::ResampleResult / evaluation. This does not include potential overhead time.

  • timestamp (POSIXct)
    Time stamp when the evaluation was logged into the archive.

  • batch_nr (integer(1))
    Hyperparameters are evaluated in batches. Each batch has a unique batch number.

  • x_domain (list())
    List of transformed hyperparameter values. By default this column is unnested.

  • x_domain_* (any)
    Separate column for each transformed hyperparameter.

  • resample_result (mlr3::ResampleResult)
    Resample result of the inner resampling.

  • task_id (character(1)).

  • learner_id (character(1)).

  • resampling_id (character(1)).

Examples

# Nested Resampling on Palmer Penguins Data Set

learner = lrn("classif.rpart",
  cp = to_tune(1e-04, 1e-1, logscale = TRUE))

# create auto tuner
at = auto_tuner(
  tuner = tnr("random_search"),
  learner = learner,
  resampling = rsmp ("holdout"),
  measure = msr("classif.ce"),
  term_evals = 4)

resampling_outer = rsmp("cv", folds = 2)
rr = resample(tsk("iris"), at, resampling_outer, store_models = TRUE)

# extract inner archives
extract_inner_tuning_archives(rr)
#>    iteration        cp classif.ce  x_domain_cp runtime_learners
#>        <int>     <num>      <num>        <num>            <num>
#> 1:         1 -2.592262       0.00 0.0748505376            0.005
#> 2:         1 -7.842829       0.00 0.0003925571            0.004
#> 3:         1 -4.167189       0.00 0.0154957620            0.005
#> 4:         1 -5.779245       0.00 0.0030910495            0.005
#> 5:         2 -4.040571       0.04 0.0175874247            0.005
#> 6:         2 -4.239391       0.04 0.0144163640            0.021
#> 7:         2 -8.942275       0.04 0.0001307433            0.005
#> 8:         2 -3.186972       0.04 0.0412967110            0.004
#>              timestamp warnings errors batch_nr  resample_result task_id
#>                 <POSc>    <int>  <int>    <int>           <list>  <char>
#> 1: 2024-07-24 10:53:07        0      0        1 <ResampleResult>    iris
#> 2: 2024-07-24 10:53:07        0      0        2 <ResampleResult>    iris
#> 3: 2024-07-24 10:53:07        0      0        3 <ResampleResult>    iris
#> 4: 2024-07-24 10:53:07        0      0        4 <ResampleResult>    iris
#> 5: 2024-07-24 10:53:07        0      0        1 <ResampleResult>    iris
#> 6: 2024-07-24 10:53:07        0      0        2 <ResampleResult>    iris
#> 7: 2024-07-24 10:53:07        0      0        3 <ResampleResult>    iris
#> 8: 2024-07-24 10:53:07        0      0        4 <ResampleResult>    iris
#>             learner_id resampling_id
#>                 <char>        <char>
#> 1: classif.rpart.tuned            cv
#> 2: classif.rpart.tuned            cv
#> 3: classif.rpart.tuned            cv
#> 4: classif.rpart.tuned            cv
#> 5: classif.rpart.tuned            cv
#> 6: classif.rpart.tuned            cv
#> 7: classif.rpart.tuned            cv
#> 8: classif.rpart.tuned            cv