Hyperparameter Tuning with Iterated Racing.

Subclass for iterated racing. Calls irace::irace() from package irace.

Source

Lopez-Ibanez M, Dubois-Lacoste J, Caceres LP, Birattari M, Stuetzle T (2016). “The irace package: Iterated racing for automatic algorithm configuration.” Operations Research Perspectives, 3, 43–58. doi:10.1016/j.orp.2016.09.002 .

Dictionary

This Tuner can be instantiated with the associated sugar function tnr():

tnr("irace")

Control Parameters

n_instances

integer(1)
Number of resampling instances.

For the meaning of all other parameters, see irace::defaultScenario(). Note that we have removed all control parameters which refer to the termination of the algorithm. Use bbotk::TerminatorEvals instead. Other terminators do not work with TunerIrace.

Archive

The ArchiveBatchTuning holds the following additional columns:

• "race" (integer(1))
  Race iteration.

• "step" (integer(1))
  Step number of race.

• "instance" (integer(1))
  Identifies resampling instances across races and steps.

• "configuration" (integer(1))
  Identifies configurations across races and steps.

Result

The tuning result (instance$result) is the best-performing elite of the final race. The reported performance is the average performance estimated on all used instances.

Progress Bars

$optimize() supports progress bars via the package progressr combined with a bbotk::Terminator. Simply wrap the function in progressr::with_progress() to enable them. We recommend to use package progress as backend; enable with progressr::handlers("progress").

Logging

All Tuners use a logger (as implemented in lgr) from package bbotk. Use lgr::get_logger("bbotk") to access and control the logger.

Optimizer

This Tuner is based on bbotk::OptimizerBatchIrace which can be applied on any black box optimization problem. See also the documentation of bbotk.

Resources

There are several sections about hyperparameter optimization in the mlr3book.

• Getting started with hyperparameter optimization.

• An overview of all tuners can be found on our website.

• Tune a support vector machine on the Sonar data set.

• Learn about tuning spaces.

• Estimate the model performance with nested resampling.

• Learn about multi-objective optimization.

• Simultaneously optimize hyperparameters and use early stopping with XGBoost.

• Automate the tuning.

The gallery features a collection of case studies and demos about optimization.

• Learn more advanced methods with the Practical Tuning Series.

• Learn about hotstarting models.

• Run the default hyperparameter configuration of learners as a baseline.

• Use the Hyperband optimizer with different budget parameters.

The cheatsheet summarizes the most important functions of mlr3tuning.

See also

Other Tuner: Tuner, mlr_tuners, mlr_tuners_cmaes, mlr_tuners_design_points, mlr_tuners_gensa, mlr_tuners_grid_search, mlr_tuners_internal, mlr_tuners_nloptr, mlr_tuners_random_search

Super classes

mlr3tuning::Tuner -> mlr3tuning::TunerBatch -> mlr3tuning::TunerBatchFromOptimizerBatch -> TunerBatchIrace

Methods

Public methods

• TunerBatchIrace$new()

• TunerBatchIrace$optimize()

• TunerBatchIrace$clone()

Inherited methods

• mlr3tuning::Tuner$format()
• mlr3tuning::Tuner$help()
• mlr3tuning::Tuner$print()

---

Method new()

Creates a new instance of this R6 class.

Usage

TunerBatchIrace$new()

---

Method optimize()

Performs the tuning on a TuningInstanceBatchSingleCrit until termination. The single evaluations and the final results will be written into the ArchiveBatchTuning that resides in the TuningInstanceBatchSingleCrit. The final result is returned.

Usage

TunerBatchIrace$optimize(inst)

Arguments

inst

(TuningInstanceBatchSingleCrit).

Returns

data.table::data.table.

---

Method clone()

The objects of this class are cloneable with this method.

Usage

TunerBatchIrace$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

Examples

# retrieve task
task = tsk("pima")

# load learner and set search space
learner = lrn("classif.rpart", cp = to_tune(1e-04, 1e-1, logscale = TRUE))
# \donttest{
# hyperparameter tuning on the pima indians diabetes data set
instance = tune(
  tuner = tnr("irace"),
  task = task,
  learner = learner,
  resampling = rsmp("holdout"),
  measure = msr("classif.ce"),
  term_evals = 42
)
#> # 2024-11-08 15:15:28 UTC: Initialization
#> # Elitist race
#> # Elitist new instances: 1
#> # Elitist limit: 2
#> # nbIterations: 2
#> # minNbSurvival: 2
#> # nbParameters: 1
#> # seed: 1855097766
#> # confidence level: 0.95
#> # budget: 42
#> # mu: 5
#> # deterministic: FALSE
#> 
#> # 2024-11-08 15:15:28 UTC: Iteration 1 of 2
#> # experimentsUsedSoFar: 0
#> # remainingBudget: 42
#> # currentBudget: 21
#> # nbConfigurations: 3
#> # Markers:
#>      x No test is performed.
#>      c Configurations are discarded only due to capping.
#>      - The test is performed and some configurations are discarded.
#>      = The test is performed but no configuration is discarded.
#>      ! The test is performed and configurations could be discarded but elite configurations are preserved.
#>      . All alive configurations are elite and nothing is discarded
#> 
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | |   Instance|      Alive|       Best|       Mean best| Exp so far|  W time|  rho|KenW|  Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x|          1|          3|          3|    0.2812500000|          3|00:00:00|   NA|  NA|    NA|
#> |x|          2|          3|          3|    0.2675781250|          6|00:00:00|+1.00|1.00|0.0000|
#> |x|          3|          3|          3|    0.2604166667|          9|00:00:00|+1.00|1.00|0.0000|
#> |x|          4|          3|          3|    0.2490234375|         12|00:00:00|+1.00|1.00|0.0000|
#> |-|          5|          1|          3|    0.2453125000|         15|00:00:00|   NA|  NA|    NA|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration:           3    mean value:     0.2453125000
#> Description of the best-so-far configuration:
#>   .ID.               cp .PARENT.
#> 3    3 -2.7229877489945       NA
#> 
#> # 2024-11-08 15:15:28 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#>                 cp
#> 3 -2.7229877489945
#> # 2024-11-08 15:15:28 UTC: Iteration 2 of 2
#> # experimentsUsedSoFar: 15
#> # remainingBudget: 27
#> # currentBudget: 27
#> # nbConfigurations: 4
#> # Markers:
#>      x No test is performed.
#>      c Configurations are discarded only due to capping.
#>      - The test is performed and some configurations are discarded.
#>      = The test is performed but no configuration is discarded.
#>      ! The test is performed and configurations could be discarded but elite configurations are preserved.
#>      . All alive configurations are elite and nothing is discarded
#> 
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | |   Instance|      Alive|       Best|       Mean best| Exp so far|  W time|  rho|KenW|  Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x|          6|          4|          4|    0.2734375000|          4|00:00:00|   NA|  NA|    NA|
#> |x|          1|          4|          4|    0.2597656250|          7|00:00:00|+0.80|0.90|0.0217|
#> |x|          4|          4|          4|    0.2447916667|         10|00:00:00|+0.40|0.60|0.2572|
#> |x|          3|          4|          4|    0.2451171875|         13|00:00:00|+0.27|0.45|0.2536|
#> |=|          5|          4|          4|    0.2421875000|         16|00:00:00|+0.20|0.36|0.2272|
#> |=|          2|          4|          4|    0.2513020833|         19|00:00:00|-0.11|0.07|0.4250|
#> |=|          7|          4|          5|    0.2522321429|         23|00:00:00|-0.15|0.02|0.4839|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration:           5    mean value:     0.2522321429
#> Description of the best-so-far configuration:
#>   .ID.                cp .PARENT.
#> 5    5 -3.17982221206359        3
#> 
#> # 2024-11-08 15:15:29 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#>                  cp
#> 5 -3.17982221206359
#> 6 -3.29396533989700
#> # 2024-11-08 15:15:29 UTC: Stopped because there is not enough budget left to race more than the minimum (2)
#> # You may either increase the budget or set 'minNbSurvival' to a lower value
#> # Iteration: 3
#> # nbIterations: 3
#> # experimentsUsedSoFar: 38
#> # timeUsed: 0
#> # remainingBudget: 4
#> # currentBudget: 4
#> # number of elites: 2
#> # nbConfigurations: 2
#> # Total CPU user time: 1.401, CPU sys time: 0.02, Wall-clock time: 1.424

# best performing hyperparameter configuration
instance$result
#>           cp configuration learner_param_vals  x_domain classif.ce
#>        <num>         <num>             <list>    <list>      <num>
#> 1: -3.179822             5          <list[2]> <list[1]>  0.2522321

# all evaluated hyperparameter configuration
as.data.table(instance$archive)
#>            cp classif.ce runtime_learners           timestamp  race  step
#>         <num>      <num>            <num>              <POSc> <num> <int>
#>  1: -8.192526  0.3085938            0.007 2024-11-08 15:15:28     1     1
#>  2: -8.612223  0.3085938            0.006 2024-11-08 15:15:28     1     1
#>  3: -2.722988  0.2812500            0.006 2024-11-08 15:15:28     1     1
#>  4: -8.192526  0.3359375            0.008 2024-11-08 15:15:28     1     1
#>  5: -8.612223  0.3359375            0.007 2024-11-08 15:15:28     1     1
#>  6: -2.722988  0.2539062            0.006 2024-11-08 15:15:28     1     1
#>  7: -8.192526  0.2851562            0.008 2024-11-08 15:15:28     1     1
#>  8: -8.612223  0.2851562            0.007 2024-11-08 15:15:28     1     1
#>  9: -2.722988  0.2460938            0.007 2024-11-08 15:15:28     1     1
#> 10: -8.192526  0.2617188            0.008 2024-11-08 15:15:28     1     1
#> 11: -8.612223  0.2617188            0.007 2024-11-08 15:15:28     1     1
#> 12: -2.722988  0.2148438            0.006 2024-11-08 15:15:28     1     1
#> 13: -8.192526  0.2382812            0.007 2024-11-08 15:15:28     1     1
#> 14: -8.612223  0.2382812            0.007 2024-11-08 15:15:28     1     1
#> 15: -2.722988  0.2304688            0.006 2024-11-08 15:15:28     1     1
#> 16: -2.722988  0.2890625            0.006 2024-11-08 15:15:28     2     1
#> 17: -3.680872  0.2734375            0.006 2024-11-08 15:15:28     2     1
#> 18: -3.179822  0.2734375            0.006 2024-11-08 15:15:28     2     1
#> 19: -3.293965  0.2734375            0.006 2024-11-08 15:15:28     2     1
#> 20: -3.680872  0.2460938            0.006 2024-11-08 15:15:28     2     1
#> 21: -3.179822  0.2539062            0.006 2024-11-08 15:15:28     2     1
#> 22: -3.293965  0.2539062            0.006 2024-11-08 15:15:28     2     1
#> 23: -3.680872  0.2148438            0.007 2024-11-08 15:15:28     2     1
#> 24: -3.179822  0.2148438            0.006 2024-11-08 15:15:28     2     1
#> 25: -3.293965  0.2148438            0.006 2024-11-08 15:15:28     2     1
#> 26: -3.680872  0.2460938            0.006 2024-11-08 15:15:29     2     1
#> 27: -3.179822  0.2460938            0.006 2024-11-08 15:15:29     2     1
#> 28: -3.293965  0.2460938            0.006 2024-11-08 15:15:29     2     1
#> 29: -3.680872  0.2304688            0.007 2024-11-08 15:15:29     2     1
#> 30: -3.179822  0.2304688            0.007 2024-11-08 15:15:29     2     1
#> 31: -3.293965  0.2304688            0.006 2024-11-08 15:15:29     2     1
#> 32: -3.680872  0.2968750            0.007 2024-11-08 15:15:29     2     1
#> 33: -3.179822  0.2968750            0.006 2024-11-08 15:15:29     2     1
#> 34: -3.293965  0.2968750            0.006 2024-11-08 15:15:29     2     1
#> 35: -2.722988  0.2500000            0.006 2024-11-08 15:15:29     2     1
#> 36: -3.680872  0.2617188            0.007 2024-11-08 15:15:29     2     1
#> 37: -3.179822  0.2500000            0.006 2024-11-08 15:15:29     2     1
#> 38: -3.293965  0.2500000            0.006 2024-11-08 15:15:29     2     1
#>            cp classif.ce runtime_learners           timestamp  race  step
#>     instance configuration warnings errors  x_domain batch_nr  resample_result
#>        <int>         <num>    <int>  <int>    <list>    <int>           <list>
#>  1:       10             1        0      0 <list[1]>        1 <ResampleResult>
#>  2:       10             2        0      0 <list[1]>        1 <ResampleResult>
#>  3:       10             3        0      0 <list[1]>        1 <ResampleResult>
#>  4:        4             1        0      0 <list[1]>        2 <ResampleResult>
#>  5:        4             2        0      0 <list[1]>        2 <ResampleResult>
#>  6:        4             3        0      0 <list[1]>        2 <ResampleResult>
#>  7:        1             1        0      0 <list[1]>        3 <ResampleResult>
#>  8:        1             2        0      0 <list[1]>        3 <ResampleResult>
#>  9:        1             3        0      0 <list[1]>        3 <ResampleResult>
#> 10:        8             1        0      0 <list[1]>        4 <ResampleResult>
#> 11:        8             2        0      0 <list[1]>        4 <ResampleResult>
#> 12:        8             3        0      0 <list[1]>        4 <ResampleResult>
#> 13:        5             1        0      0 <list[1]>        5 <ResampleResult>
#> 14:        5             2        0      0 <list[1]>        5 <ResampleResult>
#> 15:        5             3        0      0 <list[1]>        5 <ResampleResult>
#> 16:        7             3        0      0 <list[1]>        6 <ResampleResult>
#> 17:        7             4        0      0 <list[1]>        6 <ResampleResult>
#> 18:        7             5        0      0 <list[1]>        6 <ResampleResult>
#> 19:        7             6        0      0 <list[1]>        6 <ResampleResult>
#> 20:       10             4        0      0 <list[1]>        7 <ResampleResult>
#> 21:       10             5        0      0 <list[1]>        7 <ResampleResult>
#> 22:       10             6        0      0 <list[1]>        7 <ResampleResult>
#> 23:        8             4        0      0 <list[1]>        8 <ResampleResult>
#> 24:        8             5        0      0 <list[1]>        8 <ResampleResult>
#> 25:        8             6        0      0 <list[1]>        8 <ResampleResult>
#> 26:        1             4        0      0 <list[1]>        9 <ResampleResult>
#> 27:        1             5        0      0 <list[1]>        9 <ResampleResult>
#> 28:        1             6        0      0 <list[1]>        9 <ResampleResult>
#> 29:        5             4        0      0 <list[1]>       10 <ResampleResult>
#> 30:        5             5        0      0 <list[1]>       10 <ResampleResult>
#> 31:        5             6        0      0 <list[1]>       10 <ResampleResult>
#> 32:        4             4        0      0 <list[1]>       11 <ResampleResult>
#> 33:        4             5        0      0 <list[1]>       11 <ResampleResult>
#> 34:        4             6        0      0 <list[1]>       11 <ResampleResult>
#> 35:        9             3        0      0 <list[1]>       12 <ResampleResult>
#> 36:        9             4        0      0 <list[1]>       12 <ResampleResult>
#> 37:        9             5        0      0 <list[1]>       12 <ResampleResult>
#> 38:        9             6        0      0 <list[1]>       12 <ResampleResult>
#>     instance configuration warnings errors  x_domain batch_nr  resample_result

# fit final model on complete data set
learner$param_set$values = instance$result_learner_param_vals
learner$train(task)
# }