Create Tuning Callback

Function to create a CallbackTuning. Predefined callbacks are stored in the dictionary mlr_callbacks and can be retrieved with clbk().

Tuning callbacks can be called from different stages of tuning process. The stages are prefixed with on_*.

Start Tuning
     - on_optimization_begin
    Start Tuner Batch
         - on_optimizer_before_eval
        Start Evaluation
             - on_eval_after_design
             - on_eval_after_benchmark
             - on_eval_before_archive
        End Evaluation
         - on_optimizer_after_eval
    End Tuner Batch
     - on_result
     - on_optimization_end
End Tuning

See also the section on parameters for more information on the stages. A tuning callback works with bbotk::ContextOptimization and ContextEval.

Usage

callback_tuning(
  id,
  label = NA_character_,
  man = NA_character_,
  on_optimization_begin = NULL,
  on_optimizer_before_eval = NULL,
  on_eval_after_design = NULL,
  on_eval_after_benchmark = NULL,
  on_eval_before_archive = NULL,
  on_optimizer_after_eval = NULL,
  on_result = NULL,
  on_optimization_end = NULL
)

Arguments

id

(character(1))
Identifier for the new instance.

label

(character(1))
Label for the new instance.

man

(character(1))
String in the format [pkg]::[topic] pointing to a manual page for this object. The referenced help package can be opened via method $help().

on_optimization_begin

(function())
Stage called at the beginning of the optimization. Called in Optimizer$optimize(). The context available is bbotk::ContextOptimization.

on_optimizer_before_eval

(function())
Stage called after the optimizer proposes points. Called in OptimInstance$eval_batch(). The context available is bbotk::ContextOptimization.

on_eval_after_design

(function())
Stage called after design is created. Called in ObjectiveTuning$eval_many(). The context available is ContextEval.

on_eval_after_benchmark

(function())
Stage called after hyperparameter configurations are evaluated. Called in ObjectiveTuning$eval_many(). The context available is ContextEval.

on_eval_before_archive

(function())
Stage called before performance values are written to the archive. Called in ObjectiveTuning$eval_many(). The context available is ContextEval.

on_optimizer_after_eval

(function())
Stage called after points are evaluated. Called in OptimInstance$eval_batch(). The context available is bbotk::ContextOptimization.

on_result

(function())
Stage called after result are written. Called in OptimInstance$assign_result(). The context available is bbotk::ContextOptimization.

on_optimization_end

(function())
Stage called at the end of the optimization. Called in Optimizer$optimize(). The context available is bbotk::ContextOptimization.

Details

When implementing a callback, each functions must have two arguments named callback and context.

A callback can write data to the state ($state), e.g. settings that affect the callback itself. Avoid writing large data the state. This can slow down the tuning process when the evaluation of configurations is parallelized.

Tuning callbacks access two different contexts depending on the stage. The stages on_eval_after_design, on_eval_after_benchmark, on_eval_before_archive access ContextEval. This context can be used to customize the evaluation of a batch of hyperparameter configurations. Changes to the state of callback are lost after the evaluation of a batch and changes to the tuning instance or the tuner are not possible. Persistent data should be written to the archive via $aggregated_performance (see ContextEval). The other stages access ContextOptimization. This context can be used to modify the tuning instance, archive, tuner and final result. There are two different contexts because the evaluation can be parallelized i.e. multiple instances of ContextEval exists on different workers at the same time.

Examples

# write archive to disk
callback_tuning("mlr3tuning.backup",
  on_optimization_end = function(callback, context) {
    saveRDS(context$instance$archive, "archive.rds")
  }
)
#> <CallbackTuning:mlr3tuning.backup>
#> * Active Stages: on_optimization_end