Skip to contents

Logging Object for Evaluated Hyperparameter Configurations

Source: R/ArchiveTuning.R
ArchiveTuning.Rd

Container around a data.table::data.table() which stores all evaluated hyperparameter configurations and performance scores.

Data structure

The table ($data) has the following columns:

  • One column for each hyperparameter of the search space ($search_space).

  • One column for each performance measure ($codomain).

  • x_domain (list())
    Lists of (transformed) hyperparameter values that are passed to the learner.

  • runtime_learners (numeric(1))
    Sum of training and predict times logged in learners per mlr3::ResampleResult / evaluation. This does not include potential overhead time.

  • timestamp (POSIXct)
    Time stamp when the evaluation was logged into the archive.

  • batch_nr (integer(1))
    Hyperparameters are evaluated in batches. Each batch has a unique batch number.

  • uhash (character(1))
    Connects each hyperparameter configuration to the resampling experiment stored in the mlr3::BenchmarkResult.

Each row corresponds to a single evaluation of a hyperparameter configuration.

The archive stores additionally a mlr3::BenchmarkResult ($benchmark_result) that records the resampling experiments. Each experiment corresponds to to a single evaluation of a hyperparameter configuration. The table ($data) and the benchmark result ($benchmark_result) are linked by the uhash column. If the results are viewed with as.data.table(), both are joined automatically.

Analysis

For analyzing the tuning results, it is recommended to pass the archive to as.data.table(). The returned data table is joined with the benchmark result which adds the mlr3::ResampleResult for each hyperparameter evaluation.

The archive provides various getters (e.g. $learners()) to ease the access. All getters extract by position (i) or unique hash (uhash). For a complete list of all getters see the methods section.

The benchmark result ($benchmark_result) allows to score the hyperparameter configurations again on a different measure. Alternatively, measures can be supplied to as.data.table().

The mlr3viz package provides visualizations for tuning results.

S3 Methods

  • as.data.table.ArchiveTuning(x, unnest = "x_domain", exclude_columns = "uhash", measures = NULL)
    Returns a tabular view of all evaluated hyperparameter configurations.
    ArchiveTuning -> data.table::data.table()

    • x (ArchiveTuning)

    • unnest (character())
      Transforms list columns to separate columns. Set to NULL if no column should be unnested.

    • exclude_columns (character())
      Exclude columns from table. Set to NULL if no column should be excluded.

    • measures (List of mlr3::Measure)
      Score hyperparameter configurations on additional measures.

Super class

bbotk::Archive -> ArchiveTuning

Public fields

benchmark_result

(mlr3::BenchmarkResult).

Methods

Public methods

Inherited methods

Method new()

Creates a new instance of this R6 class.

Usage

ArchiveTuning$new(search_space, codomain, check_values = TRUE)

Arguments

search_space

(paradox::ParamSet)
Hyperparameter search space. If NULL (default), the search space is constructed from the TuneToken of the learner's parameter set (learner$param_set).

codomain

(bbotk::Codomain)
Specifies codomain of objective function i.e. a set of performance measures. Internally created from provided mlr3::Measures.

check_values

(logical(1))
If TRUE (default), hyperparameter configurations are check for validity.

Method learner()

Retrieve mlr3::Learner of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive. Learner does not contain a model. Use $learners() to get learners with models.

Usage

ArchiveTuning$learner(i = NULL, uhash = NULL)

Arguments

i

(integer(1))
The iteration value to filter for.

uhash

(logical(1))
The uhash value to filter for.

Method learners()

Retrieve list of trained mlr3::Learner objects of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveTuning$learners(i = NULL, uhash = NULL)

Arguments

i

(integer(1))
The iteration value to filter for.

uhash

(logical(1))
The uhash value to filter for.

Method learner_param_vals()

Retrieve param values of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveTuning$learner_param_vals(i = NULL, uhash = NULL)

Arguments

i

(integer(1))
The iteration value to filter for.

uhash

(logical(1))
The uhash value to filter for.

Method predictions()

Retrieve list of mlr3::Prediction objects of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveTuning$predictions(i = NULL, uhash = NULL)

Arguments

i

(integer(1))
The iteration value to filter for.

uhash

(logical(1))
The uhash value to filter for.

Method resample_result()

Retrieve mlr3::ResampleResult of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveTuning$resample_result(i = NULL, uhash = NULL)

Arguments

i

(integer(1))
The iteration value to filter for.

uhash

(logical(1))
The uhash value to filter for.

Method print()

Printer.

Usage

ArchiveTuning$print()

Arguments

...

(ignored).

Method clone()

The objects of this class are cloneable with this method.

Usage

ArchiveTuning$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.