Logging Object for Evaluated Hyperparameter Configurations
Source:R/ArchiveTuning.R
ArchiveTuning.Rd
Container around a data.table::data.table()
which stores all evaluated
hyperparameter configurations and performance scores.
Data structure
The table ($data
) has the following columns:
One column for each hyperparameter of the search space (
$search_space
).One column for each performance measure (
$codomain
).x_domain
(list()
)
Lists of (transformed) hyperparameter values that are passed to the learner.runtime_learners
(numeric(1)
)
Sum of training and predict times logged in learners per mlr3::ResampleResult / evaluation. This does not include potential overhead time.timestamp
(POSIXct
)
Time stamp when the evaluation was logged into the archive.batch_nr
(integer(1)
)
Hyperparameters are evaluated in batches. Each batch has a unique batch number.uhash
(character(1)
)
Connects each hyperparameter configuration to the resampling experiment stored in the mlr3::BenchmarkResult.
Each row corresponds to a single evaluation of a hyperparameter configuration.
The archive stores additionally a mlr3::BenchmarkResult
($benchmark_result
) that records the resampling experiments. Each
experiment corresponds to to a single evaluation of a hyperparameter
configuration. The table ($data
) and the benchmark result
($benchmark_result
) are linked by the uhash
column. If the results are
viewed with as.data.table()
, both are joined automatically.
Analysis
For analyzing the tuning results, it is recommended to pass the archive to
as.data.table()
. The returned data table is joined with the benchmark
result which adds the mlr3::ResampleResult for each hyperparameter
evaluation.
The archive provides various getters (e.g. $learners()
) to ease the access.
All getters extract by position (i
) or unique hash (uhash
). For a
complete list of all getters see the methods section.
The benchmark result ($benchmark_result
) allows to score the hyperparameter
configurations again on a different measure. Alternatively, measures can be
supplied to as.data.table()
.
The mlr3viz package provides visualizations for tuning results.
S3 Methods
as.data.table.ArchiveTuning(x, unnest = "x_domain", exclude_columns = "uhash", measures = NULL)
Returns a tabular view of all evaluated hyperparameter configurations.
ArchiveTuning ->data.table::data.table()
x
(ArchiveTuning)unnest
(character()
)
Transforms list columns to separate columns. Set toNULL
if no column should be unnested.exclude_columns
(character()
)
Exclude columns from table. Set toNULL
if no column should be excluded.measures
(List of mlr3::Measure)
Score hyperparameter configurations on additional measures.
Super class
bbotk::Archive
-> ArchiveTuning
Methods
Method new()
Creates a new instance of this R6 class.
Usage
ArchiveTuning$new(search_space, codomain, check_values = TRUE)
Arguments
search_space
(paradox::ParamSet)
Hyperparameter search space. IfNULL
(default), the search space is constructed from the TuneToken of the learner's parameter set (learner$param_set).codomain
(bbotk::Codomain)
Specifies codomain of objective function i.e. a set of performance measures. Internally created from provided mlr3::Measures.check_values
(
logical(1)
)
IfTRUE
(default), hyperparameter configurations are check for validity.
Method learner()
Retrieve mlr3::Learner of the i-th evaluation, by position
or by unique hash uhash
. i
and uhash
are mutually exclusive.
Learner does not contain a model. Use $learners()
to get learners with
models.
Method learners()
Retrieve list of trained mlr3::Learner objects of the i-th evaluation,
by position or by unique hash uhash
. i
and uhash
are mutually
exclusive.
Method learner_param_vals()
Retrieve param values of the i-th evaluation, by position
or by unique hash uhash
. i
and uhash
are mutually exclusive.
Method predictions()
Retrieve list of mlr3::Prediction objects of the i-th evaluation, by
position or by unique hash uhash
. i
and uhash
are mutually
exclusive.
Method resample_result()
Retrieve mlr3::ResampleResult of the i-th evaluation, by position
or by unique hash uhash
. i
and uhash
are mutually exclusive.