Rush Data Storage

The ArchiveAsyncTuning stores all evaluated hyperparameter configurations and performance scores in a rush::Rush database.

Details

The ArchiveAsyncTuning is a connector to a rush::Rush database.

Data Structure

The table ($data) has the following columns:

• One column for each hyperparameter of the search space ($search_space).

• One (list-)column for the internal_tuned_values

• One column for each performance measure ($codomain).

• x_domain (list())
  Lists of (transformed) hyperparameter values that are passed to the learner.

• runtime_learners (numeric(1))
  Sum of training and predict times logged in learners per mlr3::ResampleResult / evaluation. This does not include potential overhead time.

• timestamp_xs (POSIXct)
  Time stamp when the hyperparameter configuration was sent to a worker.

• timestamp_ys (POSIXct)
  Time stamp when the evaluation result was written to the archive.

Analysis

For analyzing the tuning results, it is recommended to pass the ArchiveAsyncTuning to as.data.table(). The returned data table contains the mlr3::ResampleResult for each hyperparameter evaluation.

S3 Methods

• as.data.table.ArchiveAsyncTuning(x, unnest = "internal_tuned_values", exclude_columns = NULL, measures = NULL)
  Returns a tabular view of all evaluated hyperparameter configurations.
  ArchiveAsyncTuning -> data.table::data.table()
  

  • x (ArchiveAsyncTuning)

  • unnest (character())
    Transforms list columns to separate columns. Set to NULL if no column should be unnested.

  • exclude_columns (character())
    Exclude columns from table. Set to NULL if no column should be excluded.

  • measures (List of mlr3::Measure)
    Score hyperparameter configurations on additional measures.

Super classes

bbotk::Archive -> bbotk::ArchiveAsync -> ArchiveAsyncTuning

Active bindings

internal_search_space

(paradox::ParamSet)
The search space containing those parameters that are internally optimized by the mlr3::Learner.

benchmark_result

(mlr3::BenchmarkResult)
Benchmark result.

Methods

Public methods

• ArchiveAsyncTuning$new()

• ArchiveAsyncTuning$learner()

• ArchiveAsyncTuning$learners()

• ArchiveAsyncTuning$learner_param_vals()

• ArchiveAsyncTuning$predictions()

• ArchiveAsyncTuning$resample_result()

• ArchiveAsyncTuning$print()

• ArchiveAsyncTuning$clone()

Inherited methods

• bbotk::Archive$format()
• bbotk::Archive$help()
• bbotk::ArchiveAsync$best()
• bbotk::ArchiveAsync$clear()
• bbotk::ArchiveAsync$data_with_state()
• bbotk::ArchiveAsync$fail_point()
• bbotk::ArchiveAsync$fail_points()
• bbotk::ArchiveAsync$finish_point()
• bbotk::ArchiveAsync$finish_points()
• bbotk::ArchiveAsync$nds_selection()
• bbotk::ArchiveAsync$pop_point()
• bbotk::ArchiveAsync$push_failed_point()
• bbotk::ArchiveAsync$push_failed_points()
• bbotk::ArchiveAsync$push_finished_point()
• bbotk::ArchiveAsync$push_finished_points()
• bbotk::ArchiveAsync$push_point()
• bbotk::ArchiveAsync$push_points()
• bbotk::ArchiveAsync$push_result()
• bbotk::ArchiveAsync$push_running_point()
• bbotk::ArchiveAsync$push_running_points()

---

ArchiveAsyncTuning$new()

Creates a new instance of this R6 class.

Usage

ArchiveAsyncTuning$new(
  search_space,
  codomain,
  rush,
  internal_search_space = NULL
)

Arguments

search_space

(paradox::ParamSet)
Hyperparameter search space. If NULL (default), the search space is constructed from the paradox::TuneToken of the learner's parameter set (learner$param_set). When using to_tune() tokens, dependencies for hierarchical search spaces are automatically handled.

codomain

(bbotk::Codomain)
Specifies codomain of objective function i.e. a set of performance measures. Internally created from provided mlr3::Measures.

rush

(Rush)
If a rush instance is supplied, the tuning runs without batches.

internal_search_space

(paradox::ParamSet or NULL)
The internal search space.

---

ArchiveAsyncTuning$learner()

Retrieve mlr3::Learner of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive. Learner does not contain a model. Use $learners() to get learners with models.

Usage

ArchiveAsyncTuning$learner(i = NULL, uhash = NULL)

Arguments

i

(integer(1))
The iteration value to filter for.

uhash

(character(1))
The uhash value to filter for.

---

ArchiveAsyncTuning$learners()

Retrieve list of trained mlr3::Learner objects of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveAsyncTuning$learners(i = NULL, uhash = NULL)

Arguments

i

(integer(1))
The iteration value to filter for.

uhash

(character(1))
The uhash value to filter for.

---

ArchiveAsyncTuning$learner_param_vals()

Retrieve param values of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveAsyncTuning$learner_param_vals(i = NULL, uhash = NULL)

Arguments

i

(integer(1))
The iteration value to filter for.

uhash

(character(1))
The uhash value to filter for.

---

ArchiveAsyncTuning$predictions()

Retrieve list of mlr3::Prediction objects of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveAsyncTuning$predictions(i = NULL, uhash = NULL)

Arguments

i

(integer(1))
The iteration value to filter for.

uhash

(character(1))
The uhash value to filter for.

---

ArchiveAsyncTuning$resample_result()

Retrieve mlr3::ResampleResult of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveAsyncTuning$resample_result(i = NULL, uhash = NULL)

Arguments

i

(integer(1))
The iteration value to filter for.

uhash

(character(1))
The uhash value to filter for.

---

ArchiveAsyncTuning$print()

Printer.

Usage

ArchiveAsyncTuning$print()

---

ArchiveAsyncTuning$clone()

The objects of this class are cloneable with this method.

Usage

ArchiveAsyncTuning$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.