Package 'epwshiftr'

Description

Query, download climate change projection data from the CMIP6 (Coupled Model Intercomparison Project Phase 6) project in the ESGF (Earth System Grid Federation) platform, and create future EnergyPlus Weather (EPW) files adjusted from climate changes using data from Global Climate Models (GCM).

Package options

• epwshiftr.verbose: If TRUE, more detailed message will be printed. Default: FALSE.

• epwshiftr.progress: If TRUE, progress bars are shown for long-running operations that support them. Default: interactive().

• epwshiftr.threshold_alpha: the threshold of the absolute value for alpha, i.e. monthly-mean fractional change, when performing morphing operations. The default value is set to 3. If the morphing methods are set "stretch" or "combined", and the absolute alpha exceeds the threshold value, warnings are issued and the morphing method fallbacks to "shift" to avoid unrealistic morphed values.

• epwshiftr.dir_store: The persistent store directory for query snapshots, dictionaries, source mirrors, downloads, extraction results, outputs, and the store manifest. If not set, tools::R_user_dir() with type "data" will be used.

• epwshiftr.cache: Controls caching behavior. TRUE enables normal caching (default), FALSE disables caching entirely, and "offline" enables offline mode where only cached data is used and no network requests are made. Default: TRUE

• epwshiftr.dir_cache: The directory for disposable cache entries. Deleting this directory can require re-fetching or re-parsing data, but should not invalidate a persistent store.

Author(s)

Hongyuan Jia

See Also

Useful links:

• https://github.com/ideas-lab-nus/epwshiftr

• Report bugs at https://github.com/ideas-lab-nus/epwshiftr/issues

Description

[ is a one-dimensional shortcut for x$slice(i).

Usage

## S3 method for class 'EsgResult'
x[i, j, ..., drop = FALSE]

Arguments

Value

A new result object of the same type, or x for x[].

Description

data_node_status() is the user-facing replacement for the legacy data-node helper name used in earlier releases.

Usage

data_node_status(
  speed_test = FALSE,
  timeout = 3,
  index_node = INDEX_NODES[["ORNL"]]
)

Arguments

Value

A data.table::data.table() of 2 or 3 (when speed_test is TRUE) columns:

Examples

## Not run: 
data_node_status()

## End(Not run)

Description

Downloader provides a general purpose file download system with:

• File status management (missing, downloading, downloaded, verified)

• Incremental checksum verification during download

• Resume capability for interrupted downloads

• Async and parallel download using mirai

• Progress tracking

• Error handling and retry logic

Active bindings

Methods

Public methods

• Downloader$new()

• Downloader$download()

• Downloader$enqueue()

• Downloader$preflight()

• Downloader$run()

• Downloader$start()

• Downloader$jobs()

• Downloader$job_status()

• Downloader$job_logs()

• Downloader$stop_job()

• Downloader$daemon_start()

• Downloader$daemon_status()

• Downloader$daemon_stop()

• Downloader$sessions()

• Downloader$tasks()

• Downloader$status()

• Downloader$events()

• Downloader$on()

• Downloader$off()

• Downloader$data_nodes()

• Downloader$reset_data_nodes()

• Downloader$record_probes()

• Downloader$retry()

• Downloader$cancel()

• Downloader$resume()

• Downloader$verify()

• Downloader$cleanup_tmp()

• Downloader$get_tasks()

• Downloader$get_task_status()

• Downloader$wait_for_tasks()

• Downloader$cancel_task()

• Downloader$list_incomplete()

• Downloader$verify_checksum()

• Downloader$print()

• Downloader$clone()

Method new()

Create a new Downloader object

Usage

Downloader$new(
  dest = NULL,
  temp = NULL,
  retries = 3L,
  timeout = 3600L,
  ssl_verifypeer = TRUE,
  proxy = NULL,
  connect_timeout = NULL,
  useragent = NULL,
  cleanup = TRUE,
  n_workers = 4L,
  node_policy = NULL,
  transfer_policy = NULL,
  resource_policy = NULL,
  manifest = NULL
)

Arguments

Returns

An Downloader object.

Examples

\dontrun{
dl <- Downloader$new()
dl <- Downloader$new(dest = "~/data")
dl <- Downloader$new(
    dest = "~/data",
    temp = "~/data/.tmp",
    n_workers = 8
)
}

Method download()

Download a single file with state management and resume support

Usage

Downloader$download(
  url,
  filename = NULL,
  subdir = NULL,
  progress = TRUE,
  overwrite = FALSE,
  checksum = NULL,
  checksum_type = "sha256",
  resume = TRUE,
  block = TRUE,
  .tmp_id = NULL
)

Arguments

Returns

If block = TRUE, returns the path to the downloaded file. If block = FALSE, returns a task ID for tracking the download.

Examples

\dontrun{
# Blocking download
path <- dl$download(url = "https://example.com/data.nc")

# Async download
task_id <- dl$download(
    url = "https://example.com/data.nc",
    block = FALSE
)
dl$wait_for_tasks(task_id)

# Multiple files (async batch)
urls <- c("https://example.com/file1.nc", "https://example.com/file2.nc")
task_ids <- sapply(urls, function(url) {
    dl$download(url = url, block = FALSE)
})
results <- dl$wait_for_tasks(task_ids)
}

Method enqueue()

Add a download plan to the persistent manifest.

Usage

Downloader$enqueue(plan, session_label = NULL)

Arguments

Returns

The created session ID.

Method preflight()

Check local resource requirements before downloading.

Usage

Downloader$preflight(
  plan = NULL,
  session_id = NULL,
  task_id = NULL,
  overwrite = FALSE
)

Arguments

Returns

A one-row data frame with byte and disk-space summary.

Method run()

Run queued persistent download tasks.

Usage

Downloader$run(
  session_id = NULL,
  task_id = NULL,
  block = TRUE,
  progress = TRUE,
  overwrite = FALSE,
  resume = TRUE
)

Arguments

Returns

If block = TRUE, a data frame of selected task records after the run. If block = FALSE, a one-row background job record.

Method start()

Start a persistent download session in the background.

Usage

Downloader$start(
  session_id = NULL,
  task_id = NULL,
  overwrite = FALSE,
  resume = TRUE,
  mode = c("process", "daemon"),
  store_path = NULL
)

Arguments

Returns

A one-row data frame describing the background job.

Method jobs()

List downloader background jobs.

Usage

Downloader$jobs(status = NULL)

Arguments

Method job_status()

Return downloader background job status.

Usage

Downloader$job_status(job_id = NULL)

Arguments

Method job_logs()

Return downloader background job log lines.

Usage

Downloader$job_logs(job_id, tail = 100L)

Arguments

Method stop_job()

Request cancellation of a background downloader job.

Usage

Downloader$stop_job(job_id, force = FALSE)

Arguments

Method daemon_start()

Start a persistent downloader daemon.

Usage

Downloader$daemon_start(port = NULL, heartbeat_interval = 5)

Arguments

Returns

A one-row data frame describing the daemon.

Method daemon_status()

Return downloader daemon status records.

Usage

Downloader$daemon_status()

Method daemon_stop()

Request the running downloader daemon to stop.

Usage

Downloader$daemon_stop(force = FALSE)

Arguments

Method sessions()

List persistent download sessions.

Usage

Downloader$sessions()

Method tasks()

List persistent download tasks.

Usage

Downloader$tasks(session_id = NULL, job_id = NULL, status = NULL)

Arguments

Method status()

Return persistent download task status.

Usage

Downloader$status(session_id = NULL, job_id = NULL, task_id = NULL)

Arguments

Returns

A data frame of matching task records.

Method events()

Return persistent downloader event logs.

Usage

Downloader$events(session_id = NULL, job_id = NULL, task_id = NULL)

Arguments

Returns

A data frame of event records.

Method on()

Register an in-session downloader event callback.

Usage

Downloader$on(event, fun)

Arguments

Returns

A callback token for ⁠$off()⁠.

Method off()

Remove a downloader event callback.

Usage

Downloader$off(token)

Arguments

Returns

TRUE when a callback was removed.

Method data_nodes()

Return historical data node download performance.

Usage

Downloader$data_nodes(service = NULL)

Arguments

Returns

A data frame of data node performance records.

Method reset_data_nodes()

Reset historical data-node health records.

Usage

Downloader$reset_data_nodes(data_node = NULL, service = NULL)

Arguments

Returns

The remaining data-node records.

Method record_probes()

Record URL probe outcomes from a download plan into node history.

Usage

Downloader$record_probes(plan, probed = TRUE)

Arguments

Returns

The current data-node records.

Method retry()

Requeue failed or cancelled persistent tasks.

Usage

Downloader$retry(
  session_id = NULL,
  task_id = NULL,
  status = c("error", "cancelled")
)

Arguments

Returns

A data frame of requeued task records.

Method cancel()

Cancel queued or in-progress persistent download tasks.

Usage

Downloader$cancel(
  session_id = NULL,
  task_id = NULL,
  status = c("queued", "downloading")
)

Arguments

Returns

A data frame of cancelled task records.

Method resume()

Resume queued or interrupted persistent tasks.

Usage

Downloader$resume(session_id = NULL, task_id = NULL, ...)

Arguments

Returns

A data frame of selected task records after the run.

Method verify()

Verify checksums for completed persistent tasks.

Usage

Downloader$verify(session_id = NULL, task_id = NULL)

Arguments

Returns

A data frame of completed task records with a checksum_ok column.

Method cleanup_tmp()

Clean up temporary files (.part and .done files)

Usage

Downloader$cleanup_tmp(all = FALSE)

Arguments

Returns

Number of files removed.

Examples

\dontrun{
n_removed <- downloader$cleanup_tmp()
n_removed <- downloader$cleanup_tmp(all = TRUE)
}

Method get_tasks()

Get all async download tasks

Usage

Downloader$get_tasks()

Returns

A list of DownloadTask objects.

Examples

\dontrun{
tasks <- downloader$get_tasks()
}

Method get_task_status()

Get status of an async download task

Usage

Downloader$get_task_status(task_id)

Arguments

Returns

A list with task information including status, progress, etc.

Examples

\dontrun{
task_id <- dl$download(url, block = FALSE)
status <- dl$get_task_status(task_id)
}

Method wait_for_tasks()

Wait for all async download tasks to complete

Usage

Downloader$wait_for_tasks(task_ids = NULL, progress = TRUE)

Arguments

Returns

A list of completed task statuses.

Examples

\dontrun{
task1 <- dl$download(url1, block = FALSE)
task2 <- dl$download(url2, block = FALSE)
results <- dl$wait_for_tasks()
}

Method cancel_task()

Cancel an async download task

Usage

Downloader$cancel_task(task_id)

Arguments

Returns

Logical TRUE if cancellation was successful, FALSE otherwise.

Examples

\dontrun{
task_id <- dl$download(url, block = FALSE)
# Cancel if needed
dl$cancel_task(task_id)
}

Method list_incomplete()

List incomplete downloads

Usage

Downloader$list_incomplete()

Returns

A data.frame with information about incomplete downloads.

Examples

\dontrun{
incomplete <- downloader$list_incomplete()
}

Method verify_checksum()

Verify file checksum

Usage

Downloader$verify_checksum(file, expected, type = "sha256")

Arguments

Returns

TRUE if checksum matches, FALSE otherwise.

Examples

\dontrun{
valid <- downloader$verify_checksum("data.nc", "abc123", "sha256")
}

Method print()

Print downloader summary

Usage

Downloader$print()

Returns

The Downloader object itself, invisibly.

Method clone()

The objects of this class are cloneable with this method.

Usage

Downloader$clone(deep = FALSE)

Arguments

Author(s)

Hongyuan Jia

Examples

## ------------------------------------------------
## Method `Downloader$new`
## ------------------------------------------------

## Not run: 
dl <- Downloader$new()
dl <- Downloader$new(dest = "~/data")
dl <- Downloader$new(
    dest = "~/data",
    temp = "~/data/.tmp",
    n_workers = 8
)

## End(Not run)

## ------------------------------------------------
## Method `Downloader$download`
## ------------------------------------------------

## Not run: 
# Blocking download
path <- dl$download(url = "https://example.com/data.nc")

# Async download
task_id <- dl$download(
    url = "https://example.com/data.nc",
    block = FALSE
)
dl$wait_for_tasks(task_id)

# Multiple files (async batch)
urls <- c("https://example.com/file1.nc", "https://example.com/file2.nc")
task_ids <- sapply(urls, function(url) {
    dl$download(url = url, block = FALSE)
})
results <- dl$wait_for_tasks(task_ids)

## End(Not run)

## ------------------------------------------------
## Method `Downloader$cleanup_tmp`
## ------------------------------------------------

## Not run: 
n_removed <- downloader$cleanup_tmp()
n_removed <- downloader$cleanup_tmp(all = TRUE)

## End(Not run)

## ------------------------------------------------
## Method `Downloader$get_tasks`
## ------------------------------------------------

## Not run: 
tasks <- downloader$get_tasks()

## End(Not run)

## ------------------------------------------------
## Method `Downloader$get_task_status`
## ------------------------------------------------

## Not run: 
task_id <- dl$download(url, block = FALSE)
status <- dl$get_task_status(task_id)

## End(Not run)

## ------------------------------------------------
## Method `Downloader$wait_for_tasks`
## ------------------------------------------------

## Not run: 
task1 <- dl$download(url1, block = FALSE)
task2 <- dl$download(url2, block = FALSE)
results <- dl$wait_for_tasks()

## End(Not run)

## ------------------------------------------------
## Method `Downloader$cancel_task`
## ------------------------------------------------

## Not run: 
task_id <- dl$download(url, block = FALSE)
# Cancel if needed
dl$cancel_task(task_id)

## End(Not run)

## ------------------------------------------------
## Method `Downloader$list_incomplete`
## ------------------------------------------------

## Not run: 
incomplete <- downloader$list_incomplete()

## End(Not run)

## ------------------------------------------------
## Method `Downloader$verify_checksum`
## ------------------------------------------------

## Not run: 
valid <- downloader$verify_checksum("data.nc", "abc123", "sha256")

## End(Not run)

Description

Get an EPW morphing backend

Usage

epw_morph_backend(name = "belcher")

Arguments

Value

An EpwMorphBackend object.

Description

EPW morphing backends

Usage

epw_morph_backends()

Value

A character vector of registered backend names.

Description

EPW morphing periods

Usage

epw_morph_periods(...)

Arguments

Value

A data.table with columns period and year.

Description

EPW morphing recipe

Usage

epw_morph_recipe(name = "belcher", backend = name, methods = NULL)

Arguments

Value

A recipe list.

Description

Register an EPW morphing backend

Usage

epw_morph_register_backend(name, backend, overwrite = FALSE)

Arguments

Value

The backend object, invisibly.

Description

Backend runner functions return epw_morph_result objects. Use epw_morph_result() in custom backends after producing complete hourly EPW weather data.

Usage

epw_morph_result(
  context,
  epw = context$epw,
  data,
  parts = list(),
  diagnostics = morpher__empty_diagnostics(),
  factors = NULL
)

Arguments

Value

An epw_morph_result object.

Description

EPW morphing variable sets

Usage

epw_morph_variables(level = c("recommended", "minimal", "extended"))

Arguments

Value

A character vector of CMIP variable IDs.

Description

Create an EPW morpher

Usage

epw_morpher(
  store,
  epw,
  site_id = NULL,
  recipe = epw_morph_recipe("belcher"),
  label = NULL
)

Arguments

Value

An EpwMorpher object.

Description

EpwMorphBackend defines a statistical downscaling backend that can be selected by epw_morph_recipe() and executed by EpwMorpher.

Public fields

Methods

Public methods

• EpwMorphBackend$new()

• EpwMorphBackend$methods()

• EpwMorphBackend$method_choices()

• EpwMorphBackend$rules()

• EpwMorphBackend$required_variables()

• EpwMorphBackend$validate_methods()

• EpwMorphBackend$rules_with_methods()

• EpwMorphBackend$run()

• EpwMorphBackend$clone()

Method new()

Create an EPW morphing backend.

Usage

EpwMorphBackend$new(
  name,
  label = NULL,
  methods = NULL,
  method_choices = NULL,
  rules,
  requires_reference = FALSE,
  runner
)

Arguments

Method methods()

Return default backend methods.

Usage

EpwMorphBackend$methods()

Method method_choices()

Return allowed backend method values.

Usage

EpwMorphBackend$method_choices()

Method rules()

Return backend rules.

Usage

EpwMorphBackend$rules()

Method required_variables()

Return required CMIP variable IDs.

Usage

EpwMorphBackend$required_variables()

Method validate_methods()

Validate and complete method overrides.

Usage

EpwMorphBackend$validate_methods(methods = NULL)

Arguments

Method rules_with_methods()

Return backend rules with methods applied.

Usage

EpwMorphBackend$rules_with_methods(methods = NULL)

Arguments

Method run()

Run this backend on a canonical EPW morphing context.

Usage

EpwMorphBackend$run(context)

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

EpwMorphBackend$clone(deep = FALSE)

Arguments

Description

EpwMorpher consumes completed EsgStore extraction outputs and creates future EPW files through a store-backed morphing workflow.

Methods

Public methods

• EpwMorpher$new()

• EpwMorpher$required_variables()

• EpwMorpher$preflight()

• EpwMorpher$summarise_climate()

• EpwMorpher$summarise_baseline()

• EpwMorpher$plan()

• EpwMorpher$preview_plan()

• EpwMorpher$diagnose()

• EpwMorpher$check()

• EpwMorpher$run()

• EpwMorpher$write_epw()

• EpwMorpher$workflow()

• EpwMorpher$status()

• EpwMorpher$outputs()

• EpwMorpher$clone()

Method new()

Create an EPW morpher.

Usage

EpwMorpher$new(
  store,
  epw,
  site_id = NULL,
  recipe = epw_morph_recipe("belcher"),
  label = NULL
)

Arguments

Method required_variables()

Return recipe-required CMIP variable IDs.

Usage

EpwMorpher$required_variables()

Method preflight()

Preflight EPW morphing inputs without writing store state.

Usage

EpwMorpher$preflight(
  plan_id = NULL,
  periods = NULL,
  reference_plan_id = NULL,
  reference_periods = NULL,
  summary_id = NULL,
  reference_summary_id = NULL,
  baseline_id = NULL,
  by = c("source_id", "experiment_id", "variant_label", "period"),
  strict = TRUE
)

Arguments

Method summarise_climate()

Summarise extracted climate data by period and month.

Usage

EpwMorpher$summarise_climate(
  plan_id,
  periods,
  strict = TRUE,
  overwrite = FALSE
)

Arguments

Method summarise_baseline()

Summarise baseline EPW weather by month.

Usage

EpwMorpher$summarise_baseline(overwrite = FALSE)

Arguments

Method plan()

Create a morphing plan and monthly factors.

Usage

EpwMorpher$plan(
  summary_id,
  reference_summary_id = NULL,
  baseline_id = NULL,
  by = c("source_id", "experiment_id", "variant_label", "period"),
  strict = TRUE,
  overwrite = FALSE
)

Arguments

Method preview_plan()

Preview a morphing plan and monthly factors without writing store state.

Usage

EpwMorpher$preview_plan(
  summary_id,
  reference_summary_id = NULL,
  baseline_id = NULL,
  by = c("source_id", "experiment_id", "variant_label", "period"),
  strict = TRUE
)

Arguments

Method diagnose()

Diagnose a morphing plan.

Usage

EpwMorpher$diagnose(morph_id)

Arguments

Method check()

Abort if a morphing plan has blocking diagnostics.

Usage

EpwMorpher$check(morph_id)

Arguments

Method run()

Execute a morphing plan and write hourly result Parquet files.

Usage

EpwMorpher$run(morph_id, overwrite = FALSE, resume = TRUE)

Arguments

Method write_epw()

Write future EPW files from morphing results.

Usage

EpwMorpher$write_epw(
  morph_id,
  dir,
  separate = TRUE,
  overwrite = FALSE,
  resume = TRUE
)

Arguments

Method workflow()

Run the store-native EPW morphing workflow.

Usage

EpwMorpher$workflow(
  plan_id,
  periods,
  reference_plan_id = NULL,
  reference_periods = NULL,
  by = c("source_id", "experiment_id", "variant_label", "period"),
  strict = TRUE,
  dir = "outputs/future-epw",
  separate = TRUE,
  overwrite = FALSE,
  resume = TRUE
)

Arguments

Method status()

Return morphing plan status rows.

Usage

EpwMorpher$status(morph_id = NULL)

Arguments

Method outputs()

Return future EPW output rows.

Usage

EpwMorpher$outputs(morph_id = NULL)

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

EpwMorpher$clone(deep = FALSE)

Arguments

Author(s)

Hongyuan Jia

Description

epwshiftr_cli() is the package-level entry point used by the optional epwshiftr launcher. It exposes a small ESGF store management interface and returns status metadata when exit = FALSE, which makes it testable from R.

Usage

epwshiftr_cli(args = commandArgs(trailingOnly = TRUE), exit = FALSE)

Arguments

Value

Invisibly, a list with status, result, and error.

Description

esg_result() creates an empty query result object of input type, so that you can load the saved JSON file via EsgResult$load().

Usage

esg_result(type = c("dataset", "file", "aggregation"))

Arguments

Value

An empty EsgResult object of given type.

Description

EsgDataset provides a unified interface for accessing NetCDF data remotely via OPeNDAP protocol. It wraps RNetCDF functions and provides convenient methods for subsetting, slicing, and reading data without downloading entire files.

The class supports three levels of interfaces:

• Basic layer: Direct wrappers around RNetCDF functions

• Middle layer: Convenient methods for subsetting by time/space

• High layer: Data manipulation and format conversion

It also supports aggregating multiple files into a single logical dataset, automatically handling time dimension concatenation.

Active bindings

Methods

Public methods

• EsgDataset$new()

• EsgDataset$open()

• EsgDataset$close()

• EsgDataset$slice()

• EsgDataset$reachable()

• EsgDataset$file_inq()

• EsgDataset$var_inq()

• EsgDataset$dim_inq()

• EsgDataset$att_get()

• EsgDataset$var_get()

• EsgDataset$get_variables()

• EsgDataset$get_dimensions()

• EsgDataset$get_time_axis()

• EsgDataset$get_spatial_grid()

• EsgDataset$read_array()

• EsgDataset$read_data_table()

• EsgDataset$read_region()

• EsgDataset$selection()

• EsgDataset$print()

• EsgDataset$clone()

Method new()

Create a new EsgDataset object

Usage

EsgDataset$new(urls)

Arguments

Returns

An EsgDataset object.

Examples

\dontrun{
# Single file
ds <- EsgDataset$new("https://example.com/data.nc")

# Multiple files
ds <- EsgDataset$new(c("url1.nc", "url2.nc"))
}

Method open()

Open OPeNDAP connection(s)

Usage

EsgDataset$open(
  async = FALSE,
  timeout = NULL,
  progress = getOption("epwshiftr.progress", interactive())
)

Arguments

Returns

The EsgDataset object itself, invisibly.

Examples

\dontrun{
ds$open()
# Returns the opened dataset directly; no Mirai/Future to collect.
ds$open(async = TRUE, timeout = 10)
}

Method close()

Close OPeNDAP connection(s)

Usage

EsgDataset$close()

Returns

The EsgDataset object itself, invisibly.

Examples

\dontrun{
ds$close()
}

Method slice()

Select files from this dataset by file position.

⁠$slice()⁠ creates a new EsgDataset with a subset of the current dataset URLs. This is a file/URL-level operation; NetCDF variable, dimension, time, and spatial slicing is still performed by ⁠$var_get()⁠, ⁠$read_array()⁠, ⁠$read_data_table()⁠, and ⁠$read_region()⁠.

Usage

EsgDataset$slice(i, reopen = FALSE)

Arguments

Returns

A new EsgDataset object.

Method reachable()

Probe whether this dataset's current files or URLs are reachable.

⁠$reachable()⁠ checks the actual URLs or local paths stored in the dataset. It does not reuse reachability checks from an EsgResult; opened datasets, fallback downloads, and manually created datasets are always evaluated from their current url values.

Usage

EsgDataset$reachable(level = c("data_node", "url"), probe = NULL)

Arguments

Returns

A data.table with columns file_index, source_index, data_node, service, url, reachable, latency_ms, error, probe_level, probe_url, and probe_cached.

Method file_inq()

Get file information

Usage

EsgDataset$file_inq(index = 1L)

Arguments

Returns

A list with file information.

Examples

\dontrun{
info <- ds$file_inq()
}

Method var_inq()

Get variable information

Usage

EsgDataset$var_inq(var, index = 1L)

Arguments

Returns

A list with variable information.

Examples

\dontrun{
var_info <- ds$var_inq("tas")
}

Method dim_inq()

Get dimension information

Usage

EsgDataset$dim_inq(dim, index = 1L)

Arguments

Returns

A list with dimension information.

Examples

\dontrun{
dim_info <- ds$dim_inq("time")
}

Method att_get()

Get attribute value

Usage

EsgDataset$att_get(var, att, index = 1L)

Arguments

Returns

The attribute value.

Examples

\dontrun{
units <- ds$att_get("tas", "units")
}

Method var_get()

Read variable data

Usage

EsgDataset$var_get(
  var,
  start = NULL,
  count = NULL,
  index = 1L,
  collapse = FALSE,
  async = FALSE,
  timeout = NULL
)

Arguments

Returns

An array with variable data.

Examples

\dontrun{
data <- ds$var_get("tas")
data_subset <- ds$var_get("tas", start = c(1, 1, 1), count = c(10, 10, 1))
# Returns the final array directly; no Mirai/Future handling required.
data_async <- ds$var_get("tas", async = TRUE, timeout = 10)
}

Method get_variables()

List all variables in the dataset

Usage

EsgDataset$get_variables(index = 1L)

Arguments

Returns

A character vector of variable names.

Examples

\dontrun{
vars <- ds$get_variables()
}

Method get_dimensions()

List all dimensions in the dataset

Usage

EsgDataset$get_dimensions(index = 1L)

Arguments

Returns

A character vector of dimension names.

Examples

\dontrun{
dims <- ds$get_dimensions()
}

Method get_time_axis()

Get time axis information

Usage

EsgDataset$get_time_axis(index = 1L)

Arguments

Returns

A list containing time values, units, and calendar.

Examples

\dontrun{
time_info <- ds$get_time_axis()
}

Method get_spatial_grid()

Get spatial grid information (latitude and longitude)

Usage

EsgDataset$get_spatial_grid(index = 1L)

Arguments

Returns

A list containing latitude and longitude values.

Examples

\dontrun{
grid <- ds$get_spatial_grid()
}

Method read_array()

Read variable data as a list of arrays (one per file)

Usage

EsgDataset$read_array(
  variable,
  start = NULL,
  count = NULL,
  collapse = FALSE,
  async = FALSE,
  timeout = NULL
)

Arguments

Returns

A list of arrays with variable data. Each element corresponds to a file in the dataset.

Examples

\dontrun{
data_list <- ds$read_array("tas")
data <- data_list[[1]]
# Returns the final list directly; no Mirai/Future handling required.
data_list_async <- ds$read_array("tas", async = TRUE, timeout = 10)
}

Method read_data_table()

Read variable data as a list of data.table (one per file)

Usage

EsgDataset$read_data_table(
  variable,
  start = NULL,
  count = NULL,
  rbind = FALSE,
  async = FALSE,
  timeout = NULL
)

Arguments

Returns

If rbind = FALSE, a list of data.table (one per file). If rbind = TRUE, a single data.table with an extra file_index column.

Examples

\dontrun{
dt_list <- ds$read_data_table("tas")
dt <- dt_list[[1]]
dt_all <- ds$read_data_table("tas", rbind = TRUE)
# Returns the final data.table directly; no Mirai/Future handling required.
dt_async <- ds$read_data_table("tas", async = TRUE, timeout = 10)
}

Method read_region()

Read variable values near a target coordinate and optional time range

Usage

EsgDataset$read_region(
  variable,
  lon,
  lat,
  time = "auto",
  method = "nearest",
  rbind = TRUE,
  async = FALSE,
  timeout = NULL
)

Arguments

Returns

A data.table or list of data.tables with columns including file_index, variable, time, lon, lat, method, and value. The "grid_sources" attribute records contributing grid coordinates and weights.

Examples

\dontrun{
dt <- ds$read_region(
    variable = c("tas", "hurs"),
    lon = 103.98,
    lat = 1.37,
    time = c("2050-01-01", "2050-12-31")
)
}

Method selection()

Return file selection provenance for this dataset.

⁠$selection()⁠ maps the current dataset file positions back to the result rows that produced the dataset when that information is available. It does not record intermediate filter steps.

Usage

EsgDataset$selection()

Returns

A list with source_count, source_num_found, and source_indices.

Method print()

Print dataset summary

Usage

EsgDataset$print()

Returns

The EsgDataset object itself, invisibly.

Method clone()

The objects of this class are cloneable with this method.

Usage

EsgDataset$clone(deep = FALSE)

Arguments

Author(s)

Hongyuan Jia

Examples

## ------------------------------------------------
## Method `EsgDataset$new`
## ------------------------------------------------

## Not run: 
# Single file
ds <- EsgDataset$new("https://example.com/data.nc")

# Multiple files
ds <- EsgDataset$new(c("url1.nc", "url2.nc"))

## End(Not run)

## ------------------------------------------------
## Method `EsgDataset$open`
## ------------------------------------------------

## Not run: 
ds$open()
# Returns the opened dataset directly; no Mirai/Future to collect.
ds$open(async = TRUE, timeout = 10)

## End(Not run)

## ------------------------------------------------
## Method `EsgDataset$close`
## ------------------------------------------------

## Not run: 
ds$close()

## End(Not run)

## ------------------------------------------------
## Method `EsgDataset$file_inq`
## ------------------------------------------------

## Not run: 
info <- ds$file_inq()

## End(Not run)

## ------------------------------------------------
## Method `EsgDataset$var_inq`
## ------------------------------------------------

## Not run: 
var_info <- ds$var_inq("tas")

## End(Not run)

## ------------------------------------------------
## Method `EsgDataset$dim_inq`
## ------------------------------------------------

## Not run: 
dim_info <- ds$dim_inq("time")

## End(Not run)

## ------------------------------------------------
## Method `EsgDataset$att_get`
## ------------------------------------------------

## Not run: 
units <- ds$att_get("tas", "units")

## End(Not run)

## ------------------------------------------------
## Method `EsgDataset$var_get`
## ------------------------------------------------

## Not run: 
data <- ds$var_get("tas")
data_subset <- ds$var_get("tas", start = c(1, 1, 1), count = c(10, 10, 1))
# Returns the final array directly; no Mirai/Future handling required.
data_async <- ds$var_get("tas", async = TRUE, timeout = 10)

## End(Not run)

## ------------------------------------------------
## Method `EsgDataset$get_variables`
## ------------------------------------------------

## Not run: 
vars <- ds$get_variables()

## End(Not run)

## ------------------------------------------------
## Method `EsgDataset$get_dimensions`
## ------------------------------------------------

## Not run: 
dims <- ds$get_dimensions()

## End(Not run)

## ------------------------------------------------
## Method `EsgDataset$get_time_axis`
## ------------------------------------------------

## Not run: 
time_info <- ds$get_time_axis()

## End(Not run)

## ------------------------------------------------
## Method `EsgDataset$get_spatial_grid`
## ------------------------------------------------

## Not run: 
grid <- ds$get_spatial_grid()

## End(Not run)

## ------------------------------------------------
## Method `EsgDataset$read_array`
## ------------------------------------------------

## Not run: 
data_list <- ds$read_array("tas")
data <- data_list[[1]]
# Returns the final list directly; no Mirai/Future handling required.
data_list_async <- ds$read_array("tas", async = TRUE, timeout = 10)

## End(Not run)

## ------------------------------------------------
## Method `EsgDataset$read_data_table`
## ------------------------------------------------

## Not run: 
dt_list <- ds$read_data_table("tas")
dt <- dt_list[[1]]
dt_all <- ds$read_data_table("tas", rbind = TRUE)
# Returns the final data.table directly; no Mirai/Future handling required.
dt_async <- ds$read_data_table("tas", async = TRUE, timeout = 10)

## End(Not run)

## ------------------------------------------------
## Method `EsgDataset$read_region`
## ------------------------------------------------

## Not run: 
dt <- ds$read_region(
    variable = c("tas", "hurs"),
    lon = 103.98,
    lat = 1.37,
    time = c("2050-01-01", "2050-12-31")
)

## End(Not run)

Description

EsgDict is an R6 class for project-specific ESG controlled vocabulary data. It stores vocabulary tables, optional request tables, normalized query indices, and source metadata used by local option discovery and legality checks.

esgdict() is a small constructor around EsgDict$new().

Usage

esgdict(project = "CMIP6")

esgdict_set_default(dict)

esgdict_get_default(project = "CMIP6")

Arguments

Value

esgdict() returns a new EsgDict object. esgdict_set_default() returns dict, invisibly. esgdict_get_default() returns the current package-level default dictionary for project, or NULL.

Supported projects

The dictionary currently supports "CMIP6", "CMIP6PLUS", "INPUT4MIP", "OBS4REF", "CORDEX-CMIP6", "CMIP7", and "EMD". CMIP6 dictionaries include both controlled vocabularies and CMOR request-table data. Other projects use vocabulary data only until a project-specific request source is registered.

Source downloads in examples

Building a dictionary may download upstream vocabulary/request sources when the parsed dictionary cache and raw source cache are missing. Most examples load a small installed CMIP6 example dictionary and run without network access. The example that calls ⁠$build()⁠ is wrapped in ⁠\dontrun{}⁠ so package checks do not depend on GitHub or upstream CV availability.

Methods

Public methods

• EsgDict$new()

• EsgDict$project()

• EsgDict$profile()

• EsgDict$version()

• EsgDict$sources()

• EsgDict$timestamp()

• EsgDict$built_time()

• EsgDict$status()

• EsgDict$has_data()

• EsgDict$is_empty()

• EsgDict$build()

• EsgDict$get()

• EsgDict$capabilities()

• EsgDict$relation_fields()

• EsgDict$fields()

• EsgDict$indices()

• EsgDict$options()

• EsgDict$check()

• EsgDict$save()

• EsgDict$load()

• EsgDict$print()

Method new()

Create a new ESG project dictionary.

The new dictionary is empty. Use $build() to fetch and parse upstream sources, or $load() to restore a saved dictionary JSON file.

Usage

EsgDict$new(project = "CMIP6")

Arguments

Returns

An EsgDict object.

Examples

dict <- EsgDict$new(project = "CMIP6")
dict$status()

Method project()

Return the normalized ESG project identifier.

Usage

EsgDict$project()

Returns

A single string.

Examples

dict <- EsgDict$new(project = "CMIP6PLUS")
dict$project()

Method profile()

Return the internal dictionary profile.

The profile determines how project-specific vocabulary sources are parsed and normalized.

Usage

EsgDict$profile()

Returns

A single string.

Examples

dict <- EsgDict$new(project = "CMIP6")
dict$profile()

Method version()

Return vocabulary and request-source versions.

Empty or partially loaded dictionaries return NULL.

Usage

EsgDict$version()

Returns

A named list with vocab and request elements, or NULL.

Examples

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
dict$version()

Method sources()

Return upstream source metadata.

Source metadata records repository, tag/ref, commit, and local source directory information for the data used to build the dictionary.

Usage

EsgDict$sources()

Returns

A named list, or NULL for an empty dictionary.

Examples

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
dict$sources()

Method timestamp()

Return source vocabulary timestamps.

Timestamps are extracted from source vocabulary metadata when available.

Usage

EsgDict$timestamp()

Returns

A named list of timestamps, or NULL.

Examples

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
dict$timestamp()

Method built_time()

Return the time when this dictionary was built.

Usage

EsgDict$built_time()

Returns

A POSIXct value, or NULL.

Examples

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
dict$built_time()

Method status()

Return the dictionary lifecycle status.

Status values are:

• "empty": no vocabulary/request payload is loaded.

• "partial": some required payload is missing.

• "built": the dictionary was built in this R session.

• "loaded": the dictionary was restored from disk.

Usage

EsgDict$status()

Returns

A single string.

Examples

dict <- EsgDict$new(project = "CMIP6")
dict$status()

Method has_data()

Check whether the dictionary contains usable data.

A dictionary has usable data after a complete $build() or $load().

Usage

EsgDict$has_data()

Returns

TRUE or FALSE.

Examples

dict <- EsgDict$new(project = "CMIP6")
dict$has_data()

Method is_empty()

Check whether the dictionary is empty.

Usage

EsgDict$is_empty()

Returns

TRUE or FALSE.

Examples

dict <- EsgDict$new(project = "CMIP6")
dict$is_empty()

Method build()

Build the dictionary from upstream source data.

⁠$build()⁠ resolves the configured project vocabulary source, downloads or reuses raw source files as needed, parses them into normalized tables, and builds query indices for option discovery and validation. If the dictionary already has data and force = FALSE, the object is returned unchanged.

Usage

EsgDict$build(
  token = NULL,
  force = FALSE,
  cv_tag = NULL,
  request_tag = NULL,
  dreq_tag = NULL,
  use_cache = TRUE,
  source_dir = dict__source_dir(project = private$m_project)
)

Arguments

Returns

The modified EsgDict object itself.

Examples

\dontrun{
    dict <- EsgDict$new(project = "CMIP6")
    dict$build()
    dict$has_data()
}

Method get()

Return raw dictionary payload data.

⁠$get("vocab")⁠ returns the full vocabulary payload list. ⁠$get("request")⁠ and ⁠$get("dreq")⁠ return the request table when available. Any other value is interpreted as a vocabulary field name, such as "experiment_id" or "source_id".

Usage

EsgDict$get(type)

Arguments

Returns

A copy of the requested data.

Examples

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
dict$get("experiment_id")
dict$get("request")

Method capabilities()

Return available dictionary capabilities.

Capabilities describe whether vocabulary data, request data, and relation indices are currently available.

Usage

EsgDict$capabilities()

Returns

A named list with vocab, request, and relations.

Examples

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
dict$capabilities()

Method relation_fields()

Return supported relation-index fields.

Relation fields describe which field combinations can be used for constrained option discovery and cross-field legality checks.

Usage

EsgDict$relation_fields()

Returns

A named list of character vectors.

Examples

dict <- EsgDict$new(project = "CMIP6")
dict$relation_fields()

Method fields()

Return normalized dictionary field names.

Empty dictionaries return character().

Usage

EsgDict$fields()

Returns

A character vector.

Examples

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
dict$fields()

Method indices()

Return normalized dictionary indices.

⁠$indices()⁠ returns all available indices. ⁠$indices(type)⁠ returns a single index table, such as "values", "variable", "activity_experiment", or "activity_source".

Usage

EsgDict$indices(type = NULL)

Arguments

Returns

A named list of indices, or a data.table::data.table() when type is supplied.

Examples

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
names(dict$indices())
dict$indices("values")

Method options()

Discover valid values for a dictionary field.

Constraints supplied through ... are used when a matching relation index exists. For example, CMIP6 experiment_id options can be constrained by activity_id.

Usage

EsgDict$options(field, ...)

Arguments

Returns

A data.table::data.table() with available values and metadata. The ignored_constraints attribute records constraints that could not be applied.

Examples

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
dict$options("experiment_id", activity_id = "CMIP")

Method check()

Check dictionary values and relationships.

⁠$check()⁠ validates supplied values against dictionary value indices and, when possible, validates cross-field combinations using relation indices.

Usage

EsgDict$check(
  ...,
  error = FALSE,
  suggest = TRUE,
  n_suggestions = 5L,
  relationship = c("any", "all_pairs")
)

Arguments

Returns

An esgdict_check_result data.table::data.table().

Examples

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
dict$check(activity_id = "CMIP", experiment_id = "historical")
dict$check(variable_id = "tas", table_id = "Amon")

Method save()

Save the dictionary to JSON.

If path = NULL, the dictionary is saved in the package store and registered in the store manifest. If path is supplied, only that JSON file is written.

Usage

EsgDict$save(path = NULL, allow_empty = FALSE)

Arguments

Returns

The normalized output path.

Examples

dict_path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(dict_path))

path <- tempfile(fileext = ".json")
dict$save(path)
file.exists(path)

Method load()

Load a dictionary from JSON.

If path = NULL, the latest stored dictionary for this project is located through the package store manifest.

Usage

EsgDict$load(path = NULL)

Arguments

Returns

The modified EsgDict object itself.

Examples

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
restored <- EsgDict$new(project = "CMIP6")
suppressMessages(restored$load(path))
restored$has_data()

Method print()

Print a dictionary summary.

Usage

EsgDict$print()

Returns

The EsgDict object itself, invisibly.

Examples

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
dict$print()

Author(s)

Hongyuan Jia

See Also

esgdict_option() and esgdict_check() for user-facing discovery and validation helpers.

Examples

example_path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- esgdict(project = "CMIP6")
suppressMessages(dict$load(example_path))
dict$project()

esgdict_set_default(dict)
identical(esgdict_get_default("CMIP6"), dict)
esgdict_option("experiment_id", activity_id = "CMIP")
esgdict_check(activity = "CMIP", experiment = "historical")

## ------------------------------------------------
## Method `EsgDict$new`
## ------------------------------------------------

dict <- EsgDict$new(project = "CMIP6")
dict$status()

## ------------------------------------------------
## Method `EsgDict$project`
## ------------------------------------------------

dict <- EsgDict$new(project = "CMIP6PLUS")
dict$project()

## ------------------------------------------------
## Method `EsgDict$profile`
## ------------------------------------------------

dict <- EsgDict$new(project = "CMIP6")
dict$profile()

## ------------------------------------------------
## Method `EsgDict$version`
## ------------------------------------------------

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
dict$version()

## ------------------------------------------------
## Method `EsgDict$sources`
## ------------------------------------------------

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
dict$sources()

## ------------------------------------------------
## Method `EsgDict$timestamp`
## ------------------------------------------------

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
dict$timestamp()

## ------------------------------------------------
## Method `EsgDict$built_time`
## ------------------------------------------------

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
dict$built_time()

## ------------------------------------------------
## Method `EsgDict$status`
## ------------------------------------------------

dict <- EsgDict$new(project = "CMIP6")
dict$status()

## ------------------------------------------------
## Method `EsgDict$has_data`
## ------------------------------------------------

dict <- EsgDict$new(project = "CMIP6")
dict$has_data()

## ------------------------------------------------
## Method `EsgDict$is_empty`
## ------------------------------------------------

dict <- EsgDict$new(project = "CMIP6")
dict$is_empty()

## ------------------------------------------------
## Method `EsgDict$build`
## ------------------------------------------------

## Not run: 
    dict <- EsgDict$new(project = "CMIP6")
    dict$build()
    dict$has_data()

## End(Not run)

## ------------------------------------------------
## Method `EsgDict$get`
## ------------------------------------------------

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
dict$get("experiment_id")
dict$get("request")

## ------------------------------------------------
## Method `EsgDict$capabilities`
## ------------------------------------------------

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
dict$capabilities()

## ------------------------------------------------
## Method `EsgDict$relation_fields`
## ------------------------------------------------

dict <- EsgDict$new(project = "CMIP6")
dict$relation_fields()

## ------------------------------------------------
## Method `EsgDict$fields`
## ------------------------------------------------

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
dict$fields()

## ------------------------------------------------
## Method `EsgDict$indices`
## ------------------------------------------------

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
names(dict$indices())
dict$indices("values")

## ------------------------------------------------
## Method `EsgDict$options`
## ------------------------------------------------

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
dict$options("experiment_id", activity_id = "CMIP")

## ------------------------------------------------
## Method `EsgDict$check`
## ------------------------------------------------

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
dict$check(activity_id = "CMIP", experiment_id = "historical")
dict$check(variable_id = "tas", table_id = "Amon")

## ------------------------------------------------
## Method `EsgDict$save`
## ------------------------------------------------

dict_path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(dict_path))

path <- tempfile(fileext = ".json")
dict$save(path)
file.exists(path)

## ------------------------------------------------
## Method `EsgDict$load`
## ------------------------------------------------

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
restored <- EsgDict$new(project = "CMIP6")
suppressMessages(restored$load(path))
restored$has_data()

## ------------------------------------------------
## Method `EsgDict$print`
## ------------------------------------------------

path <- system.file("extdata", "examples", "cmip6-dict.json", package = "epwshiftr")
dict <- EsgDict$new(project = "CMIP6")
suppressMessages(dict$load(path))
dict$print()

Description

esgdict_check() validates project parameter values against the local ESG dictionary. It checks individual field values and cross-field relationships represented in the dictionary's normalized query indices.

Usage

esgdict_check(
  ...,
  project = NULL,
  dict = NULL,
  error = FALSE,
  suggest = TRUE,
  n_suggestions = 5L,
  relationship = c("any", "all_pairs")
)

Arguments

Value

An esgdict_check_result data.table::data.table().

Description

esgdict_option() returns valid values for an ESG dictionary field, optionally constrained by other supplied fields.

Usage

esgdict_option(field, ..., project = NULL, dict = NULL, warn_ignored = TRUE)

Arguments

Value

A data.table::data.table() with at least field, value, and description columns. The ignored_constraints attribute records constraints that were not used.

Description

The Earth System Grid Federation (ESGF) is an international collaboration for the software that powers most global climate change research, notably assessments by the Intergovernmental Panel on Climate Change (IPCC).

The ESGF search service exposes RESTful APIs that can be used by clients to query the contents of the underlying search index, and return results matching the given constraints. The documentation of the APIs can be found using this link.

EsgQuery is the workhorse for dealing with ESGF search services. Start with esg_query() / EsgQuery for new workflow code. The legacy data.table-oriented API is available from the legacy branch or v0.1.4.

Usage

esg_query(index_node = "https://esgf-node.ornl.gov")

Arguments

EsgQuery object

esg_query() returns an EsgQuery object, which is an R6 object with quite a few methods that can be classified into 3 categories:

• Value listing: methods to list all possible values of facets, fields, shards, and values.

• Parameter getter & setter: methods to get the query parameter values or set them before sending the actual query to the ESGF search services.

• Query responses: methods to collect results for the query response.

Value listing

EsgQuery object provides the following value-listing methods to query available facets, fields, shards, and values from the ESGF index node:

• EsgQuery$list_facets(): List all available facet names. When called, a facet listing query is sent to the index node to get all available facets for the current project (default: CMIP6).

• EsgQuery$list_fields(): List all available field names. This is useful for bridge index nodes where facet listing is not available.

• EsgQuery$list_shards(): List all available shards (ESGF index nodes) that can be queried in distributed searches.

• EsgQuery$list_values(): List all available values of specific facets.

Parameter getter & setter

The ESGF search services support a lot of parameters. The EsgQuery contains dedicated methods to set values for most of them, including:

• Most common keywords: facets, offset, limit, fields, replica, latest, distrib and shards.

• Most common facets: project, activity_id, experiment_id, source_id, variable_id, frequency, variant_label, nominal_resolution, datetime_range, timestamp_range, version_range and data_node.

All methods act in a similar way:

• If input is given, the corresponding parameter is set and the updated EsgQuery object is returned.

  • This makes it possible to chain different parameter setters, e.g. EsgQuery$project("CMIP6")$frequency("day")$limit(1) sets the parameter project, frequency and limit sequentially.

  • For parameters that want character inputs, you can put a preceding ! to negate the constraints, e.g. EsgQuery$project(!"CMIP6") searches for all projects except for CMIP6.

• If no input is given, the current parameter value is returned. For example, directly calling EsgQuery$project() returns the current value of the project parameter. The returned value can be two types:

  • NULL, i.e. there is no constraint on the corresponding parameter

  • A QueryParam object. Use query_param__value() and query_param__negate() to inspect it.

Despite methods for specific keywords and facets, you can specify arbitrary query parameters using EsgQuery$params() method. For details on the usage, please see the documentation.

Query responses

The query is not sent unless related methods are called:

• EsgQuery$count(): Count the total number of records that match the query.

  • You can return only the total number of matched record by calling EsgQuery$count(facets = FALSE)

  • You can also count the matched records for specified facets, e.g. EsgQuery$count(facets = c("source_id", "activity_id"))

• EsgQuery$collect(): Collect the query results and format it into an EsgResultDataset object.

Bridge Index Nodes

Some ESGF index nodes are "bridge" nodes that have certain limitations compared to standard index nodes. When using a bridge index node (e.g., ⁠https://esgf-node.ornl.gov/esgf-1-5-bridge⁠), the following restrictions apply:

• The fields parameter is not supported. All available fields are always returned.

• Only Dataset and File queries are supported. Aggregation queries should use a standard ESGF search index node, such as ⁠https://esgf-data.dkrz.de⁠ or ⁠https://esgf.ceda.ac.uk⁠.

• The retracted parameter is not supported and will be ignored.

• Wget script generation is not supported. Calling ⁠$url(wget = TRUE)⁠ will result in an error.

• Facet listing is not available. ⁠$list_facets()⁠ will return a predefined set of common facets instead. Use ⁠$list_fields()⁠ to get all available fields.

Other helpers

EsgQuery object also provides several other helper functions:

• Query URL generation:

  • EsgQuery$url(): Returns the actual query URL or the wget script URL which can be used to download all files matching the given constraints.

• State persistence:

  • EsgQuery$save(): Save the query state to a JSON file for later use.

  • EsgQuery$load(): Restore the query state from a JSON file created by ⁠$save()⁠.

• Display:

  • EsgQuery$print(): Print a summary of the current EsgQuery object including the index node URL and all query parameters.

Methods

Public methods

• EsgQuery$new()

• EsgQuery$index_node()

• EsgQuery$list_facets()

• EsgQuery$list_fields()

• EsgQuery$list_shards()

• EsgQuery$list_values()

• EsgQuery$project()

• EsgQuery$activity_id()

• EsgQuery$experiment_id()

• EsgQuery$source_id()

• EsgQuery$variable_id()

• EsgQuery$frequency()

• EsgQuery$variant_label()

• EsgQuery$nominal_resolution()

• EsgQuery$data_node()

• EsgQuery$facets()

• EsgQuery$fields()

• EsgQuery$shards()

• EsgQuery$datetime_range()

• EsgQuery$timestamp_range()

• EsgQuery$version_range()

• EsgQuery$replica()

• EsgQuery$latest()

• EsgQuery$limit()

• EsgQuery$offset()

• EsgQuery$distrib()

• EsgQuery$params()

• EsgQuery$url()

• EsgQuery$count()

• EsgQuery$collect()

• EsgQuery$state()

• EsgQuery$reset()

• EsgQuery$save()

• EsgQuery$load()

• EsgQuery$print()

• EsgQuery$clone()

Method new()

Create a new EsgQuery object

Usage

EsgQuery$new(index_node = "https://esgf-node.ornl.gov")

Arguments

Returns

An EsgQuery object.

Examples

\dontrun{
q <- EsgQuery$new(index_node = "https://esgf-node.ornl.gov")
q
}

Method index_node()

Get or set the ESGF index node.

⁠$index_node()⁠ returns the current normalized index node URL. ⁠$index_node(value)⁠ updates the index node after applying the same normalization used by EsgQuery$new(). Existing query parameters are kept unchanged.

Usage

EsgQuery$index_node(value)

Arguments

Returns

If value is supplied, the modified EsgQuery object. Otherwise, a string.

Examples

\dontrun{
q$index_node()
q$index_node("https://esgf.ceda.ac.uk")
}

Method list_facets()

List all available facet names

Usage

EsgQuery$list_facets(force = FALSE)

Arguments

Returns

A character vector.

Examples

\dontrun{
q$list_facets()
}

Method list_fields()

List all available field names

Usage

EsgQuery$list_fields(force = FALSE)

Arguments

Returns

A character vector or NULL if no facet listing is found.

Examples

\dontrun{
q$list_fields()
}

Method list_shards()

List all available shards.

Usage

EsgQuery$list_shards(force = FALSE)

Arguments

Returns

A character vector or NULL if no shard listing is found.

Examples

\dontrun{
q$list_shards()
}

Method list_values()

List all available values of specific facets.

Usage

EsgQuery$list_values(facets, force = FALSE)

Arguments

Returns

If length(facets) == 1, a named integer vector giving the facet value counts. Otherwise, a list of named integer vectors of the same length as facets.

Examples

\dontrun{
q$list_values(c("activity_id", "experiment_id"))
}

Method project()

Get or set the project facet parameter.

Usage

EsgQuery$project(value = "CMIP6")

Arguments

Returns

If value is supplied, the modified EsgQuery object. Otherwise, a QueryParam object or NULL.

Method activity_id()

Get or set the activity_id facet parameter.

Usage

EsgQuery$activity_id(value)

Arguments

Returns

If value is supplied, the modified EsgQuery object. Otherwise, a QueryParam object or NULL.

Method experiment_id()

Get or set the experiment_id facet parameter.

Usage

EsgQuery$experiment_id(value)

Arguments

Returns

If value is supplied, the modified EsgQuery object. Otherwise, a QueryParam object or NULL.

Method source_id()

Get or set the source_id facet parameter.

Usage

EsgQuery$source_id(value)

Arguments

Returns

If value is supplied, the modified EsgQuery object. Otherwise, a QueryParam object or NULL.

Method variable_id()

Get or set the variable_id facet parameter.

Usage

EsgQuery$variable_id(value)

Arguments

Returns

If value is supplied, the modified EsgQuery object. Otherwise, a QueryParam object or NULL.

Method frequency()

Get or set the frequency facet parameter.

Usage

EsgQuery$frequency(value)

Arguments

Returns

If value is supplied, the modified EsgQuery object. Otherwise, a QueryParam object or NULL.

Method variant_label()

Get or set the variant_label facet parameter.

Usage

EsgQuery$variant_label(value)

Arguments

Returns

If value is supplied, the modified EsgQuery object. Otherwise, a QueryParam object or NULL.

Method nominal_resolution()

Get or set the nominal_resolution facet parameter.

Usage

EsgQuery$nominal_resolution(value)

Arguments

Returns

If value is supplied, the modified EsgQuery object. Otherwise, a QueryParam object or NULL.

Method data_node()

Get or set the data_node facet parameter.

Usage

EsgQuery$data_node(value)

Arguments

Returns

If value is supplied, the modified EsgQuery object. Otherwise, a QueryParam object or NULL.

Method facets()

Get or set the facets parameter used by ⁠$count()⁠.

Usage

EsgQuery$facets(value)

Arguments

Returns

If value is supplied, the modified EsgQuery object. Otherwise, a QueryParam object or NULL.

Method fields()

Get or set the fields parameter.

Usage

EsgQuery$fields(value = "*")

Arguments

Returns

If value is supplied, the modified EsgQuery object. Otherwise, a QueryParam object or NULL.

Method shards()

Get or set the shards parameter for distributed searches.

Usage

EsgQuery$shards(value)

Arguments

Returns

If value is supplied, the modified EsgQuery object. Otherwise, a QueryParam object or NULL.

Method datetime_range()

Get or set temporal coverage overlap constraints.

Usage

EsgQuery$datetime_range(start, stop)

Arguments

Returns

If either boundary is supplied, the modified EsgQuery object. Otherwise, a list with start and stop elements.

Method timestamp_range()

Get or set Solr index timestamp range constraints.

Usage

EsgQuery$timestamp_range(from, to)

Arguments

Returns

If either boundary is supplied, the modified EsgQuery object. Otherwise, a list with from and to elements.

Method version_range()

Get or set version range constraints.

Usage

EsgQuery$version_range(min, max)

Arguments

Returns

If either boundary is supplied, the modified EsgQuery object. Otherwise, a list with min and max elements.

Method replica()

Get or set the replica parameter.

Usage

EsgQuery$replica(value)

Arguments

Returns

If value is supplied, the modified EsgQuery object. Otherwise, a QueryParam object or NULL.

Method latest()

Get or set the latest parameter.

Usage

EsgQuery$latest(value = NULL)

Arguments

Returns

If value is supplied, the modified EsgQuery object. Otherwise, a QueryParam object or NULL.

Method limit()

Get or set the limit parameter.

Usage

EsgQuery$limit(value = 10L)

Arguments

Returns

If value is supplied, the modified EsgQuery object. Otherwise, a QueryParam object or NULL.

Method offset()

Get or set the offset parameter.

Usage

EsgQuery$offset(value = 0L)

Arguments

Returns

If value is supplied, the modified EsgQuery object. Otherwise, a QueryParam object or NULL.

Method distrib()

Get or set the distrib parameter.

Usage

EsgQuery$distrib(value = TRUE)

Arguments

Returns

If value is supplied, the modified EsgQuery object. Otherwise, a QueryParam object or NULL.

Method params()

Get or set ad hoc query parameters.

⁠$params()⁠ handles parameters without dedicated methods and can also update supported dedicated parameters by name. The type and format control parameters cannot be changed here: EsgQuery always performs Dataset queries and always parses JSON responses. Use EsgResultDataset$collect() to collect File or Aggregation records from Dataset results.

Usage

EsgQuery$params(...)

Arguments

Returns

If parameters are supplied, the modified EsgQuery object. Otherwise, a named list of QueryParam objects.

Method url()

Get the URL of actual query or wget script

The wget script URL can be used to download a bash script that contains wget commands for downloading all files matching the query constraints. This is useful for batch downloading large amounts of data.

Usage

EsgQuery$url(wget = FALSE)

Arguments

Returns

A single string.

Examples

\dontrun{
q$url()

# get the wget script URL
q$url(wget = TRUE)

# You can download the wget script using the URL directly. For
# example, the code below downloads the script and save it as
# 'wget.sh' in R's temporary folder:
download.file(q$url(TRUE), file.path(tempdir(), "wget.sh"), mode = "wb")

}

Method count()

Send a query of facet counting and fetch the results

Usage

EsgQuery$count(facets = TRUE)

Arguments

Returns

• If facets equals NULL or FALSE, or ⁠$facets()⁠ returns NULL, an integer.

• Otherwise, a named list with the first element always being total which is the total number of matched records. Other elements have the same length as input facets and are all named integer vectors.

Examples

\dontrun{
# get the total number of matched records
q$count(NULL) # or q$count(facets = FALSE)

# count records for specific facets
q$facets(c("activity_id", "source_id"))$count()

# same as above
q$count(facets = c("activity_id", "source_id"))
}

Method collect()

Send the actual query and fetch the results

⁠$collect()⁠ sends the actual query to the ESGF search services. By default it collects type=Dataset results and returns an EsgResultDataset object. If type is "File" or "Aggregation", it first collects matching Dataset results and then collects child File or Aggregation results for those datasets. The fields included depend on fields parameter. However, the following fields are always included in the results: ⁠access⁠, ⁠data_node⁠, ⁠id⁠, ⁠index_node⁠, ⁠instance_id⁠, ⁠latest⁠, ⁠master_id⁠, ⁠number_of_aggregations⁠, ⁠number_of_files⁠, ⁠replica⁠, ⁠size⁠, ⁠url⁠, ⁠version⁠. When a local EsgDict is available for the query project, ⁠$collect()⁠ also performs a warning-only dictionary check before sending the query. Missing local dictionaries are ignored and never downloaded.

Usage

EsgQuery$collect(
  all = FALSE,
  limit = TRUE,
  params = TRUE,
  type = "Dataset",
  fields = NULL,
  progress = getOption("epwshiftr.progress", interactive()),
  ...
)

Arguments

Returns

An EsgResultDataset, EsgResultFile, or EsgResultAggregation object.

Examples

\dontrun{
# by default, all fields with constrains are included in the results
query <- esg_query()$experiment_id("ssp585")$frequency("1hr")$fields("source_id")
res1 <- query$collect()
res1$fields

# set `params` to `FALSE` to exclude them
query$collect(params = FALSE)$fields

# collect all matched records with `query$limit()` records per query
res2 <- query$collect(all = TRUE, limit = TRUE)
identical(query$count(), res2$count())

# same as above, but collect all matched records with max allowed
# record limit per query
res3 <- query$collect(all = TRUE, limit = FALSE)
identical(res2$count(), res3$count())

# same as above, but collect all matched records with specified limit
# per query
res4 <- query$collect(all = TRUE, limit = 30)
identical(res2$count(), res4$count())
}

Method state()

Get the current query state.

⁠$state()⁠ returns a read-only snapshot containing the current index node and the current parameter state.

Usage

EsgQuery$state(name = NULL, null = FALSE)

Arguments

Returns

A named list with elements index_node and parameter.

Examples

\dontrun{
q$state()
q$state(null = TRUE)
}

Method reset()

Reset query parameters to their defaults.

⁠$reset()⁠ clears the current parameter store and restores the default query parameters. The current index node is kept unchanged.

Usage

EsgQuery$reset()

Returns

The modified EsgQuery object itself.

Examples

\dontrun{
q$experiment_id("ssp585")$reset()
}

Method save()

Save the query into a JSON file

⁠$save()⁠ puts main data of an EsgQuery object into a JSON file which can be loaded to restore the current state of query using EsgQuery$load().

Usage

EsgQuery$save(file = "query.json", pretty = TRUE)

Arguments

Returns

The full path of the output JSON file.

Examples

\dontrun{
q$save(tempfile(fileext = ".json"))
}

Method load()

Restore the query state from an JSON file

⁠$load()⁠ reads data of an EsgQuery object from a JSON file created using EsgQuery$save().

Usage

EsgQuery$load(file)

Arguments

Returns

The modified EsgQuery object itself.

Examples

\dontrun{
f <- tempfile(fileext = "json")

q <- esg_query()
json <- q$save(f)
q$load(f)
}

Method print()

Print a summary of the current EsgQuery object

⁠$print()⁠ gives the summary of current EsgQuery object including the index node URL and all query parameters.

Usage

EsgQuery$print()

Returns

The EsgQuery object itself, invisibly.

Examples

\dontrun{
q$print()
}

Method clone()

The objects of this class are cloneable with this method.

Usage

EsgQuery$clone(deep = FALSE)

Arguments

Note

For bridge index nodes, only predefined common facets are returned. ⁠$list_fields()⁠ can be used to get all available fields, including facets.

Author(s)

Hongyuan Jia

Examples

## ------------------------------------------------
## Method `EsgQuery$new`
## ------------------------------------------------

## Not run: 
q <- EsgQuery$new(index_node = "https://esgf-node.ornl.gov")
q

## End(Not run)

## ------------------------------------------------
## Method `EsgQuery$index_node`
## ------------------------------------------------

## Not run: 
q$index_node()
q$index_node("https://esgf.ceda.ac.uk")

## End(Not run)

## ------------------------------------------------
## Method `EsgQuery$list_facets`
## ------------------------------------------------

## Not run: 
q$list_facets()

## End(Not run)

## ------------------------------------------------
## Method `EsgQuery$list_fields`
## ------------------------------------------------

## Not run: 
q$list_fields()

## End(Not run)

## ------------------------------------------------
## Method `EsgQuery$list_shards`
## ------------------------------------------------

## Not run: 
q$list_shards()

## End(Not run)

## ------------------------------------------------
## Method `EsgQuery$list_values`
## ------------------------------------------------

## Not run: 
q$list_values(c("activity_id", "experiment_id"))

## End(Not run)

## ------------------------------------------------
## Method `EsgQuery$url`
## ------------------------------------------------

## Not run: 
q$url()

# get the wget script URL
q$url(wget = TRUE)

# You can download the wget script using the URL directly. For
# example, the code below downloads the script and save it as
# 'wget.sh' in R's temporary folder:
download.file(q$url(TRUE), file.path(tempdir(), "wget.sh"), mode = "wb")


## End(Not run)

## ------------------------------------------------
## Method `EsgQuery$count`
## ------------------------------------------------

## Not run: 
# get the total number of matched records
q$count(NULL) # or q$count(facets = FALSE)

# count records for specific facets
q$facets(c("activity_id", "source_id"))$count()

# same as above
q$count(facets = c("activity_id", "source_id"))

## End(Not run)

## ------------------------------------------------
## Method `EsgQuery$collect`
## ------------------------------------------------

## Not run: 
# by default, all fields with constrains are included in the results
query <- esg_query()$experiment_id("ssp585")$frequency("1hr")$fields("source_id")
res1 <- query$collect()
res1$fields

# set `params` to `FALSE` to exclude them
query$collect(params = FALSE)$fields

# collect all matched records with `query$limit()` records per query
res2 <- query$collect(all = TRUE, limit = TRUE)
identical(query$count(), res2$count())

# same as above, but collect all matched records with max allowed
# record limit per query
res3 <- query$collect(all = TRUE, limit = FALSE)
identical(res2$count(), res3$count())

# same as above, but collect all matched records with specified limit
# per query
res4 <- query$collect(all = TRUE, limit = 30)
identical(res2$count(), res4$count())

## End(Not run)

## ------------------------------------------------
## Method `EsgQuery$state`
## ------------------------------------------------

## Not run: 
q$state()
q$state(null = TRUE)

## End(Not run)

## ------------------------------------------------
## Method `EsgQuery$reset`
## ------------------------------------------------

## Not run: 
q$experiment_id("ssp585")$reset()

## End(Not run)

## ------------------------------------------------
## Method `EsgQuery$save`
## ------------------------------------------------

## Not run: 
q$save(tempfile(fileext = ".json"))

## End(Not run)

## ------------------------------------------------
## Method `EsgQuery$load`
## ------------------------------------------------

## Not run: 
f <- tempfile(fileext = "json")

q <- esg_query()
json <- q$save(f)
q$load(f)

## End(Not run)

## ------------------------------------------------
## Method `EsgQuery$print`
## ------------------------------------------------

## Not run: 
q$print()

## End(Not run)

Description

EsgStore manages a local DuckDB manifest and a fixed directory layout for query result snapshots, dictionaries, source files, downloaded NetCDF files, Parquet regional extracts, and generated outputs.

Active bindings

Methods

Public methods

• EsgStore$new()

• EsgStore$close()

• EsgStore$get_meta()

• EsgStore$set_meta()

• EsgStore$download_layout()

• EsgStore$set_download_layout()

• EsgStore$register_artifact()

• EsgStore$artifact_path()

• EsgStore$validate()

• EsgStore$add_query()

• EsgStore$track_query()

• EsgStore$untrack_query()

• EsgStore$tag_query()

• EsgStore$untag_query()

• EsgStore$query_tags()

• EsgStore$require_query()

• EsgStore$unrequire_query()

• EsgStore$query_graph()

• EsgStore$queries()

• EsgStore$query_files()

• EsgStore$preview_update_queries()

• EsgStore$update_queries()

• EsgStore$download_preflight()

• EsgStore$download_query()

• EsgStore$download_status()

• EsgStore$query_status()

• EsgStore$query_updates()

• EsgStore$query_changes()

• EsgStore$workflow_status()

• EsgStore$workflow_report()

• EsgStore$remove_query()

• EsgStore$remove_files()

• EsgStore$prune_orphans()

• EsgStore$storage_report()

• EsgStore$validate_files()

• EsgStore$repair_files()

• EsgStore$cleanup_downloads()

• EsgStore$retry_downloads()

• EsgStore$add_files()

• EsgStore$downloader()

• EsgStore$download_files()

• EsgStore$sync_downloads()

• EsgStore$plan_region()

• EsgStore$extract()

• EsgStore$query()

• EsgStore$summarise()

• EsgStore$coverage()

• EsgStore$assert_complete()

• EsgStore$clone()

Method new()

Create or open a local store.

Usage

EsgStore$new(path = NULL, create = TRUE, overwrite = FALSE)

Arguments

Returns

An EsgStore object.

Method close()

Close the DuckDB connection.

Usage

EsgStore$close()

Returns

The store object itself, invisibly.

Method get_meta()

Return a store metadata value.

Usage

EsgStore$get_meta(key, default = NULL)

Arguments

Returns

A single string, or default.

Method set_meta()

Set a store metadata value.

Usage

EsgStore$set_meta(key, value)

Arguments

Returns

The store object, invisibly.

Method download_layout()

Return the store download layout policy.

Usage

EsgStore$download_layout()

Returns

A named list describing how store downloads are placed under ⁠downloads/⁠.

Method set_download_layout()

Configure how store-managed ESGF downloads are placed under ⁠downloads/⁠.

Usage

EsgStore$set_download_layout(
  layout = c("flat", "dataset", "drs", "template"),
  template = NULL,
  include_version = TRUE,
  collision = c("error", "checksum", "suffix"),
  missing = c("fallback", "error")
)

Arguments

Returns

The store object, invisibly.

Method register_artifact()

Register a file artifact in the store manifest.

Usage

EsgStore$register_artifact(
  kind,
  path,
  role = NULL,
  project = NULL,
  status = "available",
  checksum = NULL,
  checksum_type = "sha256",
  size = NULL,
  query_id = NULL,
  file_key = NULL,
  dict_id = NULL,
  source_url = NULL,
  source_repo = NULL,
  source_tag = NULL,
  source_commit = NULL,
  metadata = list()
)

Arguments

Returns

The artifact ID.

Method artifact_path()

Return an artifact path from the manifest.

Usage

EsgStore$artifact_path(artifact_id)

Arguments

Returns

Absolute artifact path.

Method validate()

Validate registered artifact files against the manifest.

Usage

EsgStore$validate()

Returns

A data.table with validation results.

Method add_query()

Add an ESGF query to the long-lived store query registry.

Usage

EsgStore$add_query(query, label = NULL, track = FALSE)

Arguments

Returns

The stable query ID.

Method track_query()

Mark a stored ESGF query as tracked.

Usage

EsgStore$track_query(query_id)

Arguments

Returns

The store object, invisibly.

Method untrack_query()

Mark a stored ESGF query as untracked.

Usage

EsgStore$untrack_query(query_id)

Arguments

Returns

The store object, invisibly.

Method tag_query()

Add tags to a stored ESGF query.

Usage

EsgStore$tag_query(query_id, tag, replace = FALSE)

Arguments

Returns

A data.table of tags for the query.

Method untag_query()

Remove tags from a stored ESGF query.

Usage

EsgStore$untag_query(query_id, tag = NULL)

Arguments

Returns

A data.table of remaining tags for the query.

Method query_tags()

List stored ESGF query tags.

Usage

EsgStore$query_tags(query_id = NULL)

Arguments

Returns

A data.table of query tags.

Method require_query()

Record that one stored query depends on another stored query.

Usage

EsgStore$require_query(query_id, parent_query_id)

Arguments

Returns

A data.table of query dependency edges.

Method unrequire_query()

Remove query dependency edges.

Usage

EsgStore$unrequire_query(query_id, parent_query_id = NULL)

Arguments

Returns

A data.table of remaining dependency edges for the query.

Method query_graph()

List stored query dependency edges.

Usage

EsgStore$query_graph(
  query_id = NULL,
  direction = c("children", "parents", "both"),
  recursive = TRUE
)

Arguments

Returns

A data.table of dependency edges.

Method queries()

List stored ESGF queries.

Usage

EsgStore$queries(tracked = NULL)

Arguments

Returns

A data.table of stored query records.

Method query_files()

List files linked to a stored ESGF query.

Usage

EsgStore$query_files(query_id, status = NULL)

Arguments

Returns

A data.table of linked file records.

Method preview_update_queries()

Preview tracked ESGF query updates without changing the store.

Usage

EsgStore$preview_update_queries(
  query_id = NULL,
  tracked = TRUE,
  tag = NULL,
  children = FALSE,
  detail = FALSE,
  all = TRUE,
  limit = FALSE,
  fields = "*",
  ...
)

Arguments

Returns

A data.table summary, or a list with summary and changes when detail = TRUE.

Method update_queries()

Refresh stored ESGF queries and link their current File records.

Usage

EsgStore$update_queries(
  query_id = NULL,
  tracked = TRUE,
  tag = NULL,
  children = FALSE,
  enqueue = FALSE,
  downloader = NULL,
  replica = "auto",
  session_label = NULL,
  service = "HTTPServer",
  probe = TRUE,
  probe_concurrency = NULL,
  probe_cache_seconds = 3600L,
  strategy = c("fastest", "first", "stable"),
  all = TRUE,
  limit = FALSE,
  fields = "*",
  ...
)

Arguments

Returns

A data.table of query-file links touched by the update.

Method download_preflight()

Preview a tracked query download without changing the store.

Usage

EsgStore$download_preflight(
  query_id,
  downloader = NULL,
  replica = "auto",
  service = "HTTPServer",
  probe = TRUE,
  probe_concurrency = NULL,
  probe_cache_seconds = 3600L,
  strategy = c("fastest", "first", "stable"),
  all = TRUE,
  limit = FALSE,
  fields = "*",
  ...
)

Arguments

Returns

A list with summary, changes, files, and candidates.

Method download_query()

Refresh, enqueue, and optionally run downloads for a stored ESGF query.

Usage

EsgStore$download_query(
  query_id,
  downloader = NULL,
  replica = "auto",
  dry_run = FALSE,
  run = TRUE,
  background = FALSE,
  mode = c("process", "daemon"),
  session_label = NULL,
  service = "HTTPServer",
  probe = TRUE,
  probe_concurrency = NULL,
  probe_cache_seconds = 3600L,
  strategy = c("fastest", "first", "stable"),
  progress = TRUE,
  overwrite = FALSE,
  resume = TRUE,
  all = TRUE,
  limit = FALSE,
  fields = "*",
  ...
)

Arguments

Returns

The created downloader session ID, NA_character_ when there is no pending file to download, or a one-row background job record when run = TRUE and background = TRUE.

Method download_status()

Return downloader tasks linked to stored query files.

Usage

EsgStore$download_status(query_id = NULL, session_id = NULL, downloader = NULL)

Arguments

Returns

A data.table of downloader task rows.

Method query_status()

Summarise tracked ESGF query file and download status.

Usage

EsgStore$query_status(query_id = NULL, downloader = NULL)

Arguments

Returns

A data.table with one row per stored query.

Method query_updates()

List tracked query update runs.

Usage

EsgStore$query_updates(query_id = NULL, latest = FALSE)

Arguments

Returns

A data.table of update run summaries.

Method query_changes()

List per-file changes recorded by tracked query updates.

Usage

EsgStore$query_changes(update_id = NULL, query_id = NULL, change_type = NULL)

Arguments

Returns

A data.table of per-file query update changes.

Method workflow_status()

Summarise query, download, local, and extraction status together.

Usage

EsgStore$workflow_status(query_id = NULL, downloader = NULL)

Arguments

Returns

A data.table with one row per stored query.

Method workflow_report()

Return a compact ESGF query workflow health report.

Usage

EsgStore$workflow_report(query_id = NULL, downloader = NULL)

Arguments

Returns

A list with summary, updates, changes, downloads, and nodes.

Method remove_query()

Remove stored ESGF queries and optionally delete orphaned local files.

Usage

EsgStore$remove_query(query_id, delete = c("none", "orphaned"))

Arguments

Returns

A data.table describing removed queries.

Method remove_files()

Remove ESGF file records and optionally delete local artifacts.

Usage

EsgStore$remove_files(file_key, delete_local = FALSE, force = FALSE)

Arguments

Returns

A data.table describing removed file records.

Method prune_orphans()

Report or remove file records no longer linked to any query.

Usage

EsgStore$prune_orphans(delete_local = FALSE)

Arguments

Returns

A data.table of orphaned file records.

Method storage_report()

Summarise store download storage, registered local assets, temporary files, and cleanup candidates.

Usage

EsgStore$storage_report(detail = FALSE)

Arguments

Returns

A summary data.table, or a list when detail = TRUE.

Method validate_files()

Validate store-managed NetCDF downloads against the manifest.

Usage

EsgStore$validate_files(query_id = NULL, checksum = FALSE, layout = TRUE)

Arguments

Returns

A list with summary, files, artifacts, untracked, and actions data.tables. The method is read-only.

Method repair_files()

Repair safe store download inconsistencies reported by ⁠$validate_files()⁠.

Usage

EsgStore$repair_files(actions = NULL, dry_run = TRUE)

Arguments

Returns

A data.table describing attempted repairs.

Method cleanup_downloads()

Report or remove download cleanup candidates.

Usage

EsgStore$cleanup_downloads(
  scope = c("tmp", "orphan_records", "untracked_files", "missing_records"),
  dry_run = TRUE,
  older_than = NULL
)

Arguments

Returns

A data.table describing cleanup candidates or removals.

Method retry_downloads()

Requeue retryable downloader tasks linked to stored query files.

Usage

EsgStore$retry_downloads(
  query_id = NULL,
  session_id = NULL,
  downloader = NULL,
  status = c("error", "cancelled"),
  run = TRUE,
  ...
)

Arguments

Returns

A data.table of matching task rows after retry handling.

Method add_files()

Add File or Aggregation query results to the local file catalog.

Usage

EsgStore$add_files(files, label = NULL)

Arguments

Returns

The created or updated query ID.

Method downloader()