Compile each fixed package schema once per R process and reuse it for query, result, dictionary and shift workflow config validation, instead of recompiling the schema on every check. Validation rules and error messages are unchanged.
Coalesce live run snapshots. Within one process, ordinary changes to a run (events, case writes, heartbeats) rewrite the lock-free snapshot at most once per second, and a deferred write is delivered by the next heartbeat or change. Status changes, terminal run, step and job states, and new attempts are still written immediately, and the snapshot is always rebuilt from the manifest tables. The last reporter state is kept when a write carries none, instead of being erased.
Read a store's schema version at most once when it is opened, instead of twice. A new store reads nothing and a reopened one reads once for the version check; manifests with a missing or different version are still rejected and a matching version is not rewritten.
Append new rows to a store table by asking DuckDB only about the keys being inserted, instead of reading the table's whole key column into R. The cost of an append no longer grows with the table, and stored rows, duplicate-key and missing-key errors behave as before.
Create a new store's schema in one multi-statement DuckDB call instead of one call per table and column. The declarations and their order are unchanged, and a failing statement still leaves the earlier ones applied.
Check the store connection once per live run snapshot and once per identity-row read, instead of before every table query. A closed store is still rejected, and snapshot contents are unchanged.
Skip schema declarations that a store manifest already satisfies when it is re-opened. The store reads its table and column catalog once and runs only the missing CREATE TABLE and ADD COLUMN statements, so missing tables and additive columns are still repaired and schema-version checks are unchanged.
Look up morphing backend rules by explicit row position and apply method overrides in one match, instead of building a data.table index for every step. Duplicate steps, first-name precedence for repeated overrides, validation errors and returned columns are unchanged.
Build EPW end-of-hour timestamps from numeric Gregorian fields instead of formatting and re-parsing every hourly row. Valid dates take the numeric path; invalid, mismatched-length or missing dates keep the previous parser, so results and attributes are unchanged.
Group EPW entry points under shift_epw_*(): rename
shift_future_epw() to shift_epw_future(), shift_reference_epw() to
shift_epw_reanalysis(), shift_epw() to shift_epw_write(), and
shift_export_epw() to shift_epw_export(). Rename shift_reference_plan()
to shift_reference_from_plan() to distinguish reference configuration from
plan construction. Arguments, return values and execution behavior are
unchanged; the previous development names are no longer exported (#293).
Apply additive shifts when enhanced monthly stretching falls back, and interpolate factors in representative month/day/hour order for typical-year EPWs assembled from different source years. Enhanced monthly recipe version 3 prevents reuse of plans with the earlier arithmetic or calendar semantics (#289).
Record evaluated File-level candidates and node/service outcomes during climate input resolution. shift_selection() reads saved configuration, selected inputs and candidate explanations for completed or failed runs and shared batch children. Older runs explicitly report missing records (#288).
Validate Kernel-QDM model and observed inputs by role, retain interpolation boundary samples and allow CMIP/ScenarioMIP activity differences between roles. Save fitted CDF grids, per-value quantile mappings, direct-model signal series and bounding corrections with runtime diagnostics. Manual summaries accept input_role (#292).
Isolate observed-reference selection, result reuse, diagnostics and completion status by morphing plan and case in shared stores. Recipes requiring observed references now use version 2 to prevent reuse of earlier plans (#291).
Save backend execution factors, intermediate products, runtime diagnostics and result links for each morphing case. EpwMorpher$process_data() verifies saved checksums when reading them. Repeated identical warnings no longer block output, and explicit overwrite rebuilds completed products (#290).
Serialize background batch receipt reads and publication with a short shared lock, and report failed writes (#290).
Distinguish climate files that share an ESGF dataset-level master ID, retaining file names in store and shared-batch identities without changing raw provider metadata (#287).
Remove obsolete repository-only papers, project/environment configuration,
community templates, test notes, and local check scripts; retire the transport
comparison workflow after TCP selection and the legacy PR command workflow. Move the ERA5 reference EPW guide
into the website articles and relocate Codecov settings to .github/ (#286).
Use loopback TCP consistently for local mirai worker pools, preserving worker counts and dispatcher behaviour across platforms; clarify connection-wait diagnostics (#285).
Read CDS credentials from environment variables when configuration files are absent, and report missing required credentials with the authentication diagnostic (#284).
Keep Vim folding modelines at the end of extended test files (#283).
Route hourly options through declared component scopes and keep method-specific configuration outside the shared interpolation and pipeline layers (#283).
Decode packed ERA5 NetCDF values and keep missing per-site local inputs offline.
Standalone downloads retain their own worker startup implementation. Optional
longwave settings can be configured through hourly_transform() (#283).
Keep Vim folding modelines at the end of extended test files (#282).
Release curl handles at every wave boundary and continue checking later URLs after an individual pool error; recover only unresolved results (#282).
Check service URLs in bounded active waves and avoid large serial timeout retries. No-download workflows sample exact OPeNDAP endpoints per data node and discard HTTP-only candidates before checking catalog coverage (#282).
Keep Vim folding modelines at the end of extended test files (#281).
Resolve regional plans through query-file memberships, preserving earlier queries when a shared file is cataloged again. Filter memberships in DuckDB before materializing the selected catalog (#281).
Scope extraction plan identities to their catalog query so refreshed catalogs retain correct ownership while reusing the shared source extraction cache (#281).
Keep Vim folding modelines at the end of extended test files (#280).
Coordinate batch receipt readers and writers with a short publication lock, preventing background refresh from observing a missing replacement file (#280).
Preallocate request receipts across recursive URI retries while preserving the complete ordered request history (#280).
Batch Dataset child and replica identity queries using complete request URL lengths and retry HTTP 414 responses with smaller ID groups. Attempt individual long IDs without a local length rejection, preserving query receipts and global row limits without duplicating partial pages when a batch restarts (#280).
Coverage CI prints the total line coverage and retains per-file reports and merged counters independently of Codecov. Upload errors produce a warning without failing the job (#279).
Publish standalone-downloader.R for usethis::use_standalone(), with local checksum,
manifest, validation and process helpers. Host cache and Store integration
stay in package adapters. Downloader instances own separate worker pools.
Coverage includes complete parent and worker traces
and rejects unfinished trace writes before upload (#278).
Handwritten R sources, tests and tools include Vim marker folds for functions, classes and methods, without nested callback folds inside function bodies; modelines use the default Vim markers, with CLI string braces split to avoid ending folds prematurely (#278).
Background batch tests collect process logs only on failure; an exclusively open Windows log cannot mask the task receipt (#278).
Windows run readers recognize DuckDB sharing violations and use existing live snapshots while the worker owns the manifest. Other file-open errors remain visible (#278).
Checking a background process on Windows no longer sends a termination signal. Async lifecycle tests allow for instrumented worker startup and retain coordinator diagnostics on failure (#278).
Consolidate workflow validation, year parsing and durable run registration. Inspectors group outputs once and query relevant manifest rows; internal helpers follow their owning modules and tests are organized by behavior. Remove unused reference-argument branches (#278).
Separate workflow configuration, execution, job recovery, inspection and resolution into focused source files, alongside query-result access and replica planning and UI presentation. Preserve the existing APIs and execution behavior (#278).
Single-task and batch workflows share installed-package process launching,
scoped execution settings and attempt transitions. Internal runtime and test
hooks no longer use global options. Execution snapshots read only supported
options; background launch arguments reject invalid types and lengths, and
catalog progress callbacks belong to Dataset results. The obsolete pkgload
suggestion is removed; workflow tests share catalog fixtures and are grouped
by behavior (#278).
Batch execution resolves actual future and historical input partitions once per model/method. Optional variables selected by the ordinary resolver share bounded reads across cities, and child runs pin the same File snapshots on foreground, background and resumed execution (#278).
Batch recovery excludes completed and active children and re-resolves failed input selections once across cities. Foreground and background execution use one coordinator with durable progress and cooperative cancellation; source concurrency settings apply to the whole coordinated batch. Fully cached shared reads avoid worker startup, and collector failures drain active readers before returning an error (#278).
Dataset asynchronous operations use separate compute profiles across dataset objects (#278).
Foreground and background multi-site batches read shared physical files in bounded
native-time windows. Verified window receipts resume interrupted reads, and
child workflows reuse the same site-extraction cache keys without changing
their output and provenance contracts. Shared remote failures block dependent
children while unrelated tasks continue, without per-city retries (#278).
Recovery now records a blocked batch with its source error and can assemble
complete window receipts without reopening an unavailable source. Partial
cache recovery reads only missing site keys; duplicate method demands share
one extraction, and large site groups reuse one source connection (#278).
Shared reads preserve spatial roles in grid-source provenance. Native time
coordinates and grids are reused within each source connection; interval
bounds are read only for selected native times. Time slices now use the
actual spatial group size within the unchanged 8192-value request limit;
single-cell reads can include up to 8192 native steps. Weather subsets, CF
bounds and batch scheduling share one internal request-value limit (#278).
Ordinary extraction and shared batch reads now use a bounded source-file
worker pool, controlled by options(epwshiftr.mirai_workers = 4L). Workers
reuse each file connection when caching is enabled, while the caller writes
manifest and Parquet records. Cached ordinary extraction avoids starting
workers; failed remote reads enter the existing fallback without repeating
the OPeNDAP attempt (#278).
Ordinary point extraction now reads actual CF interval bounds only for the
requested native times and reuses them across sites. Closing a dataset clears
its metadata cache so reopening cannot reuse stale source metadata; the public
full-axis accessor continues to return all bounds (#278).
Explicit filters = list(replica = TRUE) now survives availability discovery,
allowing callers to select cataloged replicas through the existing API (#278).
Per-site extraction manifests are committed atomically; Parquet export scans
R columns directly instead of copying them through a temporary table (#278).
Added an internal bounded multi-site NetCDF reader for shared file plans. Distinct native grid/time slices are read once per acquisition while each consumer retains its site, method, CF time, actual grid cells and weights (#277). Adjacent cells share at most 2-by-2 spatial slices; native time reads are capped at 8192 native values, with explicit output-row and request limits. Grid coordinates are reused across sites; metadata failures and mismatched acquisition URLs stop the read.
Batch plans now retain a shared physical File acquisition table and separate site/method consumer links from cached discovery metadata. Overlapping source intervals are planned once; native multi-point reading remains a subsequent execution change. (#275)
shift_epw_future() now takes shift_site() objects through its first
argument, sites, and always returns a batch. Workflow configurations use
a sites array with constructor field names. Site/model/method plans share
discovery and baseline identities, with distinct output directories (#273).
Simplify the README around features and one multi-site example, recorded in a real terminal with batch status, outputs, weather summaries and diagnostics. Keep the legacy migration guide specific to CRAN v0.1.4 and remove the standalone San Francisco development script. Exclude development tools from package builds (#273).
Batch recovery now distinguishes extraction methods and validation policies; method/period constraints are checked before discovery. Reuse model/method requests and hash each unique baseline once during batch planning (#273).
Apply historical catalog requirements only when needed by the batch reference, share File coverage results across methods, and avoid repeated eligibility summaries, group counts, and input-map serialization (#269).
Shared narrow catalog normalization between variable and method availability queries, and grouped table selection without repeated full catalog scans (#269).
Reused method eligibility and normalized catalogs across batch alternatives, ranked source-file costs before choosing a model's member/grid, and simplified request construction while retaining shared catalog filter rules (#271).
Aligned method availability with executable historical/future variable combinations and reused compiled requirements for catalog matching (#269).
Fixed manifest inspection tests for DuckDB 1.5.6 by releasing fixture write connections before reopening databases read-only. Added coverage for rejected writes through read-only connections (#269).
Named future-weather registry keys, intermediate kinds, diagnostics, and
method choices after the algorithms they implement. The quantile-mapping
morphing workflow described by Arima et al. is available through
daily_transform("qm_morphing"), with the internal recipe key
quantile_mapping_morphing_daily (#165, #250).
Standalone shift_*() stages now carry their persisted run_id and
step_id into the next stage automatically. The public API does not expose a
workflow session/current-session object. Intermediate runs are waiting,
final EPW exports complete automatically, and shift_complete() explicitly
closes a workflow that intentionally stops at an intermediate artifact.
Logical progress arguments on shift_datasets() and shift_collect() were
removed in favour of ui = shift_ui(progress = ...) (#126).
Replaced public method and recipe construction with reusable
WeatherTransformSpec objects from monthly_transform(),
daily_transform(), or hourly_transform(). The task-oriented
shift_epw_future(epw, climate, periods, transform, dir, reference, observed_reference, control, ui, store, dry_run, background) interface now
keeps historical model and observed-reference inputs at execution scope.
weather_transforms() exposes each selectable method's temporal, input,
reconstruction, evidence, and output contract. Version 1 workflow plans are
rejected explicitly, and the former method constructors and low-level public
recipe discovery API have been removed (#250).
Corrected the canonical Belcher temperature implementation to use its
published combined mean-and-diurnal-range transformation. Version 2 now
requires monthly tas, tasmax, and tasmin for both historical and future
model periods, computes the EPW denominator from the monthly average of daily
temperature ranges, and aligns variable-specific CMIP6 tables by scientific
model/member/month identity. Existing outputs that previously used the
mean-only temperature shift will change.
Removed the eplusr, psychrolib, and units runtime dependencies. EPW files
are parsed and written directly, while objects inheriting from Epw remain
accepted through a dependency-free conversion to the internal EpwFile.
The required ASHRAE saturation-pressure and dew-point equations are
implemented internally, and climate-value conversion uses explicit supported
unit pairs.
Replaced the extraction nearest count option with explicit grid extraction
methods: "nearest", "idw", "bilinear", and "mean" (#123).
Replaced the legacy data.table-oriented workflow with the new store-native
workflow. Users who need v0.1.4 behavior should use the legacy branch on
GitHub or install epwshiftr v0.1.4.
Removed the legacy init_cmip6_index(), summary_database(),
match_coord(), extract_data(), morphing_epw(), future_epw(),
EsgfQuery, esgf_query(), and related helpers. Use shift_*,
EsgStore, EpwMorpher, EsgQuery, and data_node_status() instead.
Dropped the old fst, future.apply, progressr, pingr, PCICt, and
rappdirs-based implementation paths; the new implementation uses DuckDB,
mirai, S7, and store-managed manifests.
shift_epw_reanalysis() creates annual ERA5 reference weather without a
baseline EPW. It accepts existing NetCDF files or explicitly unit-labelled
point data, supports multiple sites and years, and retains input hashes,
field provenance, leap-day omissions and humidity diagnostics. Conversion
rejects incomplete or inconsistent input instead of filling template weather.
ERA5 reference inputs also support diffuse horizontal radiation from matched
SSRD/FDIR intervals. Hourly kernel QDM can opt into longwave correction and
reconstruction on the site's fixed standard clock. All reanalysis requests
use the new cache identity generation; earlier cached references are rebuilt.
macOS background workers use loopback TCP with bounded startup while retaining
asynchronous cancellation and timeouts (#283).
Added common = FALSE to shift_cmip6() for independent method-specific
model selection in future-weather batches. The default common = TRUE
preserves shared model/member/grid identities. Batch discovery shares Dataset
queries across methods and alternatives, checks File-year coverage before
selection, persists the chosen matrix, and reports differing method pools
(#271).
shift_cmip6_avail() now accepts weather methods or explicit transform
configurations, derives their variable/frequency/history requirements, and
shares one cached Dataset query across methods. Results retain rejected
candidates and missing-input reasons, with separate per-method and common
selection pools controlled by the logical common flag (default FALSE).
Catalog eligibility does not establish year coverage,
readability, or scientific quality. Both variable and method queries return
a data.table; existing variable-query columns are preserved. Internal
requirements use bulk joins for input alternatives and scenarios (#268, #269).
Added precipitation = "off" to both monthly morphing methods. This keeps
baseline EPW precipitation fields unchanged and removes CMIP pr from
discovery and input requirements when precipitation is outside the workflow
scope (#261).
Added shift_history() / shift list to find saved runs and batches without
network requests or store creation, and shift_summary() / shift summary
to inspect outputs by method, model, scenario, period, member, and grid.
Optional --weather statistics include valid-hour counts and file-read errors.
Batch dashboards now fit the terminal height, prioritize failed and active
children, show live work and update times, and count cancellations explicitly.
shift show --verbose retains complete details. Error diagnostics appear
before warnings with recovery actions, watch leaves a durable final receipt,
and CLI batch execution prints that receipt once. Calibration variables,
morphing cases, and exported EPW files use separate progress units.
Config validation, dry-run discovery, and network checks now show their intent
and progress while respecting quiet and machine-readable output settings.
Tall foreground receipts retain all output paths, and detailed batch views
reserve space for both failed and active children. Discovery method progress
is also visible in log and compact modes. Standalone staged summaries recover
method and case identity from saved results; malformed batch receipts appear
as unavailable without hiding other history records (#255).
Extended the CLI and terminal workflow dashboard to cover method/model
batches, ERA5 calibration, typed model selection, and catalog refresh.
morph describe exposes method contracts and validates candidate settings;
quoted JSON arrays in --option retain their numeric values. Added
shift_batch_get() and --batch inspection, watch, and recovery commands.
New batch receipts retain dry-run plans across sessions. Completion receipts
distinguish cases from EPW files, show warnings and field roles, and report
child reuse. Batch dashboards share the single-run box style, with separate
overview, workflow, and result sections. Local config validation stays offline;
--network explicitly checks provider coverage and readiness (#255).
Added an ergonomic shift_epw_future() workflow for multiple weather methods
and CMIP6 models. Method keys resolve through weather_transforms(), model
discovery selects common model/member/grid identities across every method,
required historical periods are created automatically, and ShiftBatch
results support the existing status, output, diagnostic, resume, and cancel
operations without imposing package-owned comparison metrics. Added
shift_era5() as a credential-free reanalysis source specification with
direct R access to official CDS services and CF weather normalization.
shift_era6() reserves the same provider-neutral interface and fails
explicitly until a stable public ERA6 dataset is available. shift_cmip6()
now expresses all selection modes through model: a positive count selects
a bounded compatible ensemble, character values select exact models, and
NULL selects all compatible models. Reanalysis sources support local and
opt-in remote credential checks, classify dataset-term failures, and have a
separate opt-in live ERA5 retrieval workflow. Batch model selection now
verifies requested future and automatic historical File-year coverage before
applying a model count. File-service resolution now validates OPeNDAP through
its DDS response, retains a compatible HTTPServer recovery candidate without
checking the unused endpoint eagerly, composes endpoints across distributed
replicas, and deduplicates node records by normalized CMIP6 DRS file
identity. Regional extraction checks and repairs HTTPServer only after an
OPeNDAP open, metadata, or read failure, retries once through the selected
download, and records both service attempts with their stage, URL, host,
duration, and original error (#254).
Identical future-weather calls now restore their persisted batch selection
and original child runs before any catalog or service request, verify that
completed EPW artifacts still exist, and avoid repeating extraction,
morphing, or export. shift_control(refresh = TRUE) provides the explicit
opt-in path for refreshing remote catalog and service information. Batch
method keys now construct only their requested transforms, and initialized
backend registries no longer rebuild every default backend, pipeline, and
component specification during each lookup. Completed batch children reuse
one authoritative store connection for run reconstruction and artifact
verification instead of opening the same manifest twice.
Dry-run ShiftBatch objects can now be passed directly to shift_run(), and
direct high-level execution first builds the complete child matrix so one
operational child failure does not prevent independent children from
running. File-service checks are applied only after scientific
model/member/table/grid selection and exact disjoint-year filtering;
automatic numeric model selection prefers complete candidates requiring
fewer physical source files. Identical ERA5 requests share one locked raw
cache across child stores while retaining store-local registered artifacts.
Equivalent site extractions from immutable climate files now share a locked,
content-addressed payload cache across method stores; store-specific plan and
query identities are attached only when the cached payload is materialized.
Failed service checks remain cached for the package cache lifetime so an
adjacent method does not repeat the same endpoint timeout.
Successful-run scientific warnings are persisted as idempotent run events
and remain available from shift_diagnostics() after refresh or resume.
Morphing cases now fail independently within one method run, with durable
per-case status and current diagnostics. BWS retains the original
Historical-to-Scenario change signal but resolves an unattainable bounded
EPW target to the nearest attainable mean and records both targets, source
means, bounds, and the adjustment reason. Every result manifest records
transformed, derived, physically closed, and inherited EPW field sets for
downstream user-owned comparisons.
Added a calculation-only EPW view that converts documented numeric and
present-weather missing codes to NA before weather algorithms run, while
preserving raw codes for exact reads and restoring canonical missing codes
on output. Original Morphing now rejects missing precipitation months in
strict mode and retains explicit diagnostics in relaxed mode instead of
treating 999 and 99 as measured precipitation. Its factor diagnostics
now distinguish missing future climate, Historical reference, and EPW
baseline months instead of reporting every unavailable factor as a generic
climate-or-baseline problem.
Kept future-weather generation contracts independent of user evaluation
designs. Package-owned comparison protocols, publication study presets, and
duplicate _comparison recipes were removed; every method now has one
canonical complete recipe. Recipe metadata continues to expose input roles,
source and target calendars, components, physical policies, diagnostics, and
provenance so users can apply consistent external evaluation designs (#248).
Added a method catalog independent of executable recipes and connected eight daily bias-adjustment methods to one reusable temperature-to-EPW adapter. Complete recipes identify their method, input roles, components, physical policies, output type, diagnostics, and provenance without embedding a paper's selected climate models or study periods (#246). Complete temperature-to-EPW compositions are classified as adapted publications when their signal kernel is published; the experimental daily EDCDFm adaptation retains its experimental evidence label.
Added variable-specific CMIP6 frequency contracts to availability discovery,
persisted workflow selection, component validation, and extraction planning.
The hourly kernel-QDM workflow now resolves point-sampled 3hrPt state and
wind fields, interval-mean 3hr radiation, and optional daily tasmin and
tasmax within one model/member/grid identity (#244).
Aligned the experimental hourly_transform("kernel_qdm") workflow with the
published raw-model input boundary: model roles now request tas, ps,
huss, uas,
vas, rsds, and rsdsdiff, then derive hourly relative humidity, scalar
wind speed, and meteorological direction before kernel QDM. Model direction
follows the corrected future speed through calendar mapping and EPW physical
closure, while scalar inputs without direction retain template compatibility.
Empty ESGF Dataset searches with maxScore: null now return a valid empty
availability result (#240).
Added the experimental hourly_transform("kernel_qdm") complete workflow
for matching
hourly observed tas, ps, hurs, sfcWind, rsds, and rsdsdiff with
three-hourly historical/future model tas, ps, huss, uas, vas,
rsds, and rsdsdiff. The seven-stage recipe reconstructs hourly model
inputs with boundary support, applies variable-specific kernel QDM, maps each
complete future year to the EPW calendar, executes the shared
absolute_model_fields physical policy, and returns one persistent EPW
member per source-model year (#238).
Added a shared EPW physical policy layer used by all built-in complete weather
methods. Method adapters submit EpwPhysicalRequest objects and each public
transform resolves one canonical internal EpwPhysicalPolicy. The registered
epw_hourly_physical_closure
component uses the absolute_model_fields policy for mapped direct-model
years, including humidity and wind alternatives, unit conversion, field
bounds, shortwave closure, inherited template fields, and typed diagnostics
(#199). Method-neutral helpers also centralize three-role signal contracts,
adjusted-result validation, future-year fitting blocks, numeric bounds,
calendar-neutral circular interpolation and target-day mapping, regular
hourly lattices, temperature conversion, parametric CDF/quantile dispatch,
and the shared Spencer solar geometry kernels while retaining each published
method's equations, temporal conventions, diagnostics, and provenance.
Added the registered direct_model_epw_calendar_mapping hourly component.
It maps complete hourly direct-model years from every supported native CF
calendar onto the baseline EPW's fixed 365-day, 8760-row lattice while
preserving each variable's time-of-day position. Already aligned 365-day
years remain exact; 360- and 366-day point variables use circular seasonal
interpolation separately at each daily hour, while interval-mean radiation
uses normalized conservative remapping separately at each daily hour. Typed
sequence members retain group identity, source calendars, mapping methods,
annual-mean error diagnostics, and upstream signal provenance for later
physical closure and output (#197).
Added the registered hourly_weather_interpolation preprocessing component
for model inputs that combine point-state variables with interval-mean
shortwave radiation. It dispatches each variable family to its existing
linear or solar-projection algorithm, optionally inserts matching daily
tasmin and tasmax as source-labelled hourly tas anchors using observed
site/month extreme-hour modes, and records deterministic hour and adjacent-
pair fallbacks. The merged historical and future roles retain variable-level
diagnostics, sampling phases, source provenance, and validated regular native-
calendar hourly coordinates (#195).
Added the registered hourly_calendar_grouping calendar component. It turns
hourly observed, historical-model, and future-model role inputs into
univariate signal groups while preserving each role's native CF chronology,
requiring complete hourly calendar years, and rejecting incompatible units,
variables, sites, or model identities before statistical adjustment. The
hourly kernel-density QDM and direct-model sequence contracts now also
accept the actual preprocessed source and sub-daily intermediate kinds
needed by this component chain (#193).
Added the registered kernel_quantile_delta_mapping_hourly signal component
for pre-aligned hourly observations, historical model output, and future
model output. It defaults to Gaussian kernel-density CDFs in centered
three-month native-calendar windows, transfers additive quantile changes for
temperature and pressure or multiplicative changes for wind, humidity, and
radiation, and retains future-model chronology in a
SubdailyAdjustedSeries. Because the source publication does not report its
KDE kernel, bandwidth, numerical grid, tails, or zero-denominator behavior,
the corresponding package defaults and supported alternatives are explicitly
validated, user-overridable, warned about, and recorded separately from the
published method settings (#191).
Added the registered solar_radiation_interpolation preprocessing component
for interval-mean three-hourly rsds and rsdsdiff. It preserves explicit
CF time bounds through regional extraction, allocates each source interval
to hourly means using positive solar projection, conserves the source mean,
and records weights and diagnostics on every supported native CF calendar.
Regular six-hourly input is supported as an explicit extension of the
published three-hourly method (#189).
Added the registered linear_temporal_interpolation preprocessing component
for regular three- and six-hourly tas, huss, hurs, ps, psl, uas,
and vas inputs. It produces an exact hourly native CF-calendar lattice,
retains source rows, timestamps, weights, grouping, diagnostics, and
provenance, and rejects extrapolation and variables requiring dedicated
temporal methods (#187).
Added the frequency-aware AdjustedWeatherSeries signal-result contract.
Existing daily methods retain their strict DailyAdjustedSeries output,
while regular sub-daily outputs carry explicit frequency, timestep, and
calendar-native time-of-day metadata. direct_model_realization now
preserves complete daily or sub-daily source-model years without resampling
(#185).
Added the deterministic direct_model_realization sequence component. It
preserves corrected daily model_future chronology, partitions complete
native CF-calendar years without selecting or resampling days, and retains
group identities and signal provenance for later hourly reconstruction
(#183).
Future-weather recipes with future_year or multi_year output types now
persist a year-addressable sequence manifest and write one independently
resumable Parquet and EPW member per weather year. Existing
representative_year outputs and identifiers remain unchanged (#181).
Weather component pipelines now pass optional variable-specific
signal_overrides to the selected signal component. Reusable downstream
temperature reconstruction components ignore signal-owned settings while
continuing to validate their own projection options (#179).
Added the native R isimip3basd_daily signal component for the complete
ISIMIP3BASD 3.0.x marginal bias-adjustment configuration. It transfers
modeled quantile changes onto historical observations, then maps the
future-model sequence through Normal, Gamma, Weibull, or empirical
distributions with variable-specific additive, mixed, or bounded trend
preservation. Published profiles cover eight direct variables plus
prsnratio, tasrange, and tasskew; reconstructing prsn, tasmin, and
tasmax, and applying MBCnSD spatial downscaling, remain explicit later
stages. Native CF calendars use the package annual-phase adapter, while
threshold randomization, missing-ratio imputation, detrending, short-wave
upper-bound scaling, fitting fallbacks, bound frequencies, seeds, and
clipping remain inspectable in result provenance (#177).
Added the native R equidistant_cdf_matching_daily signal component for
tas and pr. It applies the Li et al. additive equidistant equation using
four-parameter Beta temperature distributions and mixed dry-mass/Gamma
precipitation distributions. Although the transfer equation is
mathematically equivalent to absolute Quantile Delta Mapping, this
implementation retains the distributions and range convention of the
original method. Because Li et al. applied it to monthly fields, fitting
separate distributions from daily values in each native CF-calendar month
is recorded and warned about as an experimental epwshiftr adaptation.
Results preserve the future-model sequence and record fitted parameters,
sample coverage, probability clamping, precipitation dry counts and
non-negative clipping, resolved settings, and frequency provenance (#175).
Added the native R cdf_transform_daily signal component. It estimates the
future target CDF through the CDF-t chain
(F_{obs,hist}(F^{-1}{model,hist}(F{model,future}(x)))), then quantile
matches the future-model sequence onto that target. The method uses the
authors' additive-mean range alignment, explicit empirical-grid and
constant-correction tail conventions. Following the Famien et al. (2018)
daily Africa application, native calendar months use 17-year fitting
windows with disjoint central 9-year output blocks. The package-selected
boundary policy truncates unavailable four-year flanks and records that
provenance explicitly. Precipitation uses deterministic Singularity
Stochastic Removal below (10^{-8}) kg m-2 s-1 without changing R's global
RNG state. Results retain future-model CF-calendar coordinates and record
window coverage, target-CDF ranges, tail extensions, clipping, SSR seeds,
and resolved settings for the six variables in that application (#173).
Added the native R scaled_distribution_mapping_daily signal component. It
implements the published absolute temperature branch with linear detrending,
Normal fits, and two-tailed recurrence intervals, plus the relative
precipitation branch with a 0.1 mm/day dry threshold, zero-location Gamma
fits, expected wet-day adjustment, and one-tailed recurrence intervals.
Calendar-month groups use the published 30-year future fitting window with
disjoint 10-year retained blocks and explicit edge truncation. The result
preserves the future-model CF-calendar sequence and records resolved
settings, distribution fits, window coverage, clipping, and the method's
explicit limitation that additional future wet days are not invented
(#171).
Added the native R quantile_delta_mapping_daily signal component. It uses
each projected value's time-dependent future-model quantile to transfer the
corresponding historical-to-future model change onto the observed-reference
quantile: absolutely for interval variables and relatively for precipitation.
Calendar-neutral 91-day seasonal windows and centered 31-year future windows
make both sources of variation explicit. Published precipitation censoring
uses deterministic role-specific randomization below 0.05 mm/day without
changing R's global RNG state. The future-model sequence, resolved settings,
window coverage, quantile changes, bounds, censoring, and provenance are
retained in the DailyAdjustedSeries result (#169).
Added the native R quantile_mapping_daily signal component. It maps
future-model daily values through historical-model and observed-reference
empirical distributions in calendar-neutral circular windows, using
explicit tie, interpolation, tail, bound, and minimum-sample conventions.
Precipitation uses a mixed dry-day/positive-amount hurdle distribution with
deterministic group-specific randomization that leaves R's global RNG state
untouched. Results preserve the future-model CF-calendar sequence and record
resolved settings, sample coverage, tails, clipping, dry-day changes, and
provenance. Published method-variable profiles are limited to tas and
pr; implementation-selected defaults for other supported variables remain
experimental (#167).
Added the native R delta_change_daily signal component. It transfers
historical-to-future monthly mean changes onto the observed-reference daily
sequence: additively for tas, tasmin, and tasmax, and multiplicatively
for pr. The typed result preserves observed calendar coordinates and
variability while recording resolved settings, modeled monthly changes,
bounds, clipping, provenance, and explicit diagnostics (#163).
Added the package-native DailyAdjustedSeries signal contract and registered
native R linear_scaling_daily component. The univariate component consumes
distinct observed-reference, historical-model, and future-model daily
series, applies monthly additive temperature or multiplicative precipitation
mean corrections, and returns the corrected future sequence with canonical
CF-calendar coordinates, resolved settings, correction provenance, bounds,
and explicit diagnostics (#161).
Added daily_transform("qm_morphing") and registered the temperature-focused
quantile_mapping_morphing_daily recipe. The workflow carries baseline EPW,
historical daily model tas, future daily model tas, and multi-year
observed daily tas as four distinct input roles. It builds month-wise
historical/future inverse-CDF change functions, applies the published
endpoint-aware nine-point smoothing three times, locates each baseline daily
mean in the observed monthly CDF, and adds the selected factor to all 24
hours. Empirical-CDF conventions and endpoint clamping are recorded
explicitly. Its canonical public transform preserves the baseline humidity
fields and reports thermodynamic inconsistencies without modifying them
(#157).
Added daily_transform("ek") and registered the temperature-focused
ek_daily_factors recipe. Matching daily CMIP6 tasmin and tasmax years
are mapped from their native calendars to the 365-day EPW phase grid before
daily mean and DTR change factors are calculated and applied through the Ek
combined shift-and-stretch equation. The result records the selected
relative-DTR interpretation, zero-historical-DTR fallback, calendar mapping,
and source ambiguities. Its canonical public transform preserves baseline
humidity fields and reports thermodynamic inconsistencies without modifying
them (#155).
Added monthly_transform("bws_btws") and the bws_btws_monthly recipe. Matching
monthly CMIP6 tas, tasmin, tasmax, rsds, and clt provide the change
factors used by Eames et al. (2024). BTWS reconstructs hourly
dry-bulb temperature, while the reusable BWS kernel transforms global solar
radiation and total sky cover without changing zero or upper-bound states.
The unified physical layer then closes humidity, diffuse/direct radiation,
and opaque sky cover. Provenance records the substitution of CMIP6 monthly
factors for the paper's UKCP18 factors (#153, #251).
Added daily_transform("epwshiftr", reconstruction = "btws") and registered
the internal epwshiftr_daily_btws recipe. It combines the existing
calendar-neutral daily CMIP6 mean/minimum/maximum signal with the hourly
bounded temperature weighted
stretch from Eames et al. (2024), while reusing the baseline sequence,
specific-humidity closure, and EPW output components. The implementation
applies equations (7)--(16), records S, m, n, closure errors and
fallback reasons, and uses deterministic bisection to retain the largest
admissible exponent where the paper does not publish solver code. Its
provenance identifies the complete recipe as a combination rather than the
paper's monthly UKCP18 workflow (#151).
Added the shared specific-humidity closure used internally to test and
diagnose the Sobie-Curry thermodynamic transformation. It retains the
Sobie-Curry calendar-neutral factors, circular 21-day smoothing, baseline
sequence, and hourly temperature transformation while applying the smoothed
daily huss change through a shared specific-humidity closure. The target is
bounded at zero and saturation before relative humidity and dew point are
derived from projected temperature and pressure; closure states and clipped
targets are retained as diagnostics. The public Sobie-Curry transform keeps
the independently transformed thermodynamic fields defined by the source
method (#149).
Added the registered daily_transform("sobie_curry") method.
Its seven-stage pipeline derives daily thermodynamic factors from matching
historical and future tas, tasmin, tasmax, huss, and ps, smooths
the factors with the published circular 21-day window, preserves the
baseline EPW sequence, and independently transforms dry-bulb temperature,
dew point, relative humidity, and pressure. Dew-point standard-deviation
change is interpreted as sigma_future / sigma_historical - 1 so zero
climate change remains an identity; the interpretation, settings,
fallbacks, closure errors, and physical diagnostics are retained with the
result (#147).
Added an internal versioned registry of complete future-weather recipes. The
catalog records each method's input roles, calendar policy, seven component
stages,
execution policy, output type, diagnostics, and provenance without
serializing executable functions. The public weather_transforms() catalog
presents the corresponding method contracts, registered methods validate
their inputs before execution, and CLI/workflow round trips retain the
selected definition and policy (#145, #250).
Added explicit future-weather input roles and reusable preprocess,
calendar, signal, sequence, hourly, physics, and output
component contracts. Registered components declare their input requirements,
intermediate data kinds, dimensional scope, stochastic behavior, settings,
provenance, and diagnostics while existing EpwMorphBackend runners remain
compatible (#143).
Added the built-in daily_transform("epwshiftr") method for complete
future-EPW workflows using daily future and historical CMIP6 temperature
data. Its
seven-stage component pipeline estimates circular daily tas changes,
preserves the baseline EPW sequence, constrains each hourly profile to daily
mean/minimum/maximum targets when paired tasmin and tasmax are available,
and otherwise inherits the baseline daily range explicitly. Baseline specific
humidity is retained subject to saturation, relative humidity and dew point
are recomputed, and hourly numerical, physical, and component diagnostics
remain available in the persisted result (#141).
Added calendar-neutral daily temperature targets and constrained 24-hour
temperature projection. Matching future and historical tas, tasmin, and
tasmax climatologies provide daily mean, minimum, maximum, and DTR changes.
Feasible target statistics are closed while hourly order and extrema timing
are preserved; missing extrema and flat template days use explicit, reported
fallbacks (#139).
Added calendar-neutral circular daily-climatology primitives. Odd 21-, 31-,
and 61-day windows operate on annual_phase and map 360-, 365-, and 366-day
inputs onto a common target grid without date pairing (#137).
Added shift_cmip6_avail() to discover all CMIP6
model/member/grid identities that provide every requested Dataset variable
across future scenarios and optional historical data. The query accepts
frequency, table_id, and variable_id constraints, accepts known ESGF
node names including the ORNL/LLNL Bridge, and reports incomplete identities
without downloading NetCDF payloads (#127, #133).
Enhanced monthly morphing, now selected with
monthly_transform("epwshiftr"), uses guarded
combined temperature morphing with cyclic month-boundary smoothing, a
case-wide specific-humidity state path when complete huss + tas + ps data
are available, integrated EPW solar geometry, RBL diffuse radiation, Perez
illuminance, optional snd scaling, and recalculated ground-temperature and
typical/extreme-period headers. The original Belcher formulation remains
available through monthly_transform("original_morphing"); method-specific options
are supplied to the transform constructor (#126, #250).
shift_cmip6(table = NULL) now resolves exact variable/table/grid
partitions. Atmospheric inputs remain in Amon, snd is discovered in
LImon, and future/reference optional inputs are retained only when both
cases provide a matching partition. Scalar tables still pin every variable;
named vectors override individual variables.
Added store_reset() for deliberately incompatible store schemas. Existing
stores are moved to timestamped same-filesystem backups by default;
permanent removal requires backup = FALSE, force = TRUE. Reset targets are
validated before any filesystem change, and schema mismatch errors now show
a directly runnable recovery command.
Generalized the Future EPW reporter and run store across standalone Dataset
inspection, collect, download, extract, morph, EPW-write, and export tasks.
Each invocation records an ordered step and foreground job; the latest
returned stage continues the run, while stale or terminal inputs fork a child
lineage. shift_datasets() preserves its EsgResultDataset return value while
carrying run coordinates as lightweight attributes. Background download
sessions remain running and synchronize their completion, cancellation, and
logs through the same run ID. shift_result() rebuilds the latest successful
stage, and CLI extract/morph/download commands expose run_id and step_id
without mixing progress text into quiet, JSON, or JSONL output. Store schema
2.8 records these steps and deliberately requires a new store instead of
migrating older manifests.
ShiftRequest now follows EsgQuery's double-rule query receipt and canonical
parameter renderer. ShiftFiles preserves the established
EsgResultFile double-rule and bullet-summary header while adding a compact,
terminal-width-aware CMIP6 preview. It reads only the requested preview rows
from the store and accepts n, width, and verbose print controls; the
complete catalog remains available through shift_files() or
data.table::as.data.table().
Added a shared semantic print design for every public Shift configuration and
stage object. Receipts use double-rule titles, bullet facts, bounded
width-aware tables, compact periods and references, and common n, width,
and verbose controls. ShiftRun prints a refreshed, motion-free snapshot
through the same dashboard view used by foreground runs and shift_watch();
an unavailable store now falls back to the cached snapshot with a diagnostic
instead of failing the print method.
Added shift_ui() and a unified workflow reporter with an in-place plan
summary; a responsive stage rail, one animated current operation,
stage-specific measured progress, terse resolver attempts, and a two-item
activity feed; aggregate download bytes, speed, ETA, and active files; and
ordered normal, detail, and debug output. Resolver failover no longer
presents node attempts as a workflow percentage. ANSI terminals paint the
complete dashboard atomically through a cli-capability-aware framebuffer,
add a quiet panel border with labelled workflow and activity sections at 60
columns and above, reserve the final terminal column against autowrap, and
reflow plan, current operation, status, and diagnosis fields at semantic
boundaries instead of clipping them. Flow falls back from the complete rail
to current-plus-next and current-only forms as space contracts. Semantic
colours remain reserved for state instead of ordinary plan values. Narrow
terminals retain the same content without decorative chrome. Constrained dynamic IDE
consoles use one compact cli status row. Failed runs now commit their final
panel to terminal scrollback and replace repeated resolver exceptions with a
structured diagnosis: attempt counts, one cause, the closest CMIP6 identity,
and the first missing requirement. Recovery text distinguishes transient
retryable failures from scientific coverage failures and omits the default
store path from copyable commands. Title-like dashboard labels now use a
consistent bold accent, while failure labels use the danger colour and
secondary labels remain quiet. motion and refresh control presentation
independently from durable job heartbeats, and shift_watch() advances
cached animation frames without increasing store polling frequency. Reporter
milestones remain structured run events while throttled sidecars carry
transient state without recording animation-only events. CI, TERM=dumb,
redirected, and captured output use complete append-only logs.
Added detached Future EPW jobs with live ShiftRun refresh,
shift_watch(), shift_cancel(), and shift_logs(). Background attempts
retain their PID, heartbeat, log, cancellation, and terminal state, while a
lock-free live snapshot keeps watch/cancel responsive when the worker owns
DuckDB's process lock. shift_logs() identifies process-log and persisted-
event rows, and CLI JSONL follow emits typed snapshot, event-delta, gap, and
terminal records.
Added persisted ShiftRun workflow records, expected-case state and events,
plus shift_runs(), shift_run_get(), shift_cases(), shift_missing(), and
shift_resume() for run-ID-based inspection and cross-session recovery.
Added typed shift_cmip6() and shift_control() configuration,
complete future/reference CMIP6 resolution, strict expected-case coverage,
and explicit partial-output policy.
Added the store-native shift_request() -> shift_collect() ->
shift_download() -> shift_extract() -> shift_morph() -> shift_epw_write()
workflow, with inspection helpers such as shift_status(),
shift_diagnostics(), shift_coverage(), shift_outputs(), and
shift_data().
Added EsgStore, a DuckDB-backed local store for query snapshots,
dictionaries, source files, downloads, Parquet extracts, and generated EPW
outputs.
Reworked ESGF querying around esg_query() / EsgQuery and typed
EsgResult* objects for Dataset, File, and Aggregation records.
Added EsgDataset for remote OPeNDAP NetCDF access without downloading full
files.
Rebuilt ESG dictionaries as project-aware EsgDict objects with option
discovery and legality checks across CMIP6 and related ESG projects.
Added persistent downloader support, including download planning, resume, verification, node health, background jobs, daemon mode, and event logs.
Added EpwMorpher, morphing recipes, backend registration, historical
reference handling, resumable morphing, and manifest-backed EPW output
registration (#111, #115).
Belcher EPW downscaling now supports CMIP6 precipitation (pr) and includes it
in the recommended morphing variable set (#122).
Added the epwshiftr CLI with doctor, query, download, storage,
shift, extract, morph, and esgf command groups (#114).
Climate records now expand across every requested morphing period containing their year, including overlapping future windows. The intentional cartesian join is limited to period assignment and covered by a threshold-exceeding regression case (#263).
Extraction preflight now uses the same explicit period assignment as climate
summarisation, so overlapping future windows are validated without triggering
data.table's cartesian join guard (#265).
Registered recipe execution now validates the option-resolved input contract, keeping runtime requirements consistent with discovery and extraction when variables such as precipitation are disabled (#267).
Historical means and extrema now align with future morphing cases by
source_id, member_id, and month when ESGF catalogues use different
institution_id values across experiments. Raw climate records split across
multiple source files are concatenated before period/month statistics are
computed, with total and valid-value counts recorded separately and source
plan lineage stored independently. Legacy fragment-level mean summaries use
valid-value counts when available and retain compatible fallbacks when
planning persisted factors, including precipitation inputs (#259).
Native CF coordinates now expose cf_second_of_day directly from the parsed
clock fields, including fractional seconds. Regional extraction, empty
results, and persisted point series retain the field without reconstructing
it from annual_phase (#257).
Made workflow UI regression fixtures self-contained for installed-package checks and aligned cache progress assertions with the parsed-response event (#255).
Consolidated batch model discovery into one live panel with method and input combination ordinals, future/historical coverage scopes, requested periods, and selected common models. Nested catalog queries no longer leave separate full receipts. Catalog counts explicitly describe indexed metadata, and request waits show timeout, received records, and cache reuse separately. Standalone receipts avoid duplicate results, use explicit intermediate-step labels, and isolate terminal colour state between frame writes (#255).
Reading live workflow snapshots preserves timestamp clock times, fractional
seconds, and UTC offsets. Cancellation status overrides stale worker progress in
R, CLI, and batch views. Batch watch tracks observed events per child so
interleaved failures and terminal events are not skipped or replayed.
Single-run shift show --verbose prints all records with complete wrapped
paths and recovery actions; --debug also includes raw JSON payloads (#255).
shift_cmip6_avail(table = NULL) now discovers one CMIP6 table per variable
at the requested frequency instead of constraining every variable to the
frequency-default table. Complete rows return a named table mapping that
can be passed directly to shift_cmip6(), while the reduction keeps each
variable on one table across all requested experiments. member = NULL is
now the default, so every returned member is evaluated independently without
a preferred variant label (#242).
extract retry and morph retry now filter candidates by the requested
status instead of resolving the selector as the candidate table's status
column. Their shared status validation and preview preparation retain each
command's separate candidate source and retry execution (#232).
Point extraction now retains canonical CF calendar coordinates and annual phase, uses them for non-Gregorian range selection and yearly Parquet partitions, and prefers them in morphing summaries with legacy artifact fallback. Multi-year extraction partitions no longer duplicate every row into every year (#134, #135).
CMIP6 coverage checks now keep requested variable and grid values distinct from same-named convenience columns returned by ESGF providers, preventing complete catalog identities from being rejected (#130, #132).
ESGF result persistence now recognizes the current bridge-provided
datetime_stop, geo, and mod_time document fields. Nodes that ignore an
explicit fields request no longer fail while saving otherwise valid
Dataset or File metadata (#129, #131).
Humidity fallback now skips resolved CMIP table partitions that do not
contain complete huss, tas, and ps extraction rows, avoiding false
derivation attempts for optional partitions such as LImon snd (#126).
shift_request() now preserves provider facet values such as project and
frequency exactly for EsgDict validation instead of silently translating
non-standard aliases. A bare numeric limit now caps Dataset results instead
of silently becoming a pagination size. Empty collections are reported as
partial rather than as ready for extraction, while successful intermediate
receipts display READY instead of the internal state-machine term
waiting.
Standalone dynamic shift_*() stages now translate successful waiting and
detached running workflow states to the framebuffer's done terminal
outcome. This prevents a completed shift_collect() from failing while its
final dashboard receipt is committed.
Belcher CMIP6 resolution now keeps hurs as the canonical humidity input but
can satisfy it from complete huss, tas, and surface-pressure (ps) data
when direct relative humidity is unavailable. The derived hurs values use
the package's ASHRAE saturation-pressure equation, are persisted as normal
extraction plans and Parquet artifacts with source-plan provenance, and are
reused on resume. Mean sea-level pressure (psl) is never substituted for
surface pressure. High-level Future EPW plans also reject equal or nested
delivery/store paths so internal manifests, catalogs, and logs cannot enter
the exported-EPW directory.
Future-EPW collection now preserves ESGF File time metadata and fills missing
ranges from CMIP/DRS filenames before storing or filtering records. Resolver
coverage also repairs older cached catalogs defensively, preventing complete
member/grid combinations from being rejected as if every requested year were
absent. Closest-identity diagnostics prefer identities present in the future
catalog, and a mixed set of coverage failures plus one timed-out mirror no
longer presents an unconditional Retry action.
Historical-reference discovery no longer turns requested calendar years into exact ESGF Dataset end timestamps. This keeps complete monthly datasets whose metadata ends on a representative December date (for example December 16) eligible for resolution; the requested years are still enforced during candidate selection, extraction, and coverage checks. Empty reference catalogs now report a dedicated diagnostic before member/grid matching.
Future-EPW progress now degrades to readable line logs when a dynamic Console
region cannot be recreated, keeps the current business-unit label in sync,
freezes elapsed time for every terminal run state, and preserves every unseen
event between R or CLI watch polls. Expected resolver-node rejection no longer
creates an error diagnostic, and foreground runs expose their durable events
through shift_logs(). Extraction fallback downloads now share the workflow
reporter instead of opening competing native progress bars; remote NetCDF
reads remain visibly alive through a polled worker and fall back to synchronous
I/O when that worker cannot be launched.
Successful dynamic runs now commit a responsive final receipt to terminal scrollback instead of erasing the dashboard and leaving only two summary lines. Its Results section retains completion counts, the delivery directory, and width-wrapped output filenames; compact and log renderers keep their append-only completion summary.
Fixed precipitation morphing summaries to avoid carrying the removed legacy
dist extraction column after grid extraction methods became explicit (#124).
EsgStore now keeps ESGF query collection and downloader operations outside
the store manifest lock, reducing lock hold time during query update and
download workflows (#120).
EsgQuery$collect(all = TRUE) now warns and returns partial results when
ESGF pagination stops making progress, instead of repeatedly requesting the
same offset (#116).
shift_epw_write() and EpwMorpher$write_epw() now fill missing, out-of-range,
and special EPW values before saving generated weather files (#87).
shift_* workflow and the
store-native migration path.Consolidated temperature-only physical-policy execution across the shared Daily/BTWS/BWS component and the Arima and Ek adapters, while preserving each method's diagnostics, result schema, and selected policy (#236).
Consolidated dictionary and ESGF query-result truncation footer rendering while preserving their public print output, spacing, pluralization, and ordering (#234).
Consolidated selection-context updates across EsgDataset and EsgResult
while preserving their class-specific initial state, original source totals,
and source-index order across consecutive slices (#228).
Consolidated fixed 365- and 366-day CF calendar date/offset conversion while
preserving the 365_day/noleap and 366_day/all_leap aliases, negative
offsets, month and year boundaries, and the separate Gregorian and 360_day
algorithms (#230).
Consolidated ordered NetCDF variable and dimension name enumeration inside
EsgDataset while preserving its public methods, file-index behavior,
metadata order, return types, and closed-dataset errors (#226).
Consolidated projected and historical Belcher monthly-extrema aggregation and attachment while preserving their distinct scientific case identities, missing-input behavior, and alignment diagnostics (#224).
Consolidated absolute-target and change-factor Belcher runner assembly while retaining their separate field builders, no-reference fallback, equations, policies, diagnostics, factor metadata, part order, and result values (#222).
Added complete Belcher runner behavior snapshots covering absolute-target and change-factor execution, legacy and enhanced profiles, baseline-EPW fallback, production case isolation, every intermediate part, factors, and diagnostics (#220).
Consolidated direct-model hourly calendar traversal while retaining separate circular point interpolation and conservative interval-mean kernels, identity behavior, row order, time-of-day provenance, and diagnostics (#218).
Consolidated Shift dynamic-scope nesting and artifact-row iteration while preserving per-stage readers, metadata, column selection, diagnostics, ordering, and global row limits (#216).
Consolidated fixed-length ESGF time-field normalization and metadata-first DRS range filling while retaining query warnings, label selection, output formatting, and workflow catalog behavior (#214).
Consolidated typed weather-sequence identifier, year, member-class, provenance, uniqueness, ordering, and shared-identity validation while preserving each sequence type's calendar, variable, row, and physical rules (#212).
Consolidated concurrent data-node reachability, service URL reachability, and latency check scheduling into one callback-driven executor while preserving each check's HTTP-status, Range fallback, timing, and result semantics (#210).
Consolidated ordinary and segmented downloader worker dependencies into one explicitly serialized helper bundle while preserving their separate stream, resume, piece-scheduling, and multi-source algorithms (#207).
Consolidated Store catalog download-plan identity and base-row construction, and replaced duplicate plan decorators with one file-row implementation while preserving candidate selection and layout behavior (#205).
Consolidated the shared settings envelope, schema validation, integer and bound checks, deterministic seed normalization, and role-wise precipitation threshold randomization used by statistical signal methods while preserving each method's equations and threshold semantics (#203).
Consolidated repeated weather input validation, daily temperature preprocessing, successful signal-value extraction, native-calendar method fixtures, and Solr response fixtures into method-neutral helpers while preserving method equations and observable workflow behavior (#201).
Added regression tests for batched Dataset child collection, including
global limit handling and progress labels (#119).
Updated GitHub Actions workflow dependencies for current Actions behavior (#99).
Removed obsolete print helpers and the legacy implementation files (#109, #110).
EsgResultFile and EsgResultAggregation now share their internal download
and OPeNDAP fallback helpers while keeping the public methods unchanged
(#118).
Removed the obsolete testthat start-order override now that tests no longer depend on shared cache side effects (#117).
match_coord() has been refactored to correct the calculation method of
geographical distance. Previously, epwshiftr assumes that distance on
longitude and latitude is the same which is not true. Now it uses a spheroid
formula to calculate the tunnel distance (#39). For details, please see Tunnel
Distance
. The structure of the returned epw_cmip6_coord object has also changed.
The coord column in the coord data.table is also a data.table which
contains 6 columns describing the matched coordinates:
index: the indices of matched coordinatesind_lon, ind_lat: The value indices of longitude or latitude in the
NetCDF coordinate grids. These values are used to extract the corresponding
variable valueslon, lat: the actual longitude or latitude in the NetCDF coordinate
gridsdist: the distance in km between the coordinate values in NetCDF and input
EPWBesides, the usage of the input threshold and max_num has been changed a
little bit:
threshold: Due to the change in distance calculation, the meaning of the
threshold input has been changed. Instead of directly being used to get
the 'closest' grid points in NetCDF, the longitude and latitude threshold
is only used to help exclude grid points that are definitely too far away
from the target location. The default threshold, which is 1 degree for both
longitude and latitude, is still reasonable for common use cases and is kept
unchanged. Also threshold now can be set to NULL. In this case, the
distances between the target location and all grid points will be
calculated. But this may be only useful for rare cases.max_num: Now the value max_num is the key input to control how many grid
points are to be matched. The points will always be ordered in descending
order in terms of the distances.The data in the returned value of extract_data() has been updated to
include a new column dist which gives the spherical distance in km between
EPW location and grid coordinates (#39).
The document on the return value structure for extract_data() and
morphing_epw() has been fixed (#29). And the column order for all metadata
in the returned data.table from extract_data() and morphing_epw() are
not consistent. The columns will always be in the order below (#45):
activity_drsinstitution_idsource_idexperiment_idmember_idtable_idlonlatdistfull is added to future_epw(). When setting to TRUE,
a data.table containing information about how the
data are split by the by argument and also the generated future EPWs and
their paths are returned (#18).summary_database() can append results to the previous scan and detect if
any previously matched NetCDF files do not exist. It stores the metadata of
those missing files as a new attribute not_found in the results. Warnings
are issued if warning is set to TRUE. Also, a new parameter miss has
been added to control how to manage those invalid entries. You can set it to
"keep", which is the default, to do nothing about it or "overwrite" to
overwrite those entries based on the newly matched NetCDF files if possible
(#40).extract_data() now supports non-standard calendar, e.g. 360_day (#32).warning has been added in morphing_epw(). If set to TRUE,
warnings will be issued for cases with input data less than a decade (10 years)
. This is because using data that only covers a short period of time may not
be able to capture the average of future climate (#41).EsgDict class. Please
see ?EsgDict for details (#53).epwshiftr.threshold_alpha has been added to set the threshold
of the absolute value for alpha, i.e. monthly-mean fractional change when
performing morphing operations. The default value is set to 3. If the
morphing methods are set "stretch" or "combined", and the absolute alpha
exceeds the threshold value, warnings are issued and the morphing method is
fallbacked to "shift" to avoid unrealistic morphed values (#54).replica can be NULL in esgf_query() and init_cmip6_index(). In
this case, both the master record and replicas are all returned (#61).EsgfQuery is added to support more flexible APIs for ESGF search
services. Please see ?EsgfQuery for details (#63, #69).summary_database() when no NetCDF files are found in the
input directory (#25).epw in match_coord()
is a search string (#25).morphing_epw() can correctly fall back to use "Shift" method when any
missing values are detected in maximum and minimum prediction values of
climate variables (#25).combined method is used in
morphing_epw() (#25).get_data_node() works again (#80)fields parameter is used to directly filter the ESGF query responses (#66).LazyData in DESCRIPTION (#16).methods
argument in morphing_epw().esgf_query() can accept NULL resolution argument (#12).load_cimp6_index() can be correctly load local CMIP6 output index file when
force is TRUE (#8).esgf_query() will give an informative message when LLNL ESGF node is not
available (#3).extract_data() will automatically detect input NetCDF dimensions (#6).summary_database() now will proceed when no matched found (#6).