Cloud access, versioning, and quality assurance

library(hfutils)

This vignette covers the three responsibilities that make hfutils the trusted base layer of the stack: authenticated cloud access, dataset versioning & provenance, and staged invariant checks.

Authenticated cloud access

lynker_spatial_auth() provisions (or refreshes) an OAuth2 token and wires it into the libraries you name. With GDAL configured, any as_ogr() / sf::read_sf() call can read directly from cloud storage via a GDAL virtual file system; with DuckDB configured, the bearer token is attached as a secret.

# Interactive browser flow; sets GDAL_HTTP_BEARER for the session
token <- lynker_spatial_auth(libs = "gdal")

as_ogr("/vsis3/lynker-spatial/hydrofabric/v2.2/conus/conus_nextgen.gpkg",
  "divides")

# DuckDB over HTTP, with the token attached to the connection
con <- duckdb_connection(extensions = "httpfs")
tbl_http("https://lynker-spatial.s3.amazonaws.com/v20.1/flowpaths.parquet",
  conn = con)

Dataset versioning & provenance

gpkg_set_version() stamps a dataset version into the standard GeoPackage metadata tables — distinct from the GeoPackage specification version in PRAGMA user_version, which is left untouched. It records a machine-readable packed integer (Mmmpp), a human-readable semver string, an optional SPDX license, and an optional JSON provenance record. It is idempotent: re-stamping replaces prior entries rather than accumulating duplicates.

library(sf)
#> Linking to GEOS 3.12.1, GDAL 3.8.4, PROJ 9.4.0; sf_use_s2() is TRUE

f <- tempfile(fileext = ".gpkg")
nc <- st_read(system.file("shape/nc.shp", package = "sf"), quiet = TRUE)[1:2, ]
st_write(nc, f, "divides", quiet = TRUE)

gpkg_set_version(
  f, "2.2.0",
  provenance = list(software = "hydrofabric", vpuid = "09"),
  license    = "ODbL-1.0"
)

str(gpkg_get_version(f))
#> List of 4
#>  $ version    : chr "2.2.0"
#>  $ int_version: int 20200
#>  $ license    : chr "ODbL-1.0"
#>  $ provenance :List of 2
#>   ..$ software: chr "hydrofabric"
#>   ..$ vpuid   : chr "09"

int_version (20200 here) is the value GDAL surfaces as GPKG_METADATA_ITEM_1, convenient for machine comparison across releases.

Staged invariant checks

hf_check_invariants() is the shared QA gate used at each stage of the build pipeline. Calling it between steps means a silent failure (a duplicate id, a dangling toid, an orphan divide, a raster ring artifact) halts the run at the step that created it — not 200 lines later during visual inspection.

It supports four stages — "refactored", "reconciled", "aggregated", and "ngen" — each expecting stage-specific inputs. Here is the topology half of a refactored-stage check on a clean toy network:

refactored <- data.frame(
  flowpath_id   = c("101", "102", "103"),
  flowpath_toid = c("102", "103", "0")
)

res <- hf_check_invariants("refactored", refactored = refactored)
#> [invariants:refactored] ✓ refactored_unique                refactored flowpath_id has 0 duplicate(s)
#> [invariants:refactored] ✓ refactored_topology              0 refactored flowpath_toid values reference nonexistent IDs
res$ok
#> [1] TRUE

By default any failed check throws (strict = TRUE); set strict = FALSE to downgrade failures to warnings while still surfacing them. A dangling downstream pointer is caught:

broken <- data.frame(
  flowpath_id   = c("101", "102"),
  flowpath_toid = c("102", "999")   # 999 does not exist
)

hf_check_invariants("refactored", refactored = broken)
#> [invariants:refactored] ✓ refactored_unique                refactored flowpath_id has 0 duplicate(s)
#> Error:
#> ! [invariants:refactored] ✗ refactored_topology              1 refactored flowpath_toid values reference nonexistent IDs

The aggregated and ngen stages additionally accept coverage = TRUE to run the (more expensive) check that each flowpath lies within its assigned catchment.