Fuse Variables to a Recipient Dataset

Fuses synthetic target variables onto a recipient microdata set using a pre-trained fusion model archive (.fsn file) created by train. Generates one or more synthetic implicates (\(M\)) that attempt to preserve conditional distributions, non-linear dependencies, and covariance structures.

Usage

fuse(
  data,
  fsn,
  fsd = NULL,
  M = 1,
  retain = NULL,
  kblock = 10,
  margin = 2,
  cores = 1
)

Arguments

data

Data frame or data.table. The recipient microdata set. All categorical predictor variables must be factors (ordered where appropriate). Data types and factor levels are strictly validated against the predictor schema recorded in the trained model archive (fsn).

fsn

Character string. File path to the trained fusion model archive (.fsn extension) generated by train.

fsd

Character string. Optional path for saving the fused output directly to disk as a compressed binary file (.fsd extension). If NULL (default), the fused results are returned directly as an in-memory data.table.

M

Integer. Number of synthetic implicates to simulate (\(M \ge 1\)). Default is 1.

retain

Character vector. Column names in data to retain in the fused output (e.g., household IDs or survey weights). Retained columns are replicated across all implicates.

kblock

Integer. Fixed number of nearest neighbors (\(k\)) evaluated during candidate donor selection when fusing variable blocks (\(5 \le k \le 30\)). Applies only to multivariate block variables modeled jointly in train. Default is 10.

margin

Numeric. Safety factor applied when estimating available system memory and dynamic chunk sizes during implicate generation. Increase margin (e.g., margin = 3) if memory shortfalls occur. Alternatively, supply a negative integer (e.g., margin = -3) to force manual splitting of \(M\) implicates into a fixed number of equal chunks. Default is 2.

cores

Integer. Number of physical CPU cores used for internal LightGBM model predictions via OpenMP multithreading. Default is 1.

Value

If fsd = NULL, returns a data.table containing M * nrow(data) rows. An integer column M indicates the implicate index for each observation. The ordering of observations within each implicate strictly matches the input data.

If fsd is specified, saves the result to disk and returns the file path invisibly. Binary .fsd files should be loaded using read_fsd.

Details

Implicate Simulation and Memory Management

Data fusion generates synthetic outcomes by first predicting conditional expectations (means, probabilities, and quantiles) for recipient observations using the LightGBM boosters stored in the .fsn file. Recipient rows are then mapped to candidate donor clusters in expectation space, from which observed values are sampled proportional to donor sampling weights.

When generating multiple implicates (\(M > 1\)) on large datasets, memory demands can be substantial. fuse() automatically monitors available system RAM and splits the processing of \(M\) implicates into dynamic memory-managed chunks (nstep). Writing output directly to disk via fsd further reduces memory overhead.

Sequence Chaining across Steps

If the fusion model was trained sequentially, variables fused in earlier steps are automatically inserted into the recipient matrix to serve as predictors for subsequent fusion steps, ensuring proper conditioning across all target variables.

References

Ummel, K., et al. (2024). Multidimensional well-being of US households at a fine spatial scale using fused household surveys. Scientific Data, 11(142). doi:10.1038/s41597-023-02788-7

See also

train, read_fsd, validate

Examples

if (FALSE) { # \dontrun{
# Load sample RECS survey dataset supplied with fusionModel
data(recs)
# Define targets and predictors, then fit fusion model
fusion.vars <- c("electricity", "natural_gas", "aircon")
predictor.vars <- names(recs)[2:12]
fsn.path <- train(data = recs, y = fusion.vars, x = predictor.vars)
# 1. Generate a single implicate using original RECS data as recipient
recipient <- recs[predictor.vars]
sim <- fuse(data = recipient, fsn = fsn.path)
head(sim)
# 2. Generate multiple implicates (M = 5)
sim_multi <- fuse(data = recipient, fsn = fsn.path, M = 5)
head(sim_multi)
table(sim_multi$M)
# 3. Generate implicates and save results directly to disk (.fsd format)
fsd.path <- fuse(data = recipient, fsn = fsn.path, M = 5, fsd = "results.fsd")
sim_disk <- read_fsd(fsd.path)
head(sim_disk)
} # }