Fit a GLM to the analysis rows of each fold and score its predictions for
the held-out rows. No scales of effect are estimated. Use this function to
build null and nonspatial reference models on the same folds you give
cross_validate(), so that a scale-of-effect model has a baseline to beat.
Usage
cross_validate_glm(
formula,
data,
folds,
family = c("gaussian", "poisson", "binomial", "nbinom2", "Gamma"),
weights = NULL,
glm_control = stats::glm.control(),
metrics = NULL,
keep_fits = FALSE,
keep_selection = FALSE,
theta = NULL,
dispersion = NULL
)Arguments
- formula
Two-sided model formula, written as for
stats::glm(). The response must be a column ofdata, optionally wrapped inI(), orcbind(successes, failures)for grouped binomial data. Predictors can include factors, interactions, offsets, and the data-dependent termsscale(),poly(),splines::ns(), andsplines::bs(). Usey ~ 1for a null model. Every variable must be a column ofdata.- data
Data frame with one row per observation, holding the raw responses and every predictor. Its row names are the row IDs that the folds refer to, and they must be unique. Do not supply predictors standardized or transformed with all rows (see "What each fold fits").
- folds
Named list of explicit splits, for example from
grouped_cv_folds()orspatial_cv_folds(). Each split is a list of two character vectors of row IDs,analysisandassessment, which must not overlap. A row can be left out of a split entirely (for example, by a spatial buffer) and can be assessed in more than one split. Keep rows that share a site, animal, or other sampling unit on the same side of every split.- family
Character response family. Choose
"gaussian"(identity link, the default),"poisson"(log),"binomial"(logit),"nbinom2"(negative binomial, log), or"Gamma"(log). Lowercase"gamma"is accepted. For"nbinom2"the variance is mu + mu^2 / theta, where mu is the expected count and theta the size parameter, so a small theta means strong extra variation. For"Gamma"the variance is dispersion x mu^2. Other links and quasi-likelihood families are not supported.- weights
Optional case frequencies, one nonnegative whole number per row of
datain the same order. If the vector has names, they must equal the row names. A weight of 3 counts the row three times. A weight of 0 keeps the row in the records but out of the fit and the scores. The defaultNULLgives every row a weight of one. Gaussian models accept only weights of one. For binomial data, put the trial counts in acbind(successes, failures)response, not inweights. Weighted proportions are rejected. Some scale-fitting backends add their own weight requirements.- glm_control
Named list of
stats::glm.control()settings for the GLM fitted in each fold. The default keeps the settings ofglm().- metrics
Optional character vector of scores to record. The defaults depend on
family. They arec("rmse", "mae")for Gaussian,c("rmse", "mae", "neg_log_score", "deviance")for Poisson,c("neg_log_score", "brier")for binomial, andc("rmse", "mae", "neg_log_score")for NB2 and Gamma. You can request a subset of the defaults, but not other scores. NB2 and Gamma leave out the deviance because its value depends on theta or the dispersion, which makes it unsuitable for comparing models. Lower is better for every score, and the multiScaleR_cv page defines each one.- keep_fits
Logical. The default
FALSEkeeps the predictions, scores, fold estimates, and compact diagnostics.TRUEalso keeps the full fitted model of every fold infits, which can use much more memory. You need full fits only to inspect the fold models or to haveselect_cv()return the selected fits.- keep_selection
Logical.
TRUEkeeps a compact record of each fold's training likelihood and parameter count inselection_records, which is whatselect_cv()needs whenkeep_fits = FALSE. The records come from the fits already run, so they add no fitting time. The default isFALSE.- theta
Optional NB2 size parameter fixed in advance, one positive number. The default
NULLestimates theta in each fold withMASS::glm.nb(). A fixed theta is fitted withglm(family = MASS::negative.binomial(theta))and adds no parameter to K. Fix it only from information independent of the assessment outcomes. Other families reject it.- dispersion
Optional Gamma dispersion fixed in advance, one positive number. The default
NULLestimates it in each fold as the training deviance divided by the sum of the frequency weights, the value native Gamma AIC uses. A fixed dispersion adds no parameter to K and is built into the Gamma density before fitting. Fix it only from information independent of the assessment outcomes. Other families reject it.
Value
A list of class multiScaleR_cv, with class
multiScaleR_distribution_cv added for "nbinom2" and "Gamma". It has
the elements of a cross_validate() result, with these differences.
settings$backend is "glm". training holds only the coefficients of
each fold, plus the frozen nuisance parameter for NB2 and Gamma.
diagnostics holds the convergence record of each fold's GLM, and fits
holds ordinary glm objects when keep_fits = TRUE. See
multiScaleR_cv for every element and how to read each score.
What each fold fits
Each fold fits glm() to its analysis rows with the declared family and
frequency weights, then predicts the expected response of the assessment
rows. The families, response forms, weights, and scores are those of
cross_validate(). Factors, interactions, and offsets use the native GLM
machinery. The constants of scale(), poly(), splines::ns(), and
splines::bs() come from the analysis rows, even inside expressions and
offsets. Any other preprocessing that learns from the data must also
happen within the analysis rows of each fold, so do not supply predictors
standardized with all rows. The section "Formula terms that learn from the
data" in cross_validate() lists the supported terms and their limits.
Missing values are never dropped silently, and native warnings and failed
folds are kept.
Binomial responses
A binomial response can be numeric or logical 0/1 values, a two-level
factor (the second level counts as a success), or
cbind(successes, failures) for several trials per row. Keep trial counts
in the response and case frequencies in weights, because the two are
scored differently.
The negative log score of a row is the negative log binomial probability of its observed number of successes, given its trials. Rows are averaged with their frequency weights.
The Brier score of a row is the squared error of the predicted success probability, averaged over the row's trials. Rows are averaged with weights equal to frequency weight x trials, so the pooled Brier score is a mean per trial. A row with many trials counts more in the Brier score than in the log score.
A row with zero trials or zero weight stays in
predictionsbut contributes to neither score.An observed outcome given zero probability produces an infinite log score. It is kept as is, not clipped.
Negative binomial and Gamma responses
"nbinom2" fits MASS::glm.nb() when theta is estimated and
glm(family = MASS::negative.binomial(theta)) when it is fixed, so it
needs the MASS package. "Gamma" uses a log link. In each fold the
nuisance parameter (theta, or the Gamma dispersion) is estimated from the
analysis rows, frozen, and used to score the assessment rows. Coefficients
and their native standard errors are those of the native fit. A fixed
Gamma dispersion is built into a normalized Gamma family before fitting.
The section "Negative binomial and Gamma responses" in cross_validate()
describes how these parameters enter K and the rules they follow.
How to read the Gamma dispersion
distribution[[fold]]$dispersion is the training deviance divided by the
training sum of case frequencies, the value native Gamma AIC uses. It is
held fixed when the held-out Gamma densities are scored. Native
coefficient standard errors use the Pearson dispersion instead, so they
can rest on a different number. The density dispersion does not replace
that coefficient uncertainty. convention records the rule that produced
the value, and estimated is FALSE when you fixed it with dispersion.
Supply theta or dispersion only when the value is known or was fixed
independently of the assessment outcomes.
What the scores do and do not include
Predictions use each fold's fitted parameters as if they were known. They
do not integrate parameter uncertainty, provide prediction intervals, or
establish calibration. Use cross_validate() to estimate scales of effect
within the same folds.
Examples
data("landscape_counts") # counts and a site-level covariate
data("surv_pts") # survey locations
# Three spatial folds. The simulated coordinates are in meters.
folds <- spatial_cv_folds(crds(vect(surv_pts)), block_size = 1000, k = 3,
crs = 32617)
# Null and site-only reference models on the same folds
null_cv <- cross_validate_glm(counts ~ 1, landscape_counts, folds,
family = "poisson")
site_cv <- cross_validate_glm(counts ~ site, landscape_counts, folds,
family = "poisson")
site_cv # pooled held-out scores, lower is better
#> Cross-validation: 3 of 3 folds scored; 100 of 100 input rows assessed.
#> metric estimate unit n_assessment n_scored weight_sum complete
#> rmse 2.232857 observation 100 100 100 TRUE
#> mae 1.661678 observation 100 100 100 TRUE
#> neg_log_score 2.169778 observation 100 100 100 TRUE
#> deviance 2.372818 observation 100 100 100 TRUE
cv_tab(list(null = null_cv, site = site_cv), metric = "neg_log_score")
#> Modnames Rank Score Delta Metric Unit N_assessment
#> 1 null 1 2.163750 0.000000000 neg_log_score observation 100
#> 2 site 2 2.169778 0.006027989 neg_log_score observation 100
#> N_scored Weight_sum Warnings
#> 1 100 100 0
#> 2 100 100 0
# Negative binomial: theta is estimated in each fold and frozen for scoring
if (requireNamespace("MASS", quietly = TRUE)) {
nb_cv <- cross_validate_glm(counts ~ site, landscape_counts, folds,
family = "nbinom2")
sapply(nb_cv$distribution, function(d) d$theta) # small theta: overdispersed
}
#> fold1 fold2 fold3
#> 1.561918 1.438781 1.939947
# Gamma model for a strictly positive response
i <- 1:96
positive <- data.frame(
y = exp(0.2 + 0.35 * sin(i * 0.73)) *
rep(c(0.3, 0.6, 1.1, 2.4, 0.8, 1.8, 0.4, 1.6), 12),
x = sin(i * 0.73))
gamma_folds <- grouped_cv_folds(rep(letters[1:12], each = 8), k = 3)
gamma_cv <- cross_validate_glm(y ~ x, positive, gamma_folds, family = "Gamma")
gamma_cv$summary
#> metric estimate unit n_assessment n_scored weight_sum complete
#> 1 rmse 0.9077540 observation 96 96 96 TRUE
#> 2 mae 0.7704337 observation 96 96 96 TRUE
#> 3 neg_log_score 1.1450485 observation 96 96 96 TRUE
# Training deviance / sum of case frequencies, frozen for scoring each fold
sapply(gamma_cv$distribution, function(d) d$dispersion)
#> fold1 fold2 fold3
#> 0.4401327 0.4399048 0.4391975
