The cross-validation functions return a list of class multiScaleR_cv.
This page describes every element of that list, what each score measures
and on what scale, and which functions read the result. Several functions
add a subclass that changes or adds a few elements, and those differences
are described at the end.
Which functions return it
| Function | Class of the result |
cross_validate(), cross_validate_glm() (Gaussian, Poisson, binomial) | multiScaleR_cv |
cross_validate(), cross_validate_glm() ("nbinom2", "Gamma") | multiScaleR_distribution_cv, multiScaleR_cv |
cross_validate_clogit() | multiScaleR_choice_cv, multiScaleR_cv |
cross_validate_field() | multiScaleR_field_cv, multiScaleR_cv |
cross_validate_mixed() | multiScaleR_mixed_cv, multiScaleR_cv |
cross_validate_bayes() | multiScaleR_bayes_cv, multiScaleR_cv |
select_cv() | the class of its candidates |
A fold is one split of the data into analysis rows, which fit the model, and assessment rows, which are held out and scored. An assessment occurrence is one held-out row in one fold. A row assessed in two folds has two occurrences.
Read these elements first
complete:TRUEwhen every planned fold was fitted and scored. When any fold fails it isFALSE, and every pooled score insummaryisNA. Never rank a model on the folds that happened to finish.summary: data frame of pooled scores, one row per metric.metricnames the score, andestimateis the pooled score (lower is better).unitsays what one score refers to,"observation"or, for the Brier score,"trial".n_assessmentis the number of planned assessment occurrences, andn_scoredthe number that entered the score.weight_sumis the total frequency weight behind the score (frequency weight x trials for the Brier score).completerepeats the completion flag.
metrics: the same columns plusfold, with one row per fold and metric. A fold that finished keeps its scores even when another fold failed. The pooled RMSE is not the mean of the fold RMSEs.fold_results: data frame withfold,n_analysisandn_assessment(rows used to fit and to score), andstatus, either"scored"or"failed".errors: list with one element per fold. It isNULLfor a scored fold. For a failed fold it holdsstage, where the fold stopped (for example"initial_glm","scale_fit","prediction", or"training_selection"in aselect_cv()result), andmessage, the reason. A failed scale fit can also carrystart_resultsorradius_profile. Failed folds are never retried.warnings: list with one character vector per fold, holding the warning messages captured while that fold was fitted and predicted. They are captured rather than printed, so read them here.
Read the scores
Every score is a loss, so lower is better. Each assessment occurrence gets
its own loss, stored in a column of predictions named after the metric.
The pooled estimate is the frequency-weighted mean of those losses over
the eligible occurrences. An occurrence is eligible when its frequency
weight is positive and, for binomial data, it has at least one trial.
Ineligible occurrences stay in predictions with eligible = FALSE and
contribute nothing.
| Metric | Loss for one occurrence | Pooled score and its scale |
rmse | Squared error, (observed - predicted)^2 | Square root of the mean squared error, in response units (for example, animals per survey) |
mae | Absolute error | Mean absolute error, in response units |
neg_log_score | Negative natural log of the probability (counts, binomial successes) or density (Gamma) that the fitted model gives the observed value | Mean per observation, in natural-log units |
deviance | Poisson deviance contribution (Poisson only) | Mean per observation |
brier | Squared error of the success probability, averaged over the row's trials (binomial only) | Mean per trial, between 0 and 1 |
How to read the scores
RMSE and MAE judge only the predicted mean. RMSE weighs large misses heavily, and MAE treats every miss in proportion to its size. Note that the
rmsecolumn ofpredictionsholds squared errors, not their roots.The negative log score judges the whole predicted distribution, so it also penalizes a model that is too confident. It keeps every constant of the probability function, such as the binomial coefficient. A difference of D between two models means the better one gave the observed outcomes exp(D) times the probability, as a geometric mean per observation. A difference of 0.1 is a ratio of about 1.11. For Gamma responses the score is a density, so it depends on the response units and can be negative.
An observed outcome given zero probability produces an infinite log score. The value is kept, not clipped.
For NB2 and Gamma, the log score uses theta or the dispersion estimated from the analysis rows and frozen for scoring.
A score means little alone. Compare candidates fitted to the same folds with
cv_tab().
Other elements
predictions: data frame with one row per assessment occurrence, including the occurrences of failed folds, whose predictions and losses areNA.foldandrow_ididentify the occurrence.observedis the response (the number of successes for binomial data), andtrialsis 1 except forcbind(successes, failures)responses.predictionis the expected response on the response scale (the mean count, the mean, or the probability of success per trial).weightis the frequency weight, andeligiblemarks the occurrences that enter the scores.One loss column per metric is named after the metric.
NB2 and Gamma results add
distribution_family, the frozen trainingtheta(NB2) ordispersion(Gamma), withNAwhere it does not apply, andpoisson_limit, which isTRUEonly when a joint NB2 fit verified the exact Poisson limit.
training: list with one element per fold, describing the fits to the analysis rows. An element can be empty or partial when its fold failed early.coefficients: final regression coefficients on the link scale (log for Poisson, NB2, and Gamma, logit for binomial, identity for Gaussian). Useexp()on a log-link coefficient for a multiplicative change in the expected response. Spatial covariates are standardized with the training mean and SD, so their coefficients refer to a one training SD change, unlesscovariate_transform = "raw"was used.initial_coefficients: coefficients of the initial GLM, fitted at the starting scales.scale_est: data frame withMeanandSE, the estimated sigma and its standard error in map units, one row per optimized covariate. Similar values across folds suggest that the data identify the scales well.shape_est: estimated kernel shape (unitless) for kernels that have one, otherwiseNULL.initial_normalization,fitted_normalization: lists ofmeanandsd, the analysis-row centering and scaling of each spatial covariate at the starting and the fitted scales. Assessment rows reuse them.bin_reference: the frozen distance bins (breaks, representative distancesbin_rep, the extraction envelopedomain, andcell_size). Distances are in units ofmax_D, so 1 equalsmax_D.optimizer:par(the optimized scale parameters on the internal scale, sigma divided bymax_D),value(the minimized objective),convergence(0 means success), andmessage.start_results: the table of optimizer starts frommultiScale_optim(), when available.initial_distribution(NB2 and Gamma): the nuisance parameter of the initial fit, in the same form asdistribution.
cross_validate_glm()estimates no scales, so itstrainingholds onlycoefficients(plus the NB2 and Gamma items).distribution(NB2 and Gamma only): list with one element per fold, holding the frozen training distribution. An element isNULLwhen the training fit failed.family, withtheta(the NB2 size,Infonly for a verified Poisson limit) ordispersion(the Gamma dispersion).poisson_limit, andestimated, which isFALSEwhen the parameter was fixed in advance.convention, how the value was obtained, for example"native_gamma_deviance_scale"(the training deviance divided by the sum of the weights).rank,native_K,logLik,nobs,training_deviance, andsum_weightsof the training fit.
diagnostics: list with one element per fold.initial_glmandfitted_glmholdconverged,boundary, anditerfrom the initial and final GLM (NB2 and Gamma results also recordth.warnandSE.thetawhere the native fit has them).scaleholds the scale-fit diagnostics described indiagnostics(). Check them before ranking models, because a scored fold is not necessarily a trustworthy one.prediction_contracts: list of the prediction contracts attached by the prediction method of each fold (see prediction_contract). Each states what the prediction conditions on and which uncertainty it leaves out. It can be empty when the prediction method attaches none.folds: the split plan as supplied.assessment_counts: named integer vector, one value per input row, giving how many times the row was assessed (0 for rows never held out).targets: data frame withrow_id,observed,trials, andweightfor every row used in any split, sorted by row ID.cv_tab()andselect_cv()use it to confirm that candidates were scored on the same data. It never enters a fit.selection_records:NULLunlesskeep_selection = TRUE. Otherwise a list with one compact record per fold, read byselect_cv(). Each record holdsversion,controls,diagnostics, metadatawarnings,data, and anerrormessage if the record could not be built. Leave the records unchanged. They cannot rebuild the fitted models. A successfuldataholds these items.logLik, the training log-likelihood (natural log).native_K, the parameter count of the fitted regression as its log-likelihood reports it, including an estimated dispersion or theta.ecological_K, the number of estimated scales and shapes, andK, their total.nobsandobservation_ids, the rows that entered the likelihood.targets, the analysis-row responses, trials, and weights, andsignature, a summary of the likelihood used to check that candidates are comparable.formula(as text),family, andlink. NB2 and Gamma records addnuisance.
fits:NULLunlesskeep_fits = TRUE. Otherwise the full model of each fold, fitted to that fold's analysis rows only.raster_groups: present only for inputs fromkernel_prep_by_group(). For each fold, the map labels of the analysis and assessment rows. These are fixed inputs, not fitted random effects.settings: list withfamily,link,backend("glm"forcross_validate_glm()and"training_selection"forselect_cv()),metrics, the control lists used, and labels that state what the scores mean.normalization = "training_only"andparameter_uncertainty = "not_integrated", for example, say that standardization came from the analysis rows and that the predictions treat fitted parameters as known.cv_tab()checks these labels before comparing candidates.call: the matched call.
A select_cv() result also holds selection, with the criterion and
ties rule, the candidates, the metrics kept and metrics_dropped,
chosen (the candidate selected in each fold, NA when selection
failed), tables (the training information-criterion table of each
fold), candidate_settings, candidate_warnings, and
candidate_diagnostics. Its settings$controls_from names the candidate
whose controls fill the shared settings. NB2 and Gamma selections add
settings$fold_settings, the settings of the model chosen in each fold.
Functions that read it
print()reports how many folds were scored and how many input rows were assessed, warns when the result is incomplete, and printssummary.cv_tab()ranks complete results on one score, after checking that every candidate used identical folds, responses, trials, and weights.select_cv()scores a rule that picks a model in each fold by its training AIC, AICc, or BIC. It needsselection_recordsorfits.aic_tab()andbic_tab()refuse a cross-validation result, because held-out scores are not a maximized likelihood.
Conditional choice results (multiScaleR_choice_cv)
cross_validate_clogit() scores the probability of the used alternative
within each held-out choice set, and the elements change accordingly.
predictionshas one row per alternative.observedis 1 for the used alternative and 0 for the available ones,choiceis the set ID, andtrialsandweightare 1.linear_predictoris uncentered, so only differences within a set matter.predictionis the probability of the alternative within its set (the probabilities of a set sum to 1), andlog_probabilityis its natural log. Only the used alternative iseligible, and itsneg_log_scoreis the negative log probability of the choice. Log scores are computed directly on the log scale, so they stay finite when a printed probability rounds to zero.summaryandmetricsholdneg_log_scoreonly, averaged over choice sets, withunit = "choice_set".n_assessment,n_scored, andweight_sumcount sets, andn_alternativescounts the alternative rows. A model without covariates or offsets gives each of k alternatives a probability of 1/k and scores log(k), for example 1.099 for three alternatives.fold_resultscounts alternative rows, andtargetsaddschoice.diagnosticsholdsinitialandnative(iteration countiterand partial-likelihoodmethod) andscale.selection_recordsare version 2. Theirnobscounts choice sets (one event each), not alternatives, and they record themethod.settingsgivesfamily = "conditional_choice",link = "softmax", themethod, andbackend = "clogit"when no scales were estimated.Its
print()method reports the folds scored and printssummary.
Spatial field results (multiScaleR_field_cv)
cross_validate_field() scores each held-out block jointly, so that
correlation among neighboring held-out observations counts.
predictionsholdsfold,row_id,observed,prediction(the predicted response mean),squared_error,absolute_error, andeligible, plus four variance columns.new_location_field_varianceis the field variance that would remain even if the field were known exactly at the training locations, andfitted_training_field_varianceis the extra variance that arises because the field there is itself estimated. Both are on the linear-predictor scale and, for non-Gaussian families, describe a local Gaussian approximation.observation_noise_variance(response scale) is the observation variance, averaged over the field for non-Gaussian families.observation_predictive_SE(response scale) is the standard deviation of a new observation, combining the field and the noise. It is a predictive SD, not the standard error of the mean.joint_scoresholds, for each fold, the block sizenand the block'sneg_log_score. Gaussian blocks add thequadraticloss andlog_determinant. Laplace blocks add the analysis and combined log densities, their latent-mode diagnostics, and the gap from the native analysis likelihood.summaryhas columnsmetric,estimate,n_assessment,unit, andcomplete. The log score is the sum of the block scores divided by the number of assessment observations, with unit"joint_block_density_per_observation"or"joint_block_mass_per_observation"(counts and 0/1 data). RMSE and MAE have unit"response". There is nometricstable, so usecv_tab(by_fold = TRUE)for fold scores.trainingholds the analysisrow_ids,parameters(field_SD, the correlationrangein coordinate units,observation_SDor the NB2 or Gamma dispersion, and parameter counts),coefficients(link scale), and, with kernel inputs,normalization,scale_est,bin_reference, and the number ofrefits.selection_recordsare always kept. Each holdslogLik,K,native_K,ecological_K,nobs,training_ids, and hashes of the training responses and coordinates.diagnosticsholds the native fit record of each fold, including the covariance starts tried and the path of its audit file.targetsholdsrow_idandobserved,coordinatesholdsrow_id,x, andy, andintegrity_sha256is a hash of the recorded results.cv_tab()andselect_cv()refuse a result whose records were changed after the run.
Mixed model results (multiScaleR_mixed_cv)
cross_validate_mixed() scores all held-out rows of a group jointly,
because they share one unknown group effect.
predictionsaddsgroup, andpredictionis the response mean integrated over the group effect (the success probability per trial for binomial data).joint_scoresholds, for each scored fold, a data frame withgroup, the number of held-out rowsn, and the group's jointneg_log_score.In
summaryandmetrics, the log score is the sum of the group scores divided by the number of assessment rows (unit"joint_group_density_per_row"). RMSE and MAE have unit"response", or"proportion"for binomial data, where they compare the predicted probability with the observed fraction of successes. The Brier score has unit"trial".trainingholds the analysisrow_ids,coefficients(link scale), the random-effectcovariance, the nativedispersion(1 for Poisson and binomial), and any fittedscale_est,normalization, andbin_reference.selection_recordsare always kept, withlogLik,K,native_K,ecological_K,nobs, and theanalysisrow IDs.diagnosticsholdsconvergence,positive_hessian,native_message, andscale_convergence, andintegrity_sha256protects the records as for field results.
Bayesian results (multiScaleR_bayes_cv)
cross_validate_bayes() scores held-out blocks with the posterior
predictive density, the only score in the package that averages over
parameter uncertainty. Its summary reports the joint negative log score
per observation, and its predictions hold posterior mean responses and
marginal row scores. It has no training likelihood, so select_cv() does
not apply. See cross_validate_bayes() for its elements.
Examples
data("landscape_counts")
data("surv_pts")
# Three spatial folds. The simulated coordinates are in meters.
folds <- spatial_cv_folds(crds(vect(surv_pts)), block_size = 1000, k = 3,
crs = 32617)
cv <- cross_validate_glm(counts ~ site, landscape_counts, folds,
family = "poisson", keep_selection = TRUE)
class(cv)
#> [1] "multiScaleR_cv"
cv$complete # TRUE: every fold was fitted and scored
#> [1] TRUE
cv$summary # pooled scores, one row per metric, lower is better
#> metric estimate unit n_assessment n_scored weight_sum complete
#> 1 rmse 2.232857 observation 100 100 100 TRUE
#> 2 mae 1.661678 observation 100 100 100 TRUE
#> 3 neg_log_score 2.169778 observation 100 100 100 TRUE
#> 4 deviance 2.372818 observation 100 100 100 TRUE
cv$metrics[cv$metrics$metric == "rmse", ] # the same score in each fold
#> fold metric estimate unit n_assessment n_scored weight_sum complete
#> 1 fold1 rmse 2.091114 observation 34 34 34 TRUE
#> 5 fold2 rmse 1.620645 observation 33 33 33 TRUE
#> 9 fold3 rmse 2.824229 observation 33 33 33 TRUE
head(cv$predictions) # one row per held-out observation, with its losses
#> fold row_id observed trials prediction weight eligible rmse mae
#> 1 fold1 2 2 1 3.000225 1 TRUE 1.0004503 1.0002251
#> 2 fold1 5 2 1 2.946952 1 TRUE 0.8967175 0.9469517
#> 3 fold1 6 0 1 1.995418 1 TRUE 3.9816928 1.9954179
#> 4 fold1 8 1 1 1.690686 1 TRUE 0.4770469 0.6906858
#> 5 fold1 11 0 1 1.673290 1 TRUE 2.7998998 1.6732901
#> 6 fold1 26 1 1 2.537614 1 TRUE 2.3642583 1.5376145
#> neg_log_score deviance
#> 1 1.495998 0.3782897
#> 2 1.478556 0.3434069
#> 3 1.995418 3.9908359
#> 4 1.165552 0.3311031
#> 5 1.673290 3.3465802
#> 6 1.606390 1.2127800
cv$fold_results # rows fitted and held out, and the status of each fold
#> fold n_analysis n_assessment status
#> 1 fold1 66 34 scored
#> 2 fold2 67 33 scored
#> 3 fold3 67 33 scored
cv$training$fold1$coefficients # log-link coefficients fitted in fold 1
#> (Intercept) site
#> 0.7447308 0.1776788
# What select_cv() reads from each fold
cv$selection_records$fold1$data[c("logLik", "K", "nobs")]
#> $logLik
#> [1] -141.7151
#>
#> $K
#> [1] 2
#>
#> $nobs
#> [1] 66
#>
# The pooled RMSE is the square root of the mean squared error over all
# held-out rows, not the mean of the three fold RMSEs
sqrt(mean(cv$predictions$rmse))
#> [1] 2.232857
