Antonio Aguirre Data Modelling·Bayesian Statistics·Machine Learning

On Bayesian Methodology. Part 5/6

02 Dec 2024

VII. Understanding and Comparing Multiple Models

Visualizing Models in Relation to Each Other

A key aspect of a Bayesian Workflow is fitting multiple models for a single problem. This process is not about selecting the best model or averaging models, but rather using a series of fitted models to gain a deeper understanding of each one.

Remark:
  • Model comparison in this context is not about selecting or averaging but about exploring the process of fitting multiple models to understand them better.

    • Considerations When Fitting Multiple Models:

    • Be aware of researcher degrees of freedom:
      • Overfitting can occur if a single “best” model is chosen.
      • A set of fitted models might appear to bracket total uncertainty, but other unconsidered models could still contribute.
    • Multiverse analysis: Evaluate situations where multiple models pass all checks to ensure robustness.

    Cross-validation and Model Averaging

    Cross-validation (CV) is a powerful tool for evaluating models, but it requires careful interpretation:

    Key Principle:
  • If there is significant uncertainty in model comparisons, avoid selecting the single model with the best cross-validation result. This discards the uncertainty from the CV process.

    • Alternative Approaches:

    • Stacking:
      Combines inferences by weighting models to minimize cross-validation error. Stacking can sometimes be seen as pointwise model selection.
    • Scaffold Models:
      Include models that are deliberately simple (e.g., for comparison), experimental, or flawed (e.g., with coding errors). These models serve as benchmarks or stepping stones but are not included in final predictions.

    While a Bayesian Workflow emphasizes continuous model expansion over averaging, there are cases where averaging predictions over competing Bayesian models is reasonable.

    Comparing a Large Number of Models

    When faced with many candidate models, the goal is often to find a simpler model with comparable predictive performance to a more complex, expanded model.

    Warning:
  • Selecting one model based solely on minimizing cross-validation error risks overfitting and suboptimal choices.

    • Projection Predictive Variable Selection:

    • A stable and reliable method to identify smaller models with strong predictive performance.
    • Avoids overfitting by:
      • Projecting submodels based on the expanded model’s predictions.
      • Not fitting each model independently to the data.

    This approach enables efficient comparison of a large number of models while maintaining robust predictive performance.

    Understanding and comparing multiple models is an integral part of Bayesian Workflow. By focusing on the process of model fitting rather than rigid selection or averaging, practitioners can better navigate uncertainty and extract meaningful insights from their analyses. Tools like stacking and projection predictive variable selection ensure that model comparison remains both rigorous and practical.

    Related posts