LOP-GPN & CUQ-GNN

Extensions of Graph Posterior Networks

---

This is the official implementation of the Linear Opinion Pooled Graph Posterior Network (LOP-GPN) and Committee-based Graph Uncertainty Quantification using Posterior Networks (CUQ-GNN) models by Damke and Hüllermeier. The implementation is based on the official implementation of Graph Posterior Networks by Stadler et al. (see citation).

The following additions were made:

• Implementation of LOP-GPN
• Implementation of CUQ-GNN
• Support for additional uncertainty measures
• Accuracy-rejection curve generation
• Added the eval.clj script for automatic evaluation and summarization of the experiments
• Migrated the conda environment to Poetry
• Added and corrected type annotations

Installation

The installation requires Poetry. All dependencies can then be installed in a virtual environment simply by running:

poetry install

The evaluation and result aggregation script (eval.clj) requires the Babashka scripting runtime.

I just want to reproduce the experiments!

To replicate the results from the paper, you can simply run the following commands:

./eval.clj eval # Train and evaluate all model variants
./eval.clj acc-rej-tables # Aggregate the evaluation results into accuracy-rejection curves (CSV)
./eval.clj id-ood-table # Aggregate the OOD evaluation results into a table (CSV)

The trained model weights are written to saved_experiments and their aggregated evaluation results are written to results. Accuracy-rejection curves and the OOD evaluation table are written as *.csv files to a tables directory.

Note that the implementation has only been tested on Linux; it might not run on MacOS or Windows.

Running only parts of the evaluation

By default, the eval command will evaluate all combinations of models and datasets in different settings (standard classification, OOD with left-out-classes, OOD with Gaussian noise or OOD with Bernoulli noise). This behavior can be configured by the following command line options:

• -d, --dataset S : Datasets (CoraML, CiteSeerFull, AmazonPhotos, AmazonComputers, PubMedFull, ogbn-arxiv)
• -m, --model S : Models (appnp, matern_ggp, gdk, gpn, gpn_rw, gpn_lop, cuq_appnp, cuq_gcn, cuq_gat)
• -s, --setting S : Settings (classification, ood_loc, ood_features_normal, ood_features_ber)
• -o, --override k=v : Override key, value pairs (can be used to reconfigure/override arbitrary run parameters)
• --dry : Dry run (only print which evaluations would be performed).
• --retrain : Models will be trained, even if model parameters are already stored for a given run (in saved_experiments).
• --reeval : Models will be evaluated, even if evaluation results are already stored for a given run (in saved_experiments).
• --no-cache : Runs will be considered, even if there are already cached results (in results).
• --delete : All results for the selected runs are deleted from save_experiments and results.

The first four options (-d, -m, -s, -o) also accept multiple values, which can be provided by adding the options multiple times.

Example:

./eval.clj eval -d CoraML -d AmazonComputers -m gpn -m gpn_lop -m cuq_gat -s classification -o run.num_splits=2

This command will evaluate the GPN, LOP-GPN and GAT-based CUQ-GNN models on the CoraML and AmazonComputers datasets in the standard classification setting (no OOD transformation of the training data). Additionally, it will only run the first two (of 10) train/test splits and only compute the average of the results over those first two splits.

Generally, overrides are configured via key=value pairs. A key is of the form config_type.param_name; config_type is either run, data, model or training; given a config_type, the valid param_names are defined here.

Structure of the project

As mentioned above, this implementation is directly based on the reference implementation of GPNs by Stadler et al. The overall structure is documented in the README of their repository.

The main implementation of LOP-GPN is split across the following files:

• gpn_lop.py: Contains the core implementation of LOP-GPN, based on gpn_base.py.
• gpn_base.py: The GPN base implementation was made more generic to make it compatible with LOP-GPN.
• loss.py: The loss for LOP-GPN is computed via the mixture_uce_loss and categorical_entropy_reg functions.
• appnp_propagation.py: The APPNP implementation was extended by support for fully-sparse feature and adjacency matrices, to make LOP-GPN computationally tractable.

The implementation of CUQ-GNN is split across the following files:

• cuq_gnn.py: Contains the cor implementation of CUQ-GNN, based on gpn_base.py.
• gpn_base.py: The GPN base implementation was made more generic to make it compatible with CUQ-GNN.

Cite

If you use LOP-GPN, CUQ-GNN or this code in your own work, please cite, both, our paper(s) and that by Stadler et al.

LOP-GPN

@inproceedings{lop-gpn,
title={Linear Opinion Pooling for Uncertainty Quantification on Graphs},
author={Damke, Clemens and H{\"u}llermeier, Eyke},
booktitle={The 40th Conference on Uncertainty in Artificial Intelligence},
year={2024},
url={https://openreview.net/forum?id=qLGkfpXTSn}
}

CUQ-GNN

@inproceedings{cuq-gnn,
title={{CUQ-GNN}: Committee-Based Graph Uncertainty Quantification Using Posterior Networks},
author={Damke, Clemens and H{\"u}llermeier, Eyke},
editor={Bifet, Albert and Daniu{\v{s}}is, Povilas and Davis, Jesse and Krilavi{\v{c}}ius, Tomas and Kull, Meelis and Ntoutsi, Eirini and Puolam{\"a}ki, Kai and {\v{Z}}liobait{\.{e}}, Indr{\.{e}}},
booktitle={Machine Learning and Knowledge Discovery in Databases. Research Track and Demo Track},
publisher={Springer Nature Switzerland},
address={Cham},
pages={306--323},
year={2024}
}

GPN

@incollection{graph-postnet,
title={Graph Posterior Network: Bayesian Predictive Uncertainty for Node Classification},
author={Stadler, Maximilian and Charpentier, Bertrand and Geisler, Simon and Z{\"u}gner, Daniel and G{\"u}nnemann, Stephan},
booktitle = {Advances in Neural Information Processing Systems},
volume = {34},
publisher = {Curran Associates, Inc.},
year = {2021}
}