MOFBench#
Base classes#
Bases for MOF ML model benchmarking.
- class MOFBenchRegression(model, ds, splitter, target, name, task, k=5, model_type=None, version=None, features=None, reference=None, implementation=None, debug=False, patch_in_ds=False)[source]#
Regression benchmarking class.
Initialize the benchmarking class.
- Parameters:
model – Model to be benchmarked.
ds (AbstractStructureDataset) – Dataset to be used for benchmarking.
splitter (BaseSplitter) – Splitter to be used for benchmarking.
target (List[str]) – Target labels to be used for benchmarking. Must be included in the dataset.
name (str) – Name of the model. This will be used as filename.
task (BenchTaskEnum) – Task of the benchmark.
k (int) – Number of folds for k-fold cross-validation.
model_type (str, optional) – Model type, e.g. ‘CGCNN’, ‘BERT’, ‘XGBoost’. Defaults to None.
version (str, optional) – Version of the model. Defaults to None.
features (str, optional) – Features used in the model. If you use a graph net, report the edge and vertex features. Defaults to None.
reference (str, optional) – Reference with more details Please provide it in BibTeX form. Defaults to None.
implementation (str, optional) – Link to implementation. Defaults to None.
debug (bool) – If True, the benchmark will be run in debug mode (1% of the data).
patch_in_ds (bool) – If True, the dataset will be patched into the model class under the ds attribute.
- class BenchResult(**data)[source]#
Model for benchmark results.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
The results are reported in pydantic
model with the following fields:
BenchResult#
Model for benchmark results. |
||||
type |
object |
|||
properties |
||||
|
Start time |
|||
Start time of the benchmark |
||||
type |
string |
|||
format |
date-time |
|||
|
End time |
|||
End time of the benchmark |
||||
type |
string |
|||
format |
date-time |
|||
|
Metrics |
|||
Metrics of the benchmark |
||||
allOf |
#/definitions/RegressionMetricCollection |
|||
|
Version |
|||
Version of the benchmark |
||||
type |
string |
|||
|
Features |
|||
Features used in the model. If you use a graph net, report the edge and vertex features. |
||||
type |
string |
|||
|
Name |
|||
Name of the model. This will be used as filename. |
||||
type |
string |
|||
|
Task |
|||
Task of the benchmark |
||||
allOf |
#/definitions/BenchTaskEnum |
|||
|
Model type |
|||
Model type, e.g. ‘CGCNN’, ‘BERT’, ‘XGBoost’ |
||||
type |
string |
|||
|
Reference |
|||
Reference with more details |
||||
type |
string |
|||
|
Implementation |
|||
Link to implementation |
||||
type |
string |
|||
|
mofdscribe version |
|||
mofdscribe version |
||||
type |
string |
|||
|
Session info |
|||
Automatically captured string describing the computational environment. |
||||
type |
object |
|||
|
Logs |
|||
Additional data logged using the log method during the benchmark. |
||||
type |
array |
|||
items |
||||
type |
object |
|||
definitions |
||||
|
RegressionMetrics |
|||
Model for regression metrics. We will use this model to validate the benchmark results. |
||||
type |
object |
|||
properties |
||||
|
Mean Squared Error |
|||
type |
number |
|||
|
Mean Absolute Error |
|||
type |
number |
|||
|
R2 Score |
|||
type |
number |
|||
|
Max Error |
|||
type |
number |
|||
|
Mean Absolute Percentage Error |
|||
type |
number |
|||
|
Top 5 In Top 5 |
|||
type |
integer |
|||
|
Top 10 In Top 10 |
|||
type |
integer |
|||
|
Top 50 In Top 50 |
|||
type |
integer |
|||
|
Top 100 In Top 100 |
|||
type |
integer |
|||
|
Top 500 In Top 500 |
|||
type |
integer |
|||
|
RegressionMetricCollection |
|||
Model for regression metric collection. |
||||
type |
object |
|||
properties |
||||
|
Regression Metrics |
|||
type |
array |
|||
items |
||||
#/definitions/RegressionMetrics |
||||
|
Fit Timings |
|||
type |
array |
|||
items |
||||
type |
number |
|||
|
Inference Timings |
|||
type |
array |
|||
items |
||||
type |
number |
|||
|
BenchTaskEnum |
|||
Enum for benchmark tasks. |
||||
enum |
logKH_CO2_id, logKH_CO2_ood, pbe_bandgap_id, ch4dc_id |
logarithmic CO2 Henry coefficient#
In-dataset predictions for the logarithmitic CO2 Henry coefficients
- class LogkHCO2IDBench(model, name, version='v0.0.1', features=None, model_type=None, reference=None, implementation=None, debug=False, patch_in_ds=False)[source]#
Benchmarking models for the logarithmic CO2 Henry coefficient under in-domain conditions.
In-distribution implies that we use a cluster stratified splitter that ensures that the ratios of different clusters in the training and test set are the same.
Initialize the log KH CO2 interpolation benchmark.
- Parameters:
model (object) – The model to be benchmarked. Must implement the fit and predict methods.
name (str) – The name of the modeling approach.
version (str, optional) – Version of the dataset to use. Defaults to “v0.0.1”.
features (str, optional) – Description of the features used in the model. Defaults to None.
model_type (str, optional) – Model type (e.g. Conv-Net, BERT, XGBoost). Defaults to None.
reference (str, optional) – Reference with more details about modeling approach. Defaults to None.
implementation (str, optional) – Link to implementation. Defaults to None.
debug (bool) – If True, use a small dataset (1% of full dataset) for debugging. Defaults to False.
patch_in_ds (bool) – If True, the dataset will be patched into the model class under the ds attribute.
- class LogkHCO2OODBench(model, version='v0.0.1', features=None, name=None, model_type=None, reference=None, implementation=None, debug=False, patch_in_ds=False)[source]#
Benchmarking models for the logarithmic CO2 Henry coefficient under “out-of-domain” conditions.
“Out-of-domain” conditions means that every of the 5 training fold will only see 4 out of the 5 quantile bins. This implies that 2 runs are extrapolative and the other 3 need to “fill holes in the distribution”.
Initialize the log KH CO2 extrapolation benchmark.
- Parameters:
model (object) – The model to be benchmarked. Must implement the fit and predict methods.
name (str) – The name of the modeling approach.
version (str, optional) – Version of the dataset to use. Defaults to “v0.0.1”.
features (str, optional) – Description of the features used in the model. Defaults to None.
model_type (str, optional) – Model type (e.g. Conv-Net, BERT, XGBoost). Defaults to None.
reference (str, optional) – Reference with more details about modeling approach. Defaults to None.
implementation (str, optional) – Link to implementation. Defaults to None.
debug (bool) – If True, use a small dataset (1% of full dataset) for debugging. Defaults to False.
patch_in_ds (bool) – If True, the dataset will be patched into the model class under the ds attribute.
PBE bandgap#
In-dataset predictions for PBE bandgaps.
- class PBEBandGapIDBench(model, name, version='v0.0.1', features=None, model_type=None, reference=None, implementation=None, debug=False, patch_in_ds=False)[source]#
Benchmarking models for the PBE bandgap under in-distribution conditions.
In-distribution implies that we use a cluster stratified splitter that ensures that the ratios of different clusters in the training and test set are the same.
Initialize the PBE bandgap benchmark.
- Parameters:
model (object) – The model to be benchmarked. Must implement the fit and predict methods.
name (str) – The name of the modeling approach.
version (str, optional) – Version of the dataset to use. Defaults to “v0.0.1”.
features (str, optional) – Description of the features used in the model. Defaults to None.
model_type (str, optional) – Model type (e.g. Conv-Net, BERT, XGBoost). Defaults to None.
reference (str, optional) – Reference with more details about modeling approach. Defaults to None.
implementation (str, optional) – Link to implementation. Defaults to None.
debug (bool) – If True, use a small dataset (1% of full dataset) for debugging. Defaults to False.
patch_in_ds (bool) – If True, the dataset will be patched into the model class under the ds attribute.
Methane deliverable capacity#
In-dataset predictions for the methane deliverable capacity
- class CH4DCIDBench(model, name, version='v0.0.1', features=None, model_type=None, reference=None, implementation=None, debug=False, patch_in_ds=False)[source]#
Benchmarking models for the methane deliverable capacity under in-domain conditions.
In-distribution implies that we use a cluster stratified splitter that ensures that the ratios of different clusters in the training and test set are the same.
Initialize the CH4DC interpolation benchmark.
- Parameters:
model (object) – The model to be benchmarked. Must implement the fit and predict methods.
name (str) – The name of the modeling approach.
version (str, optional) – Version of the dataset to use. Defaults to “v0.0.1”.
features (str, optional) – Description of the features used in the model. Defaults to None.
model_type (str, optional) – Model type (e.g. Conv-Net, BERT, XGBoost). Defaults to None.
reference (str, optional) – Reference with more details about modeling approach. Defaults to None.
implementation (str, optional) – Link to implementation. Defaults to None.
debug (bool) – If True, use a small dataset (1% of full dataset) for debugging. Defaults to False.
patch_in_ds (bool) – If True, the dataset will be patched into the model class under the ds attribute.
Dummy Models#
Simple baseline models.