MOFBench#

Base classes#

Bases for MOF ML model benchmarking.

class MOFBenchRegression(model, ds, splitter, target, name, task, k=5, model_type=None, version=None, features=None, reference=None, implementation=None, debug=False, patch_in_ds=False)[source]#

Regression benchmarking class.

Initialize the benchmarking class.

Parameters:
  • model – Model to be benchmarked.

  • ds (AbstractStructureDataset) – Dataset to be used for benchmarking.

  • splitter (BaseSplitter) – Splitter to be used for benchmarking.

  • target (List[str]) – Target labels to be used for benchmarking. Must be included in the dataset.

  • name (str) – Name of the model. This will be used as filename.

  • task (BenchTaskEnum) – Task of the benchmark.

  • k (int) – Number of folds for k-fold cross-validation.

  • model_type (str, optional) – Model type, e.g. ‘CGCNN’, ‘BERT’, ‘XGBoost’. Defaults to None.

  • version (str, optional) – Version of the model. Defaults to None.

  • features (str, optional) – Features used in the model. If you use a graph net, report the edge and vertex features. Defaults to None.

  • reference (str, optional) – Reference with more details Please provide it in BibTeX form. Defaults to None.

  • implementation (str, optional) – Link to implementation. Defaults to None.

  • debug (bool) – If True, the benchmark will be run in debug mode (1% of the data).

  • patch_in_ds (bool) – If True, the dataset will be patched into the model class under the ds attribute.

class BenchResult(**data)[source]#

Model for benchmark results.

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

save_json(folder)[source]#

Save benchmark results to json file.

Return type:

None

save_rst(folder)[source]#

Prepare RST file for model card and free-text model description.

Return type:

None

The results are reported in pydantic model with the following fields:

BenchResult#

Model for benchmark results.

type

object

properties

  • start_time

Start time

Start time of the benchmark

type

string

format

date-time

  • end_time

End time

End time of the benchmark

type

string

format

date-time

  • metrics

Metrics

Metrics of the benchmark

allOf

#/definitions/RegressionMetricCollection

  • version

Version

Version of the benchmark

type

string

  • features

Features

Features used in the model. If you use a graph net, report the edge and vertex features.

type

string

  • name

Name

Name of the model. This will be used as filename.

type

string

  • task

Task

Task of the benchmark

allOf

#/definitions/BenchTaskEnum

  • model_type

Model type

Model type, e.g. ‘CGCNN’, ‘BERT’, ‘XGBoost’

type

string

  • reference

Reference

Reference with more details

type

string

  • implementation

Implementation

Link to implementation

type

string

  • mofdscribe_version

mofdscribe version

mofdscribe version

type

string

  • session_info

Session info

Automatically captured string describing the computational environment.

type

object

  • logs

Logs

Additional data logged using the log method during the benchmark.

type

array

items

type

object

definitions

  • RegressionMetrics

RegressionMetrics

Model for regression metrics.

We will use this model to validate the benchmark results.

type

object

properties

  • mean_squared_error

Mean Squared Error

type

number

  • mean_absolute_error

Mean Absolute Error

type

number

  • r2_score

R2 Score

type

number

  • max_error

Max Error

type

number

  • mean_absolute_percentage_error

Mean Absolute Percentage Error

type

number

  • top_5_in_top_5

Top 5 In Top 5

type

integer

  • top_10_in_top_10

Top 10 In Top 10

type

integer

  • top_50_in_top_50

Top 50 In Top 50

type

integer

  • top_100_in_top_100

Top 100 In Top 100

type

integer

  • top_500_in_top_500

Top 500 In Top 500

type

integer

  • RegressionMetricCollection

RegressionMetricCollection

Model for regression metric collection.

type

object

properties

  • regression_metrics

Regression Metrics

type

array

items

#/definitions/RegressionMetrics

  • fit_timings

Fit Timings

type

array

items

type

number

  • inference_timings

Inference Timings

type

array

items

type

number

  • BenchTaskEnum

BenchTaskEnum

Enum for benchmark tasks.

enum

logKH_CO2_id, logKH_CO2_ood, pbe_bandgap_id, ch4dc_id

logarithmic CO2 Henry coefficient#

In-dataset predictions for the logarithmitic CO2 Henry coefficients

class LogkHCO2IDBench(model, name, version='v0.0.1', features=None, model_type=None, reference=None, implementation=None, debug=False, patch_in_ds=False)[source]#

Benchmarking models for the logarithmic CO2 Henry coefficient under in-domain conditions.

In-distribution implies that we use a cluster stratified splitter that ensures that the ratios of different clusters in the training and test set are the same.

Initialize the log KH CO2 interpolation benchmark.

Parameters:
  • model (object) – The model to be benchmarked. Must implement the fit and predict methods.

  • name (str) – The name of the modeling approach.

  • version (str, optional) – Version of the dataset to use. Defaults to “v0.0.1”.

  • features (str, optional) – Description of the features used in the model. Defaults to None.

  • model_type (str, optional) – Model type (e.g. Conv-Net, BERT, XGBoost). Defaults to None.

  • reference (str, optional) – Reference with more details about modeling approach. Defaults to None.

  • implementation (str, optional) – Link to implementation. Defaults to None.

  • debug (bool) – If True, use a small dataset (1% of full dataset) for debugging. Defaults to False.

  • patch_in_ds (bool) – If True, the dataset will be patched into the model class under the ds attribute.

class LogkHCO2OODBench(model, version='v0.0.1', features=None, name=None, model_type=None, reference=None, implementation=None, debug=False, patch_in_ds=False)[source]#

Benchmarking models for the logarithmic CO2 Henry coefficient under “out-of-domain” conditions.

“Out-of-domain” conditions means that every of the 5 training fold will only see 4 out of the 5 quantile bins. This implies that 2 runs are extrapolative and the other 3 need to “fill holes in the distribution”.

Initialize the log KH CO2 extrapolation benchmark.

Parameters:
  • model (object) – The model to be benchmarked. Must implement the fit and predict methods.

  • name (str) – The name of the modeling approach.

  • version (str, optional) – Version of the dataset to use. Defaults to “v0.0.1”.

  • features (str, optional) – Description of the features used in the model. Defaults to None.

  • model_type (str, optional) – Model type (e.g. Conv-Net, BERT, XGBoost). Defaults to None.

  • reference (str, optional) – Reference with more details about modeling approach. Defaults to None.

  • implementation (str, optional) – Link to implementation. Defaults to None.

  • debug (bool) – If True, use a small dataset (1% of full dataset) for debugging. Defaults to False.

  • patch_in_ds (bool) – If True, the dataset will be patched into the model class under the ds attribute.

PBE bandgap#

In-dataset predictions for PBE bandgaps.

class PBEBandGapIDBench(model, name, version='v0.0.1', features=None, model_type=None, reference=None, implementation=None, debug=False, patch_in_ds=False)[source]#

Benchmarking models for the PBE bandgap under in-distribution conditions.

In-distribution implies that we use a cluster stratified splitter that ensures that the ratios of different clusters in the training and test set are the same.

Initialize the PBE bandgap benchmark.

Parameters:
  • model (object) – The model to be benchmarked. Must implement the fit and predict methods.

  • name (str) – The name of the modeling approach.

  • version (str, optional) – Version of the dataset to use. Defaults to “v0.0.1”.

  • features (str, optional) – Description of the features used in the model. Defaults to None.

  • model_type (str, optional) – Model type (e.g. Conv-Net, BERT, XGBoost). Defaults to None.

  • reference (str, optional) – Reference with more details about modeling approach. Defaults to None.

  • implementation (str, optional) – Link to implementation. Defaults to None.

  • debug (bool) – If True, use a small dataset (1% of full dataset) for debugging. Defaults to False.

  • patch_in_ds (bool) – If True, the dataset will be patched into the model class under the ds attribute.

Methane deliverable capacity#

In-dataset predictions for the methane deliverable capacity

class CH4DCIDBench(model, name, version='v0.0.1', features=None, model_type=None, reference=None, implementation=None, debug=False, patch_in_ds=False)[source]#

Benchmarking models for the methane deliverable capacity under in-domain conditions.

In-distribution implies that we use a cluster stratified splitter that ensures that the ratios of different clusters in the training and test set are the same.

Initialize the CH4DC interpolation benchmark.

Parameters:
  • model (object) – The model to be benchmarked. Must implement the fit and predict methods.

  • name (str) – The name of the modeling approach.

  • version (str, optional) – Version of the dataset to use. Defaults to “v0.0.1”.

  • features (str, optional) – Description of the features used in the model. Defaults to None.

  • model_type (str, optional) – Model type (e.g. Conv-Net, BERT, XGBoost). Defaults to None.

  • reference (str, optional) – Reference with more details about modeling approach. Defaults to None.

  • implementation (str, optional) – Link to implementation. Defaults to None.

  • debug (bool) – If True, use a small dataset (1% of full dataset) for debugging. Defaults to False.

  • patch_in_ds (bool) – If True, the dataset will be patched into the model class under the ds attribute.

Dummy Models#

Simple baseline models.

class DensityRegressor(lr_kwargs=None)[source]#

Dummy model.

Initialize the model.

Parameters:

lr_kwargs (Optional[Dict], optional) – Keyword arguments that are passed to the linear regressor. Defaults to None.

featurize(s)[source]#

You might want to use a lookup in some dataframe instead.

Or use some mofdscribe featurizers.

Parameters:

s (Structure) – Structure to featurize.

Returns:

Density of the structure.

Return type:

float