The PyCMOR Paper #179

pgierz · 2025-07-09T10:50:07Z

I think it would be a good idea for us to write a paper for PyMOR.

Target Journal is JOSS: https://joss.theoj.org

pgierz · 2025-08-06T11:10:31Z

Needs #185 and #191 before proceeding

mandresm · 2025-08-06T11:23:44Z

TODOs for 13.08.2025

@mandresm Review paper and give feedbackwith PR suggestions
@siligam Review paper and give feedback with PR suggestions

mandresm

Hi @pgierz, thanks for this great draft.

See my suggestions and comments.

I think it will also be worth explaining how is the CMIP6 standard implemented and what is the strategy for future implementations of other standards, for example CMIP7. One idea would be that we can build pycmor with options in pip (`pip install pycmor[<a_external_mip>] where this external package simply contains all the classes needed to translate those standards to the language of our tool. We could keep the standard interfaces of the higher-interest standards integrated directly into our tool, such as CMIPs, PMIPs, ... and integrate community standard interfaces if sufficiently tested and robust. We could also say, all standard interfaces are not part of the repo of pymor (separation of concerns). I don't know about this point. But a mention to how the communities will include new standard interfaces is needed, in my view.

mandresm · 2025-09-08T08:58:26Z

paper/paper.md

+    equal-contrib: true
+    affiliation: 1
+  - name: Miguel Andres-Martinez
+    orcid: ????


Suggested change

orcid: ????

orcid: 0000-0002-1525-5546

mandresm · 2025-09-08T10:30:57Z

paper/paper.md

+PyMOR is a toolbox for preparing Earth System Model (ESM) data for analysis and
+sharing with the community. PyMOR uses a simple command line interface and a


In my view it is not only about "preparing" data. It is also to standardize it. Can you add something about standardization? Like:

Suggested change

PyMOR is a toolbox for preparing Earth System Model (ESM) data for analysis and

sharing with the community. PyMOR uses a simple command line interface and a

PyCMOR is a toolbox for the preparation and standardization of Earth System Model (ESM) data, facilitating subsequent analysis and ensuring that the data can be readily shared with the community. PyCMOR uses a simple command line interface and a

We also need to explain the CMOR acronym and relate it somewhere to the name of the software

mandresm · 2025-09-08T10:32:43Z

paper/paper.md

+sharing with the community. PyMOR uses a simple command line interface and a
+clear way to manipulate NetCDF files step by step to add relevant metadata,
+transform units, combine variables, regrid, transform geometries, and more. It
+runs in parallel using Dask and SLURM, and thus is suitable to handle even


Maybe not here, but somewhere we have to state that it should be pretty easy to use for different batch systems as long as Dask has a Cluster Class defined for them.

mandresm · 2025-09-08T10:48:54Z

paper/paper.md

+Standardization of climate model outputs is crucial for preparing data for IPCC
+reports because it ensures that results from different modeling centers worldwide
+can be directly compared, combined, and analyzed in a consistent and transparent
+manner. The IPCC and associated projects like CMIP require model outputs to be
+formatted according to strict technical specifications: data must be provided
+in standardized NetCDF files, using common variable names, units, metadata
+conventions (such as the CF Metadata Conventions), and grid structures (e.g.,
+rectilinear grids for most fields, standard pressure or depth levels for
+vertical coordinates). This uniformity allows the Program for Climate Model
+Diagnosis and Intercomparison (PCMDI) to centrally archive and distribute model
+results, enabling hundreds of researchers to efficiently scrutinize, benchmark,
+and synthesize findings across models and scenarios. Without such
+standardization, the process of aggregating results for global assessments
+would be error-prone, time-consuming, and potentially unreliable, undermining
+the scientific basis for the IPCC’s policy-relevant conclusions.


Trying to improve the motivation, adding more than just CMIP and IPCC report.

Suggested change

Standardization of climate model outputs is crucial for preparing data for IPCC

reports because it ensures that results from different modeling centers worldwide

can be directly compared, combined, and analyzed in a consistent and transparent

manner. The IPCC and associated projects like CMIP require model outputs to be

formatted according to strict technical specifications: data must be provided

in standardized NetCDF files, using common variable names, units, metadata

conventions (such as the CF Metadata Conventions), and grid structures (e.g.,

rectilinear grids for most fields, standard pressure or depth levels for

vertical coordinates). This uniformity allows the Program for Climate Model

Diagnosis and Intercomparison (PCMDI) to centrally archive and distribute model

results, enabling hundreds of researchers to efficiently scrutinize, benchmark,

and synthesize findings across models and scenarios. Without such

standardization, the process of aggregating results for global assessments

would be error-prone, time-consuming, and potentially unreliable, undermining

the scientific basis for the IPCC’s policy-relevant conclusions.

Model Intercomparison projects ([MIPs](https://wcrp-cmip.org/mips/)) bring

together the international Earth system science community to address key

scientific questions by comparing results across different models and datasets.

To do that MIPs offer standardization protocols so that results from different

modeling centers worldwide can be directly compared, combined, and analyzed

in a consistent and transparent

manner. A MIP of particular importance is the Coupled Model Intercomparison Project (CMIP),

which is used in the IPCC as an estimate of future climates for different scenarios. MIPs require model outputs and observational data to be

formatted according to strict technical specifications: data must be provided

in standardized NetCDF files, using common variable names, units, metadata

conventions (such as the CF Metadata Conventions), and grid structures (e.g.,

rectilinear grids for most fields, standard pressure or depth levels for

vertical coordinates). This uniformity allows the Program for Climate Model

Diagnosis and Intercomparison (PCMDI) to centrally archive and distribute model

results, enabling hundreds of researchers to efficiently scrutinize, benchmark,

and synthesize findings across models and scenarios. Without such

standardization, the process of aggregating results for global assessments

would be error-prone, time-consuming, and potentially unreliable.

mandresm · 2025-09-08T10:50:33Z

paper/paper.md

+make it challenging to manipulate the files to conform to the requisite metadata
+standards and best-practices.
+
+We developed `pymor` to fill the need for a flexible, performant, extensible


Suggested change

We developed `pymor` to fill the need for a flexible, performant, extensible

We developed `pycmor` to fill the need for a flexible, performant, extensible

mandresm · 2025-09-08T10:57:45Z

paper/paper.md

+
+```python
+import xarray as xr
+from pymor.core.rule import Rule


Suggested change

from pymor.core.rule import Rule

from pycmor.core.rule import Rule

mandresm · 2025-09-08T10:58:28Z

paper/paper.md

+      - "pymor.core.gather_inputs.load_mfdataset"
+      - "script://./intpp_recom.py:add_pp_components"
+      - "pymor.fesom_1p4.nodes_to_levels"
+      - "script://./intpp_recom.py:vertical_integration"
+      - "script://./intpp_recom.py:set_pp_units"
+      - "pymor.std_lib.convert_units"
+      - "pymor.std_lib.time_average"
+      - "pymor.std_lib.set_global_attributes"
+      - "pymor.std_lib.trigger_compute"
+      - "pymor.std_lib.show_data"
+      - "pymor.std_lib.files.save_dataset"


Suggested change

- "pymor.core.gather_inputs.load_mfdataset"

- "script://./intpp_recom.py:add_pp_components"

- "pymor.fesom_1p4.nodes_to_levels"

- "script://./intpp_recom.py:vertical_integration"

- "script://./intpp_recom.py:set_pp_units"

- "pymor.std_lib.convert_units"

- "pymor.std_lib.time_average"

- "pymor.std_lib.set_global_attributes"

- "pymor.std_lib.trigger_compute"

- "pymor.std_lib.show_data"

- "pymor.std_lib.files.save_dataset"

- "pycmor.core.gather_inputs.load_mfdataset"

- "script://./intpp_recom.py:add_pp_components"

- "pycmor.fesom_1p4.nodes_to_levels"

- "script://./intpp_recom.py:vertical_integration"

- "script://./intpp_recom.py:set_pp_units"

- "pycmor.std_lib.convert_units"

- "pycmor.std_lib.time_average"

- "pycmor.std_lib.set_global_attributes"

- "pycmor.std_lib.trigger_compute"

- "pycmor.std_lib.show_data"

- "pycmor.std_lib.files.save_dataset"

mandresm · 2025-09-08T10:58:57Z

paper/paper.md

+the user configuration file:
+
+```yaml
+pymor:


Suggested change

pymor:

pycmor:

mandresm · 2025-09-08T10:59:13Z

paper/paper.md

+functionality once the user has defined the configuration file:
+
+```bash
+$ pymor process <path/to/config.yaml>


Suggested change

$ pymor process <path/to/config.yaml>

$ pycmor process <path/to/config.yaml>

mandresm · 2025-09-08T11:01:25Z

paper/paper.md

+Christian Stepanek for early design testing, as well as the CMIP team for
+fruitful discussions and feedback.


Suggested change

Christian Stepanek for early design testing, as well as the CMIP team for

fruitful discussions and feedback.

Christian Stepanek for early design testing, as well as the CMIP team and the WCRP ESMO Infrastructure Panel (WIP) fruitful discussions and feedback.

We also need to acknowledge here the DataHub which is the funding body for @siligam's position

pgierz and others added 9 commits July 9, 2025 12:48

wip: adds example paper

9ef1fdd

wip: readme for coauthors

c892869

Update readme.md

d2fc833

wip: adds example bib

8734d67

paper

5057581

ci: auto-build paper pdf

3fdd801

ci: auto-build paper pdf

62b712e

ci: auto-build paper pdf

7708317

included workflow diagram

da5a0e3

some links

21a919c

mandresm changed the title ~~The PyMOR Paper~~ The PyCMOR Paper Sep 8, 2025

mandresm requested changes Sep 8, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

The PyCMOR Paper #179

The PyCMOR Paper #179

Uh oh!

pgierz commented Jul 9, 2025 •

edited

Loading

Uh oh!

pgierz commented Aug 6, 2025

Uh oh!

mandresm commented Aug 6, 2025

Uh oh!

mandresm left a comment

Uh oh!

mandresm Sep 8, 2025

Uh oh!

mandresm Sep 8, 2025

Uh oh!

mandresm Sep 8, 2025

Uh oh!

mandresm Sep 8, 2025

Uh oh!

mandresm Sep 8, 2025

Uh oh!

mandresm Sep 8, 2025

Uh oh!

mandresm Sep 8, 2025

Uh oh!

mandresm Sep 8, 2025

Uh oh!

mandresm Sep 8, 2025

Uh oh!

mandresm Sep 8, 2025

Uh oh!

mandresm Sep 8, 2025

Uh oh!

Uh oh!

		PyMOR is a toolbox for preparing Earth System Model (ESM) data for analysis and
		sharing with the community. PyMOR uses a simple command line interface and a

	PyMOR is a toolbox for preparing Earth System Model (ESM) data for analysis and
	sharing with the community. PyMOR uses a simple command line interface and a
	PyCMOR is a toolbox for the preparation and standardization of Earth System Model (ESM) data, facilitating subsequent analysis and ensuring that the data can be readily shared with the community. PyCMOR uses a simple command line interface and a

	We developed `pymor` to fill the need for a flexible, performant, extensible
	We developed `pycmor` to fill the need for a flexible, performant, extensible

	from pymor.core.rule import Rule
	from pycmor.core.rule import Rule

	$ pymor process <path/to/config.yaml>
	$ pycmor process <path/to/config.yaml>

		Christian Stepanek for early design testing, as well as the CMIP team for
		fruitful discussions and feedback.

The PyCMOR Paper #179

Are you sure you want to change the base?

The PyCMOR Paper #179

Uh oh!

Conversation

pgierz commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pgierz commented Aug 6, 2025

Uh oh!

mandresm commented Aug 6, 2025

Uh oh!

mandresm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pgierz commented Jul 9, 2025 •

edited

Loading