Past Meetings

Meeting archive for the Python postprocessing development group.

Each entry includes date, participants, and a summary of decisions.

2026 Mar 30

Participants:

Illa R. Losada, Kishore G., Ramkishor Sharma, Vinay Kumar

Minutes:

1. Agenda Items:

   • Server to host python auto-tests.

   • Placement of tutorials – Read the Docs vs samples.

   • Array order convention

   • Obsolete documentation items in Decisions and Outcomes

2. Python Auto-Tests:

   • Kishore has developed the python-tests.

   • We need to set up a server to run these tests automatically on new commits.

   • Possible options include machines at Nordita. Kishore will enquire with Axel and report back at the next meeting.

3. Python Package:

   • We are moving towards an official Python package installation. This would allow users to install the package via pip.

   • Alberto is leading this effort.

   • A list of all places with hardcoded paths which would be affected by this needs to be made. Kishore and Illa will do this for python-tests and readthedocs, respectively.

4. Tutorial Placement:

   • We need to decide where to place the tutorials (dependinf on the type of tutorial).

   • Readthedocs will have all general tutorials. Sample specicfic tutorials will be placed in the samples directory.

   • We will have a contents/index page on readthedocs linking to the sample specific tutorials.

   • Jupyter notebook tutorials can be hosted in samples but also linked and rendered on readthedocs.

5. Docstrings:

   • We can add example usage of functions in the docstrings. This would be helpful for users and also for testing.

6. Array Order Convention:

   • Current array order is [z, y, x]. Consider changing to [x, y, z] (standard convention), but this may break existing scripts.

   • Not discussed further since Alberto and Simon were not around. To be continued at the next meeting.

7. Obsolete Items:

   • In python-decisions, should obsolete items be marked as obsolete or removed entirely?

   • We decided to mark them as obsolete for now, but move them to the end of the list in a separate section. This way they are still visible.

8. Export pencil-code data to other formats:

   • Vinay is developing a workflow to export data from var files into a format compatible with other codes (currently Athena). This will enable users to run part of their simulations in Pencil Code and then transfer the data for further processing or for code comparison.

   • The Python routine is being modularized into two components: (i) construction of conserved variables, and (ii) writing data in the required output format with correct processor ordering.

   • This functionality will be incorporated into the Pencil Code Python toolkit and made available to users.

9. Next Meeting:

   • Scheduled for end of May/ early June. Vinay will host and set up the Zoom link.

2026 Jan 29

Participants:

Alberto Roper Pol, Illa R. Losada, Simo Candelaresi, Kishore, Reihaneh

Minutes:

1. Agenda Items:

   • Development of Python tests for Pencil Code and exploration of code coverage tools.

   • Presentation of tutorials for specific samples, with each participant developing content for their preferred Pencil Code sample or exercise.

   • Updates on documentation (tutorials and Read the Docs improvements).

   • Inconsistencies in array formatting across different I/O options.

   • Review of pencil-code/python/pencil/TO_DO_LIST.txt (last updated June 29, 2024; check for newer updates).

   • Modification of .keys() methods (for Averages, Timeseries, etc.) to return a list instead of just printing keys.

   • Establishment of a convention for debug output control (via the quiet keyword in functions like pc.read.var and pc.read.grid). Currently, the default value varies between functions, which is confusing. Ideally, implement a module-wide flag (e.g., pc.shut_up = True) to control output globally.

   • Modification of pc.io.get_value_from_file behavior. Currently returns a boolean success indicator (like C programs); should explicitly raise error messages instead, as users typically want to know about failures rather than dealing with cryptic errors later.

   • Rename pc.sim.__Simulation__ to pc.sim.Simulation. Leading underscores typically indicate private/internal classes; if this class is intended for public use, the underscores should be removed.

   • Supported Python versions: Should we follow upstream Python support (<https://devguide.python.org/versions/>) or support versions older than 5 years?

   • The item Interactive vs. library mode in pencil-code/doc/readthedocs/python-meetings/python-decisions.rst is now obsolete due to lazy loading implementation.

   • Folders in python/ (like meetings and docs) are detected as Python modules when $PENCIL_HOME/python is added to $PYTHONPATH. Options to avoid this:

     • Move Python modules to python/src and add this to $PYTHONPATH

     • Other solutions to be explored

   • Issue with reading slices (pointed out by Axel)

   • Make Pencil Code Python routines a Python package

   • Integrate live execution of Python routines with Pencil Code (incorporating Isak’s notebook with simulation objects)

   • Default output format for var files (x, y, z coordinates)

2. Tutorial Placement:

   Current locations:

   • Read the Docs in RST format

   • Jupyter notebook tutorials in pencil-code/python/tutorials

   Action items:

   • Configure Sphinx to generate Read the Docs pages from Jupyter notebooks

   • Check for duplication between samples and Python tutorials

   • Link Pencil Code Python tutorials from the main Pencil Code homepage

3. Sample Tutorials:

   A new subsection The Pencil Code: samples has been created under Postprocessing and tutorials to host Pencil Code samples. These will include not only Python postprocessing scripts, but also scientific background and configuration explanations showing how simulations reflect the target physics.

   Example: Convection simulation: conv-slab

4. IDL vs. Python Usage:

   A new comparison document under Postprocessing and tutorials (IDL to Python Translation Guide for Pencil Code) provides guidance on postprocessing in both IDL and Python to help users choose the appropriate tool.

5. Discrepancy Between Reading Routines:

   Return values differ between direct reading functions and simulation object methods.

   Example:

   aver = pc.read.aver(simdir=sdir, datadir=os.path.join(sdir,'data/'),param=param,quiet=False)
   

   This works only when parameters are loaded via pc.read.param. Using param=sim.param produces:

   AttributeError: param.io_strategy does not exist
   

   Assigned to Simon.

6. TO_DO_LIST.txt Review:

   Simon will review python/pencil/TO_DO_LIST.txt and bring up relevant points at the next meeting.

7. Python Package Development:

   Moving toward an official Python package installation. This would allow:

   pip install pypencil
   

   Alberto is leading this effort.

8. Contributor List:

   Create a list of contributors for Pencil Code Python routines.

9. Mailing List Integration:

   Merge Python mailing list into main Pencil Code mailing list. Ask users to join the main list and discontinue the Python-specific one. This increases visibility and accessibility for new users. [Alberto]

10. README Consolidation:

    Currently there are two README files: python/README and python/pencil/README. Merge them into a single file in python/pencil.

11. Redundant keys() Functions:

    Some read functions have redundant keys() methods. Replace with direct dictionary key returns. [Kishore]

12. Debug Output Defaults:

    Reading routines currently have inconsistent quiet flag defaults (some True, some False). Standardize to quiet=False for consistency. [Kishore]

13. Error Handling Consistency:

    Some functions raise errors while others return 0 on failure. Standardize to always raise informative errors. [Kishore]

14. Next Meeting:

    Scheduled for end of March. Kishore will host and Alberto will set up the Zoom link.

15. Rename ``__Simulation__`` object.

    Rename __Simulation__ to Simulation to make it publicly accessible. [Kishore]

16. Python Version Support:

    Support Python versions that are 5 years old or newer. Update tests accordingly.

17. Redundant meetings directory:

    We have moved the contents of python/meetings to RTD. We have decided to remove this redundant folder now and keep the meetings in RTD.

18. Array Order Convention:

    Current array order is [z, y, x]. Consider changing to [x, y, z] (standard convention), but this may break existing scripts.

19. eqtools

    Move from eqtools to scipy. [Simon]

20. Obsolete Documentation Item:

    The Interactive vs. library mode item in pencil-code/doc/readthedocs/python-meetings/python-decisions.rst is obsolete due to lazy loading implementation. Currently, it has been marked as obsolete in Decisions and Outcomes. Should it be better to remove entirely? [Illa]

2025 Nov 26

Participants: Alberto Roper Pol, Carolina Ortiz, Oindrila Ghosh, Kishore, Reihaneh, Illa Losada, Simon Candelaresi, Alireza, Mgurgeni, Nordita Participants, Isak Stomberg.

Minutes:

1. Meeting Format.

1. Frequency of the Pencil Code Python Meetings should be every two months and should be kept up.

2. Rotate organizers for the meeting. Next meeting organized by Alberto.

3. Topics to be covered during the meetings: Tests, Tutorials, libraries with pip, formatting.

2. Tests.

We currently have a series of Python tests that can be run locally. - Task: Further development would include code coverage and format checking done automatically on a testing machine, maybe github: Kishore and Simon.

3. Pip: issues with library installation in Ubuntu.

• Task: Add to the documentation how to create your own Python environment. (Simon)

• Task: Upload PC Python as separate pip package. (Alberto)

4. Documentation.

• Task: Update the tutorials. (Simon, Isak)

• Task: Move the tutorials into Read The Docs. (Illa)

5. Real life simulation purely in Jupyter with life visualization.

• Task: Add it to read the docs. (Isak)

• Task: Can we use simulation objects? (Simon)

6. Tutorials.

We currently have information dispersed. - Task: Coding style should be put last. (Illa) - Task: Link tutorials in the Pencil Code from Read The Docs. (Simon)

7. Inconsistency in array format when reading with io_dist or hdf5. - Task: Make sure that the h5 file load follows same. (Kishore)

2021 Sept 17

Participants:

Minutes:

1. Rules for adding new routines and extending existing routines.

In the long term we should have test functions. (WD, SC) Build on the existing test in pc/python/pencil/tests Send regular emails to PC Python group to stick to coding rules. (IRL, SC)

2. Unified format for doc string.

Have all sections for every function: General definition, Signature, Parameters, Returns, Return Type, Examples, (Notes) Add section to the style guide about the documentation. (IRL) Send email about unified doc string. (IRL, SC)

3. Usage of Black.

Add to the style guide that everyone should run Black when making changes and additions. (IRL) Send an email to the PC Python discussion group. (WD)

4. pv plots: should we extend?

Email authors of the visu/pv_* routines. (SC)

2021 Apr 14

Participants: Illa, Fred, Simon, Wolfgang,

Minutes:

• Wolfgang: updated us on the following topics:

1. import cycle:

• he develop a script to check this isue ‘python/utils/cyclic-imports’

• he did a lot of clean-up

• extracted ‘is_sim_dir()’ to a new package ‘pencil/util’ that must not import from other packages.

• removed ‘lapack/pidly/pidly.py’

2. started to write tests

3. tried to fix ‘pencil/calc/example_shocktube.py’

4. reformatted ‘pencil/calc/streamlines’, ‘pencil/calc/tensors’ and ‘pencil/read/power’ with black

We discussed the usage of black to format the python files, and agreed on using it as a default. black can be used by default in several text editors.

5. Imports

We discussed the replacement of ‘from ..sim import simulations’ with ‘from pencil.sim import simulations’ the only problem this might have is any change of jerarchy on the different modules.

6. Backpack:

the current definition of this library is “Check for standard library” we discussed to rename it to ‘third_party’

7. Problems of ‘import pencil as pc’

Currently it takes ~4 seg to import the pencil package, and this might be a problem if one just want to run a test. We discuss the idea of having a separate package structrue for an interactive use.

8. Description of submodules:

We expend quite some time discussing if we should move the description of submodules into respective __init__.py (from pencil/__init__.py)

• use of black

• rename backpack to third party?

• Replace ‘from ..sim import simulations’ with ‘from pencil.sim import simulations’

• Distinguish between interactive and library usage ?

• Illa

9. Mailing list:

we encourage everyone interested to join out mailing list

10. Newsletter:

We discussed the contents of the newsletter and the information we want to write

11. Files Documentation:

We discussed a general idea for the formatting of a python file documentation. I suggested to follow the outline of ‘python/pencil/read/power.py’

• Simon

12. He remove the file npfile.py, as discussed in previous meetings.

13. He changed read.var

2021 Feb 24

Participants:

Minutes:

Topics: * Python <-> IDL descriptions

We added this to the wiki: https://github.com/pencil-code/pencil-code/wiki/Pencil-IDL——Python-guide Please contribute!

• Documentation: sphinx creation of the documentation

We discussed where to post the documentation and agreed on using readthedocs.org integrated with github.

• npfile.py is used in animate_slices and appears to be deprecated in scipy, Simon will check this issue.

• get_sim: Wolfgang is trying to get rid of it

2021 Jan 14

Participants:

Minutes:

Topics:

• Replace package names to non-conflicting ones?

• Documentation: use sphinx

• Imports: import in the functions or in the header

• Cyclic dependence structure in different imports.

• Newsletter

2020 Nov 23

Participants: Fred, Simon, Wladimir, Wolfgang, Alberto, Illa

Minutes:

We agree on going over the issues on the files: -pencil-code/python/meetings/Nov20_topics -pencil-code/python/pencil/TO_DO_LIST.txt

And discuss the issues:

• Where to import modules:

Right now the modules are imported inside the functions, but according to the PEP8 guidelines they should be imported at the beginning of the file.

Simon proposed to leave it as it is, since it was agreed upon on a Pencil Code meeting 2 years ago, and is cleaner. Plus it is less messy when using autocomplition in ipython.

Wolfang and Illa proposed to move to the PEP8 convention and find a solution for ipython.

• Reload:

Functions won’t reload in an ipython session. Fred pointed out that the file names are the same as functions names, making it imposible to reload, and suggested to change file names. We proposed different ways of reloading, using importlib, but it didn’s seem to work.

• Standarize formatting:

Wolfgang suggested to use “black” for formatting the code. Also as a way to standarize the formatting. We agreed to give it a try, and he would give us more info on the next meeting.

• Automatic documentation:

Illa suggested to use either Epydoc or sphinx to create automatic documentation of the python code. We agreed on the idea, and Illa will research the different options, keeping in mind doing the minimal work on the already existing documentation inside the functions.

• Tutorials:

We discuss the use of Jupyter versus plain scripts to add into the tutorials, and agreed on having both version. The Jupyter notebooks can be translated into scripts using pandoc or jupyter nboconvert –to script

Alberto will try to add new tutorials. We also agree on changing the dir names inside the tutorials to make them more relevant to the content. Creating a “Getting started” dir and examples of different topics. Simon will try to change the look of the tutorials dir.

• Functions:

We will try to add the relevant IDL functionality. We agree on adding the missing IDL functions to the TO_DO_LIST and keep a check there.