1. Preamble

See the Useful links page for official resources, including the website and the GitHub repository.

An introduction to the code and its capabilities is available in the talk: Pencil Code Introduction (2007).

1.1. Contributors list and GitHub stats

Fig. 1.1 Check-in patterns as a function of time for different subroutines. The different users are marked by different symbols and different colors.

Fig. 1.2 GitHub check-in pattern since 2002

1.2. Contributors to the code:

The up-to-date list of Pencil Code contributors can be found at GitHub.

(in inverse alphabetical order according to their user name)

Table 1.1 Contributors to the Pencil Code

Username	Name	Affiliation
xiang-yu	Xiang-Yu Li	Pacific Northwest National Laboratory
wladimir.lyra	Wladimir Lyra	New Mexico State Univ.
weezy	S. Louise Wilkin	University of Newcastle
wdobler	Wolfgang Dobler	Bruker, Potsdam
vpariev	Vladimir Pariev	University of Rochester
torkel	Ulf Torkelsson	Chalmers University
tavo-buk	Gustavo Guerrero	Univ. Minas Gerais
tgastine	Thomas Gastine	MPI for Solar System Research
tobson, theine	Tobias (Tobi) Heinemann	NBIA, Copenhagen
tarek	Tarek A. Yousef	University of Trondheim
sven.bingert	Sven Bingert	Gesellschaft für wissenschaftliche Datenverarbeitung
steveb	Steve Berukoff	UCLA
asnodin	Andrew Snodin	University of Newcastle
rplasson	Raphael Plasson	Avignon Université
qiancg	Chengeng Qian	Beijing Inst. of Technology
pkapyla	Petri Käpylä	University of Göttingen
onlymee	Antony (tOnY) Mee	Bank of America Merrill Lynch, London
nishkpph	Nishant K. Singh	IUCAA
nils.e.haugen	Nils Erland L. Haugen	SINTEF, Trondheim
NBabkovskaia	Natalia Babkovskaia	University of Helsinki
mrheinhardt	Matthias Rheinhardt	Aalto University, Espoo
mppiyali	Piyali Chatterjee	Bangalore
mkorpi	Maarit J. Korpi-Lagg (née Korpi, Mantere, Käpylä)	Aalto University
miikkavaisala	Miikka Väisälä	Academia Sinica, Inst. Astron. & Astro.
michiellambrechts	Michiel Lambrechts	Lund Observatory
Mewek	Ewa Karchniwy	Silesian University of Technology
mcmillan	David McMillan	York University, Toronto
mattias	Mattias Christensson	formerly at Nordita
luizfelippesr	Luiz Felippe S. Rodrigues	Radboud University
koenkemel	Koen Kemel	Nordita, Stockholm
karlsson	Torgny Karlsson	Nordita
jwarne	Jörn Warnecke	MPS, Göttingen
jpekkila	Johannes Pekkilä	Aalto University
jnskrueger	Jonas Krueger	Trondheim
jaarnes	Jørgen Aarnes	Trondheim
joishi	Jeff S. Oishi	Bates College
JenSchober	Jennifer Schober	EPFL, Lausanne
Illarl	Illa R. Losada	.
Iomsn1	Simon Candelaresi	University of Glasgow
grsarson	Graeme R. Sarson	University of Newcastle
fredgent	Frederick Gent	Aalto University, Espoo
fadiesis	Fabio Del Sordo	Nordita, Stockholm
dorch	Bertil Dorch	University of Copenhagen
bdintrans	Boris Dintrans	Observatoire Midi-Pyrénées, Toulouse
dhrubaditya	Dhrubaditya Mitra	Nordita, Stockholm
colinmcnally	Colin McNally	NBIA, Copenhagen
ChristerSandin	Christer Sandin	Nordita
chaochinyang	Chao-Chin Yang	The University of Alabama
Bourdin.KIS	Philippe Bourdin	Space Research Institute, Graz
AxelBrandenburg	Axel Brandenburg	Nordita, Stockholm
apichat	Apichat Neamvonk	University of Newcastle
amjed	Amjed Mohammed	University of Oldenburg
alihyder727	Ali Hyder	New Mexico State Univ.
alexanderhubbard	Alex Hubbard	American Museum of Natural History
andreas-schreiber	Andreas Schreiber	MPI Heidelberg
ajohan	Anders Johansen	GLOBE Institute, Copenhagen University

1.3. Copyright and License

Copyright and License

Copyright © 2001–2023 Wolfgang Dobler & Axel Brandenburg

Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies.

Permission is granted to copy and distribute modified versions under the same conditions.

License agreement and giving credit

The content of all files under pserver:$USER@svn.nordita.org:/var/cvs/brandenb are under the GNU General Public License (GPL).

We, the Pencil Code community, ask that in publications and presentations the use of the code (or parts of it) be acknowledged with reference to [JOSS] Pencil Code Collaboration, J. Open Source Software, 6, 2807 (2021) The Pencil Code, a modular MPI code for partial differential equations and particles: multipurpose and multiuser-maintained. This automatically gives a reference to the web sites http://www.nordita.org/software/pencil-code/ and https://github.com/pencil-code/pencil-code. As a courtesy to the people involved in developing particularly important parts of the program (use svn annotate src/*.f90 or git blame -c src/filename.f90 to find out who did what!) we suggest to give appropriate reference to one or several of the following (or other appropriate) papers (listed here in temporal order):

• Dobler, W., Haugen, N. E. L., Yousef, T. A., & Brandenburg, A., Phys. Rev., 2003, E 68, 026304, 1-8, “Bottleneck effect in three-dimensional turbulence simulations”, |astroph|{0303324}

• Haugen, N. E. L., Brandenburg, A., & Dobler, W., Astrophys. J. Lett., 2003, 597, L141, L144, “Is nonhelical hydromagnetic turbulence peaked at small scales?”, (astroph/0303372)

• Brandenburg, A., K"apyl"a, P., & Mohammed, A., Phys. Fluids, 2004, 16, 1020, 1027, “Non-Fickian diffusion and tau-approximation from numerical turbulence”, astro-ph preprint {0306521}

• Johansen, A., Andersen, A. C., & Brandenburg, A., Astron. Astrophys., 2004, 417, 361, 371, “Simulations of dust-trapping vortices in protoplanetary discs”, astro-ph preprint {0310059}

• Haugen, N. E. L., Brandenburg, A., & Mee, A. J., Monthly Notices Roy. Astron. Soc., 2004, 353, 947, 952, “Mach number dependence of the onset of dynamo action”, |astroph|{0405453}

• Brandenburg, A., & Multam"aki, T., Journal (generic), 2004, Int. J. Astrobiol., 3, 209, 219, “How long can left and right handed life forms coexist?”, q-bio preprint {0407008}

• McMillan, D. G., & Sarson, G. R., Phys. Earth Planet. Int., 2005, 153, 124, 135, “Dynamo simulations in a spherical shell of ideal gas using a high-order Cartesian magnetohydrodynamics code”

• Heinemann, T., Dobler, W., Nordlund, Å., & Brandenburg, A., Astron. Astrophys., 2006, 448, 731, 737, “Radiative transfer in decomposed domains”, astro-ph preprint {0503510}

• Dobler, W., Stix, M., & Brandenburg, A., Astrophys. J., 2006, 638, 336, 347, “Convection and magnetic field generation in fully convective spheres”, |astroph|{0410645}

• Snodin, A. P., Brandenburg, A., Mee, A. J., & Shukurov, A., Monthly Notices Roy. Astron. Soc., 2006, 373, 643, 652, “Simulating field-aligned diffusion of a cosmic ray gas”, astro-ph preprint {0507176}

• Johansen, A., Klahr, H., & Henning, T., Astrophys. J., 2006, 636, 1121, 1134, “Dust sedimentation and self-sustained Kelvin-Helmholtz turbulence in protoplanetary disc mid-planes”, astro-ph preprint {0512272}

• de Val-Borro, M. and 22 coauthors (incl. Lyra, W.), Monthly Notices Roy. Astron. Soc., 2006, 370, 529, 558, “A comparative study of disc-planet interaction”, |astroph|{0605237}

• Johansen, A., Oishi, J. S., Mac Low, M. M., Klahr, H., Henning, T., & Youdin, A., Nature, 2007, 448, 1022, 1025, “Rapid planetesimal formation in turbulent circumstellar disks”, arXiv preprint {0708.3890}

• Lyra, W., Johansen, A., Klahr, H., & Piskunov, N., Astron. Astrophys., 2008, 479, 883, 901, “Global magnetohydrodynamical models of turbulence in protoplanetary disks I. A cylindrical potential on a Cartesian grid and transport of solids”, |arXiv|{0705.4090}

• Brandenburg, A., Rädler, K.-H., Rheinhardt, M., & K"apyl"a, P. J., Astrophys. J., 2008, 676, 740, 751, “Magnetic diffusivity tensor and dynamo effects in rotating and shearing turbulence”, arXiv preprint {0710.4059}

• Lyra, W., Johansen, A., Klahr, H., & Piskunov, N., Astron. Astrophys., 2008, 491, L41, L44, “Embryos grown in the dead zone. Assembling the first protoplanetary cores in low-mass selfgravitating circumstellar disks of gas and solids”

• Lyra, W., Johansen, A., Klahr, H., & Piskunov, N., Astron. Astrophys., 2009, 493, 1125, 1139, “Standing on the shoulders of giants. Trojan Earths and vortex trapping in low-mass selfgravitating protoplanetary disks of gas and solids”

• Lyra, W., Johansen, A., Zsom, A., Klahr, H., & Piskunov, N., Astron. Astrophys., 2009, 497, 869, 888, “Planet formation bursts at the borders of the dead zone in 2D numerical simulations of circumstellar disks”, arXiv preprint {0901.1638}

• Mitra, D., Tavakol, R., Brandenburg, A., & Moss, D., Astrophys. J., 2009, 697, 923, 933, “Turbulent dynamos in spherical shell segments of varying geometrical extent”, arXiv preprint {0812.3106}

• Haugen, N. E. L., & Kragset, S., J. Fluid Mech., 2010, 661, 239, 261, “Particle impaction on a cylinder in a crossflow as function of Stokes and Reynolds numbers”

• Rheinhardt, M., & Brandenburg, A., Astron. Astrophys., 2010, 520, A28, “Test-field method for mean-field coefficients with MHD background”, arXiv preprint {1004.0689}

• Babkovskaia, N., Haugen, N. E. L., Brandenburg, A., Journal (generic), 2011, J. Comp. Phys., 230, 1, 12, “A high-order public domain code for direct numerical simulations of turbulent combustion”, arXiv preprint {1005.5301}

• Johansen, A., Klahr, H., & Henning, Th., Astron. Astrophys., 2011, 529, A62, “High-resolution simulations of planetesimal formation in turbulent protoplanetary discs”

• Johansen, A., Youdin, A. N., & Lithwick, Y., Astron. Astrophys., 2012, 537, A125, “Adding particle collisions to the formation of asteroids and Kuiper belt objects via streaming instabilities”

• Lyra, W. & Kuchner, W., Nature, 2013, 499, 184, 187, “Formation of sharp eccentric rings in debris disks with gas but without planets”

• Yang, C.-C., & Johansen, A., Astrophys. J. Suppl. Series, 2016, 224, 39, “Integration of Particle-Gas Systems with Stiff Mutual Drag Interaction”

• Roper Pol, A., Brandenburg, A., Kahniashvili, T., Kosowsky, A., & Mandal, S., Geophys. Astrophys. Fluid Dyn., 2020, 114, 130, 161, “The timestep constraint in solving the gravitational wave equations sourced by hydromagnetic turbulence”

This list is not always up-to-date. We therefore ask the developers to check in new relevant papers, avoiding however redundancies.

We are aware of the fact that certain extensions to the code may still be under intense development and no paper can be quoted yet. Again, if your work directly profits from such code, as a courtesy to those developers, we suggest to contact them, if possible, and ask whether there is anything else that can be quoted instead.

It is also sometimes nice to see that the Pencil Code is being acknowledged for having inspired certain other developments, so for example in the GALPROP program [Porter22].

1.4. Foreword

This code was originally developed at the Turbulence Summer School of the Helmholtz Institute in Potsdam (2001). While some SPH and PPM codes for hydrodynamics and magnetohydrodynamics were publicly available, this did not seem to be generally the case for higher order finite-difference or spectral codes. This has changed since 2001; examples are the SpECTRE code, which is a discontinuous Galerkin code, and there are also the Snoopy and Dedalus codes, which are spectral. Having been approached by people interested in using our code, we decided to make it as flexible as possible and as user-friendly as seems reasonable, and to put it onto a public CVS repository. Since 21 September 2008 it is distributed via https://github.com/pencil-code/pencil-code. The code can certainly not be treated as a black box (no code can), and in order to solve a new problem in an optimal way, users will need to find their own optimal set of parameters. In particular, you need to be careful in choosing the right values of viscosity, magnetic diffusivity, and radiative conductivity.

The Pencil Code is primarily designed to deal with weakly compressible turbulent flows, which is why we use high-order first and second derivatives. To achieve good parallelization, we use explicit (as opposed to compact) finite differences. Typical scientific targets include driven MHD turbulence in a periodic box, convection in a slab with non-periodic upper and lower boundaries, a convective star embedded in a fully nonperiodic box, accretion disc turbulence in the shearing sheet approximation, etc. Furthermore, nonlocal radiation transport, inertial particles, dust coagulation, self-gravity, chemical reaction networks, and several other physical components are installed, but this number increases steadily. In addition to Cartesian coordinates, the code can also deal with spherical and cylindrical polar coordinates.

Magnetic fields are implemented in terms of the magnetic vector potential to ensure that the field remains solenoidal (divergence-free). At the same time, having the magnetic vector potential readily available is a big advantage if one wants to monitor the magnetic helicity, for example. The code is therefore particularly well suited for all kinds of dynamo problems.

The code is normally non-conservative; thus, conserved quantities should only be conserved up to the discretization error of the scheme (not to machine accuracy). There is no guarantee that a conservative code is more accurate with respect to quantities that are not explicitly conserved, such as entropy. Another important quantity that is (to our knowledge) not strictly conserved by ordinary flux conserving schemes is magnetic helicity.

There are currently no plans to implement adaptive mesh refinement into the code, which would cause major technical complications. Given that turbulence is generically space-filling, local refinement to smaller scales would often not be very useful anyway. On the other hand, in some geometries turbulence may well be confined to certain regions in space, so one could indeed gain by solving the outer regions with fewer points.

In order to be cache-efficient, we solve the equations along pencils in the ``x` direction. One very convenient side-effect is that auxiliary and derived variables use very little memory, as they are only ever defined on one pencil. The domain can be tiled in the y and z directions. On multiprocessor computers, the code can use MPI (Message Passing Interface) calls to communicate between processors. An easy switching mechanism allows the user to run the code on a machine without MPI libraries (e.g., a notebook computer). Ghost zones are used to implement boundary conditions on physical and processor boundaries.

A high level of flexibility is achieved by encapsulating individual physical processes and variables in individual modules, which can be switched on or off in the file Makefile.local in the local src/ directory. This approach avoids the use of difficult-to-read preprocessor directives, at the price of requiring one dummy module for each physics module. For nonmagnetic hydrodynamics, for example, one will use the module nomagnetic.f90 and specifies:

MAGNETIC = nomagnetic

in Makefile.local, while for MHD simulations, magnetic.f90 will be used:

MAGNETIC = magnetic

Note that the term module as used here is only loosely related to Fortran modules: both magnetic.f90 and nomagnetic.f90 define an F90 module named Magnetic — this is the basis of the switching mechanism we are using.

Input parameters (which are set in the files start.in, run.in files) can be changed without recompilation. Furthermore, one can change the list of variables for monitoring (diagnostic) output on the fly, and there are mechanisms for making the code reload new parameters or exit gracefully at runtime. You may want to check for correctness of these files with the command pc_configtest.

The requirements for using the Pencil-MPI code are modest: you can use it on any Linux or Unix system with an F95 and C compiler suite, like GNU gcc and gfortran, together with the shell CSH, and the Perl interpreter are mandatory requirements.

Although the Pencil Code is mainly designed to run on supercomputers, more than 50% of the users run their code also on Macs, and the other half uses either directly Linux on their laptops or they use VirtualBox on their Windows machine on which they install Ubuntu Linux. If you have IDL as well, you will be able to visualize the results (a number of sample procedures are provided), but other tools such as Python, DX (OpenDX, data explorer) can also be used and some relevant tools and routines come with the PENCIL CODE.

If you want to make creative use of the code, this manual will contain far too little information. Its major aim is to give you an idea of the way the code is organized, so you can more efficiently read the source code, which contains a reasonable amount of comments. You might want to read through the various sample directories that are checked in. Choose one that is closest to your application and start modifying. For further enhancements that you may want to add to the code, you can take as an example the lines in the code that deal with related variables, functions, diagnostics, equations etc., which have already been implemented. Just remember: grep is one of your best friends when you want to understand how certain variables or functions are used in the code.

We will be happy to include user-supplied changes and updates to the code in future releases and welcome any feedback.

Wolfgang Dobler <wdobler@gmail.com>

Axel Brandenburg <AxelBrandenburg@gmail.com>

1.5. Acknowledgments

Many people have contributed in different ways to the development of this code. We thank first of all Åke Nordlund (Copenhagen Observatory) and Bob Stein (University of Michigan) who introduced us to the idea of using high-order schemes in compressible flows and who taught us a lot about simulations in general.

The calculation of the power spectra, structure functions, the remeshing procedures, routines for changing the number of processors, as well as the shearing sheet approximation and the flux-limited diffusion approximation for radiative transfer were implemented by Nils Erland L. Haugen (University of Trondheim). Tobi Heinemann added the long characteristics method for radiative transfer as well as hydrogen ionization. He also added and/or improved shock diffusion for other variables and improved the resulting timestep control. Anders Johansen, Wladimir (Wlad) Lyra, and Jeff Oishi contributed to the implementation of the dust equations (which now comprises an array of different components). Antony (Tony) Mee (University of Newcastle) implemented shock viscosity and added the interstellar module together with Graeme R. Sarson (also University of Newcastle), who also implemented the geodynamo set-up together with David McMillan (currently also at the University of Newcastle). Tony also included a method for outputting auxiliary variables and enhanced the overall functionality of the code and related IDL and DX procedures. He also added, together with Andrew Snodin, the evolution equations for the cosmic ray energy density. Vladimir Pariev (University of Rochester) contributed to the development and testing of the potential field boundary condition at an early stage. The implementation of spherical and cylindrical coordinates is due to Dhrubaditya (Dhruba) Mitra and Wladimir Lyra. Wlad also implemented the global set-up for protoplanetary disks (as opposed to the local shearing sheet formalism). He also added a N-body code (based on the particle module coded by Anders Johansen and Tony), and implemented the coupled evolution equations of neutrals and ions for two-fluid models of ambipolar diffusion. Boris Dintrans is in charge of implementing the anelastic and Boussinesq modules. Philippe-A. Bourdin implemented HDF5 support and wrote the optional IO-modules for high-performance computing featuring various communication strategies. He also contributed to the solar-corona module and worked on the IDL GUI, including the IDL routines for reading and working with large amounts of data. Again, this list contains other recent items that are not yet fully documented and acknowledged.

Use of the PPARC supported supercomputers in St Andrews (Mhd) and Leicester (Ukaff) is acknowledged. We also acknowledge the Danish Center for Scientific Computing for granting time on Horseshoe, which is a 512+140 processor Beowulf cluster in Odense (Horseshoe).

[JOSS]

Pencil Code Collaboration, J. Open Source Software, 6, 2807 (2021) The Pencil Code, a modular MPI code for partial differential equations and particles: multipurpose and multiuser-maintained.

[Porter22]

Porter, T. A., Jóhannesson, G., & Moskalenko, I. V. (2022). Astrophys. J. Supp., 262, 30. “The GALPROP Cosmic-ray Propagation and Nonthermal Emissions Framework: Release v57.”