pencil.io

Input und output functions, like saving data.

Submodules

Attributes

pc_print

Exceptions

GetValueError

Common base class for all non-exit exceptions.

Classes

npfile

Class for reading and writing numpy arrays to/from files

Functions

mkdir(destination[, lfs, MB, count])

debug_breakpoint()

Python debug breakpoint.

timestamp()

open_h5(filename, status[, driver, comm, overwrite, ...])

This function opens hdf5 file in serial or parallel.

group_h5(h5obj, groupname[, status, delete, ...])

This function adds/removes hdf5 group objects.

dataset_h5(h5obj, dataname[, status, data, shape, ...])

This function adds/removes hdf5 dataset objects.

write_snapshot(snapshot[, file_name, datadir, nprocx, ...])

Write a snapshot given as numpy array.

write_h5_snapshot(snapshot[, file_name, datadir, ...])

Write a snapshot given as numpy array.

write_h5_grid([file_name, datadir, precision, nghost, ...])

Write the grid information as hdf5.

write_h5_averages(aver[, file_name, datadir, nt, ...])

Write an hdf5 format averages dataset given as an Averages object.

write_h5_slices(vslice, coordinates, positions[, ...])

Write an hdf5 format slices dataset given as an Slices object.

get_systemid()

This function returnes informations over the used system.

exists_file(file[, folder])

Better version of exists, compared with os.path.exists!

remove_files(path[, do_it, do_it_really])

This method clears path COMPLETELY.

get_value_from_file(filename, quantity[, ...])

Use to read in a quantity from

change_value_in_file(filename, quantity, newValue[, ...])

Use to change a quantity in

rename_in_submit_script(new_name[, ...])

pkl_load(name[, folder, sim])

This scripts loads an pkl-file. It automatically checks known folders if no folder is specified.

pkl_save(obj, name[, folder, ascii])

This scripts saves any kind of object as a pkl file in folder.

pkl_exists(name[, folder])

This scripts checks if a certain pkl-file already exists.

walklevel(some_dir[, level])

Walklevel walks through higher level dirs to a certain depth.

Package Contents

pencil.io.pc_print
class pencil.io.npfile(file_name, permission='rb', endian='dtype', order='C')

Bases: object

Class for reading and writing numpy arrays to/from files

Inputs:
file_name – The complete path name to the file to open

or an open file-like object

permission – Open the file with given permissions: (‘r’, ‘w’, ‘a’)

for reading, writing, or appending. This is the same as the mode argument in the builtin open command.

format – The byte-ordering of the file:

([‘native’, ‘n’], [‘ieee-le’, ‘l’], [‘ieee-be’, ‘B’]) for native, little-endian, or big-endian respectively.

endian   -- default endian code for reading / writing
order    -- default order for reading writing
Type:

‘C’ or ‘F’

file     -- file object containing read / written data
seek, tell, close  -- as for file objects
rewind             -- set read position to beginning of file
read_raw           -- read string data from file (read method of file)
write_raw          -- write string data to file (write method of file)
read_array         -- read numpy array from binary file data
write_array        -- write numpy array contents to binary file

Example use: >>> from StringIO import StringIO >>> import numpy as np >>> from scipy.io import npfile >>> arr = np.arange(10).reshape(5,2) >>> # Make file-like object (could also be file name) >>> my_file = StringIO() >>> npf = npfile(my_file) >>> npf.write_array(arr) >>> npf.rewind() >>> npf.read_array((5,2), arr.dtype) >>> npf.close() >>> # Or read write in Fortran order, Big endian >>> # and read back in C, system endian >>> my_file = StringIO() >>> npf = npfile(my_file, order=’F’, endian=’>’) >>> npf.write_array(arr) >>> npf.rewind() >>> npf.read_array((5,2), arr.dtype)

endian = 'dtype'
order = 'C'
get_endian()
set_endian(endian_code)
parse_endian(endian_code)

Returns valid endian code from wider input options

close()
seek(*args)
tell()
rewind(howmany=None)

Rewind a file to its beginning or by a specified amount.

read_raw(size=-1)

Read raw bytes from file as string.

write_raw(str)

Write string to file as raw bytes.

remaining_bytes()
write_array(data, endian=None, order=None)

Write to open file object the flattened numpy array data

Inputs data - numpy array or object convertable to array endian - endianness of written data

(can be None, ‘dtype’, ‘<’, ‘>’) (if None, get from self.endian)

order - order of array to write (C, F)

(if None from self.order)

read_array(dt, shape=-1, endian=None, order=None)

Read data from file and return it in a numpy array.

Inputs

dt - dtype of array to be read shape - shape of output array, or number of elements

(-1 as number of elements or element in shape means unknown dimension as in reshape; size of array calculated from remaining bytes in file)

endian - endianness of data in file

(can be None, ‘dtype’, ‘<’, ‘>’) (if None, get from self.endian)

order - order of array in file (C, F)

(if None get from self.order)

Outputs arr - array from file with given dtype (dt)

fort_write(data, endian=None, order=None, head_size=4)

Write a Fortran binary record from a numpy array

Inputs:

fmt – If a string then it represents the same format string as

used by struct.pack. The remaining arguments are passed to struct.pack.

If fmt is an array, then this array will be written as a Fortran record using the output type args[0].

*args – Arguments representing data to write.

fort_read(dt, shape=-1, endian=None, order=None, head_size=4)

Read data from a fortran binary record and return it in a numpy array.

note that fortran records give a 4-byte (or 8-byte if you use gfortran) header describing the number of bytes in a record. if shape does not agree with this, an error is raised.

Inputs

dt - dtype of array to be read shape - shape of output array, or number of elements

(-1 as number of elements or element in shape means unknown dimension as in reshape; size of array calculated from remaining bytes in file)

endian - endianness of data in file

(can be None, ‘dtype’, ‘<’, ‘>’) (if None, get from self.endian)

order - order of array in file (C, F)

(if None get from self.order)

head_size - the size in bytes of the fortran record header.

(almost always 4, except gfortran uses 8 by default)

Outputs arr - array from file with given dtype (dt)

pencil.io.mkdir(destination, lfs=False, MB=1, count=1)
pencil.io.debug_breakpoint()

Python debug breakpoint.

pencil.io.timestamp()
pencil.io.open_h5(filename, status, driver=None, comm=None, overwrite=False, size=1, rank=0, lfs=False, MB=1, count=1)

This function opens hdf5 file in serial or parallel.

Keyword Arguments:

  • filename – relative or absolute path string for name of hdf5 file.

  • status – open state of file ‘w’: write, ‘r’: read or ‘a’/’r+’: append.

  • driver – ‘mpio’ required for parallel: version but absent for serial.

  • comm – only present for parallel version of h5py.

  • overwrite – flag to replace existing file.

  • rank – processor rank with root = 0.

pencil.io.group_h5(h5obj, groupname, status='r', delete=False, overwrite=False, rank=0, size=1, comm=None)

This function adds/removes hdf5 group objects.

Keyword Arguments:

  • h5obj – h5 object, may be the file or a sub group within the file

  • groupname – string for name of the group.

  • status – open state of file ‘w’: write, ‘r’: read or ‘a’/’r+’: append.

  • delete – flag to remove existing group from h5 object.

  • overwrite – flag to replace existing group from h5 object.

  • rank – processor rank with root = 0.

  • comm – only present for parallel version of h5py.

pencil.io.dataset_h5(h5obj, dataname, status='r', data=None, shape=None, dtype=None, overwrite=False, delete=False, rank=0, size=1, comm=None)

This function adds/removes hdf5 dataset objects.

Keyword Arguments:

  • h5obj – h5 object, may be the file or a sub group within the file

  • dataname – string for name of the dataset.

  • status – open state of file ‘w’: write, ‘r’: read or ‘a’/’r+’: append.

  • data – h5 compatible data object; float, integer, string, array

  • shape – data shape tuple of length > 0

  • dtype – h5 compatible data type, eg. np.float64

  • delete – flag to remove existing group from h5 object.

  • overwrite – flag to replace existing group from h5 object.

  • rank – processor rank with root = 0.

  • comm – only present for parallel version of h5py.

pencil.io.write_snapshot(snapshot, file_name='VAR0', datadir='data', nprocx=1, nprocy=1, nprocz=1, precision='d', nghost=3, persist=None, param=None, t=None, x=None, y=None, z=None, lshear=False, dx=None, dy=None, dz=None)

Write a snapshot given as numpy array.

call signature:

write_snapshot(snapshot, file_name=’VAR0’, datadir=’data’,

nprocx=1, nprocy=1, nprocz=1, precision=’d’, nghost=3, persist=None, param=None, t=None, x=None, y=None, z=None, lshear=False, dx=None, dy=None, dz=None)

Keyword arguments:

snapshot:

Numpy array containing the snapshot. Must be of shape [nvar, nz, ny, nx] (without boundaries).

file_name:

Name of the snapshot file to be written, e.g. VAR0 or var.dat.

datadir:

Directory where the data is stored.

nproc[xyz]:

Number of processors in xyz.

precision:

Single ‘f’ or double ‘d’ precision.

nghost:

Number of ghost zones.

persist:

Structure of persistent values

param:

Optional Param object.

t:

Time of the snapshot.

xyz:

xyz arrays of the domain without ghost zones.

lshear:

Flag for the shear.

dx, dy, dzfloat

Grid spacing (for calculation in the case of nonequidistant grids, see subroutine grid.f90/construct_grid)

pencil.io.write_h5_snapshot(snapshot, file_name='VAR0', datadir='data/allprocs', precision='d', nghost=3, persist=None, settings=None, param=None, grid=None, lghosts=False, indx=None, proc=None, ipx=None, ipy=None, ipz=None, procdim=None, unit=None, t=None, x=None, y=None, z=None, state='a', quiet=True, lshear=False, driver=None, comm=None, overwrite=False, rank=0, size=1, sim_datadir='data')

Write a snapshot given as numpy array. We assume by default that a run simulation directory has already been constructed and start completed successfully in h5 format so that files dim, grid and param files are already present. If not the contents of these will need to be supplied as dictionaries along with persist if included.

call signature:

write_h5_snapshot(snapshot, file_name=’VAR0’, datadir=’data/allprocs’,

precision=’d’, nghost=3, persist=None, settings=None, param=None, grid=None, lghosts=False, indx=None, unit=None, t=None, x=None, y=None, z=None, procdim=None, quiet=True, lshear=False, driver=None, comm=None)

Keyword arguments:

snapshot:

Numpy array containing the snapshot. Must be of shape [nvar, nz, ny, nx] without boundaries or. Must be of shape [nvar, mz, my, mx] with boundaries for lghosts=True.

file_name:

Name of the snapshot file to be written, e.g. VAR0 or var.

datadir:

Directory which the snapshot will be written into.

precision:

Single ‘f’ or double ‘d’ precision.

persist:

optional dictionary of persistent variable.

settings:

optional dictionary of persistent variable.

param:

optional Param object.

grid:

optional Pencil Grid object of grid parameters.

nghost:

Number of ghost zones.

lghosts:

If True the snapshot contains the ghost zones.

indx

Index object of index for each variable in f-array

unit:

Optional dictionary of simulation units.

quiet:

Option to print output.

t:

Time of the snapshot.

xyz:

xyz arrays of the domain with ghost zones. This will normally be obtained from Grid object, but facility to redefine an alternative grid value.

lshear:

Flag for the shear.

driver

File driver for hdf5 io for use in serial or MPI parallel.

comm

MPI objects supplied if driver is ‘mpio’.

overwrite

flag to replace existing h5 snapshot file.

rank

rank of process with root=0.

sim_datadir

Datadir of the destination simulation. Used to get param, dim, and index info when they are not provided.

pencil.io.write_h5_grid(file_name='grid', datadir='data', precision='d', nghost=3, settings=None, param=None, grid=None, unit=None, quiet=True, driver=None, comm=None, overwrite=False, rank=0)

Write the grid information as hdf5. We assume by default that a run simulation directory has already been constructed, but start has not been executed in h5 format so that binary sim files dim, grid and param files are already present in the sim directory, or provided from an old binary sim source directory as inputs.

call signature:

write_h5_grid(file_name=’grid’, datadir=’data’, precision=’d’, nghost=3,

settings=None, param=None, grid=None, unit=None, quiet=True, driver=None, comm=None)

Keyword arguments:

file_name:

Prefix of the file name to be written, ‘grid’.

datadir:

Directory where ‘grid.h5’ is stored.

precision:

Single ‘f’ or double ‘d’ precision.

nghost:

Number of ghost zones.

settings:

Optional dictionary of persistent variable.

param:

Optional Param object.

grid:

Optional Pencil Grid object of grid parameters.

unit:

Optional dictionary of simulation units.

quiet:

Option to print output.

pencil.io.write_h5_averages(aver, file_name='xy', datadir='data/averages', nt=None, precision='d', indx=None, trange=None, quiet=True, append=False, procdim=None, dim=None, aver_by_proc=False, proc=-1, driver=None, comm=None, rank=0, size=1, overwrite=False, nproc=1)

Write an hdf5 format averages dataset given as an Averages object. We assume by default that a run simulation directory has already been constructed and start completed successfully in h5 format so that files dim, grid and param files are already present. If not the contents of these will need to be supplied as dictionaries along with persist if included.

call signature:

write_h5_averages(aver, file_name=’xy’, datadir=’data/averages’,

precision=’d’, indx=None, trange=None, quiet=True)

Keyword arguments:

aver:

Averages object. Must be of shape [n_vars, n1] for averages across ‘xy’, ‘xz’ or ‘yz’. Must be of shape [n_vars, n1, n2] for averages across ‘y’, ‘z’.

file_name:

Name of the snapshot file to be written, e.g. ‘xy’, ‘xz’, ‘yz’, ‘y’, ‘z’.

datadir:

Directory where the data is stored.

precision:

Single ‘f’ or double ‘d’ precision.

indx

Restrict iterative range to be written.

trange:

Restrict time range to be written.

append

For large binary files the data may need to be appended iteratively.

dim

Dim object required if the large binary files are supplied in chunks.

pencil.io.write_h5_slices(vslice, coordinates, positions, datadir='data/slices', precision='d', indx=None, trange=None, quiet=True, append=False, dim=None)

Write an hdf5 format slices dataset given as an Slices object. We assume by default that a run simulation directory has already been constructed and start completed successfully in h5 format so that files dim, grid and param files are already present. If not the contents of these will need to be supplied as dictionaries along with persist if included.

call signature:

write_h5_slices(vslice, coordinates, positions,

datadir=’data/slices’, precision=’d’, indx=None, trange=None, quiet=True)

Keyword arguments:

vslice:

Slices object. Object with attributes ‘t’, extensions e.g, ‘xy’, ‘xy2’, ‘xz’, ‘yz’ and data fields of shape [nt, n1, n2] e.g ‘uu1’, ‘uu2’, ‘uu3’, …

coordinates

Dictionary of lmn indices of all slices in the object n for ‘xy’, etc. Obtained from ‘data/positions.dat’ in source binary simulation

positions

Dictionary of xyz values of all slices in the object z for ‘xy’, etc. Obtained from source binary simulation grid at coordinates.

datadir:

Directory where the data is stored.

precision:

Single ‘f’ or double ‘d’ precision.

indx

Restrict iterative range to be written.

trange:

Restrict time range to be written.

append

For large binary files the data may need to be appended iteratively.

dim

Dim object required if the large binary files are supplied in chunks.

pencil.io.get_systemid()

This function returnes informations over the used system.

Please add further functionallity by yourself!

Return Values:
system_name: name of computer/cluster system as identified. Default return is ‘pc’
Known systems are: - THEO (Garchingen)

  • HYDRA (Garchingen)

  • JUQUEEN

raw_name: raw system name job_name_key: to identify job name in submit script, e.g. ‘-N’, ‘job_name’ submit_scriptfile: default name of submit script submit_line: batch line to submit the submit.csh/.sh

pencil.io.exists_file(file, folder=False)

Better version of exists, compared with os.path.exists!

Parameters:

  • file – filename to look for in folder, or complete filepath string

  • folder – optinal, folder in which file should be

Returns:

True if file in folder of path specified in file exists.

pencil.io.remove_files(path, do_it=False, do_it_really=False)

This method clears path COMPLETELY. Meaning, if you start this method my using path on your root dir your whole computer would be gone! So we implemented some safety mechanism.

Parameters:

  • path – path do clear

  • do_it – to activate pass True, True

  • do_it_really – to activate pass True, True

pencil.io.get_value_from_file(filename, quantity, change_quantity_to=None, sim=None, filepath=None, DEBUG=False, silent=False)
Use to read in a quantity from

  • *.in

  • *.local

  • submit*, i.e. submit.sh, submit.csh, files, only works if computer is readily specified in pc.io.get_systemid

Please add further functionallity by yourself!

Parameters:

  • filename – can be “run.in”, “start.in”, “cparam.local”, path to that file is extraced from filepath or sim object

  • quantity – variable to read in from file

  • sim – put simulation object here, file will be found by filename automatically

  • filepath – normally not needed, specify here where to find the file with filename, can be a list of paths if unshure

  • DEBUG – make dry run, tell me what you would do but dont change anything!

  • silent – suppress certain output by setting True

Returns:

Returns None if not successful

exception pencil.io.GetValueError

Bases: Exception

Common base class for all non-exit exceptions.

Initialize self. See help(type(self)) for accurate signature.

pencil.io.change_value_in_file(filename, quantity, newValue, sim=None, filepath=None, DEBUG=False)
Use to change a quantity in

  • *.in

  • *.local

  • submit*, i.e. submit.sh, submit.csh, files, only works if computer is readily specified in pc.io.get_systemid

Please add further functionallity by yourself!

Parameters:

  • filename – can be “run.in”, “start.in”, “cparam.local”

  • quantity – variable to read in from file

  • sim – put simulation object here, file will be found by filename automatically

  • filepath – normally not needed, specify here where to find the file with filename, can be a list of paths if unshure

  • DEBUG – make dry run, tell me what you would do but dont change anything!

  • silent – suppress certain output by setting True

pencil.io.rename_in_submit_script(new_name, submit_script_path=False, sim=False, old_name=False)
pencil.io.pkl_load(name, folder=False, sim=False)

This scripts loads an pkl-file. It automatically checks known folders if no folder is specified.

Parameters:

  • name – Name of pkl-file (<name>.pkl)

  • folder – Folder containing pkl file

  • sim – Simulation for checking automatically folders for

Example

to read “.pc/sim.pkl” use: pkl_load(‘sim’, ‘.pc’) or simply pkl_load(‘sim’), since pkl_load checks for following folders automatically: ‘.pc’, ‘data/.pc’

pencil.io.pkl_save(obj, name, folder='.pc', ascii=True)

This scripts saves any kind of object as a pkl file in folder.

Args: obj: object you want to save in an pkl file name: name of pkl file, ‘.pkl’ will be added automatically if missing ascii: if you dont want to save pkl file in ascii set this to false

pencil.io.pkl_exists(name, folder=False)

This scripts checks if a certain pkl-file already exists.

Parameters:

  • name – Name of pkl file (<name>.pkl)

  • folder – Folder containing pkl file

pencil.io.walklevel(some_dir, level=1)

Walklevel walks through higher level dirs to a certain depth. Is used as an interator in loops, e.g.:

sim_paths = [] for path,dirs,files in walklevel(‘.’, depth):

if ‘start.in’ in files and ‘run.in’ in files:

print ‘## Found Simulation in ‘+path sim_paths.append(path)

Parameters:

  • some_dir – at which dir to start the walklevel algorithm

  • level – how deep this algorithm shall walk through dirs, default is 1

Yields:

yield root, dirs, files