particles_mpicomm_blocks

Module

Description

$Id$

This module deals with communication of particles between processors.

This version is for block domain decomposition of particles. See particles_map_blocks.f90 for documentation.

Quick access

Variables:

communicate_fpbuf, cumulate_counts, dfb, dist_particles_evenly_procs, dvol1xb, dvol1xbrick, dvol1yb, dvol1ybrick, dvol1zb, dvol1zbrick, dx1b, dx1brick, dy1b, dy1brick, dz1b, dz1brick, fb, get_brick_index, ibrick_parent_block, inearblock, input_blocks_dist, input_blocks_mpi, iproc_foster_brick, iproc_foster_brick_part, iproc_foster_list, iproc_parent_block, iproc_parent_list, it1_loadbalance, k1_iblock, k2_iblock, l1b, l2b, ladopt_own_light_bricks, lbrick_partition, lfill_blocks_density, lfill_blocks_gpotself, lfill_blocks_velocity, lfill_bricks_velocity, lreblock_particles_run, m1b, m2b, migrate_particles_block_to_proc, migrate_particles_proc_to_block, migrate_particles_proc_to_proc, mxb, myb, mzb, n1b, n2b, nblock_loc, nbrick_foster, nbricks, nbx, nby, nbz, nghostb, npar_iblock, npart_brick_max, nproc_foster, nproc_parent, nxb, nyb, nzb, output_blocks_dist, output_blocks_mpi, report_missing_particles, xb, xbrick, xref_par, yb, ybrick, yref_par, zb, zbrick, zref_par

Routines:

communicate_fpbuf(), get_brick_index(), initialize_particles_mpicomm(), input_blocks(), load_balance_particles(), migrate_particles(), output_blocks(), sort_blocks()

Needed modules

Variables

  • particles_mpicomm/cumulate_counts [private]
  • particles_mpicomm/dfb (2 * nghostb + nxb,2 * nghostb + nyb,2 * nghostb + nzb,mvar,nblockmax) [real,public]
  • particles_mpicomm/dist_particles_evenly_procs [private]
  • particles_mpicomm/dvol1xb (2 * nghostb + nxb,nblockmax) [real,public]
  • particles_mpicomm/dvol1xbrick (2 * nghostb + nxb,nbx * nby * nbz) [real,public]
  • particles_mpicomm/dvol1yb (2 * nghostb + nyb,nblockmax) [real,public]
  • particles_mpicomm/dvol1ybrick (2 * nghostb + nyb,nbx * nby * nbz) [real,public]
  • particles_mpicomm/dvol1zb (2 * nghostb + nzb,nblockmax) [real,public]
  • particles_mpicomm/dvol1zbrick (2 * nghostb + nzb,nbx * nby * nbz) [real,public]
  • particles_mpicomm/dx1b (2 * nghostb + nxb,nblockmax) [real,public]
  • particles_mpicomm/dx1brick (2 * nghostb + nxb,nbx * nby * nbz) [real,public]
  • particles_mpicomm/dy1b (2 * nghostb + nyb,nblockmax) [real,public]
  • particles_mpicomm/dy1brick (2 * nghostb + nyb,nbx * nby * nbz) [real,public]
  • particles_mpicomm/dz1b (2 * nghostb + nzb,nblockmax) [real,public]
  • particles_mpicomm/dz1brick (2 * nghostb + nzb,nbx * nby * nbz) [real,public]
  • particles_mpicomm/fb (2 * nghostb + nxb,2 * nghostb + nyb,2 * nghostb + nzb,mfarray,nblockmax) [real,public]
  • particles_mpicomm/ibrick_parent_block (nblockmax) [integer,public]
  • particles_mpicomm/inearblock (mpar_loc) [integer,public]
  • particles_mpicomm/input_blocks_dist [private]
  • particles_mpicomm/input_blocks_mpi [private]
  • particles_mpicomm/iproc_foster_brick (nbx * nby * nbz) [integer,public]
  • particles_mpicomm/iproc_foster_list (ncpus) [integer,public]
  • particles_mpicomm/iproc_parent_block (nblockmax) [integer,public]
  • particles_mpicomm/iproc_parent_list (ncpus) [integer,public]
  • particles_mpicomm/it1_loadbalance [integer,public/optional/default=100]
  • particles_mpicomm/k1_iblock (nblockmax) [integer,public]
  • particles_mpicomm/k2_iblock (nblockmax) [integer,public]
  • particles_mpicomm/l1b [integer,parameter=2]
  • particles_mpicomm/l2b [integer,parameter=l1b+nxb-1]
  • particles_mpicomm/ladopt_own_light_bricks [logical,public/optional/default=.false.]
  • particles_mpicomm/lbrick_partition [logical,public/optional/default=.false.]
  • particles_mpicomm/lfill_blocks_density [logical,public/optional/default=.false.]
  • particles_mpicomm/lfill_blocks_gpotself [logical,public/optional/default=.false.]
  • particles_mpicomm/lfill_blocks_velocity [logical,public/optional/default=.false.]
  • particles_mpicomm/lfill_bricks_velocity [logical,public/optional/default=.false.]
  • particles_mpicomm/lreblock_particles_run [logical,public/optional/default=.false.]
  • particles_mpicomm/m1b [integer,parameter=2]
  • particles_mpicomm/m2b [integer,parameter=m1b+nyb-1]
  • particles_mpicomm/migrate_particles_block_to_proc [private]
  • particles_mpicomm/migrate_particles_proc_to_block [private]
  • particles_mpicomm/migrate_particles_proc_to_proc [private]
  • particles_mpicomm/mxb [integer,parameter=nxb+2*nghostb]
  • particles_mpicomm/myb [integer,parameter=nyb+2*nghostb]
  • particles_mpicomm/mzb [integer,parameter=nzb+2*nghostb]
  • particles_mpicomm/n1b [integer,parameter=2]
  • particles_mpicomm/n2b [integer,parameter=n1b+nzb-1]
  • particles_mpicomm/nblock_loc [integer,public]
  • particles_mpicomm/nbrick_foster [integer,public]
  • particles_mpicomm/nbricks [integer,parameter=nbx*nby*nbz]
  • particles_mpicomm/nbx [integer,parameter=nbrickx/nprocx]
  • particles_mpicomm/nby [integer,parameter=nbricky/nprocy]
  • particles_mpicomm/nbz [integer,parameter=nbrickz/nprocz]
  • particles_mpicomm/nghostb [integer,parameter=1]
  • particles_mpicomm/npar_iblock (nblockmax) [integer,public]
  • particles_mpicomm/nproc_foster [integer,public]
  • particles_mpicomm/nproc_parent [integer,public]
  • particles_mpicomm/nxb [integer,parameter=nxgrid/nbrickx]
  • particles_mpicomm/nyb [integer,parameter=nygrid/nbricky]
  • particles_mpicomm/nzb [integer,parameter=nzgrid/nbrickz]
  • particles_mpicomm/output_blocks_dist [private]
  • particles_mpicomm/output_blocks_mpi [private]
  • particles_mpicomm/report_missing_particles [private]
  • particles_mpicomm/xb (2 * nghostb + nxb,nblockmax) [real,public]
  • particles_mpicomm/xbrick (2 * nghostb + nxb,nbx * nby * nbz) [real,public]
  • particles_mpicomm/xref_par [real,private/optional/default=0.0]
  • particles_mpicomm/yb (2 * nghostb + nyb,nblockmax) [real,public]
  • particles_mpicomm/ybrick (2 * nghostb + nyb,nbx * nby * nbz) [real,public]
  • particles_mpicomm/yref_par [real,private/optional/default=0.0]
  • particles_mpicomm/zb (2 * nghostb + nzb,nblockmax) [real,public]
  • particles_mpicomm/zbrick (2 * nghostb + nzb,nbx * nby * nbz) [real,public]
  • particles_mpicomm/zref_par [real,private/optional/default=0.0]

Subroutines and functions

subroutine  particles_mpicomm/initialize_particles_mpicomm(f)

Perform any post-parameter-read initialization i.e. calculate derived parameters.

31-oct-09/anders: coded

Parameters:

f (mx,my,mz,mfarray) [real,in]

Use :

mpicomm (mpi_precision(), mpibcast_real())

Called from:

particles_register_modules(), particles_rprint_list(), particles_initialize_modules()

Call to:

fatal_error_local(), fatal_error_local_collect(), fatal_error(), input_blocks(), get_brick_index(), warning(), sort_blocks(), mpiwait()

subroutine  particles_mpicomm/migrate_particles(fp, ipar[, dfp[, linsert]])

Migrate particles between processors.

Migration is divided into three steps for block domain decomposition:

  1. Particles that are no longer in any blocks adopted by the considered processor are migrated to their parent processor.

  2. The parent processor either keeps the particle, if it is really the parent, or migrates the particle to the actual parent. The actual parent may differ from the registred parent if the particle has moved away from the latter.

  3. The parent processor either keeps the particle, if the particle is in one of its blocks, or migrates the particle to the foster parent.

The three-step division is necessary to have directed communication between processors, because it is impractical at high processor numbers to allow all processors to exchange particles.

31-oct-09/anders: coded

Parameters:

  • fp (mpar_loc,mparray) [real]

  • ipar (mpar_loc) [integer]

  • dfp (mpar_loc,mpvar) [real]

  • linsert [logical]

Called from:

input_particles(), output_particles(), append_npvar(), append_npaux(), boundconds_particles()

Call to:

get_brick_index(), fatal_error(), fatal_error_local(), fatal_error_local_collect(), warning(), sort_blocks(), mpiwait()

subroutine  particles_mpicomm/load_balance_particles(f, fp, ipar)

This subroutine counts particles in the bricks at the local processor and distributes the bricks in such a away that there is approximately equal number of particles per processor.

Alternative method:

  • Integrate np on processors (should be possible to do MPI integral)

  • Each processor finds out if it has the boundary between two procs

  • Use MPI_ANY_SOURCE to communicate boundary to correct processor

To avoid stacking light bricks at a single processor, this can be done first for heavy bricks and then for light bricks.

12-oct-09/anders: coded

Parameters:
Use :

mpicomm (mpirecv_int(), mpirecv_nonblock_int(), mpirecv_nonblock_real(), mpirecv_real(), mpisend_int(), mpisend_nonblock_int(), mpisend_nonblock_real(), mpisend_real(), mpiwait())

Called from:

particles_register_modules(), particles_rprint_list(), particles_initialize_modules(), particles_init(), particles_finalize(), particles_read_snapshot(), particles_write_snapshot(), particles_write_dsnapshot(), particles_write_pdim(), particles_write_block(), particles_write_rmv(), particles_timestep_first(), particles_timestep_second(), split_update_particles(), particles_load_balance(), read_all_particles_init_pars(), read_all_particles_run_pars(), write_dim_particles(), write_snapshot_particles(), read_snapshot_particles()

Call to:

fatal_error_local(), fatal_error_local_collect(), mpiwait(), sort_blocks(), fatal_error()

subroutine  particles_mpicomm/output_blocks(filename)

Dispatches to a blocks writer according to the IO strategy.

07-oct-22/ccyang: coded

Parameters:

filename [character,in]

Use :

io (io_strategy())

Called from:

particles_register_modules(), particles_rprint_list(), particles_initialize_modules(), particles_init(), particles_finalize(), particles_read_snapshot(), particles_write_snapshot(), particles_write_dsnapshot(), particles_write_pdim(), particles_write_block(), particles_write_rmv(), particles_timestep_first(), particles_timestep_second(), split_update_particles(), particles_load_balance(), particles_boundconds(), particles_pencil_criteria(), particles_pencil_interdep(), particles_calc_pencils(), particles_calc_selfpotential(), particles_before_boundary(), particles_special_bfre_bdary(), particles_special_after_dtsub(), particles_pde(), particles_pde_pencil(), particles_calc_pencil_diags(), particles_pde_blocks(), read_all_particles_init_pars(), write_all_particles_init_pars(), read_all_particles_run_pars(), write_all_particles_run_pars(), write_dim_particles(), write_snapshot_particles(), read_snapshot_particles()

Call to:

fatal_error(), fatal_error_local(), fatal_error_local_collect()

subroutine  particles_mpicomm/input_blocks(filename)

Dispatches to a blocks reader according to the IO strategy.

07-oct-22/ccyang: coded

Parameters:

filename [character,in]

Use :

io (io_strategy())

Called from:

initialize_particles_mpicomm()

Call to:

fatal_error(), fatal_error_local(), fatal_error_local_collect()

subroutine  particles_mpicomm/sort_blocks()

Sort the blocks by parent processor and by parent brick index.

04-nov-09/anders: coded

Called from:

initialize_particles_mpicomm(), migrate_particles(), load_balance_particles()

Call to:

fatal_error(), fatal_error_local()

subroutine  particles_mpicomm/get_brick_index(xxp, iproc_rec, ibrick_rec[, ineargrid[, status]])

Find the parent processor and brick of a given position.

09-jan-12/ccyang: adapted from the original version in various

routines migrate_particles_*_to_*

Parameters:

  • xxp (3) [real,in]

  • iproc_rec [integer,out]

  • ibrick_rec [integer,out]

  • ineargrid (3) [integer,out,]

  • status [integer,out,]

Use :

sub (find_index_by_bisection())

Called from:

initialize_particles_mpicomm(), migrate_particles(), initialize_particles_map(), map_nearest_grid()

Call to:

fatal_error_local(), fatal_error()

subroutine  particles_mpicomm/communicate_fpbuf(to_neigh, from_neigh, her_npbuf, my_npbuf)
Parameters:

  • to_neigh [integer,in]

  • from_neigh [integer,in]

  • her_npbuf [integer,in]

  • my_npbuf [integer,in]