parallel¶

Contains a parallel runner implementation, with support for various backends

class microscopes.kernels.parallel.runner(runners, backend='multiprocessing', **kwargs)¶

A parallel runner. Note the parallelism is applied across runners (currently, each runner is assumed to be single-threaded).

Parameters:

Parameters:	runners : list of runner objects backend : string, one of {‘multiprocessing’, ‘multyvac’} Indicates the parallelization strategy to be used across runners. Note for the ‘multiprocessing’ backend, the only valid kwarg is ‘processes’. For the ‘multyvac’ backend, the valid kwargs are ‘layer’, ‘core’, and ‘volume’. processes : int, optional For the ‘multiprocessing’ backend, the number of processes in the process pool. Defaults to the number of processes on the current machine. layer : string The multyvac layer which has the datamicroscopes dependencies installed. core : string The type of multyvac core to use. Defaults currently to ‘f2’ (the most expensive, but powerful core type). volume : string, optional The volume is highly recommended to work around multyvac’s limitations regarding passing around large objects (e.g. dataviews). The volume must be created beforehand. The runner uses the root directory of the volume as a cache.

runners : list of runner objects

backend : string, one of {‘multiprocessing’, ‘multyvac’}

Indicates the parallelization strategy to be used across runners. Note for the ‘multiprocessing’ backend, the only valid kwarg is ‘processes’. For the ‘multyvac’ backend, the valid kwargs are ‘layer’, ‘core’, and ‘volume’.

processes : int, optional

For the ‘multiprocessing’ backend, the number of processes in the process pool. Defaults to the number of processes on the current machine.

layer : string

The multyvac layer which has the datamicroscopes dependencies installed.

core : string

The type of multyvac core to use. Defaults currently to ‘f2’ (the most expensive, but powerful core type).

volume : string, optional

The volume is highly recommended to work around multyvac’s limitations regarding passing around large objects (e.g. dataviews). The volume must be created beforehand. The runner uses the root directory of the volume as a cache.

Notes

To use the multyvac backend, you must first authenticate your machine (e.g. by running multyvac setup) beforehand.

get_latents()¶: Returns a list of the current state of each of the runners.

run(r, niters=10000)¶

Run each runner for niters, using the backend supplied in the constructor for parallelism.

Parameters:

Parameters:	r : rng niters : int

r : rng

niters : int