lasif.tools.parallel

Helpers for embarrassingly parallel calculations using MPI. All functions works just fine when running on one core and not started with MPI.

copyright:Lion Krischer (krischer@geophysik.uni-muenchen.de), 2014-2015
license:GNU Lesser General Public License, Version 3 (http://www.gnu.org/copyleft/lesser.html)
class lasif.tools.parallel_helpers.FunctionInfo[source]

Namedtuple used to collect information about a function execution.

It has the following fields: func_args, result, warnings, exception, and traceback.

lasif.tools.parallel_helpers.distribute_across_ranks(function, items, get_name, logfile)[source]

Calls a function once for each item.

It will be distributed across MPI ranks if launched with MPI.

Parameters:
  • function – The function to be executed for each item.
  • items – The function will be executed once for each item. It expects a list of dictionaries so that function(**item) can work. Only rank 0 needs to pass this. It will be ignored coming from other ranks.
  • get_name – Function to extract a name for each item to be able to produce better logfiles.
  • logfile – The logfile to write.
lasif.tools.parallel_helpers.function_info(traceback_limit=10)[source]

Decorator collecting information during the execution of a function.

This is useful for collecting information about a function runnning on a number of processes/machines.

It returns a FunctionInfo named tuple with the following fields:

  • func_args: Dictionary containing all the functions arguments and values.
  • result: The return value of the function. Will be None if an exception has been raised.
  • warnings: A list with all warnings the function raised.
  • exception: The exception the function raised. Will be None, if no exception has been raised.
  • traceback: The full traceback in case an exception occured as a string. A traceback object is not serializable thus a string is used.
>>> @function_info()
... def test(a, b=2):
...     return a / b
>>> info = test(4, 1)
>>> info.func_args
{'a': 4, 'b': 1}
>>> info.result
4

warnings is empty if no warning has been raised. Otherwise it will collect all warnings.

>>> info.warnings
[]

exception and traceback are None if the function completed successfully.

>>> info.exception
>>> info.traceback