halodrops.pipeline

Module Contents

Functions

get_mandatory_args	Get a list of all arguments that do not have a default value for each function in the list.
get_mandatory_values_from_config	Extracts mandatory values from the ‘MANDATORY’ section of a configuration file.
get_nondefaults_from_config	Get the non-default arguments for a given function from a ConfigParser object.
get_args_for_function	Get the arguments for a given function.
get_platforms	Get platforms based on the directory names in data_directory or the user-provided platforms values.
create_and_populate_flight_object	Creates a Flight object and populates it with A-files.
iterate_Sonde_method_over_dict_of_Sondes_objects	Iterates over a dictionary of Sonde objects and applies a list of methods to each Sonde.
sondes_to_gridded
iterate_method_over_dataset
gridded_to_pattern	The flight-phase segmentation file must be provided via the config file.
run_substep	This function applies a specified function to the input data and stores the output for use in subsequent steps.
run_pipeline	Executes a pipeline of processing steps.

Data

pipeline

API

halodrops.pipeline.get_mandatory_args(function)

Get a list of all arguments that do not have a default value for each function in the list.

Parameters

list_of_functions : list A list of functions to inspect.

Returns

list A list of argument names that do not have a default value.

Examples

def func1(a, b=2): … pass def func2(c, d=4, e=5): … pass mandatory_args([func1, func2]) [‘a’, ‘c’]

halodrops.pipeline.get_mandatory_values_from_config(config, mandatory_args)

Extracts mandatory values from the ‘MANDATORY’ section of a configuration file.

Parameters

config : ConfigParser The configuration file parser. mandatory_args : list A list of argument names that are expected to be in the ‘MANDATORY’ section of the config file.

Returns

dict A dictionary where the keys are the argument names and the values are the corresponding values from the config file.

Raises

ValueError If the ‘MANDATORY’ section is not found in the config file or if a mandatory argument is not found in the ‘MANDATORY’ section.

Examples

import configparser config = configparser.ConfigParser() config.read_string(‘[MANDATORY]\narg1=value1\narg2=value2’) mandatory_values_from_config(config, [‘arg1’, ‘arg2’]) {‘arg1’: ‘value1’, ‘arg2’: ‘value2’}

halodrops.pipeline.get_nondefaults_from_config(config: configparser.ConfigParser, obj: callable) → dict

Get the non-default arguments for a given function from a ConfigParser object.

Parameters

config : configparser.ConfigParser A ConfigParser object containing configuration settings. obj : callable The function for which to get the non-default arguments.

Returns

dict A dictionary of non-default arguments for the function.

halodrops.pipeline.get_args_for_function(config, function)

Get the arguments for a given function.

This function first checks if the qualified name of the function exists in the nondefaults dictionary. If it does, it uses the corresponding value as the arguments for the function. Otherwise, it initializes an empty dictionary as the arguments.

Then, it gets the list of mandatory arguments for the function. If there are any mandatory arguments, it gets their values from the config and updates the arguments dictionary with them.

Parameters

function : function The function for which to get the arguments. nondefaults : dict A dictionary mapping function qualified names to dictionaries of arguments.

Returns

dict A dictionary of arguments for the function.

halodrops.pipeline.get_platforms(config)

Get platforms based on the directory names in data_directory or the user-provided platforms values.

Parameters

config : ConfigParser instance The configuration file parser.

Returns

dict A dictionary where keys are platform names and values are Platform objects.

Raises

ValueError If platforms is specified in the config file but platform_directory_names is not, or if a value in platform_directory_names does not correspond to a directory in data_directory.

halodrops.pipeline.create_and_populate_flight_object(config: configparser.ConfigParser)

Creates a Flight object and populates it with A-files.

Parameters

config : configparser.ConfigParser A ConfigParser object containing configuration settings.

Returns

Flight A Flight object.

halodrops.pipeline.iterate_Sonde_method_over_dict_of_Sondes_objects(obj: dict, functions: list, config: configparser.ConfigParser) → dict

Iterates over a dictionary of Sonde objects and applies a list of methods to each Sonde.

For each Sonde object in the dictionary, this function applies each method listed in the ‘functions’ key of the substep dictionary. If the method returns a value, it stores the value in a new dictionary. If the method returns None, it does not store the value in the new dictionary.

The arguments for each method are determined by the get_args_for_function function, which uses the nondefaults dictionary and the config object.

Parameters

obj : dict A dictionary of Sonde objects. functions : list a list of method names. nondefaults : dict A dictionary mapping function qualified names to dictionaries of arguments. config : configparser.ConfigParser A ConfigParser object containing configuration settings.

Returns

dict A dictionary of Sonde objects with the results of the methods applied to them (keys where results are None are not included).

halodrops.pipeline.sondes_to_gridded(sondes: dict) → xarray.Dataset

halodrops.pipeline.iterate_method_over_dataset(dataset: xarray.Dataset, functions: list) → xarray.Dataset

halodrops.pipeline.gridded_to_pattern(gridded: xarray.Dataset, config: configparser.ConfigParser) → xarray.Dataset

The flight-phase segmentation file must be provided via the config file.

halodrops.pipeline.run_substep(previous_substep_output, substep: dict, config: configparser.ConfigParser)

This function applies a specified function to the input data and stores the output for use in subsequent steps.

Parameters

previous_substep_output : dict A dictionary storing the output data from previous steps. The input data for this step is retrieved from this dictionary using the key specified in substep[‘intake’], and the output of this step is stored in this dictionary under the key(s) specified in substep[‘output’]. substep : dict A dictionary containing information about the current processing step. It should have the following keys: - ‘apply’: a function to apply to the input data. - ‘intake’: the key in the previous_substep_output dictionary that corresponds to the input data for this step. - ‘output’: the key(s) under which to store the output of this step in the previous_substep_output dictionary. If this is a list, the function should return a list of outputs of the same length. - ‘functions’ (optional): a list of functions to apply to the input data. If this key is present, the ‘apply’ function should take this list as an additional argument. config : object A configuration object used by the function.

Returns

dict The updated dictionary with the output data from the current step. The output data is stored under the key(s) specified in substep[‘output’].

Notes

This function assumes that the ‘apply’ function returns a list of outputs if substep[‘output’] is a list, and a single output otherwise. If substep[‘output’] is a list but the ‘apply’ function does not return a list of outputs, or if the lengths of the two lists do not match, this function will raise an exception.

halodrops.pipeline.run_pipeline(pipeline: dict, config: configparser.ConfigParser)

Executes a pipeline of processing steps.

Parameters:

pipeline : dict A dictionary representing the pipeline where each key is a substep and the value is a dictionary with the configurations of that substep. config : configparser.ConfigParser Configuration settings for the package.

Returns:

dict: The output of the last substep in the pipeline.

halodrops.pipeline.pipeline

None