dagrunner.plugin_framework.md

module: dagrunner.plugin_framework

Source

see class: dagrunner.utils.Singleton

see function: dagrunner.utils.data_polling

see function: dagrunner.utils.process_path

see function: dagrunner.utils.stage_to_dir

class: DataPolling

Source

Call Signature:

DataPolling(timeout=120, polling=1, file_count=None, verbose=False)

A trigger plugin that completes when data is successfully polled.

function: __call__

Source

Call Signature:

__call__(self, *args)

The main method of the plugin (abstract method).

Positional arguments represent the plugin's inputs (dependencies), while keyword arguments represent the plugin's parameters.

Args:

• *args: Positional arguments.
• **kwargs: Keyword arguments.

Returns:

• Any: The output of the plugin.

function: __init__

Source

Call Signature:

__init__(self, timeout=120, polling=1, file_count=None, verbose=False)

Initialize self. See help(type(self)) for accurate signature.

_IgnoreEvent: IGNORE_EVENT

class: Input

Source

Call Signature:

Input()

An abstract base class plugin that is of type that instructs the plugin executor to pass it node parameters. This enables the definition of plugins that are 'node aware'.

function: __call__

Source

Call Signature:

__call__(self, filepath, node_properties=None, **kwargs)

Given a string, expand it and return this expanded string.

Expand the provided string (typically representing a filepath) using the keyword arguments and environment variables. Note that this plugin is 'node aware' since it is derived from the NodeAwarePlugin.

Args:

• filepath (str): The filepath to be expanded.
• node_properties: node properties passed by the plugin executor.
• **kwargs: Keyword arguments to be used in the expansion.

Returns:

• str: The expanded filepath.

Raises:

• ValueError: If positional arguments are provided.

class: Load

Source

Call Signature:

Load(staging_dir=None, ignore_missing=False, verbose=False)

Abstract base class to define our plugin UI

function: __call__

Source

Call Signature:

__call__(self, *args, **kwargs)

Load data from a file or list of files.

Args:

• *args: List of filepaths to load. <hostname>:<path> syntax supported for loading files from a remote host.
• **kwargs: Keyword arguments to pass to.

function: __init__

Source

Call Signature:

__init__(self, staging_dir=None, ignore_missing=False, verbose=False)

Load data from a file.

The load method must be implemented by the subclass.

Args:

• staging_dir: Directory to stage files in.
• verbose: Print verbose output.

function: load

Source

Call Signature:

load(self, *args, **kwargs)

Load data from a file.

Args:

• *args: Positional arguments.
• **kwargs: Keyword arguments.

Returns:

• Any: The loaded data.

Raises:

• NotImplementedError: If the method is not implemented.

class: LoadJson

Source

Call Signature:

LoadJson(staging_dir=None, ignore_missing=False, verbose=False)

Load json file.

function: __call__

Source

Call Signature:

__call__(self, *args, **kwargs)

Load data from a file or list of files.

Args:

• *args: List of filepaths to load. <hostname>:<path> syntax supported for loading files from a remote host.
• **kwargs: Keyword arguments to pass to.

function: __init__

Source

Call Signature:

__init__(self, staging_dir=None, ignore_missing=False, verbose=False)

Load data from a file.

The load method must be implemented by the subclass.

Args:

• staging_dir: Directory to stage files in.
• verbose: Print verbose output.

function: load

Source

Call Signature:

load(self, *args)

Load data from a file.

Args:

• *args: Positional arguments.
• **kwargs: Keyword arguments.

Returns:

• Any: The loaded data.

Raises:

• NotImplementedError: If the method is not implemented.

class: LoadPickle

Source

Call Signature:

LoadPickle(staging_dir=None, ignore_missing=False, verbose=False)

Load pickle file.

function: __call__

Source

Call Signature:

__call__(self, *args, **kwargs)

Load data from a file or list of files.

Args:

• *args: List of filepaths to load. <hostname>:<path> syntax supported for loading files from a remote host.
• **kwargs: Keyword arguments to pass to.

function: __init__

Source

Call Signature:

__init__(self, staging_dir=None, ignore_missing=False, verbose=False)

Load data from a file.

The load method must be implemented by the subclass.

Args:

• staging_dir: Directory to stage files in.
• verbose: Print verbose output.

function: load

Source

Call Signature:

load(self, *args)

Load data from a file.

Args:

• *args: Positional arguments.
• **kwargs: Keyword arguments.

Returns:

• Any: The loaded data.

Raises:

• NotImplementedError: If the method is not implemented.

class: NodeAwarePlugin

Source

Call Signature:

NodeAwarePlugin()

An abstract base class plugin that is of type that instructs the plugin executor to pass it node parameters. This enables the definition of plugins that are 'node aware'.

function: __call__

Source

Call Signature:

__call__(self, *args, **kwargs)

The main method of the plugin (abstract method).

Positional arguments represent the plugin's inputs (dependencies), while keyword arguments represent the plugin's parameters.

Args:

• *args: Positional arguments.
• **kwargs: Keyword arguments.

Returns:

• Any: The output of the plugin.

class: Plugin

Source

Call Signature:

Plugin()

Abstract base class to define our plugin UI

function: __call__

Source

Call Signature:

__call__(self, *args, **kwargs)

The main method of the plugin (abstract method).

Positional arguments represent the plugin's inputs (dependencies), while keyword arguments represent the plugin's parameters.

Args:

• *args: Positional arguments.
• **kwargs: Keyword arguments.

Returns:

• Any: The output of the plugin.

_SkipEvent: SKIP_EVENT

class: SaveJson

Source

Call Signature:

SaveJson()

An abstract base class plugin that is of type that instructs the plugin executor to pass it node parameters. This enables the definition of plugins that are 'node aware'.

function: __call__

Source

Call Signature:

__call__(self, *args, filepath, node_properties=None, **kwargs)

Save data to a JSON file

Save the provided data to a JSON file at the specified filepath. The filepath is expanded using the keyword arguments and environment variables. Note that this plugin is 'node aware' since it is derived from the NodeAwarePlugin.

Args:

• *args: Positional arguments (data) to be saved.
• filepath: The filepath to save the data to.
• node_properties: node properties passed by the plugin executor.
• **kwargs: Keyword arguments to be used in the expansion.

Returns:

• None

class: SavePickle

Source

Call Signature:

SavePickle()

An abstract base class plugin that is of type that instructs the plugin executor to pass it node parameters. This enables the definition of plugins that are 'node aware'.

function: __call__

Source

Call Signature:

__call__(self, *args, filepath, node_properties=None, **kwargs)

Save data to a Pickle file

Save the provided data to a pickle file at the specified filepath. The filepath is expanded using the keyword arguments and environment variables. Note that this plugin is 'node aware' since it is derived from the NodeAwarePlugin.

Args:

• *args: Positional arguments (data) to be saved.
• filepath: The filepath to save the data to.
• node_properties: node properties passed by the plugin executor.
• **kwargs: Keyword arguments to be used in the expansion.

Returns:

• None

class: Shell

Source

Call Signature:

Shell()

Abstract base class to define our plugin UI

function: __call__

Source

Call Signature:

__call__(self, *args, **kwargs)

Execute a subprocess command.

Args:

• *args: The command to be executed.
• **kwargs: Additional keyword arguments to be passed to subprocess.run

Returns:

• CompletedProcess: An object representing the completed process.

Raises:

• CalledProcessError: If the command returns a non-zero exit status.