The Federated Execution Manager (FEM) is a software component designed to coordinate and manage the execution of distributed processes in a federated leaning (FL) or decentralised compute environment where multiple devices or clients, i.e. the federated data nodes (FDN) provide compute and data resources.
➡️ Source code moved at https://gitlab.bsc.es/fl/fem-client
FEM is responsible for controlling and orchestrating the lifecycle of the federated tasks. It consists of two parts:
- an orchestrator module (FEM-orchestrator) that ensures tasks reach the multiple FDNs in a secure and coordinated manner
- a client module (FEM-client) installed at the participating FDN that pulls such tasks and allocates them in the local infrastructure.
In short, the FEM-client interact with the FEM-orchestrator via a message broker (RabbitMQ) and pulls the federated task details. across multiple Federated Data Nodes (FDNs). The FEM Client works in conjunction with the central FEM Orchestrator to manage distributed computing workflows, such as federated learning (FL) experiments and other data processing tasks.
The key functions of the FEM client include:
- Task reception: it ensures secure interactions between the FDN and the central orchestrator via a message broker (RabbitMQ) and pulls the federated task details.
- Task stage-in: it prepares the working enviroment, handling data volumes, container image pulling and source code Git syncronizations.
- Task execution: it executes tasks in the local infraestructure delegating the resources managment the customizable executors or launchers. Default executor: "docker".
- Task stage-out: it exchanging the necessary task outfiles and logging files with the central orchestrator.
- FEM-orchestrator: https://github.com/DataTools4Heart/FEM-orchestrator
- dt4h-FEM-client-config: https://github.com/DataTools4Heart/dt4h-FEM-client-config