Merge branch 'main' into Dataset-from-torch-to-torch-does-not-write-i…

…n-storage

.github/workflows/python-publish.yml

@@ -0,0 +1,39 @@
+# This workflow will upload a Python Package using Twine when a release is created
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python#publishing-to-package-registries
+
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+
+name: Upload FLEXible
+
+on:
+  release:
+    types: [published]
+
+permissions:
+  contents: read
+
+jobs:
+  pypi-publish:
+    name: Publish to PyPI
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python
+      uses: actions/setup-python@v3
+      with:
+        python-version: '3.9'
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install build
+    - name: Build package
+      run: python -m build
+    - name: Publish package
+      uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29
+      with:
+        user: AlArgente
+        password: ${{ secrets.PYPI_API_TOKEN }}

.readthedocs.yaml

@@ -0,0 +1,40 @@
+# Read the Docs configuration file for Sphinx projects
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the OS, Python version and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.9"
+    # You can also specify other tool versions:
+    # nodejs: "20"
+    # rust: "1.70"
+    # golang: "1.20"
+
+# Build documentation in the "docs/" directory with Sphinx
+sphinx:
+  configuration: docs/source/conf.py
+  # You can configure Sphinx to use a different builder, for instance use the dirhtml builder for simpler URLs
+  builder: html
+  # builder: "dirhtml"
+  # Fail on all warnings to avoid broken references
+  fail_on_warning: true
+
+# Optionally build your docs in additional formats such as PDF and ePub
+# formats:
+#   - pdf
+#   - epub
+
+# Optional but recommended, declare the Python requirements required
+# to build your documentation
+# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
+python:
+  install:
+    - method: pip
+      path: .
+      extra_requirements:
+        - docs
+    - requirements: docs/requirements.txt

.trunk/config/ruff.toml

@@ -2,4 +2,4 @@
 select = ["B", "D3", "D4", "E", "F"]
 
 # Never enforce `E501` (line length violations). This should be handled by formatters.
-ignore = ["E501", "D417", "D401", "B905"]
+ignore = ["E501", "D417", "D401", "B905", "D409", "D301", "E402"]

README.md

@@ -1,72 +1,134 @@
-![](https://twemoji.maxcdn.com/v/latest/72x72/1f938.png)
+<img src="FLEXible_Logo.png" width="100">
 
 # FLEXible
 
 [![Tests](https://github.com/FLEXible-FL/FLEX-framework/actions/workflows/pytest.yml/badge.svg)](https://github.com/FLEXible-FL/FLEX-framework/actions/workflows/pytest.yml)
 [![Linter](https://github.com/FLEXible-FL/FLEX-framework/actions/workflows/trunk.yml/badge.svg)](https://github.com/FLEXible-FL/FLEX-framework/actions/workflows/trunk.yml)
 
-The documentation of the package was generated using [pdoc](https://pdoc3.github.io/pdoc/). The steps to generate the documentation are the following ones:
+FLEXible (Federated Learning Experiments) is an open source Federated Learning (FL) framework that provides a set of tools and utilities to work with deep learning and machine learning models in a federated scenario. It is designed to federate data and models in a simple and easy way, and it is compatible with the most popular deep learning frameworks such as PyTorch and TensorFlow. It also provides a set of federated datasets to test the models.
 
-``
-pdoc flex -o=docs --docformat markdown --footer-text "FLEXible: Federad Learning Experiments"
-``
+FLEXible let the user to customize the federated scenario, from the bottom to the top. FLEXible has the following tools to carry out the federated learning experiments:
 
-In order to install this repo locally for development:
+- Datasets: FLEXible provides a set of federated datasets to test the models. Some datasets are: MNIST, CIFAR10, Shakespeare, etc.
+- Data: FLEXible provides a set of tools to federate your own data. You can import your own data our even import the data from other libraries such as `torchvision`, `torchtext`, `datasets` or `tensorflow_datasets`.
+- Architecture: In FLEXible you can create custom federated architectures. You can quickly deploy a client-server architecture or a peer-to-peer architecture, or easily create your own federated architecture.
+- Roles: FLEXible provides a set of roles to define the federated scenario. Usually, you will work with the `Server`, `Aggregator` and the `Client` roles, but you can create nodes with multiple roles, such as `Server` and `Client` at the same time, or `Server` and `Aggregator` at the same time. The last one is used in the client-server architecture.
+- FLEXible defines the [`FlexPool`](https://github.com/FLEXible-FL/FLEXible/blob/main/flex/pool/pool.py) as the orchestrator of the federated scenario.
+- FLEXible provides its own [decorators](https://github.com/FLEXible-FL/FLEXible/blob/main/flex/pool/decorators.py) to define the federated functions. Also, FLEXible provides a set of primitives for different frameworks.[PyTorch primitives](https://github.com/FLEXible-FL/FLEXible/blob/main/flex/pool/primitives_pt.py) and [TensorFlow primitives](https://github.com/FLEXible-FL/FLEXible/blob/main/flex/pool/primitives_tf.py), that let the user adapt their centralized experiments to a federated scenario.
+- FLEXible algo provides some [aggregators](https://github.com/FLEXible-FL/FLEXible/blob/main/flex/pool/aggregators.py), such as FedAVG or WeightedFedAVG, but you can create your own aggregators.
 
-``
-    pip install -e ".[develop]"
-``
+## Installation
 
-With only tensorflow support:
+We recommend Anaconda/Miniconda as the package manager. To install the package, you can use the following commands:
 
-``
-    pip install -e ".[tf]"
-``
+### Using pip
 
-With only pytorch support:
+```bash
+pip install flexible-fl
+```
 
-``
-    pip install -e ".[pt]"
-``
+### Download the repository and install it locally
 
-Without support for any particular framework
+First download the repository:
 
-``
+```bash
+git clone git@github.com:FLEXible-FL/FLEXible.git
+cd FLEXible
+```
+
+Then, install the package:
+
+- Without support for any particular framework
+
+    ```bash
     pip install -e .
-``
+    ```
+
+- With only pytorch support:
+
+    ```bash
+    pip install -e ".[pt]"
+    ```
+
+- With only tensorflow support:
+
+    ```bash
+    pip install -e ".[tf]"
+    ```
+
+- In order to install this repo locally for development:
+
+    ```bash
+    pip install -e ".[develop]"
+    ```
+
+## Getting started
+
+To get started with **FLEXible**, you can check the [notebooks](https://github.com/FLEXible-FL/FLEXible/tree/main/notebooks) available in the repository. These notebooks have examples for how to federate data, or how to integrate deep learning frameworks such as PyTorch or TensorFlow. Some notebooks are:
+- [Create custom federated architectures using FLEXible](https://github.com/FLEXible-FL/FLEXible/blob/main/notebooks/Create%20custom%20architectures%20using%20FLEXible.ipynb).
+- [Federate data with FLEXible](https://github.com/FLEXible-FL/FLEXible/blob/main/notebooks/Federating%20data%20with%20FLEXible.ipynb).
+- [Federate a simple neural network with FLEXible using the FLEXible primitives and using the MNIST dataset](https://github.com/FLEXible-FL/FLEXible/blob/main/notebooks/Federated%20MNIST%20PT%20example%20with%20simplified%20primitives.ipynb).
+- [Federate a simple neural network with FLEXible using the FLEXible primitives and using the IMDb dataset](https://github.com/FLEXible-FL/FLEXible/blob/main/notebooks/Federated%20Text%20Classification%20TF%20with%20simplified%20primitives.ipynb).
+- [Vertical federated learning using FLEXible with a tabular dataset](https://github.com/FLEXible-FL/FLEXible/blob/main/notebooks/Vertical%20Federated%20Adult-income%20Classification%20PT.ipynb).
+
+## Documentation
+
+The documentation is available at the [ReadTheDocs repositoy](https://flexible.readthedocs.io/en/latest/index.html).
+
+To build the documentation you will need to `sphinx` and `sphinx-rtd-theme`. You can install them using pip:
+
+```bash
+pip install sphinx sphinx-rtd-theme
+```
+
+Once you have installed the dependencies, you can build the documentation using the following commands:
+
+```bash
+cd docs/
+make clean html # Generate the documentation in HTML format.
+```
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Citation
+
+If you use this package, please cite the following paper:
+
+``` TODO: Add citation ```
 
 ### Others
 **Authors order**: The order of the authors was chosen by asking ChatGPT to make a program to do it.
 
 The conversation was:
 
-Q: "Escribe en Python un programa que elija entre dos personas aleatoriamente, Alberto y Daniel, el orden entre estos dada una semilla aleatoria que tú elijas. Además, ejecuta el programa y dime el resultado."
+Q: "Write a program in Python that chooses between two people at random, Alberto and Daniel, the order between them given a random seed that you choose. Also, run the program and tell me the result."
 
-A: "
-Aquí tienes un programa en Python que selecciona aleatoriamente el orden entre dos personas, Alberto y Daniel, utilizando una semilla aleatoria predefinida.
+A: "Here is a Python program that randomly selects the order between two people, Alberto and Daniel, using a predefined random seed.
 
 ```python
 import random
 
-# Semilla aleatoria predefinida
+# Predefined random seed
 random.seed(123)
 
-# Lista con los nombres de las personas
-personas = ["Alberto", "Daniel"]
+# List with people's names
+people = ["Alberto", "Daniel"]
 
-# Elección aleatoria del orden
-orden = random.sample(personas, k=2)
+# Random choice of order
+order = random.sample(people, k=2)
 
-# Imprimir el resultado
-print("El orden seleccionado es:", orden)
+# Print the result
+print("The selected order is:", order)
 ```
 
-Al ejecutar este programa, el resultado puede variar dependiendo de la semilla aleatoria seleccionada. Sin embargo, dado que hemos establecido una semilla específica (`random.seed(123)`), el resultado será el mismo cada vez que se ejecute con esa semilla.
+When running this program, the result may vary depending on the random seed selected. However, since we have set a specific seed (`random.seed(123)`), the result will be the same every time it is run with that seed.
 
-Ejemplo de resultado:
+Example result:
 
-```
-El orden seleccionado es: ['Daniel', 'Alberto']
+```bash
+The selected order is: ['Daniel', 'Alberto']
 ```
 
-Recuerda que si cambias la semilla aleatoria o ejecutas el programa en otro momento, el resultado será diferente."
+Remember that if you change the random seed or run the program at another time, the result will be different."

docs/requirements.txt

@@ -0,0 +1,9 @@
+Sphinx==7.2.6
+sphinx-rtd-theme==2.0.0
+sphinxcontrib-applehelp==1.0.8
+sphinxcontrib-devhelp==1.0.6
+sphinxcontrib-htmlhelp==2.0.5
+sphinxcontrib-jquery==4.1
+sphinxcontrib-jsmath==1.0.1
+sphinxcontrib-qthelp==1.0.7
+sphinxcontrib-serializinghtml==1.1.10

docs/source/conf.py

@@ -1,3 +1,19 @@
+"""
+Copyright (C) 2024  Instituto Andaluz Interuniversitario en Ciencia de Datos e Inteligencia Computacional (DaSCI).
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU Affero General Public License as published
+    by the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU Affero General Public License for more details.
+
+    You should have received a copy of the GNU Affero General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+"""
 import os
 import sys
 
@@ -16,7 +32,7 @@
 
 project = "FLEXible"
 copyright = "2022, FLEXible Team"
-author = "FLEXible Team"
+author = "Jimenez-Lopez Daniel and Argente-Garrido Alberto"
 
 # -- General configuration ---------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
@@ -43,4 +59,11 @@
 
 # html_theme = "alabaster"
 html_theme = "sphinx_rtd_theme"
+
+# Would be nice to add the text color option to the theme, so the table in FedDatasetConfig
+# can be properly displayed.
+html_theme_options = {}
+
 html_static_path = ["_static"]
+# html_static_path = []
+html_logo = "_static/FLEXibleLogo.png"

docs/source/flex.actors.rst

@@ -12,6 +12,14 @@ flex.actors.actors module
    :undoc-members:
    :show-inheritance:
 
+flex.actors.architectures module
+---------------------------------
+
+.. automodule:: flex.actors.architectures
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 flex.actors.role module
 -----------------------
 

docs/source/flex.common.rst

@@ -0,0 +1,21 @@
+flex.common package
+===================
+
+Submodules
+----------
+
+flex.common.utils module
+------------------------
+
+.. automodule:: flex.common.utils
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+Module contents
+---------------
+
+.. automodule:: flex.common
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/flex.data.rst

@@ -8,6 +8,7 @@ flex.data.dataset module
 ------------------------
 
 .. automodule:: flex.data.dataset
+   :no-index:
    :members:
    :undoc-members:
    :show-inheritance:
@@ -32,6 +33,7 @@ flex.data.fed\_dataset\_config module
 -------------------------------------
 
 .. automodule:: flex.data.fed_dataset_config
+   :no-index:
    :members:
    :undoc-members:
    :show-inheritance:
@@ -52,14 +54,6 @@ flex.data.preprocessing\_utils module
    :undoc-members:
    :show-inheritance:
 
-flex.data.utils module
-----------------------
-
-.. automodule:: flex.data.utils
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
 Module contents
 ---------------
 

docs/source/flex.pool.rst

@@ -28,10 +28,18 @@ flex.pool.pool module
    :undoc-members:
    :show-inheritance:
 
-flex.pool.primitives module
----------------------------
+flex.pool.primitives_pt module
+------------------------------
+
+.. automodule:: flex.pool.primitives_pt
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+flex.pool.primitives_tf module
+------------------------------
 
-.. automodule:: flex.pool.primitives
+.. automodule:: flex.pool.primitives_tf
    :members:
    :undoc-members:
    :show-inheritance: