Merge pull request #11 from alalain/main

upgrades

.github/workflows/unit_test.yml

@@ -57,14 +57,14 @@ jobs:
 
     - name: Run tests with coverage
       run: |
-        coverage run -m pytest
+        coverage run -m pytest --ignore=tests/integration/mysql/base
         coverage xml
 
-    - name: Upload coverage to Codecov
-      uses: codecov/codecov-action@v4.0.1
-      with:
-        token: ${{ secrets.CODECOV_TOKEN }}
-        files: ./coverage.xml
-        flags: unittests
-        name: codecov-umbrella
-        fail_ci_if_error: true
+#    - name: Upload coverage to Codecov
+#      uses: codecov/codecov-action@v4.0.1
+#      with:
+#        token: ${{ secrets.CODECOV_TOKEN }}
+#        files: ./coverage.xml
+#        flags: unittests
+#        name: codecov-umbrella
+#        fail_ci_if_error: true

.gitignore

@@ -166,3 +166,10 @@ cython_debug/
 # Pypi configs
 .pypirc
 tests/unit/data/config.yaml
+
+data/
+configs/
+
+plots/plots
+plots/test
+plots/live/

README.md

@@ -23,10 +23,14 @@ This is a cli tool to extract table data from a fits file and write it down in a
 | Postgres |  under validation |
 
 ## Installation 
-For installation you can use pip
+For installation up to version 0.0.3 can be installed with pip
 ```bash 
 pip install fits2db
 ```
+For newer verions use the git repository
+```bash
+pip install git+https://github.com/pmodwrc/fits2db.git@main
+```
 check if you got the right version with 
 ```bash 
 fits2db --version
@@ -54,9 +58,13 @@ fits_files:
   paths:
     - path/to_your_file/2021-07-07_L1a.fits
     - path_to_your_folder
+
+# Delete rows from above listed files from tables which are not listed below
+delete_rows_from_missing_tables: True
+
 tables:
     - name: HOUSEKEEPING
-      target_table: RAW_HOUSEKEEPING # This will be the table_name in the db
+      date_column: timestamp # This column will be interpreted as a datetime variable
     - name: OTHER_TABLE # If no table name given it will use the orignal name
 ```
 
@@ -74,6 +82,14 @@ This will create the following tables:
 | OTHER_TABLE |   Contains the data of your fits files tables HOUSEKEEPING  merged|
 | OTHER_TABLE_META |  Contains the Column information from the fits files|
 
-
-
-
+To add new Files to a existing database use
+```
+fits2db update <path_to_config_yaml>
+```
+Files which are not yet in the database are added. 
+Already exisitng files are only updated, if their last change time is newer than
+the already existing one.
+To force the update of already uploaded files use
+```
+fits2db update <path_to_config_yaml> -f
+```

docs/contribution/contribution.md

@@ -6,7 +6,7 @@ There are many ways you can contribute to our project:
 
 - **Bug Reports**: Report issues you encounter on our [Issues page](https://github.com/pmodwrc/fits2db/issues).
 - **Feature Requests**: Suggest new features or improvements in our [Issues page](https://github.com/pmodwrc/fits2db/issues).
-- **Start a discussion**: If you have some discussion point or suggestions yo ucan start a [Discussion](https://github.com/pmodwrc/fits2db/discussions).
+- **Start a discussion**: If you have some discussion point or suggestions you can start a [Discussion](https://github.com/pmodwrc/fits2db/discussions).
 - **Documentation**: Help us improve or expand our documentation.
 - **Code**: Contribute bug fixes or implement new features.
 

docs/contribution/setup.md

@@ -4,7 +4,7 @@ To contribute to fits2db, follow these steps:
 
 1. Fork the repository by clicking "Fork" on the top right of this page.
 2. Clone your fork to your local machine
-3. Set up your local development environment.
+3. Set up your local development environment. 
 
 ## Docs setup
 ```bash

docs/contribution/tests.md

@@ -44,4 +44,5 @@ to show the results you can run either of these
     coverage html
     ```
 
+The tests of the base.py script are excluded in the github actions, but will run when pytest is called locally.
 

docs/user-guide/installation.md

@@ -3,7 +3,7 @@
 It is recommended to use a virtual environment when running this tool. Utilizing [`venv`](https://docs.python.org/3/library/venv.html) or [`conda`](https://www.anaconda.com/) ensures a clean and isolated environment, which is a best practice.
 
 ## Install from [`PYPI`](https://pypi.org/project/fits2db/)
-
+For Versions <= 0.0.3 fits2db can be installed from PYPI. Newer Versions must be installed from source
 
 !!! tip "You can use pip to install this library "
     ```bash 
@@ -24,8 +24,14 @@ Clone this repo to your local machine. Once cloned navigate to the `root` direct
 With this the `fits2db` lib should be installed. You can test if its properly installed by running the version command to check if you got the right version
 ```bash title="cmd"
 >fits2db --version
->>> fits2db, version 0.0.1
+>>> fits2db, version 0.1.0
+```
+
+Alternatively it can be installed from git with 
+```bash
+pip install git+https://github.com/pmodwrc/fits2db.git@main
 ```
+For this method a local [`git`](https://git-scm.com/downloads/win) client installation is required.
 
 **Happy coding :sparkles:**
 

docs/user-guide/overview.md

@@ -12,7 +12,9 @@
 ### Installation
 To install fits2db, follow these steps:
 
-1. Dependencies: Ensure you have Python installed on your system along with necessary libraries. You might need libraries like astropy for handling FITS files and SQLAlchemy for database interactions.
+1. Dependencies: Ensure you have Python installed on your system.
+    The Required libraries will be installed alongside fits2db.
+    These dependencies include libraries like astropy for handling FITS files and SQLAlchemy for database interactions. 
 
 2. Install fits2db:
 
@@ -39,11 +41,14 @@ Before using fits2db, you need to create a configuration yaml file. This file co
     paths:
         - tests\unit\data\2021-07-07_L1a.fits
         - test\unit\data # If just a path is given it will search recursive for fits files
+
+    delete_rows_from_missing_tables: False # [optional] by default false
 
     tables: # The tables from the fitsfiles you want to upload
         - name: HOUSEKEEPING
-        target_table: RAW_HOUSEKEEPING
+          date_column: timetsamp
         - name: JTSIM_BROADCAST
+          date_column: timestamp
 
     ```
 ### Configuration Parameters
@@ -57,12 +62,15 @@ Before using fits2db, you need to create a configuration yaml file. This file co
 
 - fits_files: A list of directories or paths containing your FITS files.
     - path: Path to the directory with FITS files.
+    - delete_rows_from_missing_tables: Sets wheter entries in tables not specified in the config on updated files are removed during the update of a file.
     - tables: Names of the tables to create and populate in the database.
+        - date_column: column in the table that contains a datetime value. This column will be uploaded to the database as type 'datetime'
 
 ## Usage
 ### Initial Setup
 
-To initialize the database and load data from the FITS files, use the rebuild command. This command creates the necessary tables in the SQL database and populates them with data from the FITS files.
+To initialize the database and load data from the FITS files, use the rebuild command. 
+This command creates the necessary tables in the SQL database and populates them with data from the FITS files.
 
 ## Additional Resources
 
@@ -77,4 +85,4 @@ fits2db is an open-source project licensed under the MIT License. Contributions
 - Create a new branch for your feature or bugfix.
 - Submit a pull request with a detailed description of your changes.
 
-For more details, see the [contribution guide](../contribution/contribution.md).
+For more details, see the [contribution guide](../contribution/contribution.md).

docs/user-guide/trouble.md

@@ -1,4 +0,0 @@
-# Troubleshooting
-!!! tipp
-    If you run the Raspberry Pi and it suddently shuts down. Check if the power supply has enough power! If it drains to much it can just shutdown. We recommend using the original power supply that runs at 5V@5A. 
-

docs/user-guide/usage.md

@@ -1,7 +1,7 @@
 # Usage
 
 ## __Getting help__
-If you need some information about a command you can always pass the `--help` flag to get some information
+If you need some information about a command you can always pass the `--help` flag to get some information.
 
 ## __Generate a config file__
 First you can generate a template config file:
@@ -43,15 +43,23 @@ fits_files:
   paths:
     - path/to_your_file/2021-07-07_L1a.fits
     - path_to_your_folder
+
+# Delete rows from above listed files from tables which are not listed below. By default False
+delete_rows_from_missing_tables: True
+
 tables:
     - name: HOUSEKEEPING
-      target_table: RAW_HOUSEKEEPING # This will be the table_name in the db
-    - name: JTSIM_BROADCAST # If no table name given it will use the orignal name
+      date_column: timestamp # This column will be interpreted as a datetime variable
+    - name: IRRADIANCE # If no table name given it will use the orignal name
+      date_column: irradiance_timeutc
 ```
 
 
 !!! note
-    if a folder is given all fits files under this folder will be taken for upload.
+    if a folder is given all fits files under this folder will be taken recursively for upload.
+
+!!! note 
+    if the date column is not 'timestamp', a copy of the date column called 'timestamp' is created. This is due to backward compatibility reasons.
 
 ## __Check if the right files are taken__
 You can check if you get the right fits files with 
@@ -88,17 +96,34 @@ Now upload the data into our data base we use the build command
 $ fits2db build <path_to_config_file> 
 ```
 this will upload all the fits tables into your data base and create the meta tables to keep track on changes of the files
+When running thei build command the user is prompted to confirm the action to remove all existing data from the configured database.
+If the user denies, the build command is aborted.
 
 !!! warning
-    If you rerun the build command it acts as an reset. it will drop the tables and reupload all data to have a fresh start. This is only recommend to use when you lost track of some changes in the data you have done manually and you are not sure you corrupted the data.
+    If you rerun the build command it acts as an reset.
+    It will drop the tables and reupload all data to have a fresh start.
+    This is only recommend to use when you lost track of some changes in the data you have done manually and you are not sure you corrupted the data.
 
 
 ## __Update db__
 
-Once builded and you get new files or changes you can update the database. This command will check if there a new files in your defnied folders and upload them to the db. If the timestamp of your file changed to a newer date. Like when you changed a file it will also update this file to the newer version. This way the fits files and the db stay in sync. To update just run 
+Once builded and you get new files or changes you can update the database. 
+This command will check if there a new files in your defnied folders and 
+upload them to the db. If the timestamp of your file changed to a newer
+date. Like when you changed a file it will also update this file to the 
+newer version. This way the fits files and the db stay in sync. To update just run 
 
 ```bash
 $ fits2db update <path_to_config_file> 
 ```
 !!! note
-    If you add a new table in your config file the update command will check trough the older files too if this table is in this file and upload accordingly.
+    If you want to add new tables from a already updated file, you can use the ```-f``` flag
+    to force update all files specified in the config.
+    This can for example be used to add additional tables from a already uploaded file to the database.
+
+
+### Remove files from tables
+With the `remove_rows_from_missing_tables` option, one can remove entries form columns.
+For example if a upladed file has entries in Table `a` and `b` and the update command is excecuted with only the 
+table `a` configured and `remove_rows_from_missing_tables` set to True, then the rows from all configured files from 
+table `b` will be removed.