Merge pull request #67 from ThibTrip/add_compatibility_for_sqlalchemy2

[FEAT] Add compatibility for sqlalchemy2

Author: ThibTrip

.circleci/config.yml

@@ -29,6 +29,13 @@ jobs:
           python3 -m venv venv
           . venv/bin/activate
           pip install -r requirements.txt
+
+    - run:
+        name: check linting with flake8
+        command: |
+          pip install flake8
+          flake8 . --exclude venv
+
     - save_cache:
         paths:
         - ./venv
@@ -40,14 +47,17 @@ jobs:
           # install package (fetches setup.py in current directory)
           pip install .
           # we need cryptography for MySQL
-          pip install codecov coverage numpy pytest pytest-benchmark pytest-cov
+          pip install codecov coverage flake8 numpy pytest pytest-benchmark pytest-cov
           pip install aiosqlite aiomysql asyncpg psycopg2 pymysql cx_Oracle cryptography tabulate npdoc_to_md
           # use pytest
-          ## first test with sqlalchemy latest i.e. sqlalchemy>=1.4 (after API changes, notably engine.has_table being deprecated)
+          ## first test with sqlalchemy latest i.e. sqlalchemy==2
           pytest -sxv pangres --cov=pangres --doctest-modules --sqlite_conn=sqlite:// --async_sqlite_conn=sqlite+aiosqlite:///test.db --pg_conn=postgresql://circleci_user:password@localhost:5432/circleci_test?sslmode=disable --async_pg_conn=postgresql+asyncpg://circleci_user:password@localhost:5432/circleci_test --mysql_conn=mysql+pymysql://circleci_user:password@127.0.0.1:3306/circleci_test --async_mysql_conn=mysql+aiomysql://circleci_user:password@127.0.0.1:3306/circleci_test --benchmark-group-by=func,param:engine,param:nb_rows --benchmark-columns=min,max,mean,rounds --benchmark-sort=name --benchmark-name=short
           ## second test with sqlalchemy<1.4 (before API changes)
           pip install sqlalchemy==1.3.24
           pytest -sxv pangres --cov=pangres --cov-append --doctest-modules --sqlite_conn=sqlite:// --pg_conn=postgresql://circleci_user:password@localhost:5432/circleci_test?sslmode=disable --mysql_conn=mysql+pymysql://circleci_user:password@127.0.0.1:3306/circleci_test --benchmark-group-by=func,param:engine,param:nb_rows --benchmark-columns=min,max,mean,rounds --benchmark-sort=name --benchmark-name=short
+          ## third test with sqlalchemy==1.4.46
+          pip install sqlalchemy==1.4.46
+          pytest -sxv pangres --cov=pangres --cov-append --doctest-modules --sqlite_conn=sqlite:// --pg_conn=postgresql://circleci_user:password@localhost:5432/circleci_test?sslmode=disable --mysql_conn=mysql+pymysql://circleci_user:password@127.0.0.1:3306/circleci_test --benchmark-group-by=func,param:engine,param:nb_rows --benchmark-columns=min,max,mean,rounds --benchmark-sort=name --benchmark-name=short
           codecov
 workflows:
   version: 2

.flake8

@@ -0,0 +1,4 @@
+[flake8]
+ignore = E731
+max-line-length = 120
+

.gitignore

@@ -1,4 +1,5 @@
-# Jupyter LSP
+# Custom
+.idea
 .virtual_documents
 
 # Byte-compiled / optimized / DLL files

README.md

@@ -10,7 +10,7 @@
 _Thanks to [freesvg.org](https://freesvg.org/) for the logo assets_
 
 Upsert with pandas DataFrames (<code>ON CONFLICT DO NOTHING</code> or <code>ON CONFLICT DO UPDATE</code>) for PostgreSQL, MySQL, SQlite and potentially other databases behaving like SQlite (untested) with some additional optional features (see features). Upserting can be done with **primary keys** or **unique keys**.
-Pangres also handles the creation of non existing SQL tables and schemas.
+Pangres also handles the creation of non-existing SQL tables and schemas.
 
 
 # Features
@@ -31,10 +31,22 @@ Pangres also handles the creation of non existing SQL tables and schemas.
 * Python >= 3.6.4
 * See also ./pangres/requirements.txt
 
+## Requirements for sqlalchemy>=2.0
+
+For using `pangres` together with **`sqlalchemy>=2.0`** (sqlalchemy is one of pangres dependencies
+listed in requirements.txt) - you will need the following base requirements:
+* `alembic>=1.7.2`
+* `pandas>=1.4.0`
+* Python >= 3.8 (`pandas>=1.4.0` only supports Python >=3.8)
+
+## Requirements for asynchronous engines
+
+For using asynchronous engines (such as `aiosqlite`, `asyncpg` or `aiomysql`) you will need **Python >= 3.8**.
+
 # Gotchas and caveats
 
 ## All flavors
-1. We can't create JSON columns automatically but we can insert JSON like objects (list, dict) in existing JSON columns.
+1. We can't create JSON columns automatically, but we can insert JSON like objects (list, dict) in existing JSON columns.
 
 ## Postgres
 
@@ -102,10 +114,10 @@ Note:
 
 The wiki is generated with a command which uses my library [npdoc_to_md](https://github.com/ThibTrip/npdoc_to_md).
 It must be installed with `pip install npdoc_to_md` and you will also need the extra dependency `fire` which you
-can install with `pip install fire`.
+can install with `pip install fire`. Replace `$DESTINATION_FOLDER` with the folder of you choice in the command below:
 
 ```bash
-npdoc-to-md render-folder ./wiki/templates ./wiki
+npdoc-to-md render-folder ./wiki_templates $DESTINATION_FOLDER
 ```
 
 # Contributing
@@ -124,6 +136,8 @@ thanks to [**nb_conda_kernels**](https://github.com/Anaconda-Platform/nb_conda_k
 
 # Testing
 
+## Pytest
+
 You can test one or multiple of the following SQL flavors (you will of course need a live database for this): PostgreSQL, SQlite or MySQL.
 
 NOTE: in one of the tests of `pangres` we will try to drop and then create a PostgreSQL schema called `pangres_create_schema_test`. If the schema existed and was not empty an error will be raised.
@@ -156,3 +170,15 @@ Additionally, the following flags could be of interest for you:
 * `-x` for stopping at the first failure
 * `--benchmark-only` for only testing benchmarks
 * `--benchmark-skip` for skipping benchmarks
+
+## flake8
+
+flake8 must run without errors for pipelines to succeed.
+If you are not using the conda environment, you can install flake8 with: `pip install flake8`.
+
+To test flake8 locally you can simply execute this command:
+
+```
+flake8 .
+```
+

demos/gotchas_asynchronous_pangres.ipynb

@@ -1309,7 +1309,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.7"
+   "version": "3.11.0"
   }
  },
  "nbformat": 4,

demos/pangres_demo.ipynb

@@ -599,7 +599,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.7"
+   "version": "3.11.0"
   }
  },
  "nbformat": 4,

environment.yml

@@ -1,15 +1,20 @@
 name: pangres-dev
+channels:
+  - conda-forge
 dependencies:
   - pip
-  - pytest
-  - pytest-cov
-  - pytest-benchmark
   - psycopg2
-  - pymysql
-  - tabulate
   - pip:
     - asyncpg
     - aiosqlite
     - aiomysql
     - cx_Oracle
-    - npdoc_to_md
+    - cryptography
+    - flake8
+    - mypy
+    - npdoc_to_md
+    - pymysql
+    - pytest
+    - pytest-benchmark
+    - pytest-cov
+    - tabulate

pangres/__init__.py

@@ -1,5 +1,8 @@
-from pangres.core import aupsert, upsert
-from pangres.utils import adjust_chunksize, fix_psycopg2_bad_cols
-from pangres.examples import DocsExampleTable
-from pangres._version import __version__
-from pangres.exceptions import *
+from pangres.core import aupsert, upsert  # noqa: F401
+from pangres.utils import adjust_chunksize, fix_psycopg2_bad_cols  # noqa: F401
+from pangres.examples import DocsExampleTable  # noqa: F401
+from pangres._version import __version__  # noqa: F401
+from pangres.exceptions import (BadColumnNamesException, HasNoSchemaSystemException,  # noqa: F401
+                                UnnamedIndexLevelsException,  # noqa: F401
+                                DuplicateValuesInIndexException, DuplicateLabelsException,  # noqa: F401
+                                MissingIndexLevelInSqlException, TooManyColumnsForUpsertException)  # noqa: F401

pangres/_version.py

@@ -1 +1 @@
-__version__ = "4.1.2"
+__version__ = "4.1.3"

pangres/core.py

@@ -6,30 +6,31 @@
 that will be directly exposed to its users.
 """
 import pandas as pd
-from sqlalchemy.engine.base import Connectable
-from typing import Optional, Union
+from sqlalchemy.engine import Connectable
+from typing import Union
 
 # local imports
 from pangres.executor import Executor
 from pangres.helpers import validate_chunksize_param
+from pangres.pangres_types import AsyncConnectable, AUpsertResult, UpsertResult
 
 
 # -
 
 # # upsert
 
-def upsert(con:Connectable,
-           df:pd.DataFrame,
-           table_name:str,
-           if_row_exists:str,
-           schema:Optional[str]=None,
-           create_schema:bool=False,
-           create_table:bool=True,
-           add_new_columns:bool=False,
-           adapt_dtype_of_empty_db_columns:bool=False,
-           chunksize:Optional[int]=None,
-           dtype:Union[dict,None]=None,
-           yield_chunks:bool=False):
+def upsert(con: Connectable,
+           df: pd.DataFrame,
+           table_name: str,
+           if_row_exists: str,
+           schema: Union[str, None] = None,
+           create_schema: bool = False,
+           create_table: bool = True,
+           add_new_columns: bool = False,
+           adapt_dtype_of_empty_db_columns: bool = False,
+           chunksize: Union[int, None] = None,
+           dtype: Union[dict, None] = None,
+           yield_chunks: bool = False) -> UpsertResult:
     """
     Insert updates/ignores a pandas DataFrame into a SQL table (or
     creates a SQL table from the DataFrame if it does not exist).
@@ -210,7 +211,7 @@ def upsert(con:Connectable,
     ...        if_row_exists='update',
     ...        dtype=dtype,
     ...        create_table=False)
-    >>> 
+    >>>
     >>> # Now we read from the database to check what we got and as you can see
     >>> # John Travolta was updated and Arnold Schwarzenegger was added!
     >>> with engine.connect() as connection:
@@ -299,32 +300,33 @@ def upsert(con:Connectable,
     # execute SQL operations
     if not yield_chunks:
         executor.execute(connectable=con, if_row_exists=if_row_exists, chunksize=chunksize)
+        return None
     else:
         return executor.execute_yield(connectable=con, if_row_exists=if_row_exists, chunksize=chunksize)
 
 
 # # Async upsert
 
-async def aupsert(con,
-                  df:pd.DataFrame,
-                  table_name:str,
-                  if_row_exists:str,
-                  schema:Optional[str]=None,
-                  create_schema:bool=False,
-                  create_table:bool=True,
-                  add_new_columns:bool=False,
-                  adapt_dtype_of_empty_db_columns:bool=False,
-                  chunksize:Optional[int]=None,
-                  dtype:Union[dict,None]=None,
-                  yield_chunks:bool=False):
+async def aupsert(con: AsyncConnectable,
+                  df: pd.DataFrame,
+                  table_name: str,
+                  if_row_exists: str,
+                  schema: Union[str, None] = None,
+                  create_schema: bool = False,
+                  create_table: bool = True,
+                  add_new_columns: bool = False,
+                  adapt_dtype_of_empty_db_columns: bool = False,
+                  chunksize: Union[int, None] = None,
+                  dtype: Union[dict, None] = None,
+                  yield_chunks: bool = False) -> AUpsertResult:
     """
     Asynchronous variant of `pangres.upsert`. Make sure to read its docstring
     before using this function!
 
     The parameters of `pangres.aupsert` are the same but parameter `con`
     will require an asynchronous connectable (asynchronous engine or asynchronous connection).
 
-    For example you can use PostgreSQL asynchronously with `sqlalchemy` thanks to
+    For example, you can use PostgreSQL asynchronously with `sqlalchemy` thanks to
     the library/driver `asyncpg`, or SQLite with `aiosqlite` or Mysql with `aiomysql`.
 
     **WARNING**
@@ -389,12 +391,12 @@ async def aupsert(con,
     >>> df = DocsExampleTable.df
     >>>
     >>> # Create table before inserting! This will avoid race conditions mentionned above
-    >>> # (here we are lazy so we'll use pangres to do that but we could also use a sqlalchemy ORM model)
-    >>> # By using `df.head(0)` we get 0 rows but we have all the information about columns, index levels
+    >>> # (here we are lazy, so we'll use pangres to do that, but we could also use a sqlalchemy ORM model)
+    >>> # By using `df.head(0)` we get 0 rows, but we have all the information about columns, index levels
     >>> # and data types that we need for creating the table.
     >>> # And in a second step (see coroutine `execute_upsert` that we define after)
     >>> # we will set all parameters that could cause structure changes
-    >>> # to False so we can run queries in parallel without worries!
+    >>> # to False, so we can run queries in parallel without worries!
     >>> async def setup():
     ...     await aupsert(con=engine, df=df.head(0),
     ...                   table_name='example',
@@ -455,6 +457,7 @@ async def aupsert(con,
     # execute SQL operations
     if not yield_chunks:
         await executor.aexecute(async_connectable=con, if_row_exists=if_row_exists, chunksize=chunksize)
+        return None
     else:
         # IMPORTANT! NO `await` because this returns an asynchronous generator
         return executor.aexecute_yield(async_connectable=con, if_row_exists=if_row_exists, chunksize=chunksize)

pangres/docs/fix_changelog.py

@@ -4,23 +4,27 @@
 
 Execute the command below first. Make sure to replace the following variables:
 * $PATH_TO_PANGRES -> path to pangres repo on your computer (you have to clone it)
-* `-t $GITHUB_TOKEN` -> optionally give a github token (for much higher API quota)
+* `-t $GITHUB_TOKEN` -> optionally give a GitHub token (for much higher API quota)
 * $OUTPUT_PATH -> where to put the CHANGELOG.md file
 
-sudo docker run -it --rm -v "$(pwd)":$PATH_TO_PANGRES githubchangeloggenerator/github-changelog-generator -u ThibTrip -p pangres -t $GITHUB_TOKEN -o $OUTPUT_PATH --release-url https://github.com/ThibTrip/pangres/releases/tag/%s
+sudo docker run -it --rm -v "$(pwd)":$PATH_TO_PANGRES githubchangeloggenerator/github-changelog-generator\
+-u ThibTrip -p pangres -t $GITHUB_TOKEN -o $OUTPUT_PATH\
+--release-url https://github.com/ThibTrip/pangres/releases/tag/%s
 
 Usage:
 
 python fix_changelog.py $PATH_TO_CHANGELOG
 """
 import argparse
+import logging
 import re
 import sys
 from pathlib import Path
 
-# # Helpers
 
-# +
+logging.basicConfig(level=logging.INFO, format='%(asctime)s %(message)s')
+
+# region helpers
 re_section_release_notes = re.compile(r'^# [A-Z]{1,}')  # e.g. "# New Features" -> "# N"
 re_release_title_md = re.compile(r'## \[(?P<version>v[\d\.]+)\]')  # see https://regex101.com/r/g6yRM8/1
 
@@ -43,12 +47,10 @@ def get_release_notes(github_token=None):
     kwargs = dict(headers={'Authorization': f'token {github_token}'}) if github_token else {}
     response = requests.get('https://api.github.com/repos/ThibTrip/pangres/releases', **kwargs)
     response.raise_for_status()
-    return {d['tag_name']:adjust_levels_release_notes(d['body']) for d in response.json()}
+    return {d['tag_name']: adjust_levels_release_notes(d['body']) for d in response.json()}
 
 
 def add_release_notes_to_changelog(filepath, github_token=None, dryrun=False):
-    from loguru import logger  # pip install loguru
-
     # open original file
     with open(filepath, mode='r', encoding='utf-8') as fh:
         ch = fh.read()
@@ -69,10 +71,10 @@ def add_release_notes_to_changelog(filepath, github_token=None, dryrun=False):
         version = match_version['version']
         try:
             notes = release_notes[version]
-            logger.info(f'Adding release notes for version {version}')
+            logging.info(f'Adding release notes for version {version}')
             new_ch.extend([line, '\n', '**Release Notes**', '\n', '___', notes, '___', '\n'])
         except KeyError:
-            logger.warning(f'No release notes found for version {version}!')
+            logging.warning(f'No release notes found for version {version}!')
             continue
     new_ch = '\n'.join(new_ch)
     if not dryrun:
@@ -81,19 +83,17 @@ def add_release_notes_to_changelog(filepath, github_token=None, dryrun=False):
     else:
         print(new_ch)
     return new_ch
+# endregion
 
 
-# -
-
-# # Main
-
-# +
 def main():
     # parse arguments
     parser = argparse.ArgumentParser(description=sys.modules['__main__'].__doc__)
     parser.add_argument('filepath_change_log', metavar='filepath_change_log', type=str, help="Path to the changelog")
-    parser.add_argument('--github_token', action="store", type=str, default=None, help='Optional github token for higher API quota')
-    parser.add_argument('--dryrun', action="store_true", default=False, help='If True, simply prints what we would save otherwise overwrites the changelog')
+    parser.add_argument('--github_token', action="store", type=str, default=None,
+                        help='Optional github token for higher API quota')
+    parser.add_argument('--dryrun', action="store_true", default=False,
+                        help='If True, simply prints what we would save otherwise overwrites the changelog')
     args = parser.parse_args()
     add_release_notes_to_changelog(filepath=Path(args.filepath_change_log).resolve(), github_token=args.github_token,
                                    dryrun=args.dryrun)