v3.1

Bug Fixes

• fixed wrong version number for SQLite when checking how many parameters are allowed (see commit a60c61e)

Improvements

• when using pangres.upsert all operations will be done within a transaction
• when using pangres.upsert all operations will be done using a single connection

v3.0

New Features

• added option create_table to pangres.upsert for disabling table creation/check (CREATE TABLE IF NOT EXISTS... statement) that was always issued by this function. This can speed things up a little bit if you are sure that the target SQL table already exists
• pangres should already be compatible with the future sqlalchemy version 2 (unless something in the API changes in the future of course). You can try this by using the flag future=True in sqlalchemy.create_engine. For instance create_engine('sqlite:///', future=True)

Improvements

• allowed more SQL parameters for newer versions of SQLite (>=3.22.0): from 999 to 32766 (see issue #43)
• improved error messages (e.g. showing duplicated labels)

Bug Fixes

• when using parameter yield_chunks=True in pangres.upsert with an empty DataFrame we will now return an empty generator instead of None in order to ensure data type consistency
• fixed problem with log levels not being respected when pangres logs something (see commit #c494c95)
• fixed problem with the logger not filtering properly when the environment variable "PANGRES_LOG_LEVEL" is set (see commit #c494c95)

Breaking Changes

These changes are all related to new exceptions in pangres. If you weren't catching specific exceptions from pangres before this should not change anything for you.

• when create_schema=True, pangres.upsert will now raise the custom exception pangres.exceptions.HasNoSchemaSystemException if given database does not support schemas (AFAIK this only exist in PostgreSQL) but a schema was provided (schema is not None)
• in the presence of problematic column names (e.g. column names with "(", ")" or "%" for PostgreSQL) pangres.upsert will now raise the custom exception pangres.exceptions.BadColumnNamesException
• in the presence of duplicated labels (duplicates amongst columns/index levels or both) pangres.upsert and pangres.fix_psycopg2_bad_cols will now raise the custom exception pangres.exceptions.DuplicateLabelsException
• in the presence of unnamed index levels pangres.upsert and pangres.fix_psycopg2_bad_cols will now raise the custom exception pangres.exceptions.UnnamedIndexLevelsException
• in the presence of duplicates amongst the index values pangres.upsert will now raise the custom exception pangres.exceptions.DuplicateValuesInIndexException
• in pangres.upsert, when add_missing_columns is True but one of the columns to be added in the SQL table is part of the df's index, pangres will now raise the custom exception pangres.exceptions.MissingIndexLevelInSqlException

Documentation

• added notes on logging
• added a demo in the form of a notebook

Development

• rewrote the upsert module and separated the creation of the queries from their execution. The code should be easier to maintain and understand

Testing

• made it possible to test one or more databases types instead of always all of them
• added testing with future=True flag
• further improved coverage thanks to many new tests, ignoring coverage for tests (some functionalities of pytest caused coverage to be missed) and ignoring some lines
• added ids for tests to better read parameters used in pytest
• added context managers to ensure tables don't exist before and after tests in the database

v2.3.1

Bugfixes

• Having a column named values will not raise errors anymore when using pangres.upsert. See issue #34

v2.3

New Features

• Added yield_chunks parameter to the main function pangres.upsert. When True, this will yield the result of inserted chunks (sqlalchemy.engine.cursor.LegacyCursorResult objects). This allows you to notably count upserted rows for each chunk. See issue #32.

Improvements

• The context manager for the connection to the database to make upserts was being called unnecessarily early. I have now put it at the very last step to minimize the connection time. See commit 4573588

Bugfixes

• The value of the chunksize parameter was mistakenly being modified when upserting to a SQlite database because of a wrong indentation in the code. See commit da2e9aa

v2.2.4

Bugfixes

• pangres is now also compatible with sqlalchemy>=1.4. Important: make sure to get pandas>=1.2.4 as well to avoid deprecation warnings

v2.2.3

Bugfixes

• Fixed case where upsert was not possible with DataFrames that have no columns (only index) (see #26)
• Fixes failed release 2.2.2 (version commit was done afterwards)

v2.2.1

Bugfixes

• fixed logging issue where pangres' logging formatting would take over once imported (it would override your own configuration, see #20)
• fixed logging issue where pangres' logs would not be written to file (see #18)
• fixed conda environment file (the library npdoc_to_md has to be installed via pip, see commit 44d5423)

Improvements

• Added version attribute (see PR #22):

import pangres
pangres.__version__

2.2.1

v2.2

Breaking changes

• Removal of previously deprecated function pangres.pg_upsert (use pangres.upsert instead)
• In function pangres.upsert the arguments create_schema and add_new_columns have been set to False by default (they were previously both set to True by default) as this would be the common expectation (i.e. failing on missing schema or missing column). See #15. Thanks to @lsloan and @rajrohan.

v2.1

Bugfixes

• Fixed bug where the "ON...CONFLICT" statement would be repeated leading to a syntax error in SQL insert statements

Changes for developers

• Testing instructions changed
• Added necessary tools to generate the documentation (see folder pangres/docs)

v2

New features

• Added support for MySQL and SQlite 🎉 !
• Completely SQL injection safe (everything is escaped or parameterized including schema, table and column names)
• Documentation improved
• Logo added ! (I am not much of a graphic designer but I did my best 🙈)

Deprecations

• pangres.pg_upsert became pangres.upsert (to reflect the fact that pangres can now handle other databases than postgres). pangres.pg_upsert will be removed in the next version!
• the argument "if_exists" of the old function pangres.pg_upsert was removed in the new pangres.upsert in favor of the argument "if_row_exists" whose functionnality is clearer. The equivalent of if_exists="upsert_overwrite" is now if_row_exists="update" and if_exists="upsert_keep" is now if_row_exists="ignore"

Breaking changes

• Contrary to the old function pangres.pg_upsert the new pangres.upsert function does not clean "bad" column names automatically for postgres. An error will be raised if any column contains "%" or "(" or ")" but you can use pangres.fix_psycopg2_bad_cols to fix such problems.

Changes for developers

• Testing improved! You can provide a connection string in the command line directly 👍 !