Release Spark Connector 2.8.4

Added some pushdown supports and fixed some critical issues.

1. Fixed an issue that occurred when writing a DataFrame with Map type column if the name of the column contains spaces.
2. Fixed an issue that occurred when writing to a table with overwrite mode if the schema name isn't specified as part of the table name, the table is not in the schema specified by the “sfSchema” option, and the PUBLIC schema contains a table with the same name.
3. Added pushdown support for the COALESCE() function when reading from the Snowflake database.
4. Added pushdown support for the windows functions of RANK() and DENSE_RANK() when reading from the Snowflake database.
5. Updated the connector to use the latest version of the Snowflake JDBC Driver (3.12.17).
6. Updated the connector to use the latest version of the Snowflake Ingest SDK (0.10.1).

Release Spark Connector 2.8.3

Fixed some critical issues and added more test cases for stability.

1. Reduced the amount of memory needed when uploading large partitions of data to the Snowflake database.
2. Fixed an issue with pushdowns when the Spark aggregation function AGG() was used without groups and functions.
3. Fixed an issue that occurred when using the overwrite mode with a table in a different schema than the one specified by “sfSchema”. The issue occurred when the user had no CREATE TABLE privilege in the schema specified by "sfSchema".
4. Fixed an issue with pushdowns where NULLs were incorrectly converted to the default values for value-type objects.
5. Use the latest version of the JDBC Driver (3.12.15).

Snowflake For Spark Connector 2.8.2 Release Notes

Fixed some critical issues and added test cases for stability.

1. Fixed an issue that occurred when reading a large result set with Arrow format from a Snowflake database through AWS PrivateLink.
2. Added support for quoted table names that contain special characters when writing to the Snowflake database.
3. Fixed an issue with writing data to the Snowflake database via an external stage.
4. Fix an issue with pushdowns and the InSet expression when reading from the Snowflake database.
5. Use the latest version of the JDBC Driver (3.12.12).

Note:

1. Spark Connector 2.8.2 is NOT compatible with version 3.12.11 (and earlier versions) of the JDBC Driver.
2. On Linux, when retrieving the Snowflake GPG public key to verify the Snowflake Connector for Spark package signature, use the GPG key ID 37C7086698CB005C.

Release Spark Connector 2.8.1

Do some enhancements and bug fixes.

1. Support nano second for writing Timestamp to snowflake
2. Reduce memory usage for writing to snowflake tables to avoid Out-Of-Memory issue.
3. Don’s send “spark_plan” telemetry message.
4. Support to pushdown for “Case when/otherwise” statement to snowflake.
5. Use the latest version of the JDBC Driver (3.12.8).

Release Spark Connector 2.8.0 for spark 3.0 support

Support spark 3.0 and some bug fixes.

1. Fix one data-loss issue when writing data to snowflake table.
2. Fix one issue that the returned ResultSet for Utils.runQuery() is always empty.
3. Use JDBC 3.12.8

Note:

1. Spark 3.0 is supported, but Spark 3.0 preview/preview2 are not compatible.
2. No binary is released for spar 2.2 from spark connector 2.8.0.

Release Spark Connector 2.7.2

Release Spark Connector for some critical enhancements and bug fixes.

1. Support pushdown for length()/trunc()/date_trunc()
2. Log WARN messages if the runtime JDBC version is not the certified JDBC version.
3. Send some telemetry messages for statistic and diagnostic purpose.
4. Use JDBC 3.12.7

Release Spark Connector 2.7.1

Release Spark Connector for some critical enhancements and bug fixes.

1. Support to write Spark BinaryType column to Snowflake Binary type column.
2. Log diagnostic information when writing to snowflake.
3. Log diagnostic information for Snowflake Azure deployment accounts.
4. Implement backoff and retry when uploading data to snowflake internal stage to tolerate some intermediate network failures or cloud storage service throttling.
5. Allow Spark Connector option: “sfurl” to begin with "https://“.
6. Close the JDBC connection created in Utils.runQuery().
7. Use JDBC 3.12.5.

NOTE:
DO NOT use JDBC 3.12.4 with any Spark Connector because of the compatibility issue.

Release Spark Connector 2.7.0

Spark Connector is enhanced to support Snowflake GCP account and OAuth authentication. This release also includes below minor enhancements and bug fixes:

• Log diagnostic statistic data size in pretty format
• Use JDBC 3.12.2

Release Spark Connector 2.6.0

Spark Connector is enhanced to leverage Snowflake internal arrow result format feature for excellent read performance. This release also includes below minor enhancements and bug fixes:

1. Enable column name to support dot(.) character.
2. Support special values of Inf/-Inf/NaN for Double and Float type.
3. Move to use latest JDBC driver 3.12.1

Release snowflake spark connector 2.5.9

Upgrade to use JDBC 3.12.0
Introduce Max File Count Per Partition to avoid Out-Of-Memory exception when reading big data from snowflake.
Enable pushdown for FULL OUTER JOIN to snowflake.
Wrap TRUNCATE TABLE and COPY INTO statements in one transaction when writing data to snowflake in Overwrite mode, "usestagingtable" is "false" and "truncate_table" is "on".
Fix "Unparseable number exceptions" issue when reading from snowflake.