Version 0.12.0

Major changes

All Processing Operators Updated to Column-Based Events

All processing operators now use the column-based event format, leading to better performance.
However, some sources and sinks for connectors still use the row-based event format.
Additionally, a few functions continue to rely on the row-based event format.

MongoDB Source Support

This release adds support to read data from MongoDB, this includes using
MongoDBs change stream to directly react on data changes.

SQL Server Support for Stream State Persistence

You can now store the stream state in SQL Server. For setup instructions, refer to the documentation:
https://koralium.github.io/flowtide/docs/statepersistence#sql-server-storage

Timestamp with Time Zone Data Type

A new data type for timestamps has been added.
This ensures that connectors can correctly use the appropriate data type, especially when writing.
For example, writing to MongoDB now uses the BSON Date type.

Minor Changes

Virtual Table Support

Static data selection is now supported. Example usage:

INSERT INTO output 
SELECT * FROM 
(
  VALUES 
  (1, 'a'),
  (2, 'b'),
  (3, 'c')
)

What's Changed

• Add better exception on persistent storage read failure by @Ulimo in #586
• Bump cross-spawn from 7.0.3 to 7.0.6 in /src/FlowtideDotNet.AspNetCore/ClientApp by @dependabot in #589
• Add mongoDB source by @Ulimo in #588
• Add new set operator based on column events by @Ulimo in #593
• Refactor buffer operator to use column data structure instead by @Ulimo in #590
• Fix bug in spicedb and fix tests by @Ulimo in #594
• Fix failing test in substrait by @Ulimo in #595
• Refactor timestamp provider to use column based events by @Ulimo in #603
• Refactor table function operator to use column based events by @Ulimo in #608
• Remove unwrap operator and relation by @Ulimo in #610
• Fix bugs in aggregate operator and null column by @Ulimo in #615
• Fix bug where a variable was reused and could contain the wrong information by @Ulimo in #617
• Add support to run substrait test files to check compliance with substrait functions by @Ulimo in #618
• Fix bug where emit with brackets caused an exception by @Ulimo in #619
• Add substrait test for substring and fix compliance with specification by @Ulimo in #620
• Add support for aggregate substrait tests with count, min and max to start by @Ulimo in #621
• Add substrait tests for concat trims and upper by @Ulimo in #622
• Add final substrait tests for implemented string functions by @Ulimo in #623
• Fix bug where a union containing map or list failed deserialization by @Ulimo in #624
• push emit list through buffer by @Ulimo in #626
• Add support for strpos by @Ulimo in #627
• Remove seed script from sql server to sql server sample by @Ulimo in #628
• Change to use concurrent dictionary in frontend cache by @Ulimo in #625
• Refactor TopN to use column based events by @Ulimo in #611
• Add substrait tests for comparison and boolean functions by @Ulimo in #629
• Add column based rounding functions by @Ulimo in #630
• Nested Block Join refactor and right and full outer support for joins by @Ulimo in #616
• Fix acceptance tests for set operator and add debug logging for block join by @Ulimo in #634
• add virtual table operator by @Ulimo in #632
• Validate that stop types exist in the zanzibar schema when converting by @Ulimo in #635
• Add elasticsearch column based sink by @Ulimo in #567
• Add support to use concat(..) in addition to || by @Ulimo in #638
• Add storage sqlserver by @bpfz in #591
• Add Sql Server Storage solution to nuget release by @Ulimo in #645
• Refactor mongodb sink to use column based format by @Ulimo in #640
• Add support for serializers to write and read metadata pages by @Ulimo in #643
• Change spicedb connector to use column based events by @Ulimo in #647
• Add test case for mongodb where primary key is not in position 0 by @Ulimo in #648
• Add new data type column for timestamp with offset by @Ulimo in #646
• Change sql server connector to use column based events in the sink by @Ulimo in #649
• Update SQL Server storage documentation by @bpfz in #651
• [Snyk] Security upgrade next from 14.2.13 to 14.2.21 by @Ulimo in #652
• Bump nanoid from 3.3.7 to 3.3.8 in /src/FlowtideDotNet.AspNetCore/ClientApp by @dependabot in #641
• Bump next from 14.2.13 to 14.2.21 in /src/FlowtideDotNet.AspNetCore/ClientApp by @dependabot in #650
• Bump nanoid from 3.3.7 to 3.3.8 in /docs by @dependabot in #654
• Bump FluentAssertions from 6.12.0 to 7.0.0 by @dependabot in #633
• Add better logging for fasterkv storage where logger is injected by @Ulimo in #655
• Fix timestamp size to include alignment, also fix timestamp in union by @Ulimo in #656
• Bug fix: fix so save page saves page correctly if resize is required by @Ulimo in #657
• Add that memory allocated by data columns are reduced if the usage is 2.5 less than allocated by @Ulimo in #660
• Reenable the append tree and change metrics to use the append tree by @Ulimo in #661
• Fix InsertAt in map with null values by @Ulimo in #663
• change sharepoint source to retry on task cancelled by @Ulimo in #662
• Add search to docs, fixes #659 by @Ulimo in #664
• Fix failing tests for append tree and mongodb by @Ulimo in #665
• Remove dependency on FluentAssertions by @Ulimo in #666
• Bug fix: fix union deserialize special case with only null values by @Ulimo in #667
• Add support for binary literals by @Ulimo in #668
• Update for version 0.12.0 by @Ulimo in #669

Full Changelog: v0.11.2...v0.12.0

Version 0.12.0 alpha 8

What's Changed

• Add search to docs, fixes #659 by @Ulimo in #664
• Fix failing tests for append tree and mongodb by @Ulimo in #665
• Remove dependency on FluentAssertions by @Ulimo in #666
• Bug fix: fix union deserialize special case with only null values by @Ulimo in #667

Full Changelog: v0.12.0-alpha7...v0.12.0-alpha8

Version 0.12.0 alpha 7

What's Changed

• Bug fix: fix so save page saves page correctly if resize is required by @Ulimo in #657
• Add that memory allocated by data columns are reduced if the usage is 2.5 less than allocated by @Ulimo in #660
• Reenable the append tree and change metrics to use the append tree by @Ulimo in #661
• Fix InsertAt in map with null values by @Ulimo in #663
• change sharepoint source to retry on task cancelled by @Ulimo in #662

Full Changelog: v0.12.0-alpha6...v0.12.0-alpha7

Version 0.12.0 alpha 6

What's Changed

• Add better logging for fasterkv storage where logger is injected by @Ulimo in #655
• Fix timestamp size to include alignment, also fix timestamp in union by @Ulimo in #656

Full Changelog: v0.12.0-alpha5...v0.12.0-alpha6

Version 0.12.0 alpha 5

What's Changed

• Change spicedb connector to use column based events by @Ulimo in #647
• Add test case for mongodb where primary key is not in position 0 by @Ulimo in #648
• Add new data type column for timestamp with offset by @Ulimo in #646
• Change sql server connector to use column based events in the sink by @Ulimo in #649
• Update SQL Server storage documentation by @bpfz in #651
• [Snyk] Security upgrade next from 14.2.13 to 14.2.21 by @Ulimo in #652
• Bump nanoid from 3.3.7 to 3.3.8 in /src/FlowtideDotNet.AspNetCore/ClientApp by @dependabot in #641
• Bump next from 14.2.13 to 14.2.21 in /src/FlowtideDotNet.AspNetCore/ClientApp by @dependabot in #650
• Bump nanoid from 3.3.7 to 3.3.8 in /docs by @dependabot in #654
• Bump FluentAssertions from 6.12.0 to 7.0.0 by @dependabot in #633

Full Changelog: v0.12.0-alpha4...v0.12.0-alpha5

Version 0.12.0 alpha 4

What's Changed

• Add support for strpos by @Ulimo in #627
• Remove seed script from sql server to sql server sample by @Ulimo in #628
• Change to use concurrent dictionary in frontend cache by @Ulimo in #625
• Refactor TopN to use column based events by @Ulimo in #611
• Add substrait tests for comparison and boolean functions by @Ulimo in #629
• Add column based rounding functions by @Ulimo in #630
• Nested Block Join refactor and right and full outer support for joins by @Ulimo in #616
• Fix acceptance tests for set operator and add debug logging for block join by @Ulimo in #634
• add virtual table operator by @Ulimo in #632
• Validate that stop types exist in the zanzibar schema when converting by @Ulimo in #635
• Add elasticsearch column based sink by @Ulimo in #567
• Add support to use concat(..) in addition to || by @Ulimo in #638
• Add storage sqlserver by @bpfz in #591
• Add Sql Server Storage solution to nuget release by @Ulimo in #645
• Refactor mongodb sink to use column based format by @Ulimo in #640
• Add support for serializers to write and read metadata pages by @Ulimo in #643

Full Changelog: v0.12.0-alpha3...v0.12.0-alpha4

Version 0.12.0 alpha 3

What's Changed

• Add substrait test for substring and fix compliance with specification by @Ulimo in #620
• Add support for aggregate substrait tests with count, min and max to start by @Ulimo in #621
• Add substrait tests for concat trims and upper by @Ulimo in #622
• Add final substrait tests for implemented string functions by @Ulimo in #623
• Fix bug where a union containing map or list failed deserialization by @Ulimo in #624
• push emit list through buffer by @Ulimo in #626

Full Changelog: v0.12.0-alpha2...v0.12.0-alpha3

Version 0.12.0 alpha 2

What's Changed

• Fix bug where a variable was reused and could contain the wrong information by @Ulimo in #617
• Add support to run substrait test files to check compliance with substrait functions by @Ulimo in #618
• Fix bug where emit with brackets caused an exception by @Ulimo in #619

Full Changelog: v0.12.0-alpha1...v0.12.0-alpha2

Version 0.12.0 alpha 1

What's Changed

• Add better exception on persistent storage read failure by @Ulimo in #586
• Bump cross-spawn from 7.0.3 to 7.0.6 in /src/FlowtideDotNet.AspNetCore/ClientApp by @dependabot in #589
• Add mongoDB source by @Ulimo in #588
• Add new set operator based on column events by @Ulimo in #593
• Refactor buffer operator to use column data structure instead by @Ulimo in #590
• Fix bug in spicedb and fix tests by @Ulimo in #594
• Fix failing test in substrait by @Ulimo in #595
• Refactor timestamp provider to use column based events by @Ulimo in #603
• Refactor table function operator to use column based events by @Ulimo in #608
• Remove unwrap operator and relation by @Ulimo in #610
• Fix bugs in aggregate operator and null column by @Ulimo in #615

Full Changelog: v0.11.2...v0.12.0-alpha1

Version 0.11.2

Minor changes

Fix so a SqlServerSource can be limited to only use the database defined in the connection string.

Pull requests

• Bump cross-spawn from 7.0.3 to 7.0.5 in /docs by @dependabot in #584
• [Snyk] Fix for 1 vulnerabilities by @Ulimo in #573
• Bump AspNetCore.HealthChecks.Publisher.ApplicationInsights from 8.0.0 to 8.0.1 by @dependabot in #564
• [Snyk] Upgrade reactflow from 11.10.4 to 11.11.0 by @Ulimo in #448
• Bump Bogus from 35.4.0 to 35.6.1 by @dependabot in #509
• Fix issue where changing db throws access exception in table provider by @Ulimo in #587

Full Changelog: v0.11.1...v0.11.2