Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

source-postgres: use flat arrays rather than array objects w/ dimensions #2460

Open
willdonnelly opened this issue Feb 27, 2025 · 0 comments
Assignees
Labels
change:planned This is a planned change

Comments

@willdonnelly
Copy link
Member

willdonnelly commented Feb 27, 2025

Backstory:

Currently source-postgres captures array columns as a JSON object with dimensions and elements properties, each of which is an array. This is because PostgreSQL arrays are inherently multidimensional, and we're trying to preserve them as faithfully as possible.

(The JSON schema for a recursive "array whose elements can each be type X or another nested array" type is so incredibly terrible that it was never a realistic option, so "a flat array of values plus a list of array dimensions" was the best we could do to represent the original value faithfully)

But it turns out that nobody ever wants that, they just want simple, boring one-dimensional arrays of values to translate into a JSON array of equivalent values. So we should do that.

Plan:

  1. Add a feature flag flattened_arrays.
    a. When flattened_arrays is set, value translation will return just the elements array instead of the array-with-dimensions object.
    b. When flattened_arrays is set, discovered schemas will of course reflect that change in behavior.
  2. Set no_flattened_arrays on all source-postgres captures in production.
  3. Merge a followup PR which toggles the default for new captures.
@willdonnelly willdonnelly self-assigned this Feb 27, 2025
@willdonnelly willdonnelly added the change:planned This is a planned change label Feb 27, 2025
@willdonnelly willdonnelly changed the title source-postgres: use flat arrays rather than array objections w/ dimensions source-postgres: use flat arrays rather than array objects w/ dimensions Feb 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
change:planned This is a planned change
Projects
None yet
Development

No branches or pull requests

1 participant