pymongoarrow replaces existing codecs

[norrbom]

pymongoarrow replaces existing codecs

The api.write() method replaces any existing codecs.

The collection TypeRegistry is replaced with a new instance, effectively removing any existing custom codecs.
https://github.com/mongodb-labs/mongo-arrow/blob/1.6.0/bindings/python/pymongoarrow/api.py#L419

A related issue is that pymongoarrow uses pyarrow.to_pylist() to convert a pyarrow.Table to Python objects as an intermediate step before converting to raw BSON.

Pyarrow converts Arrow decimal128 types to Python Decimal objects. However, BSON cannot handle Decimal types, necessitating the use of a custom codec.
https://github.com/mongodb-labs/mongo-arrow/blob/1.6.0/bindings/python/pymongoarrow/api.py#L385

[ShaneHarvey]

Thanks, for reporting. I'm curious, how did you discover this? Specifically, what type codecs are you configuring (Decimal -> Decimal128, something else)? I'm wondering if those converters should be enabled by default or if we should really inherit the current TypeRegistry.

[blink1073]

I created https://jira.mongodb.org/browse/INTPYTHON-497, which will require a change to pymongo to enable extending an existing TypeRegistry.

[norrbom]

Thanks, for reporting. I'm curious, how did you discover this? Specifically, what type codecs are you configuring (Decimal -> Decimal128, something else)? I'm wondering if those converters should be enabled by default or if we should really inherit the current TypeRegistry.

My use case is to read Parquet and write to MongoDB.

This is what happens:

• Read from Parquet using pyarrow.parquet.read_table() -> pyarrow.Table
• PyMongoArrow converts Arrow to Python (via pyarrow.RecordBatch.to_pylist()). Decimal128 becomes Decimal(), which is correct
• PyMongoArrow converts Python to BSON using the Python BSON module → this is where it goes wrong; BSON cannot handle Decimal().

Yes, I believe it makes sense to have the Decimal → Decimal128 conversion as the default. Ideally, PyMongoArrow should manage all type conversions from Parquet to BSON.

Do you have any plans to open-source the Parquet-to-BSON conversion code (and vice versa) used in Atlas, its Go right?

[blink1073]

Hi @norrbom, I'd consider it a bug for any Parquet types to not be handled by PyMongoArrow. I opened https://jira.mongodb.org/browse/INTPYTHON-520 to track the bug. I can't speak to Atlas Data Federation, that's a whole different part of MongoDB 😄.

[blink1073]

In #278 we're adding support for the PyArrow Decimal128 type.