ENH(string dtype): fallback for HDF5 with UTF-8 surrogates #60993
+96
−39
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rhshadrach
added
IO HDF5
rhshadrach
commented
Feb 23, 2025
| @@ -5224,7 +5246,7 @@ def _convert_string_array(data: np.ndarray, encoding: str, errors: str) -> np.nd | |||
| # encode if needed | |||
| if len(data): | |||
| data = ( | |||
| Series(data.ravel(), copy=False) | |||
| Series(data.ravel(), copy=False, dtype="object") | |||
There was a problem hiding this comment.
We go immediately back to NumPy, so no reason to use string dtypes here.
rhshadrach
commented
Feb 23, 2025
Comment on lines
+5289
to
+5291
| ser = Series(data, copy=False).str.decode( | ||
| encoding, errors=errors, dtype="object" | ||
| ) |
There was a problem hiding this comment.
We go immediately back to NumPy, so no reason to use string dtypes here.
rhshadrach
commented
Feb 23, 2025
| and isinstance(values, np.ndarray) | ||
| and is_string_array(values, skipna=True) | ||
| ): | ||
| result = result.astype(StringDtype(na_value=np.nan)) |
There was a problem hiding this comment.
This was added back in #54431. I do not believe it is no longer necessary - we will infer string in the Series constructor.
rhshadrach
changed the title
rhshadrach
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.One oddity I encountered here: only the index is currently encoded / decoded on write / read operations respectively. Columns on the other hand are written and read as strings. I haven't looked into why this is, and if we can avoid encode/decode for index.
It seemed best to only fallback when
errors="surrogatepass", though that use is a bit odd since there is no actual decode occurring. If there are other approaches (perhaps always falling back?), I'm certainly open to them.Another option here is to fallback to
objectinstead ofStringDtype(storage="python"), but it seems withinfer_string=Truethe latter is more appropriate.