Skip to content

Commit

Permalink
Bug: Fix convert_dtypes not preserving timezone details for ArrowDtype (
Browse files Browse the repository at this point in the history
#60952)

* modified the files according to bug#60237

* Update doc/source/whatsnew/v3.0.0.rst

Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com>

* moved test case to frame and serier folders

* fix pyarrow import error

---------

Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com>
  • Loading branch information
Anurag-Varma and mroeschke authored Feb 18, 2025
1 parent 5b2cddb commit d4dff29
Show file tree
Hide file tree
Showing 4 changed files with 28 additions and 1 deletion.
1 change: 1 addition & 0 deletions doc/source/whatsnew/v3.0.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -670,6 +670,7 @@ Conversion
- Bug in :meth:`DataFrame.astype` not casting ``values`` for Arrow-based dictionary dtype correctly (:issue:`58479`)
- Bug in :meth:`DataFrame.update` bool dtype being converted to object (:issue:`55509`)
- Bug in :meth:`Series.astype` might modify read-only array inplace when casting to a string dtype (:issue:`57212`)
- Bug in :meth:`Series.convert_dtypes` and :meth:`DataFrame.convert_dtypes` removing timezone information for objects with :class:`ArrowDtype` (:issue:`60237`)
- Bug in :meth:`Series.reindex` not maintaining ``float32`` type when a ``reindex`` introduces a missing value (:issue:`45857`)

Strings
Expand Down
2 changes: 1 addition & 1 deletion pandas/core/dtypes/cast.py
Original file line number Diff line number Diff line change
Expand Up @@ -1113,7 +1113,7 @@ def convert_dtypes(
else:
inferred_dtype = input_array.dtype

if dtype_backend == "pyarrow":
if dtype_backend == "pyarrow" and not isinstance(inferred_dtype, ArrowDtype):
from pandas.core.arrays.arrow.array import to_pyarrow_type
from pandas.core.arrays.string_ import StringDtype

Expand Down
15 changes: 15 additions & 0 deletions pandas/tests/frame/methods/test_convert_dtypes.py
Original file line number Diff line number Diff line change
Expand Up @@ -196,3 +196,18 @@ def test_convert_dtypes_from_arrow(self):
result = df.convert_dtypes()
expected = df.astype({"a": "string[python]"})
tm.assert_frame_equal(result, expected)

def test_convert_dtype_pyarrow_timezone_preserve(self):
# GH 60237
pytest.importorskip("pyarrow")
df = pd.DataFrame(
{
"timestamps": pd.Series(
pd.to_datetime(range(5), utc=True, unit="h"),
dtype="timestamp[ns, tz=UTC][pyarrow]",
)
}
)
result = df.convert_dtypes(dtype_backend="pyarrow")
expected = df.copy()
tm.assert_frame_equal(result, expected)
11 changes: 11 additions & 0 deletions pandas/tests/series/methods/test_convert_dtypes.py
Original file line number Diff line number Diff line change
Expand Up @@ -297,3 +297,14 @@ def test_convert_dtypes_pyarrow_null(self):
result = ser.convert_dtypes(dtype_backend="pyarrow")
expected = pd.Series([None, None], dtype=pd.ArrowDtype(pa.null()))
tm.assert_series_equal(result, expected)

def test_convert_dtype_pyarrow_timezone_preserve(self):
# GH 60237
pytest.importorskip("pyarrow")
ser = pd.Series(
pd.to_datetime(range(5), utc=True, unit="h"),
dtype="timestamp[ns, tz=UTC][pyarrow]",
)
result = ser.convert_dtypes(dtype_backend="pyarrow")
expected = ser.copy()
tm.assert_series_equal(result, expected)

0 comments on commit d4dff29

Please sign in to comment.