Render PPL time column using the correct time zone

[abbyhu2000]

Description

When sending PPL requests, the time based fields are rendered in wrong time zone, and it looks like DQL and PPL are getting different results. We should format the PPL search response to the same format as DQL search response to indicate it is in UTC so the time fields can be formatted and rendered correctly in the discover table according to user defined time zone in advanced setting.

For example, previously

// PPL search response
"order_date": "2025-02-13 00:51:50"

// DQL search response
"order_date": "2025-02-13T00:51:50+00:00”

Issues Resolved

resolves #9104

Screenshot

Before:

PPL render time field in the wrong timezone

Screen.Recording.2025-02-12.at.5.22.01.PM.mov

After:

PPL and DQL return and render same results for default timezone or user defined timezone

Screen.Recording.2025-02-12.at.5.17.57.PM.mov

Testing the changes

Changelog

• fix: Make PPL time column respect time zone and date format

Check List

• All tests pass
  • yarn test:jest
  • yarn test:jest_integration
• New functionality includes testing.
• New functionality has been documented.
• Update CHANGELOG.md
• Commits are signed per the DCO using --signoff

[codecov]

Codecov Report

Attention: Patch coverage is 78.57143% with 6 lines in your changes missing coverage. Please review.

Project coverage is 61.74%. Comparing base (6cf56f0) to head (61e085b).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
src/plugins/data/common/data_frames/utils.ts	87.50%	2 Missing and 1 partial ⚠️
.../data/common/search/search_source/search_source.ts	0.00%	2 Missing ⚠️
...ic/application/view_components/utils/use_search.ts	50.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9379      +/-   ##
==========================================
+ Coverage   61.71%   61.74%   +0.02%     
==========================================
  Files        3817     3817              
  Lines       91862    91898      +36     
  Branches    14552    14562      +10     
==========================================
+ Hits        56697    56741      +44     
+ Misses      31509    31497      -12     
- Partials     3656     3660       +4     

Flag	Coverage Δ
Linux_1	28.99% <3.57%> (-0.02%)	⬇️
Linux_2	56.41% <3.84%> (-0.05%)	⬇️
Linux_3	39.26% <78.57%> (+0.05%)	⬆️
Linux_4	28.88% <3.57%> (-0.02%)	⬇️
Windows_1	29.00% <3.57%> (-0.02%)	⬇️
Windows_2	56.37% <3.84%> (-0.05%)	⬇️
Windows_3	39.25% <78.57%> (+0.04%)	⬆️
Windows_4	28.88% <3.57%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Maosaic]

Any unit test we want to add or update?

[kavilla]

i can't remember this is before we know it is a date field correct? or do we already know it's a date field? as in field.type === 'date'.

YYYY-MM-DD HH:mm:ss seems too specific to PPL, correct me if im wrong.

would it make sense to do it something like

const tzConfig = opensearchDashboards.services.uiSettings!.get(UI_SETTINGS.DATE_FORMAT_TIMEZONE);
const momentParsedValue = moment(value).tz(tzConfig);
if (momentParsedValue.isValid()) return momentParsedValue?.format('YYYY-MM-DDTHH:mm:ss.SSSZ');

or does it need to be +00:00 because to me it feels like we would be forcing any date value of YYYY-MM-DD HH:mm:ss to have a value we statically append +00:00

[abbyhu2000]

yea at first i used field.type === 'date' but then i realized this will skip those nested field which does not have a field.type yet, therefore i switch to use moment.

[abbyhu2000]

We do not need to get or change the timezone here because opensearch backend always store and return time value as UTC, no matter which timezone we define in the advanced setting. And then the table formatter will reformat all the values according to the advanced setting later. So here we should not change time zone.

We could potentially do:

if (moment.utc(value).isValid())  or if(moment(value).isValid())

but this don't work for some reason: when the value type is object, it will always go into the if loop as it is a date. I tried multiple methods for determining if a string is a date, so far only if (moment(value, ['YYYY-MM-DD HH:mm:ss'], true).isValid()) does the correct job. We could potentially do if (moment(value, ['YYYY-MM-DD HH:mm:ss', 'YYYY-MM-DD HH:mm:ss.SSS'], true).isValid()) to include more types here? i know only PPL doesn't include miliseconds

[abbyhu2000]

i changed return value.replace(' ', 'T') + '+00:00'; to return moment.utc(value).format('YYYY-MM-DDTHH:mm:ssZ'); so it it more general

[kavilla]

wow. great find!

do we have a manual test or could we add some cypress/unit test for this one. when first implementing this extensibility i had completely neglected the responses could be coming back in a different timezone format. if developers wanted to add new langauges and new data sources it might be useful for them to ensure they properly are formatting their responses or that we do it for all cases. or at least documented

[abbyhu2000]

Added unit test for convertResult() function @Maosaic @kavilla

[abbyhu2000]

Reconstruct this PR to use OSDfieldsList to get the type; also add a config in the language config so PPL will format its date fields

[kavilla]

could you clean this up

[abbyhu2000]

removed

[abbyhu2000]

make this formatter more general so in the future it can be utilized for other cases too

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch-Dashboards/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch-Dashboards/backport-2.x
# Create a new branch
git switch --create backport/backport-9379-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 601d36745ab09bb34852e0dad5e3bc66370a3b94
# Push it to GitHub
git push --set-upstream origin backport/backport-9379-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch-Dashboards/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-9379-to-2.x.