Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(stepfunctions): various JSONata query language fix #33404

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

GavinZZ
Copy link
Contributor

@GavinZZ GavinZZ commented Feb 12, 2025

Issue # (if applicable)

Closes #33403 and #33374 and #33396.

Reason for this change

There are three issues here:

  1. For summary, the first issue is basically that assign property cannot be accessed with using Map.jsonata(...) but available if we directly create map through new Map(...) using JSONATA query language.
  2. For summary, the second issue is that JSONATA main PR added the outputs and assign property in the CatchProps interface for AddCatch functionality. But I don't think it's being used in the actual addCatch call https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk-lib/aws-stepfunctions/lib/states/state.ts#L398.
  3. Result writer and item reader class do not support using JSONATA. Deployment will fails due to if SFN is set to use JSONATA, it expects Arguments in the ASL instead of Parameters.

Description of changes

Fix both issues by fixing the interface inheritance and added the props to AddCatch method.
Support JSONATA as the query language.

Description of how you validated changes

Added integ test and unit test to make sure that

Checklist


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@github-actions github-actions bot added bug This issue is a bug. p2 labels Feb 12, 2025
@aws-cdk-automation aws-cdk-automation requested a review from a team February 12, 2025 00:55
@mergify mergify bot added the contribution/core This is a PR that came from AWS. label Feb 12, 2025
Copy link

codecov bot commented Feb 12, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.00%. Comparing base (aebb331) to head (a39ac45).
Report is 9 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #33404      +/-   ##
==========================================
+ Coverage   80.90%   81.00%   +0.09%     
==========================================
  Files         236      238       +2     
  Lines       14261    14269       +8     
  Branches     2493     2492       -1     
==========================================
+ Hits        11538    11558      +20     
+ Misses       2438     2425      -13     
- Partials      285      286       +1     
Flag Coverage Δ
suite.unit 81.00% <ø> (+0.09%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
packages/aws-cdk 79.92% <ø> (+0.18%) ⬆️
packages/aws-cdk-lib/core 82.16% <ø> (ø)

@GavinZZ GavinZZ added the pr/do-not-merge This PR should not be merged at this time. label Feb 12, 2025
@GavinZZ
Copy link
Contributor Author

GavinZZ commented Feb 12, 2025

Marking this PR as pr/do-not-merge. Pending a review from @WinterYukky prior to merge this change.

Also update the PR label to correctly reflect the issue priority.

@GavinZZ GavinZZ added p1 and removed p2 labels Feb 12, 2025
@aws-cdk-automation aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Feb 12, 2025
…#33423)

### Issue # (if applicable)

Closes #33403 and #33374 and #33396.

### Reason for this change

There are three issues here:
1. For summary, the first issue is basically that assign property cannot
be accessed with using Map.jsonata(...) but available if we directly
create map through new Map(...) using JSONATA query language.
2. For summary, the second issue is that JSONATA main PR added the
outputs and assign property in the CatchProps interface for AddCatch
functionality. But I don't think it's being used in the actual
`addCatch` call
https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk-lib/aws-stepfunctions/lib/states/state.ts#L398.
3. Result writer and item reader class do not support using JSONATA.
Deployment will fails due to if SFN is set to use JSONATA, it expects
`Arguments` in the ASL instead of `Parameters`.

### Description of changes

Fix both issues by fixing the interface inheritance and added the props
to `AddCatch` method.
Support `JSONATA` as the query language.

### Description of how you validated changes

Added integ test and unit test to make sure that 

### Checklist
- [ ] My code adheres to the [CONTRIBUTING
GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and
[DESIGN
GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md)

----

*By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache-2.0 license*
@github-actions github-actions bot added the effort/medium Medium work item – several days of effort label Feb 13, 2025
@GavinZZ GavinZZ changed the title fix(stepfunctions): support JSONATA for result writer and item reader fix(stepfunctions): various JSONata query language fix Feb 13, 2025
Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(This review is outdated)

@aws-cdk-automation aws-cdk-automation dismissed their stale review February 13, 2025 01:04

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

Copy link
Contributor

@WinterYukky WinterYukky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, thank you so much for fixing this bug. I'm very honored that you started working on this fix right away.

assign and _addCatch fix seems good! On the other hand, itemReader and itemWriter may need to be changes. In itemReader and itemWriter, whether to use Arguments or Parameters is determined only by the top-level or Map state queryLanguage. In this PR, users can use s3objectsitemReader.jsonata() and s3objectsitemReader.jsonPath(), but for example, s3objectsitemReader.jsonPath() (i.e. Parameters) can be used when the Map state uses JSONata we can't.

// invalid cofigure
 const distributedMap = sfn.DistributedMap.jsonPath(this, 'DistributedMap', {
    itemReader: sfn.S3ObjectsItemReader.jsonata({
      bucketNamePath: '{% $bucketName %}',
      prefix: '{% $prefix %}',
    }),
 });

This should probably be processed projectively with CDK without relying on user specifications. Also, since the properties used in JSONata and JSONPath are not different, there is probably no need to use different constructors.

 const distributedMap = sfn.DistributedMap.jsonPath(this, 'DistributedMap', {
    itemReader: new sfn.S3ObjectsItemReader({
      bucketName: '{% $bucketName %}',
      prefix: '{% $prefix %}',
    }),
 });

@GavinZZ
Copy link
Contributor Author

GavinZZ commented Feb 13, 2025

@WinterYukky I addressed all your comments. Please give it another review once you're available.

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: a39ac45
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. contribution/core This is a PR that came from AWS. effort/medium Medium work item – several days of effort p1 pr/do-not-merge This PR should not be merged at this time. pr/needs-maintainer-review This PR needs a review from a Core Team Member
Projects
None yet
Development

Successfully merging this pull request may close these issues.

step functions: Map.jsonata does not support assign
3 participants