Fix a bug to unflatten the doc with list of map with multiple entries correctly

[bzhangam]

Description

Fix a bug to unflatten the doc with list of map with multiple entries correctly.

Let's say we have a doc like:

{
   "test.products":[
      {
         "product_description":"product_description 1",
         "price":10
      },
      {
         "product_description":"product_description 2",
         "price":20
      }
   ]
}

If we define an ingest pipeline to create the embedding for the product_description we will try to unflatten the doc first to ensure the . in the field name will be handled properly. Before my fix the doc will be unflattened like:

{
   "test":{
      "products":[
         {
            "price":10
         },
         {
            "price":20
         }
      ]
   }
}

We lose the product_description because when we handle the map in a list we incorrectly do targetList.set(targetList.size() - 1, tempMap);. With my fix the doc will be unflattened correctly:

{
   "test":{
      "products":[
         {
            "product_description":"product_description 1",
            "price":10
         },
         {
            "product_description":"product_description 2",
            "price":20
         }
      ]
   }
}

Related Issues

There is no related issue.

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.71%. Comparing base (9cc8959) to head (8f7b230).
Report is 1 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1204      +/-   ##
============================================
- Coverage     81.77%   81.71%   -0.06%     
+ Complexity     2512     1255    -1257     
============================================
  Files           190       95      -95     
  Lines          8564     4282    -4282     
  Branches       1436      718     -718     
============================================
- Hits           7003     3499    -3504     
+ Misses         1003      506     -497     
+ Partials        558      277     -281     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[will-hwang]

looks good to me. signoff your commit

[heemin32]

Please fix the failed DCO checks.

[martin-gaievski]

few questions here:

• what does it mean for end-user if we loose that prefix part in the flattened document?
• what the impact of this change, which exact processors can be affected by this change? To me it looks like all inference processors and that includes both single and batch modes, is this correct?

[will-hwang]

few questions here:

• what does it mean for end-user if we loose that prefix part in the flattened document?
• what the impact of this change, which exact processors can be affected by this change? To me it looks like all inference processors and that includes both single and batch modes, is this correct?

text embedding processor and sparse encoding processor would be affected by this change since they both extend from InferenceProcessor, where unflatten is used for pre-processing the ingest document.

[will-hwang]

@bzhangam an alternative to consider is:

we could treat

{
"product_description":"product_description 1",
"price":10
}

as a single item to be pushed to stack, and and modify unflattenSingleItem. That will allow the existing logic to work as is. I'll leave it up to you to decide, as this will require additional changes

[bzhangam]

few questions here:

• what does it mean for end-user if we loose that prefix part in the flattened document?
• what the impact of this change, which exact processors can be affected by this change? To me it looks like all inference processors and that includes both single and batch modes, is this correct?

Yeah all the inference processors are impacted by this bug. If we run into the bug the doc will lose some data and the missing data will not be stored to the source or index. The missing data is removed from the ingestDoc.

[bzhangam]

@bzhangam an alternative to consider is:

we could treat

{
"product_description":"product_description 1",
"price":10
}

as a single item to be pushed to stack, and and modify unflattenSingleItem. That will allow the existing logic to work as is. I'll leave it up to you to decide, as this will require additional changes

I think that one line fix should be good enough. I feel it should be done that way. The bug is more like a typo thing.

[heemin32]

LGTM. Thanks!

[junqiu-lei]

Thanks for the fixing, LGTM.