Deal with Control Characters in DynamoDB

[Galactus22625]

Description

Manually Parse DDB Records

• Tested by adding control characters to test_writeSingleRecordToBuffer_with_other_data in StreamRecordConveterTest
• Tested by running DynamoDB source pipeline on local and ingesting records with control characters successfully.
• Tested by running DynamoDB source pipeline on local and ingesting records with more complex data structures like lists, maps, lists and maps in lists, and lists and maps in maps as well as every other ddb data type. Compared output to output of unchanged dataprepper.

Issues Resolved

Resolves #5027 . Bug ingesting control characters from dynamo db

Check List

• New functionality includes testing.
• New functionality has a documentation issue. Please link to it in this PR.
  • New functionality has javadoc added
• Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[Galactus22625]

Not sure if I need to make the same change in Export Record Converter because I'm not sure what the export option does for dynamo db source

[dlvenable]

Thank you @Galactus22625 for working this.

I tend to think this is not the right approach. Here are a few reasons.

1. The DDB records do in fact contain properly escaped control characters. So the ultimate bug appears to be that we are unescaping them (probably through the EnhancedDocument).
2. This may result in allowing leniency that we don't actually want to allow. Again, this is not a problem with the input data.
3. There may be other issues with the serialization we miss from this.
4. Looking at the code, there is a bit of overhead in this EnhancedDocument approach anyway. We take the DDB record, make a JSON string, then deserialize it.

See this comment in particular:

#5027 (comment)

It appears that the best options are:

1. Contact the DDB team to see if this is the expected behavior of EnhancedDocument. Perhaps by opening a GitHub issue.
2. or; Change the implementation to parse the DDB record using the approach I took in the proof-of-concept that I shared in the comment above.

[Galactus22625]

ok, I will try implementing a solution in the style of the proof-of-concept from the comment above

[dlvenable]

Please add unit tests to cover all these data types to verify that we handle them correctly.

[Galactus22625]

unit tests added

[dlvenable]

Is this the current behavior - base64 encoding the string?

[Galactus22625]

yes, i made sure the output matches what was being outputted before

[Galactus22625]

and also dynamo db only takes base64 binary values

[dlvenable]

You could consolidate this code with the code above.

private Object processAttributeValue(AttributeValue attributeValue) {
  switch (attributeValue.type()) {
    case N:
       return new BigDecimal(attributeValue.n());
    case B:
       return BASE64_ENCODER.encodeToString(attributeValue.b().asByteArray());
   ....
  }
}

And then use it. The following should work (or be close to working)

 convertListData(...) {
   return data.stream.map(this::processAttributeValue).collect(toList());
 }

 convertMapData(...) {
   return data.stream.collect(toMap(attributeValue::type, this::processAttributeValue));
  }

[Galactus22625]

ok, will consolidate

[dlvenable]

This should probably be InvalidArgumentException.

[Galactus22625]

ok, will change. I was matching the error message that was being sent before.

[dlvenable]

Very nice. Thank you for this improvement!