Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix : Cannot read Optionals written with StdTypeResolverBuilder, from [jackson-modules-java8#86] #4838

Closed
wants to merge 19 commits into from
Closed
Show file tree
Hide file tree
Changes from 12 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions release-notes/VERSION-2.x
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,15 @@ Project: jackson-databind
#4773: `SerializationFeature.ORDER_MAP_ENTRIES_BY_KEYS` should not apply to Maps
with uncomparable keys
(requested by @nathanukey)
#4850: Cannot read reference types written with `StdTypeResolverBuilder`
(reported by @isopov)
(fix by Joo-Hyuk K)

2.18.3 (not yet released)

#4827: Subclassed Throwable deserialization fails since v2.18.0 - no creator
index for property 'cause'
(reported by @nilswieber)
(fix by Joo-Hyuk K)
#4844: Fix wrapped array hanlding wrt `null` by `StdDeserializer`
(fix by Stanislav S)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -257,6 +257,9 @@ public Object deserializeWithType(JsonParser p, DeserializationContext ctxt,
if (_valueTypeDeserializer == null) {
return deserialize(p, ctxt);
}
return referenceValue(_valueTypeDeserializer.deserializeTypedFromAny(p, ctxt));
// [modules-java8#86] Since 2.19.0, Cannot read `Optional`s written with `StdTypeResolverBuilder`
return typeDeserializer.deserializeTypedFromAny(p, ctxt);
// Previously was...
// return referenceValue(_valueTypeDeserializer.deserializeTypedFromAny(p, ctxt));
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
import com.fasterxml.jackson.annotation.JsonInclude;

import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.core.JsonToken;
import com.fasterxml.jackson.core.type.WritableTypeId;
import com.fasterxml.jackson.databind.*;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
import com.fasterxml.jackson.databind.introspect.Annotated;
Expand Down Expand Up @@ -408,17 +410,14 @@ public void serializeWithType(T ref,
// (which is what non-polymorphic serialization does too), we will need
// to simply delegate call, I think, and NOT try to use it here.

// Otherwise apply type-prefix/suffix, then std serialize:
/*
typeSer.writeTypePrefixForScalar(ref, g);
serialize(ref, g, provider);
typeSer.writeTypeSuffixForScalar(ref, g);
*/
// [modules-java8#86] Since 2.19.0, Bringing back the prefix and suffix writing part
WritableTypeId typeId = typeSer.writeTypePrefix(g, typeSer.typeId(ref, JsonToken.VALUE_STRING));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. I think this makes sense, except for one problem -- JsonToken.VALUE_STRING works for some cases, not for others. Problem is, we do not actually know the expected shape of contained value. Later on, test can be broken; I'll add a note where and how, below.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... I am really not sure how to fix this. I understand why type id would be needed, for AtomicReference / Optional wrapper. Unfortunately there is a key problem: nothing else will be written for these wrappers.
Unless... unless I guess if we decide that for the specific case of polymorphic value, there is.

And I think the only possibility, there, is extra wrapper array. I'll see if that works.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sigh. Nope. Was not able to make this work overall.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least if we have more test case 👍🏼👍🏼

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cowtowncoder I modified the test via de92568, to configure StdTypeResolverBuilder to use appropriate inclusion scheme for each mapper. Seems like that's way to go?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps? I am not sure which cases to focus most; ones with @JsonTypeInfo introduced ones or Default Typing (I know, it's reported wrt Default Typing but in a way @JsonTypeInfo cases are more important -- Default Typing having security concerns).

Another confounding factor is that for String and Integer, type id MUST NOT be written ("natural" types), nor expected.
It's a mess. :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But no matter what, we really cannot use JsonToken.VALUE_STRING for all cases because some values are serialized as Objects (Maps, most POJOs), others as Arrays (Collections) and those cases would not work with this type id.

Yet there is also no way for ReferecenTypeSerializer to know for sure value shape.

This is why I think we actually must use JSON Array wrapper for Type Id case to know exact shape for just this case (in theory JSON Object but that would require bogus key for single entry; array seems cleaner).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am starting to think this situation (Default Typing overrides@JsonTypeInfo) is inevitable, due to Jackson does heavy lifiting during serialization.
I don't think there is easy, sensible way to introspect all the way from Foo<T> -> AtomicRference<T> -> Animal to get configured type information, during runtime.

So maybe we can either...

  1. separate concerns, resolve very specific issue here around Optional<String> and create separate milestone for this override case by pushing some tests as tofix.. or
  2. Just add all tests to tofix and come back later in the future when we are in best shape to do so.

Copy link
Member

@cowtowncoder cowtowncoder Dec 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the problem is more with the "missing" serialization for Optional, actually. For non-polymorphic case it is sort of ok, as long as POJO typing is defined. Basically following 2 serialize to identical JSON:

static class Required {
   public MyPojo value = new MyPojo();
}

static class Optional {
  public Optional<MyPojo> value = Optional.of(new MyPojo());
}

// Both become something like:

{ "value": {
   "myPojoProp1": 1,
   "myPojoProp2" : 2
}

// Compared to say

static class Multiple {
   public List<MyPojo> values = List.of(new MyPojo());'
}

// which becomes

{ "values": [
   {
   "myPojoProp1": 1,
   "myPojoProp2" : 2
  }
]}

this is ok because no Type Id needs to be added anywhere.
But if (polymorphic) Type Id needs to be added, there is only place for one of Type Ids (MyPojo) -- for Optional there is no JSON value to modify/include because it is invisible, unlike List and others for which there is JSON Array.
If there was JSON Array to represent Optional level things would be easier.

So for Collection case, As.WRAPPER_ARRAY we might have something like:

[ "com.xxx.Multiple",
   { "values": [ "java.util.ArrayList",  [
    [ "com.xxx.MyPojo",  {
        "myPojoProp1": 1,
       "myPojoProp2" : 2
      }
    ]
  ]
  ]
]}]

where extra "wrapper Array" is added around Multiple, List as well as MyPojo element)s_s (depending on details of @JsonTypeInfo or Default Typing -- here assume it is applicable to all of Multiple, List and MyPojo).

This is not really possible with Optional because nothing is written to represent Optional

Now: in theory we could consider doing something similar to EnumSet where Class name as type id is actually polymorphic -- and we might be able to do the same with optional. But it would have to replace Type Id of thing contained (and only works for Class Name type id, but that's probably to be expected).
But I suspect that would be even more work.

As to "solving" reported case... I am still worried I don't fully understand the behavior before and after change. We'd need to, I think, verify exact JSON being produced and not simply checking round-tripping.
My concern is specifically that we end up changing serialization output and breaking other usage that is not covered by test cases.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, this is getting bigger than I imagined I agree with your concerns wrt serialization output. Knowing that the serialization output changed versions back, (2.8 -> 2.9) complicates things I think.

We might even consider current output as expected behavior and suggest to try work around or something, for cases where they update [ 2.8 -> 2.9 or later like 2.18 .

In any case, I will close this off for now and would like to get back to taking steps toward 3.0 -- I will leave some closing comments

JsonSerializer<Object> ser = _valueSerializer;
if (ser == null) {
ser = _findCachedSerializer(provider, value.getClass());
}
ser.serializeWithType(value, g, provider, typeSer);
typeSer.writeTypeSuffix(g, typeId);
}

/*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
import com.fasterxml.jackson.databind.jsontype.BasicPolymorphicTypeValidator;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;

import static com.fasterxml.jackson.databind.testutil.DatabindTestUtil.jsonMapperBuilder;

Expand Down Expand Up @@ -48,5 +49,6 @@ public void testPolymorphicDeserialization4214() throws Exception
String json = mapper.writeValueAsString(enumSetHolder);
EnumSetHolder result = mapper.readValue(json, EnumSetHolder.class);
assertEquals(result, enumSetHolder);
assertTrue(result.enumSet instanceof EnumSet<?>);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
package com.fasterxml.jackson.databind.deser.jdk;

import java.util.Objects;
import java.util.concurrent.atomic.AtomicReference;

import org.junit.jupiter.api.Test;

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.annotation.JsonTypeInfo;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.jsontype.impl.StdTypeResolverBuilder;
import com.fasterxml.jackson.databind.testutil.DatabindTestUtil;

import static org.junit.jupiter.api.Assertions.assertEquals;

// Reported via [modules-java8#86] Cannot read `Optional`s written with `StdTypeResolverBuilder`
public class AtomicReferenceWithStdTypeResolverBuilder4838Test
extends DatabindTestUtil
{

public static class Foo<T> {
public AtomicReference<T> value;

@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
Foo<?> foo = (Foo<?>) o;
return Objects.equals(value.get(), foo.value.get());
}
}

public static class Pojo86 {
public String name;

public static Pojo86 valueOf(String name) {
Pojo86 pojo = new Pojo86();
pojo.name = name;
return pojo;
}

@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
Pojo86 pojo86 = (Pojo86) o;
return Objects.equals(name, pojo86.name);
}
}

@JsonTypeInfo(use = JsonTypeInfo.Id.CLASS, include = JsonTypeInfo.As.WRAPPER_OBJECT)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If changing As.WRAPPER_OBJECT to any other choice -- specifically, As.PROPERTY or As.WRAPPER_ARRAY, test fails, because of earlier write of TypeId assuming output value is a scalar (need not be JSON String).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

@cowtowncoder cowtowncoder Dec 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick note: Default Typing does not override @JsonTypeInfo since annotation is more specific than general setting. This is why we probably need separate types.
Although it also gets bit more complicated to reason about combinations.

We also need to try As.PROPERTY as mechanism.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, general one should not take effect if there is more specific one.

Copy link
Member Author

@JooHyukKim JooHyukKim Dec 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverting test modification that I made.

public static abstract class Animal {
public String name;

protected Animal(String name) {
this.name = name;
}
}

public static class Dog extends Animal {
@JsonCreator
public Dog(@JsonProperty("name") String name) {
super(name);
}

@Override
public boolean equals(Object obj) {
return (obj instanceof Dog) && name.equals(((Dog) obj).name);
}
}

@Test
public void testRoundTrip() throws Exception {
_test(new AtomicReference<>("MyName"), String.class);
_test(new AtomicReference<>(42), Integer.class);
_test(new AtomicReference<>(Pojo86.valueOf("PojoName")), Pojo86.class);
}

@Test
public void testPolymorphic() throws Exception {
_test(new AtomicReference<>(new Dog("Buddy")), Animal.class);
}

private <T> void _test(AtomicReference<T> value, Class<?> type) throws Exception {
ObjectMapper mapper = configureObjectMapper();

// Serialize
Foo<T> foo = new Foo<>();
foo.value = value;
String json = mapper.writeValueAsString(foo);

// Deserialize
Foo<T> bean = mapper.readValue(json, mapper.getTypeFactory().constructParametricType(Foo.class, type));

// Compare the underlying values of AtomicReference
assertEquals(foo.value.get(), bean.value.get());
}

private ObjectMapper configureObjectMapper() {
return jsonMapperBuilder()
// this is what's causing failure in later versions.....
.setDefaultTyping(
new StdTypeResolverBuilder()
.init(JsonTypeInfo.Id.CLASS, null)
.inclusion(JsonTypeInfo.As.WRAPPER_OBJECT)
).build();
}
}