Composite Aggregation after key ignored

[assafcoh]

What is the bug?

when searching with composite aggregation the supplied afterKey is ignored and we always get back the same initial afterKey.
Therefore, iterating with composite aggregation until the afterKey is null, leads to an infinite loop

client version

we encountered this bug with client version 2.6.0, 2.70 and also after upgrading to the latest 2.8.1 client version

How can one reproduce the bug?

search with composite aggregation in a loop until the afterKey is null.
Since the afterKey is ignored, it will always search from the beginning and repeatedly return the same afterKey

What is the expected behavior?

the afterKey should should be considered, so that after each search we get back the "next" afterKey

workaround

Unfortunately we switched back to the rest client :(

Code sample to reproduce infinite loop

List<Map<String, CompositeAggregationSource>> compositeAggSourceList = new ArrayList<>();

compositeAggSourceList.add(Collections.singletonMap("tenant_id", new CompositeAggregationSource.Builder()
				.terms(new TermsAggregation.Builder()
						.field("tenant_id")
						.build())
		.build()));
compositeAggSourceList.add(Collections.singletonMap("user_id", new CompositeAggregationSource.Builder()
		.terms(new TermsAggregation.Builder()
				.field("user_id")
				.build())
		.build()));
					
CompositeAggregate compositeAggregate = null;
do {
	// update the composite aggregation after key
	Map<String, String> afterKey;
	if(compositeAggregate !=null){
		afterKey = compositeAggregate.afterKey().entrySet().stream().collect(Collectors.toMap(Map.Entry::getKey,
				e -> new String(e.getValue().toString())));
	} else {
		afterKey = null;
	}

	SearchRequest searchRequest = new SearchRequest.Builder()
			.index("example-index")
			.size(0)
			.aggregations("composite-agg", new Aggregation.Builder()
					.composite(c-> afterKey !=null ? c.sources(compositeAggSourceList).size(10).after(afterKey) : c.sources(compositeAggSourceList).size(10))
					.aggregations("incidents",
							a -> a.sum(t -> t.field("incidents")))
					.aggregations("activities",
							a -> a.sum(t -> t.field("activities")))
					.build())
			.build();

	SearchResponse<Map> searchResponse = openSearchClient.get().search(searchRequest, Map.class);

	compositeAggregate = searchResponse.aggregations().get("composite-agg").composite();
	
	// our internal logic...

} while (compositeAggregate.afterKey()!=null);

[assafcoh]

composite aggregation is the only way to paginate aggregations. This is a major bug,
Is there a fix estimate?

[dblock]

@assafcoh Nobody has picked this up yet, care to help? Maybe start by adding a (failing) test?

[assafcoh]

Unfortunately, i am not able to contribute here.

However, this bug means there is no way to paginate aggregations with this java client. In my opinion, this is a central feature and should be given higher priority.

[amcgreevy-r7]

@dblock Has there been any movement on this? This is a core feature that is really needed.

[dblock]

@amcgreevy-r7 Would love your help, see above.

[assafcoh]

hi guys, I am really surprised this bug is still lurking in the code.
Composite aggregation is the only way to paginate aggregations. This is a major bug,
Is there a fix estimate?

[Xtansia]

So, there's a couple things going on here:
One actual issue is that CompositeAggregation.after() should be changed to accept a Map<String, JsonData> to match CompositeAggregate.afterKey().

Separately a quirk of the client is that List and Map types are never actually null, there's just a sentinel undefined map/list type. As such you should not be checking for agg.afterKey() != null but instead using ApiTypeHelepr.isDefined(agg.afterKey()) or !agg.afterKey().isEmpty().

[assafcoh]

@Xtansia thank you for your reply.

I decided to give this another try with the new java opensearch client version 2.19.0.
I realized that the problem was that method CompositeAggregate.afterKey() returned values enclosed inside redundant double quotes (see example below):

{ 
 "tenant_id"": ""123"",
 "user_id"": ""222""
}

Apparently, when populating these double-quoted-values into CompositeAggregation.after(), they are ignored, and the search response contained the same initial afterkey every time.

To fix this, whilst paginating the composite aggregation, i had to remove all the redundant double quotes (see code snippet below)

if(compositeAggregate !=null){
                afterKey = compositeAggregate.afterKey().entrySet().stream()
                        .collect(Collectors.toMap(Map.Entry::getKey,
                        e -> new String(e.getValue().toString().**replaceAll("\"", "")**)));    // remove redundant double quotes
            } else {
                afterKey = null;
            }

So the good news is that paginating composite aggregation works.

But as you mentioned, to prevent such misunderstandings, the CompositeAggregation.after() should be changed to accept a Map<String, JsonData> to match CompositeAggregate.afterKey()

[Xtansia]

@Xtansia thank you for your reply.

I decided to give this another try with the new java opensearch client version 2.19.0. I realized that the problem was that method CompositeAggregate.afterKey() returned keys/value enclosed inside redundant double quotes (see example below):

{ 
 ""tenant_id"": ""123"",
 ""user_id"": ""222""
}

If you could create a complete minimal code reproduction (ie. client setup, create index, ingest data, perform aggregate search) of this that would be ideal, as it may be a bug in OpenSearch itself or may be an oddity of the JsonData implementation.

[assafcoh]

I fixed typo in my previous comment. Only the values returned from CompositeAggregate.afterKey() are enclosed in redundant double quotes