[CLOSED] Build performance test comparing reflection and MethodHandles

[aionbot]

Issue created by jeff-aion (on Wednesday Oct 17, 2018 at 21:43 GMT)

In the cases where we don't hit the SoftCache, we spend a lot of time looking up reflected elements from the DApp. Even in the cases where we do hit it, we still see reflection-related operations in the profile.

There are many claims that MethodHandle-based solutions perform better than using reflection (due to a change to when some checks are made, extra inlining opportunity, etc). We need to study this, in isolation, to see if/when to replace our reflection-based solutions with MethodHandles.

We have 3 points we need to research with this:

1. Constructor lookup (for instantiating the IHelper).
2. Method lookup (for invoking avm_main and also in the ABIDecoder).
3. Field lookup (for persistence-related cases).

We need to study each of these questions in a few different ways:

1. Cost to lookup/resolve the artifact on the same class instance.
2. Cost to lookup-resolve the artifact on a unique class instance (load the same class bytecode in several loaders)
3. Access the artifact once resolved.

We should be able to test access to each example using the different mechanisms. Cases we need to test:

• Constructors
• Instance methods
• Static methods
• Instance fields
• Static fields

[aionbot]

Comment by jeff-aion (on Friday Nov 09, 2018 at 22:18 GMT)

This is a somewhat low-priority item but we eventually need the answer and, depending on what that answer is, this may give way to a new item to redesign our AVM-DApp boundary layer or may inform #284.

[aionbot]

Comment by aionick (on Thursday Nov 15, 2018 at 23:41 GMT)

I have some benchmarks sitting on my reflection-benchmark branch, I want to try running them again under some different conditions, some of these numbers seem a little dubious to me at the moment, but so far it seems that:

• Using the same class loader and only doing resolution/lookup: reflection is ~37x faster
• Using unique class loaders and only doing resolution/lookup: reflection is ~32x faster
• Using the same class loader and doing resolution/lookup + access: reflection is ~22x faster
• Using the same class loader and doing access only: method handle is ~2.5x faster
• Using unique class loaders and doing resolution/lookup + access: reflection is ~16x faster
• Using unique class loaders and doing access only: method handle is ~2x faster for static field write, instance field write and instance method invocation, and reflection is faster in everything else by a factor of ~3.8

If anything, it does seem that the lookup phase of MethodHandles is considerably slow, and once this has been done we get about a ~2x speed up. In the "cold case" method handle does even worse, it's access speed up appears only in 3 cases.

[aionbot]

Comment by jeff-aion (on Friday Nov 16, 2018 at 14:11 GMT)

It will definitely be good to get a test we can check in so we can go back to this on different JDK versions to see if anything changes or tweak it to check other cases.

The order of the differences is surprising, and definitely needs to be verified/explained, but the general trend sounds about in line with what I suspected: based on the vague MethodHandle understanding I had, I figured it just pushed some verification checks to the point of lookup, instead of invocation, so I figured it would only be a win if we were re-invoking the same handle (which is not really the case is our non-cached example - which is the slow one).

Still, we will need to check this against other binding behaviour and the different invoke variants to make sure we have a clear picture.

[aionbot]

Comment by jeff-aion (on Friday Nov 16, 2018 at 14:13 GMT)

There is also early discussion around MethodHandle performance from 2014, here, which may still be relevant: http://chriskirk.blogspot.com/2014/05/which-is-faster-in-java-reflection-or.html

[aionbot]

Comment by aionick (on Tuesday Nov 20, 2018 at 16:43 GMT)

The main findings are as follows:

• Resolution is always substantially faster using reflection, regardless of what is being resolved, and regardless of whether or not the resolution is applied to the same class or a unique class.
• Invocation is almost always faster using MethodHandle. In the worst case (invoking a constructor) reflection and MethodHandle perform nearly identically, but in all other cases MethodHandle is 2-4x faster than reflection.
• invokeExact() is almost always faster than invoke() and in the worst case performs nearly identically.
• For a unique class, performing one resolution followed by n invocations, MethodHandle is preferable to reflection in all cases except for invoking constructors, so long as n > 50 or so. For the same class, n must be in the hundreds and sometimes thousands to see any advantage.

Why the large difference in resolution times?

Typically, resolution using reflection is about 36-50x faster than MethodHandle.
Reflection doesn't do a whole lot to resolve something.

Consider the case of resolving a field.

For reflection, it consults the SecurityManager, grabs all the declared fields (a lazily initialized soft cache is used, and if a value is not in the cache it is loaded in a class loader).

MethodHandle on the other hand doesn't cache its "reflection data", it begins with class lookup privilege checks, symbolic reference checks, it then checks the calling class's access permissions by loading its module, it then resolves the field with a native resolve call and a class type check, then it grabs the field, goes through more accessibility checks, consults the SecurityManager, then creates a new DirectMethodHandle and returns it.

Benchmarks were done on a target class that had 4 static fields, 4 instance fields, 3 constructors, 4 instance methods and 4 static methods, all pretty mocked up and meaningless.
Benchmarks on the same class instances were run 15 million times each, and run 1 million times each on the unique class instances (since this also carries the overhead of creating 1 million ClassLoaders and Class references).