Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds more benchmarks #169

Merged
merged 1 commit into from
Feb 6, 2024
Merged

Adds more benchmarks #169

merged 1 commit into from
Feb 6, 2024

Conversation

ncordon
Copy link
Collaborator

@ncordon ncordon commented Jan 31, 2024

Why

To test out more improvements on auto-completion and signature help, especially when we are moving the cursor in the query

Comment on lines +23 to +24
Benchmark.options.minSamples = 10;
Benchmark.options.maxTime = 1;
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't have much effect in the results of the benchmark, except on the parsing time, which I believe is cause the first iteration is always going to be really slow

Copy link
Collaborator

@OskarDamkjaer OskarDamkjaer Feb 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could be wrong but I think the semantic analysis for the really large query takes more than a second per cycle, so won't setting the maxTime effectively make that benchmark useless?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the 1s limit:

simple - validate syntax x 5,016 ops/sec ±4.13% (26 runs sampled)
movies - validate syntax x 0.38 ops/sec ±2.84% (10 runs sampled)
tictactoe - validate syntax x 126 ops/sec ±6.65% (23 runs sampled)

Without:

simple - validate syntax x 4,893 ops/sec ±1.38% (93 runs sampled)
movies - validate syntax x 0.39 ops/sec ±2.01% (10 runs sampled)
tictactoe - validate syntax x 124 ops/sec ±3.25% (74 runs sampled)

I don't think that 1s is a hard limit, so if the benchmarks overrun that it detects it well

@ncordon ncordon merged commit 2eff826 into main Feb 6, 2024
4 checks passed
@ncordon ncordon deleted the more-benchmarks branch February 6, 2024 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants