Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmarker improvements + more benchmarks #581

Merged
merged 13 commits into from
Jul 15, 2022

Conversation

rctcwyvrn
Copy link
Contributor

@rctcwyvrn rctcwyvrn commented Jul 15, 2022

Taking the stuff not waiting for legal approval from #515 and adding some more stuff

Benchmarker improvements

  • Fixing some benchmark names
  • Add suite filtering by patterns instead of names
  • Exclude firstMatch benchmarks by default because they were running too quickly and not providing information the matches benchmarks weren't
  • Add flag for excluding the NSRegex comparison benchmarks
  • Make save and compare take file paths like a normal CLI instead of the weirdness it was doing before
  • Add some basic statistics to the runner (standard deviation computation, t-test to determine notable improvements/regressions)
  • Add a benchmark type for running a whole match regex on each item in a list of strings

New benchmarks

  • Emojis
  • Dice notation
  • Swiftc error messages
  • ipv4, ipv6, Mac addresses

(If you saw the bit I had here about the interesting list benchmark results, ignore it. It was just a bug :C)

@rctcwyvrn rctcwyvrn requested a review from milseman July 15, 2022 00:18
@rctcwyvrn
Copy link
Contributor Author

@swift-ci test

Copy link
Member

@milseman milseman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@milseman
Copy link
Member

@swift-ci please test

result = measure(benchmark: b, samples: result.samples*2)
print("N = \(result.samples), median: \(result.median), stdev: \(Time(result.stdev))")
if result.stdev > Stats.maxAllowedStdev {
fatalError("Benchmark \(b.name) is too variant")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this print the error instead of trapping the program?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Being too variant generally indicates that the system is running stuff in the background and making the benchmarker variant, so in those cases I thought it would probably best to just stop everything and make the user restart it

@rctcwyvrn rctcwyvrn merged commit 96fb215 into swiftlang:main Jul 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants