-
Notifications
You must be signed in to change notification settings - Fork 47
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Benchmarker improvements + more benchmarks #581
Benchmarker improvements + more benchmarks #581
Conversation
@swift-ci test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
@swift-ci please test |
result = measure(benchmark: b, samples: result.samples*2) | ||
print("N = \(result.samples), median: \(result.median), stdev: \(Time(result.stdev))") | ||
if result.stdev > Stats.maxAllowedStdev { | ||
fatalError("Benchmark \(b.name) is too variant") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this print the error instead of trapping the program?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Being too variant generally indicates that the system is running stuff in the background and making the benchmarker variant, so in those cases I thought it would probably best to just stop everything and make the user restart it
Taking the stuff not waiting for legal approval from #515 and adding some more stuff
Benchmarker improvements
save
andcompare
take file paths like a normal CLI instead of the weirdness it was doing beforeNew benchmarks
(If you saw the bit I had here about the interesting list benchmark results, ignore it. It was just a bug :C)