-
Notifications
You must be signed in to change notification settings - Fork 3
/
Copy pathbenchmarking-swift-code-properly-with-attabench.html
295 lines (252 loc) · 15.5 KB
/
benchmarking-swift-code-properly-with-attabench.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
<!DOCTYPE html>
<html lang="en">
<head>
<script src="https://use.fontawesome.com/afd448ce82.js"></script>
<!-- Meta Tag -->
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<!-- SEO -->
<meta name="author" content="Bruno Rocha">
<meta name="keywords" content="Software, Engineering, Blog, Posts, iOS, Xcode, Swift, Articles, Tutorials, OBJ-C, Objective-C, Apple">
<meta name="description" content="Hmmmm, how fast is this piece of code? Let's find out!">
<meta name="title" content="Benchmarking Swift Code Properly with Attabench">
<meta name="url" content="https://swiftrocks.com/benchmarking-swift-code-properly-with-attabench">
<meta name="image" content="https://swiftrocks.com/images/thumbs/thumb.jpg?4">
<meta name="copyright" content="Bruno Rocha">
<meta name="robots" content="index,follow">
<meta property="og:title" content="Benchmarking Swift Code Properly with Attabench"/>
<meta property="og:image" content="https://swiftrocks.com/images/thumbs/thumb.jpg?4"/>
<meta property="og:description" content="Hmmmm, how fast is this piece of code? Let's find out!"/>
<meta property="og:type" content="website"/>
<meta property="og:url" content="https://swiftrocks.com/benchmarking-swift-code-properly-with-attabench"/>
<meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="https://swiftrocks.com/images/thumbs/thumb.jpg?4"/>
<meta name="twitter:image:alt" content="Page Thumbnail"/>
<meta name="twitter:title" content="Benchmarking Swift Code Properly with Attabench"/>
<meta name="twitter:description" content="Hmmmm, how fast is this piece of code? Let's find out!"/>
<meta name="twitter:site" content="@rockbruno_"/>
<!-- Favicon -->
<link rel="icon" type="image/png" href="images/favicon/iconsmall2.png" sizes="32x32" />
<link rel="apple-touch-icon" href="images/favicon/iconsmall2.png">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Source+Sans+3:ital,wght@0,200..900;1,200..900&display=swap" rel="stylesheet">
<link rel="canonical" href="https://swiftrocks.com/benchmarking-swift-code-properly-with-attabench"/>
<!-- Bootstrap CSS Plugins -->
<link rel="stylesheet" type="text/css" href="css/bootstrap.css">
<!-- Prism CSS Stylesheet -->
<link rel="stylesheet" type="text/css" href="css/prism4.css">
<!-- Main CSS Stylesheet -->
<link rel="stylesheet" type="text/css" href="css/style48.css">
<link rel="stylesheet" type="text/css" href="css/sponsor4.css">
<!-- HTML5 shiv and Respond.js support IE8 or Older for HTML5 elements and media queries -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]-->
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "BlogPosting",
"mainEntityOfPage": {
"@type": "WebPage",
"@id": "https://swiftrocks.com/benchmarking-swift-code-properly-with-attabench"
},
"image": [
"https://swiftrocks.com/images/thumbs/thumb.jpg"
],
"datePublished": "2021-07-20T14:00:00+02:00",
"dateModified": "2021-07-20T14:00:00+02:00",
"author": {
"@type": "Person",
"name": "Bruno Rocha"
},
"publisher": {
"@type": "Organization",
"name": "SwiftRocks",
"logo": {
"@type": "ImageObject",
"url": "https://swiftrocks.com/images/thumbs/thumb.jpg"
}
},
"headline": "Benchmarking Swift Code Properly with Attabench",
"abstract": "Hmmmm, how fast is this piece of code? Let's find out!"
}
</script>
</head>
<body>
<div id="main">
<!-- Blog Header -->
<!-- Blog Post (Right Sidebar) Start -->
<div class="container">
<div class="col-xs-12">
<div class="page-body">
<div class="row">
<div><a href="https://swiftrocks.com">
<img id="logo" class="logo" alt="SwiftRocks" src="images/bg/logo2light.png">
</a>
<div class="menu-large">
<div class="menu-arrow-right"></div>
<div class="menu-header menu-header-large">
<div class="menu-item">
<a href="blog">blog</a>
</div>
<div class="menu-item">
<a href="about">about</a>
</div>
<div class="menu-item">
<a href="talks">talks</a>
</div>
<div class="menu-item">
<a href="projects">projects</a>
</div>
<div class="menu-item">
<a href="software-engineering-book-recommendations">book recs</a>
</div>
<div class="menu-item">
<a href="games">game recs</a>
</div>
<div class="menu-arrow-right-2"></div>
</div>
</div>
<div class="menu-small">
<div class="menu-arrow-right"></div>
<div class="menu-header menu-header-small-1">
<div class="menu-item">
<a href="blog">blog</a>
</div>
<div class="menu-item">
<a href="about">about</a>
</div>
<div class="menu-item">
<a href="talks">talks</a>
</div>
<div class="menu-item">
<a href="projects">projects</a>
</div>
<div class="menu-arrow-right-2"></div>
</div>
<div class="menu-arrow-right"></div>
<div class="menu-header menu-header-small-2">
<div class="menu-item">
<a href="software-engineering-book-recommendations">book recs</a>
</div>
<div class="menu-item">
<a href="games">game recs</a>
</div>
<div class="menu-arrow-right-2"></div>
</div>
</div>
</div>
<div class="content-page" id="WRITEIT_DYNAMIC_CONTENT">
<!--WRITEIT_POST_NAME=Benchmarking Swift Code Properly with Attabench-->
<!--WRITEIT_POST_HTML_NAME=benchmarking-swift-code-properly-with-attabench-->
<!--Add here the additional properties that you want each page to possess.-->
<!--These properties can be used to change content in the template page or in the page itself as shown here.-->
<!--Properties must start with 'WRITEIT_POST'.-->
<!--Writeit provides and injects WRITEIT_POST_NAME and WRITEIT_POST_HTML_NAME by default.-->
<!--WRITEIT_POST_SHORT_DESCRIPTION=Hmmmm, how fast is this piece of code? Let's find out!-->
<!--DateFormat example: 2020-04-12T14:00:00+02:00-->
<!--WRITEIT_POST_SITEMAP_DATE_LAST_MOD=2021-07-20T14:00:00+02:00-->
<!--WRITEIT_POST_SITEMAP_DATE=2021-07-20T14:00:00+02:00-->
<title>Benchmarking Swift Code Properly with Attabench</title>
<div class="blog-post">
<div class="post-title-index">
<h1>Benchmarking Swift Code Properly with Attabench</h1>
</div>
<div class="post-info">
<div class="post-info-text">Published on 20 Jul 2021</div>
</div>
<p>Hmmmm, how fast is this piece of code? Let's find out!</p>
<div class="sponsor-article-ad-auto hidden"></div>
<p>When it comes to benchmarking the speed of code, it's common for people to boot a playground, throw in a <code>Date</code> object and calculate the time difference after a piece of code runs. That may give you a rough estimate, but it can also be very misleading. The difference between code running in debug builds versus release ones can be massive, and the size of the input (assuming we're benchmarking an algorithm) can also make a huge difference in the speed of a function. In this article, we'll see how to properly benchmark your Swift code. (Note that we're not talking about things like iOS startup time -- we're talking about the speed to run a specific piece of code.)</p>
<h2>Introducing Attabench</h2>
<p><b>Attabench</b> is an open-source benchmarking tool created by Apple engineer Karoy Lorentey back in 2017 and has been my favorite benchmarking utility ever since I found it. The tool itself is a wrapper on top of Swift Package Manager that includes a benchmarking framework in where you can set up the code you'd like to measure, accompanied by a GUI app that lets you to see and customize how the result is presented:</p>
<div class="post-image">
<img src="https://i.imgur.com/4p4iHrb.png" alt="Attabench">
</div>
<p>The measuring process is similar to how one would do with a playground, but Attabench's SPM abstraction not only compiles your app in the release configuration but also runs your code multiple times with different inputs, later providing a graph showing how your code behaves in function of the size of its input.</p>
<p>The project has technically been abandoned in favor of the <a href="https://github.com/apple/swift-collections-benchmark">Swift Collections Benchmark package</a> that is roughly the same thing as Attabench, but the package is heavily command-line based which makes it a lot harder to use than Attabench. For that reason, I think that Attabench is still the superior benchmarking tool.</p>
<p>You can download Attabench's source code in <a href="https://github.com/attaswift/Attabench">its repo</a>, but since the project has been abandoned a long time ago you may have issues compiling it with newer Xcode versions. My friend Rafael Machado was able to fix the compilation issues and provide a binary, <a href="https://drive.google.com/file/d/1ItV_YGGs8puCI1vvazmlttAydK1n20iv/view?usp=sharing">which you can download here for simplicity.</a></p>
<p>Before you open Attabench, we must first create a benchmarking file. It doesn't seem like you can create this from the tool itself, so we'll do it by copying an existing one. The zip downloaded I linked earlier has a <b>Sorting.attabench</b> file -- copy it somewhere and open it with Attabench.</p>
<p>When you open the file, you'll see a benchmark that compares Swift's default sorting method with a custom Quicksort. If you click the <b>Run</b> button, Attabench will start continuously running the two methods with different input sizes and update the graph as it proceeds:</p>
<div class="post-image">
<img src="https://i.imgur.com/Gk4PxMV.png" alt="Alt">
</div>
<p>It takes a massive amount of time for Attabench to fully complete its run, but you don't need for all of it. I normally wait just a couple of seconds until the graph appears to be stable enough and stop it.</p>
<p>One interesting configuration you can play with is the <b>input size</b> in the top left of the app, which controls the minimum and maximum value that Attabench will use in its runs. In this case, the value between 1 and 1 million dictates the number of elements in the array that is going to be parsed.</p>
<p>Another important configuration is <i>how</i> is shown to you, available at the top right. By default the graph shows the average of how long it takes and on a logarithmic scale, but I'm not smart enough to understand what I'm supposed to do with that information. I personally like to disable everything and see precisely how long the code takes to run for each input size.</p>
<p>Now that you know how to use Attabench, let's see how you can create your own benchmarks. The <b>.attabench</b> file is a workspace that you can open as a folder, and inside you'll find a SPM <b>Package.swift</b> file. If you open it in Xcode, you'll be able to see and modify the benchmark:</p>
<pre><code>let benchmark = Benchmark<[Int]>(title: "Sorting")
benchmark.descriptiveTitle = "Time spent sorting elements"
benchmark.descriptiveAmortizedTitle = "Average time spent sorting elements"
benchmark.addSimpleTask(title: "Swift.sort") { input in
_ = input.sorted()
}
benchmark.addTimerTask(title: "Quicksort") { (input, timer) in
var input = input
timer.measure {
input.quicksort()
}
}
benchmark.start()</code></pre>
<p>As you can see from the snippet, the benchmarking process is as simple as defining a benchmark object and registering the code that you want to measure.</p>
<p>The generic argument of the <code>Benchmark</code> represents the type of the input, which in this case is an array. You can modify it to be anything you want, as long you provide a function that generates the input for a given provided size. For example, if you'd like to create a benchmark where the input is a number, here's how we could define the <code>Benchmark</code> object:</p>
<pre><code>let benchmark = Benchmark<Int>(title: "Calculate number of digits in a number") { size in
return size
}</code></pre>
<p>As you might've realized by now, <code>size</code> is a random number between the range you defined previously in Attabench.</p>
<p>To define a block of code to be measured, simply add a call to <code>addSimpleTask</code> (heh) with the measured content inside a closure:</p>
<pre><code>benchmark.addSimpleTask(title: "Swift.sort") { input in
_ = input.sorted()
}</code></pre>
<p>By default, the entire content of the closure will be measured. If you'd like to have finer-grained control over what's being measured, you can use one of the lower-level abstractions like <code>addTimerTask</code>. In this example case, I use a timer task to make sure Quicksort's benchmarking includes only the time it takes to run the algorithm itself, and not the setup needed to run it (creating a mutable copy):</p>
<pre><code>benchmark.addTimerTask(title: "Quicksort") { (input, timer) in
var input = input
timer.measure {
input.quicksort()
}
}</code></pre>
<div class="sponsor-article-ad-auto hidden"></div>
<p>Finally, you can call <code>benchmark.start()</code> to run the benchmark. To update the Attabench app after changing the code, simply click the refresh button at the top-left corner.</p>
</div>
</div>
<div class="blog-post footer-main">
<div class="footer-logos">
<a href="https://swiftrocks.com/rss.xml"><i class="fa fa-rss"></i></a>
<a href="https://twitter.com/rockbruno_"><i class="fa fa-twitter"></i></a>
<a href="https://github.com/rockbruno"><i class="fa fa-github"></i></a>
</div>
<div class="footer-text">
© 2025 Bruno Rocha
</div>
<div class="footer-text">
<p><a href="https://swiftrocks.com">Home</a> / <a href="blog">See all posts</a></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<!-- Blog Post (Right Sidebar) End -->
</div>
</div>
</div>
<!-- All Javascript Plugins -->
<script type="text/javascript" src="js/jquery.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/js/bootstrap.bundle.min.js" integrity="sha384-MrcW6ZMFYlzcLA8Nl+NtUVF0sA7MsXsP1UyJoMp4YLEuNSfAP+JcXn/tWtIaxVXM" crossorigin="anonymous"></script>
<script type="text/javascript" src="js/prism4.js"></script>
<!-- Main Javascript File -->
<script type="text/javascript" src="js/scripts30.js"></script>
<!-- Google tag (gtag.js) -->
<script async src="https://www.googletagmanager.com/gtag/js?id=G-H8KZTWSQ1R"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'G-H8KZTWSQ1R');
</script>
</body>
</html>