GH Actions/quicktest: use Linux Arm64

[jrfnl]

Description

ℹ️ PR title and description has been updated to reflect the latest state of the PR.

---

GitHub has made Linux arm64 runners generally available and running tasks on these images instead of the traditional images can deliver up to a 40% performance boost.

Based on our testing with this, the performance benefit is minimal to non-existent in our test setup and as the code base contains no arm64 specific code/conditions, there is also no functional benefit to running the tests on arm64.

Having said that, having some test runs on arm64 can serve as an early detection system in case code changes would be needed in the future.

With this in mind and after some discussion about this, this commit ONLY introduces arm64 in the quicktest workflow which is run for branch pushes (with the exception of pushes to master).

That means that in practice, it will run:

• When contributors with push rights push a branch directly to this server.
• When contributors without push rights, but who have enabled workflow runs on their own fork of the repo, push a branch to their fork.

The "normal" (PR) test runs will not use arm64 at this time.

This does mean that a test run failure against arm64 for a PR contribution via a fork could go unnoticed, but as stated above: as the code contains no arm64 specific conditions, this is deemed unlikely and is seen as an acceptable risk.

If needs be, this can be revisited in a later iteration.

Refs:

• https://github.blog/news-insights/product-news/arm64-on-github-actions-powering-faster-more-efficient-build-systems/
• https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources
• https://fraudmarc.com/post/arm64-vs-x86-64-for-php

Suggested changelog entry

N/A

[fredden]

The changes here look good to me. I think we can switch over all the PHP 5.x builds without much impact.

I see that we'll need to set COVERAGE_REPORTER_PLATFORM so that the correct flavour of binary gets installed for the Coveralls part to work successfully. Ideally that fix gets pushed to the script itself rather than having to set the variable, as uname -m will give the value that they're looking for. See coverallsapp/github-action#218

Edit: This change should make the tests pass here. coverallsapp/github-action#238

[fredden]

While there are more places where this change can be applied, this is a good first step.

[jrfnl]

The changes here look good to me. I think we can switch over all the PHP 5.x builds without much impact.

I see that we'll need to set COVERAGE_REPORTER_PLATFORM so that the correct flavour of binary gets installed for the Coveralls part to work successfully. Ideally that fix gets pushed to the script itself rather than having to set the variable, as uname -m will give the value that they're looking for. See coverallsapp/github-action#218

Edit: This change should make the tests pass here. coverallsapp/github-action#238

@fredden I was just about to look at the build failure when I got interrupted by a phone call. Thanks for finding the root cause so quickly!

coverallsapp/github-action#238 looks like magic to me. 🙌🏻
I've also opened a loosely related PR - coverallsapp/github-action#239 , though that one may not be needed once coverallsapp/github-action#238 gets merged.

I'm going to give it a few days, but may just update this PR with the coverage-reporter-platform input to make it mergeable. We can always remove it again later once the upstream PR(s) have been merged.

[afinetooth]

@fredden @jrfnl Thanks for your PRs. I've merged them both and am cutting new release of coverallsapp/github-action@v2 right now. Should be v2.3.5 but @v2 will continue pointing to the latest release.

[jrfnl]

@afinetooth That's awesome! Thank you for working with us on this. Once the release is out, I will rebase this PR to retrigger the build and 🤞🏻

[afinetooth]

@jrfnl (Cc: @fredden)

• v2.3.5 release of coverallsapp/github-action is now available, by using coverallsapp/github-action@v2. 🚀

Let me know if you have any issues! 🤙

Have a great weekend! 🎉

[jrfnl]

Thanks @afinetooth ! I've rebased the PR without changes. Build is running.

[jrfnl]

@fredden @afinetooth Uh-oh... looks like the build is still failing: https://github.com/PHPCSStandards/PHP_CodeSniffer/actions/runs/12958904548/job/36150203750

@fredden Also: for the "quicktest" workflow (just test, no Xdebug or code coverage), the PHP 5.4 run is about 25% faster (~6 vs 8 minutes), but for the "coverage" job, running on arm64 unfortunately doesn't look to make any difference... If you have any bright ideas about this, I'd be happy to hear them.

[afinetooth]

@jrfnl OK, I'll check it out...

[jrfnl]

@afinetooth Let me know if you want me to do a run with debug turned on or anything.

[afinetooth]

@jrfnl yes, debug: true would probably be helpful.

The Exec format error error indicates an architecture mismatch between the binary called and the runner.

(I assume arm64 for the runner and x86_64 for the binary. In fact, I can better than assume for the binary since the value is shown on Line 129: COVERAGE_REPORTER_PLATFORM: x86_64.)

debug: true should clarify which binary was actually called, though.

But right now, I can't tell for sure what the architecture of the runner was, so I would love your help adding a debug line showing what the value of (uname -m) was. Or else what GitHub Actions exposes as RUNNER_ARCH, or both, which should give us that as a certainty.

Unless I'm missing it somewhere in your workflow, you are not explicitly setting coverage-reporter-platform, yet it is somehow getting set to x86_64 by virtue of COVERAGE_REPORTER_PLATFORM: x86_64 on Line 129. 🤷‍♂️

---

A little more on RUNNER_ARCH:

As I say I assume its arm64 from the flag-name chosen, os-ubuntu-24.04-arm-php-5.4-custom-ini-false, though that doesn't necessarily guarantee it.

I can't tell from the "set up job" step---can you?:

• Included Software is close to certainty, but it's not the same as RUNNER_ARCH:

The closest hint after that is this line in the raw logs:

2025-01-24T22:50:24.6442845Z Acquiring 20.18.2 - arm64 from https://github.com/actions/node-versions/releases/download/20.18.2-12900461873/node-20.18.2-linux-arm64.tar.gz

But that's just the version of node being downloaded for arm64 architecture, right?

---

This should expose the "sure things" (RUNNER_ARCH and (uname -m):

jobs:
  check-runner-arch:
    runs-on: ubuntu-latest  # Change to your actual runner OS
    steps:
      - name: Print Runner Architecture
        run: |
          echo "RUNNER_ARCH: $RUNNER_ARCH"
          echo "uname -m: $(uname -m)"

[afinetooth]

👀 🙏

[jrfnl]

@jrfnl yes, debug: true would probably be helpful.

@afinetooth I've added a debug commit which adds the "Print Runner Architecture" step as per above as the first step in the coverage runs + has debug: true for the coverallsapp/github-action steps.

This will be the job to examine once it finishes in 25 minutes or so: https://github.com/PHPCSStandards/PHP_CodeSniffer/actions/runs/12960113668/job/36153483913

Unless I'm missing it somewhere in your workflow, you are not explicitly setting coverage-reporter-platform, yet it is somehow getting set to x86_64 by virtue of COVERAGE_REPORTER_PLATFORM: x86_64 on Line 129. 🤷‍♂️

Correct as, if I understood things correctly, that should be set automagically now PR coverallsapp/github-action#238 has been merged & released.

[afinetooth]

Correct as, if I understood things correctly, that should be set automagically now PR coverallsapp/github-action#238 has been merged & released.

Same here.

So if it wasn't set explicitly, that suggests (uname -m) returned x86_64---despite all other indications.

We'll see in ~25 min! 🍿

[jrfnl]

@afinetooth I may need to bow out for some 💤 before the run finishes, but will happily make any further debug adjustment tomorrow.

[jrfnl]

[afinetooth]

So either:

1. Our new code isn't working; OR
2. COVERAGE_REPORTER_PLATFORM is getting set, to x86_64, somehow, ahead of the run.

Re-examining the code, and the run...

[fredden]

So either:

1. Our new code isn't working; OR

2. COVERAGE_REPORTER_PLATFORM is getting set, to x86_64, somehow, ahead of the run.

Re-examining the code, and the run...

I've just had a look at this now. The variable has a default value set of "x86_64". To fix this, we can either change the default (to an empty string), or remove the variable altogether. @afinetooth, which option would you prefer? I'm happy to open a pull request to sort this out.

https://github.com/coverallsapp/github-action/blob/773b6d8e80fa7862da56a7664bd747c91255b2e2/action.yml#L79

Edit: I have opened coverallsapp/github-action#240 in anticipation of the "change the default" option. I'm happy to adapt this into "remove the parameter" if you would prefer.

[afinetooth]

@fredden removing the default value of x86_64 for the coverage-reporter-platform option seems a good approach, since it retains the ability to force-set the value.

Reviewing your PR...

[jrfnl]

@fredden @afinetooth Thank you both for working with me on this. I've retriggered the build for PHP 5.4, so let's see in 25 minutes or so 🤞🏻

[afinetooth]

@fredden @jrfnl v2.3.6 of coverallsapp/github-action is now used by coverallsapp/github-action@v2. 🚀 🍿 🤞

[fredden]

Checks are now passing. 🎉

[jrfnl]

Looks like the upload was successful now! 🎉

@fredden There's still the question about how useful this is for now. Have you got an opinion ?

Also: for the "quicktest" workflow (just test, no Xdebug or code coverage), the PHP 5.4 run is about 25% faster (~6 vs 8 minutes), but for the "coverage" job, running on arm64 unfortunately doesn't look to make any difference... If you have any bright ideas about this, I'd be happy to hear them.

I would also be happy to update/expand the PR, to have a more balanced mix of Linux vs ARM64 build, though I do also want to continue to have at least some runs on "plain" Linux.

[afinetooth]

Thank you and congratulations, you guys. 🎉

[fredden]

the question about how useful this is for now.

It does look like the performance (ie, time) benefit is minimal. Looking at a recent run with and without this change, it looks like the arm64 version actually took one minute longer.

x86_64: 25m40s (unrelated merged pull request)
arm64: 26m38s (this pull request)

This page has an interesting analysis of arm64 versus x86_64: https://fraudmarc.com/post/arm64-vs-x86-64-for-php

While this doesn't achieve the performance boost that we had hoped for, I think that there is value with this change in that we are now testing the software on a slightly wider set of platforms. Perhaps we can change one or two other builds over to arm64 too, and then say that we run on a selection for "coverage" but not for performance.

[jrfnl]

@fredden Thanks for sharing your thoughts.

the question about how useful this is for now.

It does look like the performance (ie, time) benefit is minimal. Looking at a recent run with and without this change, it looks like the arm64 version actually took one minute longer.

This page has an interesting analysis of arm64 versus x86_64: https://fraudmarc.com/post/arm64-vs-x86-64-for-php

I've read the article you linked.

Test/coverage builds run with parallel turned off, while there is a plain phpcs run in the test workflow which runs with parallel=75. Might be that the effect might be more noticeable with that, but the benefits would be minimal all the same as that step typically runs in ~4 (PHP 8.4) to ~15 (PHP 5.4) seconds...

While this doesn't achieve the performance boost that we had hoped for, I think that there is value with this change in that we are now testing the software on a slightly wider set of platforms. Perhaps we can change one or two other builds over to arm64 too, and then say that we run on a selection for "coverage" but not for performance.

While I can see the benefit from a "we've tested with this" point of view, from a functional testing point of view, there is no real benefit as the code base doesn't contain any code which is expected to behave differently on x86_64 vs arm64 (unlike testing against Windows, for which the code base contains a number of specific conditions).

My current thinking considering the minimal performance benefits:

• Switch (all) the *nix jobs in quicktest workflow over to arm64 to ensure that all PRs/ branch pushes have at least some runs against arm64 - kind of a "just in case" thing.
• Don't use arm64 in the test workflow.

What say you ?

[fredden]

Yes, that sounds like a good plan to me. 👍

[jrfnl]

Thanks for working with me on this @fredden !

I've now:

• Removed the "Debug" commit.
• Updated the original commit to reflect the proposal to only use this in the quicktest workflow.
• Updated the commit title and description to match.
• Updated the PR title and description to match.

I believe this is now ready for a final review.