vim-test integration

[gfontenot]

Thesis

• vim-xcode is currently able to build a complete command to build or test an Xcode project
• vim-test is able to look at files and pick test runners based on those files in order to run a suite of tests, a single file's tests, or even a single test.
• xcodebuild now (I don't know when this was added) has the ability to run individual test bundles/classes/cases with a flag: -only-testing:Bundle[/Class[/Case]].

We should be able to hook these up nicely.

Implementation

Building the command building logic into vim-test itself probably doesn't make sense since there's plenty of intricacy involved and we'd end up re-implementing the bulk of vim-xcode. Likewise, vim-test has a lot of logic for parsing the bundle/class/case from a single file and building a flag (or list of flags) to pass to a test runner command accordingly that doesn't really make sense to duplicate here. That being said, we should be able expose the plumbing needed in vim-xcode and then end-users can add some local config in order to expose it to vim-test.

The gist would be to ship a test new runner integration with vim-xcode. This basically just means adding an autoload/test/xcode/xcode.vim file to this plugin and following the extension steps laid out by vim-test. End-users would then need to add let test#custom_runners = {'Xcode': ['Xcode']} to their local vimrc in order for vim-test to see our runner. We'd also need to open up some of the internals of vim-xcode via exposed functions (xcode#test_command(), for example) so that we can access them from the vim-test integration.

Problems

Some issues I've run into this while testing that would need to be figured out (this list might change over time):

• How do we disambiguate between testing with Xcode and testing with swift test when they are both technically valid options? We might need to add something similar to the explicit runner selection config they have for Python and Go, but I'm not entirely sure how that would/should work since the only "official" runner is swiftpm.
• How do we determine the test bundle properly? If the bundle doesn't match the name of the directory on disk, the command will end up erroring. For example, parsing the test bundle for Argo based on the file structure would result in ArgoTests. This is fine if you're testing the Argo-iOS scheme because that happens to be the name of the test bundle in Xcode. But if you test the Argo-Mac scheme, it fails with a semi cryptic error because there are no available tests. To make things worse, while you can run xcodebuild -list to get a list of possible targets, there's really no way to 100% tell which target is a test target, and no way at all to tell which targets are linked to which schemes.

[gfontenot]

@janko-m: If you happen to have any thoughts about this I'd love to hear them.

[janko]

Thanks for the initiative 👍

How do we disambiguate between testing with Xcode and testing with swift test when they are both technically valid options?

Yeah, trying to determine which runner the user wants to use can sometimes be very challenging, that's why for some languages in vim-test you need to explicitly specify the runner. If you find that there is no clear way of distinguishing the two, I'd just go with requiring users to explicitly choose whether they want to use switftpm or xcodebuild runner.