Proc Macros to support and simplify testing

[MusicalNinjaDad]

I'm just getting started learning rust with a view to porting a (currently private) mathematical library from native python to rust and making it public. Pyo3 is wonderful and has really made life easy for what I was expecting to be one of the most complicated aspects of the task.

One thing I have noticed is that unit testing the rust code after it is wrapped by pyo3 contains quite some boilerplate and was not so easy to work out how to do it.

Testing strategy I'm following:

I have tests on all 3 sections of code, each with a specific focus:

1. rust library providing base functionality: interface and functionality tested by what rust terms "integration tests"
2. pyo3 library providing python-bound functionality: functionality binding, type-mapping & error-handling tested by rust "unit tests"
3. python package containing the bound functionality: overall functionality & integration tested with pytest tests

In order to simplify point (2) I'm about to start creating a few proc-macros, initially something like:

#[pyo3test(module,function)]

which will prefix the test code with:

pyo3::append_to_inittab!(module...
pyo3::prepare_freethreaded_python();
        Python::with_gil(|py| {
... .import_bound(module...
... .get_attr(function....

maybe later also wrap let result = function.call... in match ... unwrap

#[pyo3raises(...)]

which will be the equivalent of:

with pytest.raises(...):
    ....

My Question

Is this something you would be interested in potentially integrating into pyo3? In which case I will try to follow the same pattern you have for segregating proc_macros and their implementations, write the documentation in the same style etc... I'd be happy to help support maintaining and extending the testing functionality and documentation.

Or is this not really something you see in pyo3 in the near future? In which case I will think about packaging it up in a separate way and documenting it more in line with my other projects, which I am just beginning to make public.

I'm guessing that integration into pyo3 will lead to a faster uptake and interest in extending and improving the functionality due to the wider reach and will add value to both pyo3 and the testing macros by not expecting users to find and use an external solution for testing their code. At the same time it would add further complexity to the project, ...

What are your thoughts?

[davidhewitt]

Hello, thanks for the suggestions! I don't know if we necessarily need this in the main pyo3 crate, but we could support it as a pyo3-testing crate if these look like they provide a lot of value.

The #[pyo3test] macro - it would definitely be nice to remove some boilerplate. I'm not totally sure whether all tests need (module, function) pairs. Avoiding the rightward drift of Python::with_gil would be 👍 from me.

The #[pyo3raises(...)] macro - I think probably an attribute is the wrong form and you'd want something more like py_assert_raises!(some_expression, PyValueError, match='foo'). We could make a lot of use of this in PyO3's test suite too. Probably this doesn't need to be a proc-macro even; a regular macro_rules! would probably handle this fine.

[MusicalNinjaDad]

Thanks for the quick answer and good pointers ... I'll keep you updated with progress via this discussion.

[davidhewitt]

Some prior art in #[tokio::test]: https://docs.rs/tokio-macros/latest/tokio_macros/attr.test.html

[MusicalNinjaDad]

I've done some experimenting in a sandbox (and learnt a lot about proc-macros, syn and parse along the way)...

I agree that the better solution is not to include any import info in the proc_macro attribute. I need to pull it out of the remaining function though, or trying to put the code initiating python in the right place becomes tricky and restrictive.

The approach that looks most promising to me would be to turn:

...
#[pymodule]
#[pyo3(name = "fizzbuzzo3")]
fn py_fizzbuzzo3(module: &Bound<'_, PyModule>) -> PyResult<()> {
    module.add_function(wrap_pyfunction!(py_fizzbuzz, module)?)?;
    Ok(())
}
...
#[cfg(test)]
mod tests {
    use pyo3::exceptions::PyTypeError;

    use super::*;

    #[test]
    fn test_fizzbuzz() {
        pyo3::append_to_inittab!(py_fizzbuzzo3);
        pyo3::prepare_freethreaded_python();
        Python::with_gil(|py| {
            let fizzbuzzo3 = py
                .import_bound("fizzbuzzo3")
                .expect("Failed to import fizzbuzzo3");
            let fizzbuzz = fizzbuzzo3
                .getattr("fizzbuzz")
                .expect("Failed to get fizzbuzz function");
            let result: PyResult<String> = match fizzbuzz.call1((1i32,)) {
                Ok(r) => r.extract(),
                Err(e) => Err(e),
            };
            let result = result.unwrap();
            let expected_result = "1";
            assert_eq!(result, expected_result);
        });
    }
}

into:

...
#[pymodule]
#[pyo3(name = "fizzbuzzo3")]
fn py_fizzbuzzo3(module: &Bound<'_, PyModule>) -> PyResult<()> {
    module.add_function(wrap_pyfunction!(py_fizzbuzz, module)?)?;
    Ok(())
}
...
#[cfg(test)]
mod tests {
    use pyo3::exceptions::PyTypeError;

    use super::*;

    #[pyo3test]
    #[pyo3(import = py_fizzbuzzo3: "from fizzbuzzo3 import fizzbuzz")]
    fn test_fizzbuzz() {
            let result = pyo3_call!(fizzbuzz, 1i32);
            let expected_result = "1";
            assert_eq!(result, expected_result);
        });
    }
}

Question 1: I ended up starting to rebuild a lot of what you already have in pyo3-macros-backend/src/attributes.rs so would find it best to add an "import" keyword in there if that's OK?

Question 2: Would that fit your ideas and overall design?

Question 3: I would intend to put the macro_rules and a reexport of the proc_macro into pyo3-testing, the proc_macro into pyo3-testing-macros and add the custom keyword directly to pyo3-macros-backend to avoid a third crate. Does that make sense?

All of these ideas can change, based on how the code progresses and your thoughts on the drafts, just looking for some starting guidance if there is anything you feel strongly about...

I'll have a look at the raises options afterwards. Right now I'm not sure how successfully I could get a macro_rules! to cope with nesting a pyo3_call!() inside pyo3_raises!(), but I'm sure that will become clear when working on it (maybe it'll just work "out of the box").

[MusicalNinjaDad]

I finally had a good chance to sit down an look at this... I'd really welcome your views on rough implementation here: https://github.com/MusicalNinjaDad/pyo3/tree/pyo3-testing

• I'm not completely happy with the naming
• it's happy path only, no error or edge case handling (e.g. all cases have exactly one import, everything is using plain unwrap, ...)
• there is no proc-macro wrapper yet - the whole thing is backend-only using TokenStream2 as starting and ending-point. The goal is for the proc-macro to simply be a one-liner impl_pyo3test(attr.into(), input.into()).into()
• the best place to start reading is probably the final test case (currently test_macro at line 162)

I'm completely new to rust, so not yet up to speed on the main idioms and it's been over 20 years since I last did any real work with a low-level language - so feedback is very welcome. Any pointers on the kind of error to raise in unexpected cases, ability to streamline the code etc., improvements to the API to make it easy to recognise for both rust and python coders alike, ...

[MusicalNinjaDad]

Hi @davidhewitt - I've refactored, polished and finished the first basic version end-to-end. I'd really welcome your views on the overall design, how to best integrate it (I was thinking a feature which is enabled by default; a subsection 2.2.3 in the docs "Testing python functions") and which other functionality would be needed for this to be worthy of a PR as pyo3-testing v0.1.0

What's the easiest way to share with you? Raising a PR and then continuing to work or sharing my working branch directly?

[MusicalNinjaDad]

Hi @davidhewitt , I understand you're busy and that the next major work is on technical improvements rather than polishing this bit of the UI.

I hope you don't mind: I broke out pyo3-testing into an independent crate (for now) and added in the raises functionality you were interested in... Any comments more than welcome

I'd be honoured if you did consider

• a mention in the docs
• moving the project under the pyo3 organisation or even
• bringing the functionality into the main codebase

and have no problems "handing over the keys", so to speak.