CellCharter permutation test

[milddaa]

Description of feature

Hello! I was wondering about the permutation test implemented by CellCharter in differential neighbourhood enrichment. 1) What is the number of permutations I should be using given my sample size? I have two groups with 8 samples in each. 2) Should the derived p-values be additionally adjusted for multiple testing, and if so, how would you recommend to do it? I have 9 clusters total (cell types). Thank you very much in advance!

[marcovarrone]

Hi @milddaa, thank you for the interesting questions!

The number of permutations depends on a lot of factors. Since it is an empirical pvalue I would say at least 100 permutations to be able to obtain pvalues down to 0.01, but since you have many samples in the two conditions you can give further (e.g. 1000 permutations), but if your dataset is very big, it may take a while.

Regarding multiple testing yes, I would suggest something like Bonferroni correction based on the number of values in the neighborhood enrichment matrix. That would be 99-9=72 values if you only inter-cluster enrichment or 99=81 if you also have intra-cluster enrichment.

You can get the matrix of pvalues and differential enrichment by passing copy=True to diff_nhood_enrichment.

[milddaa]

Hi @marcovarrone , thank you so much for your response. I am working to implement CellCharter in two projects, in another the sample size is 4+5, would 1000 permutations still work in such case? I also noticed that p-values change when setting a different seed, which is of course expected but in practice it means that some of them can cross the significance threshold with different seed. How should I deal with that in your opinion? Should I try setting it a few times and only report the consistent findings, or perhaps increasing the number of permutations would help? Finally, does p-value adjustment still make sense when the empirical p-value is 0?
Sorry for so many questions! And thanks a lot for your time!