ImportError: cannot import name 'dedup_cython' from partially initialized module 'pairtools.lib' #157
Replies: 18 comments
-
Hi @georgette-femerling , how did you install pairtools? pip install numpy pysam cython
pip install pairtools one after another? |
Beta Was this translation helpful? Give feedback.
-
Hi @agalitsyna , I am having the same issue and error message using python 3.7.7 - I had installed pairtools the way you described. |
Beta Was this translation helpful? Give feedback.
-
Hi @aosakwe , @georgette-femerling, I was unable to reproduce your problem in a fresh environment with python 3.7.7 (also 3.8.10). It runs fine if appropriately installed. Two modifications/checks that I can propose:
pip install pysam numpy cython --no-cache-dir
pip install pairtools --no-cache-dir
which pip
which python
which pairtools These three should sit in the same directory, e.g. /home/username/anaconda3/envs/test/bin/ Explanation: However, if pip and python have different locations, they might also have different paths for searching for libraries like cython and pysam. Another problem might be that you have tried to install pairtools before, so |
Beta Was this translation helpful? Give feedback.
-
Hi, I tried reinstalling them in a new environment (using the same code you have provided) and all three are in the same directory. I still encounter the same issue. I did try to install pairtools before, how could I make resolve this issue? |
Beta Was this translation helpful? Give feedback.
-
You may explicitly remove cache of pip, e.g. Another option might be installing from source: git clone https://github.com/open2c/pairtools.git
cd pairtools
pip install -e . |
Beta Was this translation helpful? Give feedback.
-
There seems to be no residual cache. I get the following installation error when installing from source: Installing collected packages: pairtools
|
Beta Was this translation helpful? Give feedback.
-
Hi @aosakwe , this is some pysam-related problem, not pairtools. As you can see in your output, there is no htslib file, which is part of pysam. I don't know why it might happen on your computer, and looking through the pysam issues on github might be the best idea. Here is an answer that might guide you: pysam-developers/pysam#262 My other suggestions are:
|
Beta Was this translation helpful? Give feedback.
-
Hi @aosakwe - thanks so much for the updates tool, I'm eager to try it out but am running into the same issue. One possible solution is to use pep518 to specify the build dependencies in the pyproject.toml file. That way pip would know to install numpy, cython and pysam before trying to build pairtools. There's some details on how it would work for a Cython package here: https://levelup.gitconnected.com/how-to-deploy-a-cython-package-to-pypi-8217a6581f09. Failing that, would it be possible to update the version on bioconda? That would allow users to avoid the build-time issues. |
Beta Was this translation helpful? Give feedback.
-
hi @eharr, good suggestion. I don't think it works without some package managers like The problem is that pyproject.toml and pep518 require building the package in isolated environment, and cannot include the libraries linked from outside like Here is the record of why pyproject.toml does not work with pysam dependency: https://github.com/open2c/pairtools/commits/remove-query-scaling |
Beta Was this translation helpful? Give feedback.
-
Thanks @agalitsyna I'll take a look at the pyproject.toml if I get a chance. I think I've narrowed down the issue to some difference between conda and virtualenv. I noticed from the thread above that you're using a conda environment as a base for your |
Beta Was this translation helpful? Give feedback.
-
I spent a bit more time looking at this today (python packaging is a mess!) and I'm not sure there's an easy way to make this pip-installable. It might be that something like cibuildwheel would fix the linking issue but I can't be sure. Another option might be to vendor in the pysam package. In either case it's probably a lot of work and might not be worth the additional complexity. That said, I think it makes it more important to update the version available through bioconda so that folks can start to use it in their pipelines. |
Beta Was this translation helpful? Give feedback.
-
Hi @eharr, thanks for looking into this. We've pinged bioconda developers to review and merge the most recent pairtools. Recent pairtools should be added to bioconda soon. |
Beta Was this translation helpful? Give feedback.
-
Awesome - thank you @agalitsyna! |
Beta Was this translation helpful? Give feedback.
-
Hi, I'm also finding the same issue when installing pairtools from scratch in a plain Ubuntu:22:04 docker (docker file attached bellow). Is there a solution in sight? This is somewhat critical for our analytical pipelines ###Dockerfile:
|
Beta Was this translation helpful? Give feedback.
-
Hi @mblanche , have you tried separating pip installation for cython/numpy/pysam and pairtools?
|
Beta Was this translation helpful? Give feedback.
-
Yeah, that’s what I just did (I putt an && and did two pip install) and it seems to be solving the problem. I’ll wait for the job to complete and post the docker solution back later.
…Sent from my iPad
-- Marco
On Oct 5, 2022, at 10:51 AM, agalitsyna ***@***.***> wrote:
Hi @mblanche , have you tried separating pip installation for cython/numpy/pysam and pairtools?
I find no trouble with this version of your Dockerfile:
FROM ubuntu:22.04
USER root
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update -y\
&& DEBIAN_FRONTEND=noninteractive \
&& apt-get install -y \
python3 \
python3-pip \
git \
samtools \
tabix \
libbz2-dev \
lz4 \
zlib1g-dev \
build-essential \
autotools-dev \
automake \
&& apt-get clean \
&& apt-get purge \
&& rm -rf /var/lib/apt/lists/* /tmp/*
RUN pip3 install \
cython\
numpy\
nose\
click\
scipy\
pandas\
pysam\
pyyaml\
bioframe
RUN pip3 install pairtools
RUN git clone https://github.com/nh13/pbgzip.git \
&& cd pbgzip \
&& sh autogen.sh \
&& ./configure \
&& make -j \
&& make install \
&& rm -rf /pbgzip
RUN git clone https://github.com/4dn-dcic/pairix \
&& mv /pairix/util/bam2pairs/bam2pairs /usr/bin/ \
&& rm -rf /pairix
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.
|
Beta Was this translation helpful? Give feedback.
-
Very similar to @agalitsyna suggestion. I'm forcing the installation of the prerequisite python libraries first and I finish with installing pairtools if the first install is successful with
|
Beta Was this translation helpful? Give feedback.
-
Hi, Georgette and I were able to fix the issue. It seemed to be an issue with the HPC cluster being used. |
Beta Was this translation helpful? Give feedback.
-
I am getting an import error when I try to run pairtools in a cluster:
I'm using Python 3.8.10.
Beta Was this translation helpful? Give feedback.
All reactions