Using Kinect point cloud as input

[kfarivar]

Hi

I gave the non parametric part of the IPNet a point cloud captured by Azure kinect. (running test_IPNet.py ) But I feel like the results are not very promising. The results are for the body file. I wanted to check with you to see if I'm doing something wrong. (I know I have to fit the SMPL model to the non-parametric reconstruction to get the final result, But I feel like if the SMPL fitting code used these results as input it wouldn't work very well. ) In all the following I use -batch_points 100000 since otherwise I would get a GPU out of memory error.

This is the point cloud file I have used as a csv file (just three columns of x,y,z first row is column names.)
frame_20.zip
I added these lines in the test_IPNet.py under the main function to read the point cloud:

def main(args):
# Load kinect point cloud from csv file
    kinect_pc = np.genfromtxt('kiya_data/frame_20.csv', delimiter=',')
    kinect_pc = kinect_pc[1:]

so instead of sending pc.vertices to pc2vox I sent kinect_pc which is a numpy array.

I have tried this point cloud as input : (it is around 36k points)

my first attempt: (I reversed the direction of the Y axis since I thought that might help the network since your sample was also like that)
python kiya_test_IPNet.py assets/scan.obj experiments/IPNet_p5000_01_exp_id01/checkpoints/checkpoint_epoch_249.tar kiya_out_dir -m IPNet -batch_points 100000

Then I tried reversing the direct of the Z axis as well and I got :

Finally I also tried increasing the resolution of the input voxelized figure to the network but it actually made the result worse:
python kiya_test_IPNet.py assets/scan.obj experiments/IPNet_p5000_01_exp_id01/checkpoints/checkpoint_epoch_249.tar kiya_out_dir -m IPNet -batch_points 100000 -res 200

[kfarivar]

I have also now tired the SMPL fitting part and as I expected the result is not good. this corresponds to the first reconstruction's body where I reversed the direction of the Y axis.

I also noticed that my SMPL result is not centered while the result I get when running your sample is.

I should say that the script was not initially working because of a missing faces file (and some incorrect paths). to get the faces this is what I did (in the smpl_paths.py):

@staticmethod
    def get_faces():
        from lib.smpl_layer import SMPL_Layer
        model = SMPL_Layer(center_idx=0, gender='male', model_root='smplpytorch/smplpytorch/native/models')
        return model.th_faces.numpy()

I don't know if this is relevant to the problem but that was the only major thing I changed in the code (besides some paths).

[kfarivar]

So I just went through your paper again and noticed this line:

We show that IP-Net can be trained to process sparse point clouds from a single view (such as from Kinect)

I think according to this, it makes sense that the input in training data should include incompleteness to learn to reconstruct such examples. and it seems for the weights and sample you have uploaded the input was already complete.

So in case my assumption is right, I was wondering if you could upload the weights of the model that was trained on incomplete input data ?

[ashwath98]

Hey regarding the SMPL fitting issue, Their registration code works really well if you initialize with an SMPL mesh with roughly aligned pose. I did this by doing something similar to how they mention they generate data in their paper. By doing the foll

• Rendering Scan in Multiple Views
  -Generating Openpose Keypoints for the scan
  -Optimizing 3D keypoints to match the rendered 2D keypoints.
  -fitting SMPL on those 3D keypoints.

This I give as initialization to their SMPL registration code and it works really well

I presume you can face some issues with generating keypoints for your scan, but it can be worth a shot

[Bingoang]

@kfarivar @ashwath98 @bharat-b7
Hi, all~
I ran the demo commands(using the pre-trained model "checkpoint_epoch_249.tar" in README), but met some problems:
My input is a 3D scan, and the IPNet predictions(correspondences labled parts and outersurface) look good.
But when fitting SMLP and SMPL+D to the IPNet predictions, the final outputs are so strange...
Here are my input 3Dscan, IPNet predictions(body.ply & full.ply), and final fitting outputs:

What' wrong with it? How can I get reasonable SMPL/SMPL+D fitting output?
Hope you can give me some advice~
Many thanks

[ashwath98]

Afaik the code shared tries to fit SMPL -> SMPLD from scratch using scan2mesh distance and other terms. In my experiments I found much better results if you have a good SMPL initialization ( which I got Estimating 3D joints using openpose in multiple views -> Optimizing SMPL on those joints). This repo has code for joint optimization, but you will have to get 3D joints yourself (by method I described or any other method like VNECT etc).

Also your scan seems to be really good I think using SMPL -> SMPLD directly should give better results (opinion) than running it on IPNet surface

[Bingoang]

@ashwath98
Thanks very much for your quick and explicit reply! I agree that the method in your experiments will output better results, especially in some challenge views or poses~

But what puzzles me most now is:
I use the "scan.obj" the author offers in "assets" file, the IPNet predictions and SMPL/SMPL+D fitting outputs are both good. My input 3D scan is also complete and its pose is really simple; Moreover, the IPNet predictions(correspondences labled parts and outersurface) are complete and look good, too. Based on the above 2 conditions, the SMPL/SMPL+D fitting outputs should be good, but why are my fitting outputs so poor(even the pose is so horrible) ?

I really want to know if I'm doing something wrong with my own 3D scan input?

[Daixinxin-cc]

@Bingoang
Have you solved this problem? I met the same problem when I used my own 3D scan input, and I used fit_SMPL_IPNet.py. I saw that in this file, the input would do SMPL fitting first and SMPLD fitting then. However, the results I got were not good enough even terrible for dressed scan data. Also, the result of 'scan.obj' performed well. I have no idea what's the problem leading to the bad results for my data.

[Bingoang]

@Daixinxin-cc
Hi~
Maybe you have to do some preprocessing to your own 3D scan input: translate and rotate your input model, making sure that the 3D model's center is roughly at the coordinate system's origin，and the head of model face to the y-axis' positive direction, the front of model face to the z-axis' positive direction, the x-axis' positive direction point to the model's left side.