Dimensions of the NFT marker in the NFTMarkerInfo to fix marker position #42

kalwalt · 2020-02-03T15:53:05Z

Discussion started in issue #41.
List of feature to add:

printing in the console of width, height, dpi of the loaded NFT marker.
width, height, dpi as data members in the arController struct inside ARToolKitJS.cpp
inject width, height, dpi to NFTMarkerInfo to transmit the data through the event mecahnism.

The model is positioned with this formula:

model.position.y = (msg.height / msg.dpi * 2.54 * 10)/2.0;
model.position.x = (msg.width / msg.dpi * 2.54 * 10)/2.0;

…dpi of the NFT marker

kalwalt · 2020-02-03T15:54:58Z

This is the result in the log console, you can see the width height and dpi:

THREE.WebGLRenderer 103
artoolkit.min.js:1 onload callback should be defined
artoolkit.min.js:1 onerror callback should be defined
artoolkit.min.js:1 [warning] *** Camera Parameter resized from 640, 480. ***
artoolkit.min.js:1 [info] Allocated videoFrameSize 307200
artoolkit.min.js:1 [info] Reading /markerNFT_0.fset3
artoolkit.min.js:1 [info]   Assigned page no. 0.
artoolkit.min.js:1 [info]   Done.
artoolkit.min.js:1 [info] Reading /markerNFT_0.fset
artoolkit.min.js:1 [info] 
artoolkit.min.js:1 ### Surface No.1 ###
artoolkit.min.js:1 [info]   Read ImageSet.
artoolkit.min.js:1 [info] Imageset contains 9 images.
artoolkit.min.js:1 [info]     end.
artoolkit.min.js:1 [info]   Read FeatureSet.
artoolkit.min.js:1 [info]     end.
artoolkit.min.js:1 [info] NFT num. of ImageSet: 9
artoolkit.min.js:1 [info] NFT marker width: 893
artoolkit.min.js:1 [info] NFT marker width: 1117
artoolkit.min.js:1 [info] NFT marker dpi: 120
artoolkit.min.js:1 [info]   Done.
artoolkit.min.js:1 [info] points-636
artoolkit.min.js:1 [info] points-648
artoolkit.min.js:1 [info] points-615
artoolkit.min.js:1 [info] points-607
artoolkit.min.js:1 [info] points-590
artoolkit.min.js:1 [info] points-540
artoolkit.min.js:1 [info] points-508
artoolkit.min.js:1 [info] points-437
artoolkit.min.js:1 [info] points-269
artoolkit.min.js:1 [info] Loading of NFT data complete.
artoolkit.worker.js:37 loadNFTMarker ->  0

kalwalt · 2020-02-05T14:10:12Z

Maybe we could do the calculation of the center of the marker inside the C++ code what do you think @nicolocarpignoli ?

nicolocarpignoli · 2020-02-05T14:47:29Z

I think it has to be done outside it. It is enough what you did. We may add on this PR, before merging it, the use of these new parameters, so remove the y = 100 x = 100 from where we decide statically the position, and calculate it dynamically inside the code.

after that we can merge this :D !

kalwalt · 2020-02-06T20:20:32Z

Maybe we can find the center position from the matrix data, i have an idea in mind, will try tomorrow for sure.

nicolocarpignoli · 2020-02-09T09:34:43Z

Ok so after some tests, We found out that width/height of marker are not enough: we also need width/height of 3D objects.

With that, we can fix the center the 3D object at the center of the Marker.

Practically talking, we can calculate the blue lines (distances).

If knowing exactly the 3D object position is not doable in a way so that we have certain data, I think it is not a matter on waste other time/effort.

kalwalt · 2020-02-09T10:15:59Z

@nicolocarpignoli have you tried doing this? If this not works, i'm thinking for another approach. Maybe we can find the center of the marker from the four corners of the markers. If we have the matrix of the marker we can do this. We have already some code like this see

jsartoolkit5/examples/nft_improved_worker/main_worker.js

Lines 152 to 189 in 244b2b2

    
               if (!lastmsg) return; 
        
               var proj = JSON.parse(lastmsg.proj); 
        
               var world = JSON.parse(lastmsg.matrixGL_RH); 
        
               var mat = multiplyMatrices(proj, world); 
        
               function glpointToCanvas(xyz) { 
        
                   return { 
        
                       x: (xyz.x + 1) * 0.5 * pw / pscale * sscale - ox / pscale * sscale, 
        
                       y: (1 - xyz.y) * 0.5 * ph / pscale * sscale - oy / pscale * sscale, 
        
                   } 
        
               } 
        
               function drawpoint(x, y, z) { 
        
                   var r = transformPoint(mat, {x: x, y: y, z: z}); 
        
                   var c = glpointToCanvas(r); 
        
                   return c; 
        
               } 
        
               var width = marker.width; 
        
               var height = marker.height; 
        
               var dpi = marker.dpi; 
        
               var w = width / dpi * 2.54 * 10; 
        
               var h = height / dpi * 2.54 * 10; 
        
               var p1 = drawpoint(0, 0, 0); 
        
               var p2 = drawpoint(w, 0, 0); 
        
               var p3 = drawpoint(w, h, 0); 
        
               var p4 = drawpoint(0, h, 0); 
        
               context_draw.beginPath(); 
        
               context_draw.moveTo(p1.x, p1.y); 
        
               context_draw.lineTo(p2.x, p2.y); 
        
               context_draw.lineTo(p3.x, p3.y); 
        
               context_draw.lineTo(p4.x, p4.y); 
        
               context_draw.closePath(); 
        
               context_draw.strokeStyle = "red"; 
        
               context_draw.stroke(); 
        
           };

nicolocarpignoli · 2020-02-10T09:36:32Z

@kalwalt I have tried to calculate the 3D object dimensions and use them, along with NFT marker width height and dpi.

I have to be honest: the calculation I did does not make too much sense to me, it's solely based on empirics.
But it works for Flamingo + Pinball Image. We need other tests before think it is correct. Are you able to do some test with 1-2 different 3D models and 1-2 different images as nft marker?

nicolocarpignoli · 2020-02-10T09:37:18Z

I got the 3D model sizes like this (don't know if it's correct)

   var dimensions = new THREE.Box3().setFromObject(model);
            objPositions = {
                width: dimensions.max.x - dimensions.min.x,
                height: dimensions.max.y - dimensions.min.y,
            };

kalwalt · 2020-02-10T09:39:33Z

i will give a try when i have time. 😄

kalwalt · 2020-02-10T15:07:19Z

@nicolocarpignoli it seems that for the sphere not works in the same way for the flamingo model. I will try to solve this. In the meantime i will upload another example with a different model.

…sartoolkit5 into dimensions-NFT-feature

ThorstenBux · 2020-03-29T20:09:43Z

Tested on iOS,

Portrait works awesome and model is in the centre.
Landscape: Model vanishes and I can't find it anymore. (Not sure if that is expected to be like that)

kalwalt · 2020-03-29T20:39:41Z

Landscape: Model vanishes and I can't find it anymore. (Not sure if that is expected to be like that

That's very bad. I have not checked so much in landscape mode. I assumed that if it works fine on portrait. should work fine also on landcape mode.

kalwalt · 2020-03-29T21:02:55Z

@ThorstenBux the model is wrong placed when you switch from portrait to landscape mode. The model is placed on the origin of the marker ( the left bottom of it). If you reload the page while in landscape mode the model is correctly placed in the center of the marker. I need to solve this, maybe forcing to update the matrix? or does it is caused by another factor?

nicolocarpignoli · 2020-03-30T08:38:20Z

On my Hawuei, it is repositioning good switching from landscape/portrait and back.

Apart from the strange rotation (that we can change from the application, no problem) I see a weird animation flow. It is not the correct one. But I guess it does not have anything to do with this PR

nicolocarpignoli · 2020-03-30T08:39:40Z

see video: https://www.dropbox.com/s/wirx7sgre1jne9g/VID_20200330_103505.mp4?dl=0

kalwalt · 2020-03-30T08:50:08Z

that's good that with the flamingo model switching from portrait to landscape mode and back the position is correct, maybe there are some differences in the code that make react in a different way.

I see a weird animation flow.

it is possible, i will look into this if it is caused by the actual implementation.

i rotated the model on the X axis because this should be the right position. ( you should see in a viewer) but this is less important...

ThorstenBux · 2020-03-30T09:39:02Z

I was using the alterra example with the robot. When it broke Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Walter Perdan <notifications@github.com> Sent: Monday, March 30, 2020 9:50:22 PM To: kalwalt/jsartoolkit5 <jsartoolkit5@noreply.github.com> Cc: Thorsten Bux <thorsten.bux@outlook.com>; Mention <mention@noreply.github.com> Subject: Re: [kalwalt/jsartoolkit5] Dimensions of the NFT marker in the NFTMarkerInfo to fix marker position (#42) that's good that with the flamingo model switching from portrait to landscape mode and back the position is correct, maybe there are some differences in the code that make react in a different way. I see a weird animation flow. it is possible, i will look into this if it is caused by the actual implementation. i rotated the model on the X axis because this should the right position. ( you should see in a viewer) but this is less important... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkalwalt%2Fjsartoolkit5%2Fpull%2F42%23issuecomment-605868090&data=02%7C01%7C%7Cf6e799d1a50d4680ff5808d7d4876244%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637211550233780964&sdata=vFphoHLembXp173faqUlvjQbVTFtUWpGjK9BQZqvdWA%3D&reserved=0>, or unsubscribe<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD765PF2TNHM5AIVH466UULRKBMM5ANCNFSM4KPHRKHA&data=02%7C01%7C%7Cf6e799d1a50d4680ff5808d7d4876244%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637211550233790959&sdata=j6U20rU56Cyr8Bd9at2KKnQzIXGbLjSS%2BQx4R5W6SDM%3D&reserved=0>.

kalwalt · 2020-03-30T09:42:57Z

i have this problem, the wrong position of the mesh while switching from portrait to landscape mode, also for other examples. I tried to put the code to adjust the position in the loader:

threeGLTFLoader.load("../Data/models/Flamingo.glb", function (gltf) {
// other code...
// width=2100 height=1576 dpi=150
model.position.y = (2100 / 150 * 2.54 * 10)/2.0;
model.position.x = (1576 / 150 * 2.54 * 10)/2.0; 
// other code...
}

but not solve anything...

kalwalt · 2020-03-30T19:16:21Z

I will try another approach, I will try to do a translation matrix inside the C++, it should be possible manipulating the trans matrix...

kalwalt · 2020-03-31T17:08:04Z

i leave this link as a reminder https://stackoverflow.com/questions/8146082/how-to-disable-toggling-of-landscape-portrait-on-a-mobile-web-app-android-brows/18737262

kalwalt · 2020-03-31T18:51:45Z

Maybe we can find a simpler solution, we could force a page to reload with location.reload() see this stackoverflow article, maybe creating an EventListener that triggers a reload when changing the window mode?

kalwalt · 2020-03-31T21:23:21Z

i addedd in the flamingo example before the end of the </script> tag this snippet:

// Listen for orientation changes
window.addEventListener("orientationchange", function() {
  location.reload();
  return false;
}, false);

This make to reload the page when the orientation change and consequently the model is placed correctly. I would prefer to find another solution, as reloading is a waste of time.

kalwalt · 2020-04-01T09:40:18Z

Inside the found function (flamingo example) i did this, but do not help:

var found = function(msg) {
       if (!msg) {
           world = null;
       } else {
           world = JSON.parse(msg.matrixGL_RH);

           if (!window.firstPositioning) {
               window.firstPositioning = true;
               model.position.y = (msg.height / msg.dpi * 2.54 * 10)/2.0;
               model.position.x = (msg.width / msg.dpi * 2.54 * 10)/2.0;
           }

           window.addEventListener("orientationchange", function() {
             // Announce the new orientation number
             console.log(screen.orientation);
             model.updateWorldMatrix(true, true);          
           }, false);

           console.log("NFT width: ", msg.width);
           console.log("NFT height: ", msg.height);
           console.log("NFT dpi: ", msg.dpi);
           var o_view = scene.getObjectByName('Flamingo');
           console.log(o_view);
       }
   };

kalwalt · 2020-04-14T14:11:57Z

I think that enter the data ( width, height, dpi of the marker) in the dispatch event, maybe has no sense. It's better maybe if we retrieve these data while we load the NFT marker. See this code: https://github.com/kalwalt/FeatureSET-Display/blob/7aaab29930a2069ad3ea783f4f467f4f8c761fc7/emscripten/ARimageFsetDisplay.cpp#L99
It loads the marker and fill the struct with the data. I thnk that is a better option. Probably i will close this PR and will open a new one.

evaristoc · 2020-04-20T10:33:29Z

#42 (comment)

Not sure if good idea... If someone has an animation that shouldn't stop no matter the view, reloading could restart the animation instead and the flow of the animation would not be as expected.

evaristoc · 2020-04-20T10:37:50Z

#42 (comment)

I guess the marker is still detected after the change in landscape/portrait view?

kalwalt · 2020-04-20T10:42:46Z

Not sure if good idea... If someone has an animation that shouldn't stop, reloading could restart the animation instead.

@evaristoc i agree with you, i left this idea.

I guess the marker is still detected after the change in landscape/portrait view?

I think yes and the 3d model is always displayed, I think that is something related to updating the matrix but still not found a solution.

evaristoc · 2020-04-20T10:46:14Z

Hi @kalwalt

what are these values?
https://github.com/kalwalt/FeatureSET-Display/blob/7aaab29930a2069ad3ea783f4f467f4f8c761fc7/emscripten/ARimageFsetDisplay.cpp#L147

evaristoc · 2020-04-20T10:51:53Z

And @kalwalt

What do you see in the ARLOGi when log-consoling the same value (#42 (comment))?

kalwalt · 2020-04-20T11:16:57Z

Hi @kalwalt

what are these values?
https://github.com/kalwalt/FeatureSET-Display/blob/7aaab29930a2069ad3ea783f4f467f4f8c761fc7/emscripten/ARimageFsetDisplay.cpp#L147

Thiis link refer to another project, the Faeture Set displayer, but the values is the width and height of image to be displayed for the size of the pointer.

And @kalwalt

What do you see in the ARLOGi when log-consoling the same value (#42 (comment))?

As i remember, this give always the width and height of the pinball marker. I think that outputting always in the stream these values is not necessary. It would be better retrieve these values only when loading the NFT marker and store them in a variable. I tried for https://github.com/kalwalt/FeatureSET-Display and it worked...

evaristoc · 2020-04-20T11:21:41Z

this give always the width and height of the pinball marker

So what is the difference between this value and the values of width and height obtained at #42 (comment)?

My question is based on the possibility that the interpretation of width and height might change when changing the view. It might not be the issue based on @nicolocarpignoli video: it appears that the object rotated accordingly. But I want to be sure that is not the problem you are seeing.

Different devices treat the change of view differently.

evaristoc · 2020-05-14T21:53:30Z

This formula, @kalwalt :
model.position.y = (msg.height / msg.dpi * 2.54 * 10)/2.0;
model.position.x = (msg.width / msg.dpi * 2.54 * 10)/2.0;

The value of 2.54, what is exactly that? EDIT: 2.54 is the conversion from centimeters to inches. It appears this is a conversion into centimeters?

I am not really sure but I suspect using dot density (dpi == dots per inch) will render different sizes in different devices in the directions of x and y. The larger the dpi, the smaller the relation msg.LENGTH/msg.dpi. I wonder if that explains what happens with the different results from the different tests when comparing different devices? Larger dpis might give a smaller object, if I am correct.

If that is true, the approach might not be generalizable: different devices will see different scenes sizes.

Can anyone confirm this claim?

Regarding the positioning: In my opinion, the ideal would be to calculate the appropriate coordinate system and then render the figure on top on that system. I am not sure if the suggestion by @nicolocarpignoli (#42 (comment)) corresponds to a standard. As I see it, the final result should be a square which center would correspond to the center of the marker, in clip coordinates (2d(-1,1)). As far as I know, keeping aspect ratio of the square is relatively standard too. However, this wouldn't correct for scene size per device if my claim is true.

h3retic-EEL · 2021-10-03T18:54:00Z

I premise by thanking you for your efforts, guys.
@kalwalt @nicolocarpignoli Is there any chance to have a nightly ar-with-nft.js build with the image tracking position fixed? Thank you.

kalwalt · 2021-10-11T17:19:12Z

I premise by thanking you for your efforts, guys. @kalwalt @nicolocarpignoli Is there any chance to have a nightly ar-with-nft.js build with the image tracking position fixed? Thank you.

Thank you @Hadrhune, I'm working to the AR.js feature in this PR AR-js-org/AR.js#114 though i haven't yet a working solution for Aframe.

display in the log console the number of ImageSet, width, height and …

735b091

…dpi of the NFT marker

kalwalt added C/C++ coding with C and C++ language code design NFT nft markerless technology labels Feb 3, 2020

kalwalt self-assigned this Feb 3, 2020

kalwalt added 2 commits February 3, 2020 17:10

added new width, height and dpi data members to arController struct

0013322

passing data to the Event listener (NFTMarkerInfo)

d32573f

kalwalt mentioned this pull request Feb 5, 2020

Out of Memory issue using NFT #37

Closed

prettify inside getNFTMarker

50719ce

trying to set the position of the model with the BoundingBox size

032c26a

added calculation of position for flamingo, need more tests

a695e97

trying calculation of position with the sphere, not works as expected

13b810e

kalwalt and others added 8 commits February 10, 2020 19:13

new Duck and CesiumMan examples, in the scene but can not be viewed

1d2e4fb

Duck now is displayed, but without the re-positioning stuff

181d21f

console log for 3d model size

7a2abb3

Duck is displayed now!

cf3bddb

Merge branch 'dimensions-NFT-feature' of https://github.com/kalwalt/j…

6162b55

…sartoolkit5 into dimensions-NFT-feature

CesiumMan visible now

052954d

sphere is not needed

4566f63

Alterra NFT marker

0c05e43

updateWorldMatrix inside orientationchange Event listener

cabe736

This was referenced Apr 5, 2020

Enhancement in the tools/makem.js script and other fixes artoolkitx/jsartoolkit5#96

Open

Q's re hosting and NFT without AR webarkit/jsartoolkitNFT#9

Closed

kalwalt mentioned this pull request Apr 14, 2020

[ WIP ] width, height, dpi from NFT marker. webarkit/jsartoolkitNFT#12

Closed

kalwalt mentioned this pull request Apr 26, 2020

nftMarker struct with addNftMarker returning it webarkit/jsartoolkitNFT#31

Merged

evaristoc mentioned this pull request May 15, 2020

Fix positioning of content #41

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dimensions of the NFT marker in the NFTMarkerInfo to fix marker position #42

Dimensions of the NFT marker in the NFTMarkerInfo to fix marker position #42

kalwalt commented Feb 3, 2020 •

edited

Loading

kalwalt commented Feb 3, 2020 •

edited

Loading

kalwalt commented Feb 5, 2020

nicolocarpignoli commented Feb 5, 2020

kalwalt commented Feb 6, 2020

nicolocarpignoli commented Feb 9, 2020

kalwalt commented Feb 9, 2020

nicolocarpignoli commented Feb 10, 2020

nicolocarpignoli commented Feb 10, 2020

kalwalt commented Feb 10, 2020

kalwalt commented Feb 10, 2020

ThorstenBux commented Mar 29, 2020

kalwalt commented Mar 29, 2020

kalwalt commented Mar 29, 2020 •

edited

Loading

nicolocarpignoli commented Mar 30, 2020

nicolocarpignoli commented Mar 30, 2020

kalwalt commented Mar 30, 2020 •

edited

Loading

ThorstenBux commented Mar 30, 2020 via email

kalwalt commented Mar 30, 2020

kalwalt commented Mar 30, 2020

kalwalt commented Mar 31, 2020

kalwalt commented Mar 31, 2020

kalwalt commented Mar 31, 2020

kalwalt commented Apr 1, 2020

kalwalt commented Apr 14, 2020

evaristoc commented Apr 20, 2020 •

edited

Loading

evaristoc commented Apr 20, 2020

kalwalt commented Apr 20, 2020 •

edited

Loading

evaristoc commented Apr 20, 2020

evaristoc commented Apr 20, 2020

kalwalt commented Apr 20, 2020 •

edited

Loading

evaristoc commented Apr 20, 2020 •

edited

Loading

evaristoc commented May 14, 2020 •

edited

Loading

h3retic-EEL commented Oct 3, 2021

kalwalt commented Oct 11, 2021 •

edited

Loading

Dimensions of the NFT marker in the NFTMarkerInfo to fix marker position #42

Are you sure you want to change the base?

Dimensions of the NFT marker in the NFTMarkerInfo to fix marker position #42

Conversation

kalwalt commented Feb 3, 2020 • edited Loading

kalwalt commented Feb 3, 2020 • edited Loading

kalwalt commented Feb 5, 2020

nicolocarpignoli commented Feb 5, 2020

kalwalt commented Feb 6, 2020

nicolocarpignoli commented Feb 9, 2020

kalwalt commented Feb 9, 2020

nicolocarpignoli commented Feb 10, 2020

nicolocarpignoli commented Feb 10, 2020

kalwalt commented Feb 10, 2020

kalwalt commented Feb 10, 2020

ThorstenBux commented Mar 29, 2020

kalwalt commented Mar 29, 2020

kalwalt commented Mar 29, 2020 • edited Loading

nicolocarpignoli commented Mar 30, 2020

nicolocarpignoli commented Mar 30, 2020

kalwalt commented Mar 30, 2020 • edited Loading

ThorstenBux commented Mar 30, 2020 via email

kalwalt commented Mar 30, 2020

kalwalt commented Mar 30, 2020

kalwalt commented Mar 31, 2020

kalwalt commented Mar 31, 2020

kalwalt commented Mar 31, 2020

kalwalt commented Apr 1, 2020

kalwalt commented Apr 14, 2020

evaristoc commented Apr 20, 2020 • edited Loading

evaristoc commented Apr 20, 2020

kalwalt commented Apr 20, 2020 • edited Loading

evaristoc commented Apr 20, 2020

evaristoc commented Apr 20, 2020

kalwalt commented Apr 20, 2020 • edited Loading

evaristoc commented Apr 20, 2020 • edited Loading

evaristoc commented May 14, 2020 • edited Loading

h3retic-EEL commented Oct 3, 2021

kalwalt commented Oct 11, 2021 • edited Loading

kalwalt commented Feb 3, 2020 •

edited

Loading

kalwalt commented Feb 3, 2020 •

edited

Loading

kalwalt commented Mar 29, 2020 •

edited

Loading

kalwalt commented Mar 30, 2020 •

edited

Loading

evaristoc commented Apr 20, 2020 •

edited

Loading

kalwalt commented Apr 20, 2020 •

edited

Loading

kalwalt commented Apr 20, 2020 •

edited

Loading

evaristoc commented Apr 20, 2020 •

edited

Loading

evaristoc commented May 14, 2020 •

edited

Loading

kalwalt commented Oct 11, 2021 •

edited

Loading