Cluster and DPD modes

[arpruss]

Do you by any chance know the returned data format for the IR camera in the clustering or the DPD modes? I've been looking at it, and while I'm expecting to see coordinates of clusters or blobs, the hex data doesn't look like it's encoding coordinates (I'm expecting coordinates to be something like the 0-319 and 0-239 ranges.)

An example of data in clustering mode that I'm seeing is:
318c2e0000000000002af86e0b000000000000000000000000000000000000000000000000000000000000000000000000030006d937022439000001007eeb71424408441f00004d000000ef0000540200400ea000390039000200030065ff5d7bc333d66a7d01ef017600ff00aef66bfa6f4e3e379f019f012200df007c527387557cc91f7e01eb01ba00be00ffbb5fcfde03237eeb01b30162004900f5e7ebefef578f57f501bd001f00bf00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000b203df

I can tell that the 0x06 in position 51 is the mode (0x06 = clustering), and the next byte is the fragment number. I am guessing that the 0x01 in position 59 may be the noise level, and that soon after that come the clusters, 32 bytes per cluster, but I can't figure out what that data is.

Any hints would be appreciated. Is there maybe some sort of encryption here?

[arpruss]

I've figured out the clustering mode, I think. The data starts at offset 61, and it is a sequence of 16-byte items, one per cluster, each consisting of 8 littleendian uint16 entries:
average brightness
number of pixels in cluster
center of mass x-coordinate in units of 1/64 pixel
center of mass y-coordinate in units of 1/64 pixel
x start
x end
y start
y end

It would be really cool if visualizing the clusters got added to the toolkit.

[arpruss]

I figured out the DPD mode. It is also a sequence of clusters, in the same format as in clustering mode, starting at offset 61. However, there is an extra byte inserted after every third cluster. Thus, the first entry starts at 61, the second at 61+16, the third at 61+32, but the fourth is at 61+49, the fifth at 61+49+16, the sixth at 61+49+32, the seventh at 61+49+49, and so on.

Maybe the extra padding byte carries some information sometimes, but I think I've only seen 00 for it. It's a strange padding!

[CTCaer]

Hey, have in mind that JC toolkit is in indefinite hiatus.
I may sometime start cleaning it up again and improving it but dunno when.

Just wanted to chime in since you did actual research on the protocol!

So don't fret if you don't get an answer or I won't work again with jctoolkit right away.
Documenting these and having a PoC in PRs are still very good for the general public until then.

About that, most of my latest but old research is in the fork that I used to make PRs for the main dekunukem repo https://github.com/CTCaer/Nintendo_Switch_Reverse_Engineering.
Internally I used to have notes for the extra IR modes but dunno where they are after 6+ years.
Btw, nintendo added more IR modes/options since then. So it's good to also document on what jc firmware you are.

[arpruss]

Thanks for the response!

I will try to remember to document the firmware when I get the joycon back from Nintendo (today I sent it for drift repair).

[arpruss]

BTW, do you happen to have a recommendation for a good library for communicating with the joycon over BT in python? I have a project which turns a Wiimote into a lightgun for retro emulation (e.g., Duck Hunt), using four IR LEDs around the TV, and I am looking to move the project to a joycon for greater precision.

[CTCaer]

it's the same lib for everything; hidapi

btw it's better to use a compiled lang for such things instead of python to reduce latency and cpu usage.

I also wonder if there's an accessory that makes joycon more ergonomic to hold with IR camera at front.

[arpruss]

I am thinking of 3D printing a holder for the joycon to attach to a wooden grip, with the joycon upside down so that the analog stick could be used as a trigger.

[CTCaer]

@arpruss btw if you put that code around WaitForSingleObject for hid_read_timeout in hid.c and enable the console at the bottom of jctool.cpp, what times do you get for spi dumping?

char time[32];
struct timespec ts, te;
timespec_get(&ts, TIME_UTC);

res = WaitForSingleObject(ev, milliseconds);

timespec_get(&te, TIME_UTC);
sprintf(time, "%d\n", te.tv_nsec / 1000 - ts.tv_nsec / 1000);
printf(time);

I want to see if my old bt4 adapter is bad or windows is just bad in usb-bt hid latencies.
I see 28ms myself and it shows in wireshark. 14ms for the hci packet received event and 14ms for the packet.
Where it should be only 14-15ms (1x ssr).

[arpruss]

I can check once I get my right joycon back from Nintendo (I only own one; our Switch is a salvage unit with a broken screen and it didn't come with joycons).

[arpruss]

Do you happen to know how to get the joycon into the mode where it's forwarding strain data from the ringcon? Someone on the web says that there is a mode for that, and then the ringcon data replaces the gyro data.

[arpruss]

@arpruss btw if you put that code around WaitForSingleObject for hid_read_timeout in hid.c and enable the console at the bottom of jctool.cpp, what times do you get for spi dumping?

char time[32];
struct timespec ts, te;
timespec_get(&ts, TIME_UTC);

res = WaitForSingleObject(ev, milliseconds);

timespec_get(&te, TIME_UTC);
sprintf(time, "%d\n", te.tv_nsec / 1000 - ts.tv_nsec / 1000);
printf(time);

I want to see if my old bt4 adapter is bad or windows is just bad in usb-bt hid latencies. I see 28ms myself and it shows in wireshark. 14ms for the hci packet received event and 14ms for the packet. Where it should be only 14-15ms (1x ssr).

I'm seeing about 28000.

[CTCaer]

Do you happen to know how to get the joycon into the mode where it's forwarding strain data from the ringcon? Someone on the web says that there is a mode for that, and then the ringcon data replaces the gyro data.

subcmd: 3 arg: 0x32 to enable it
this enables attachment input report (ringcon and spaceship accessories)
0x12 as output report id to send stuff to attachment.

typedef struct _attachment_output_report_t
{
	u8 id; // report id 0x12
	u8 pkt_id;
	u8 rumble[8];
	u8 data[38]; // attachment data.
} attachment_output_report_t;

typedef struct _attachment_input_report_t
{
	// Start - Same as 0x30/0x31.
	u8 id;     // report id 0x32
	u8 pkt_id; // latency timer
	u8 info;   // conn_info/power_info
	u8 btn_right;
	u8 btn_shared;
	u8 btn_left;
	u8 stick_left[3];
	u8 stick_right[3];
	u8 vib_decider;
	u8 sixaxis_acc0[6];
	u8 sixaxis_gyro0[6];
	u8 sixaxis_acc1[6];
	u8 sixaxis_gyro1[6];
	u8 sixaxis_acc2[6];
	u8 sixaxis_gyro2[6];
	// End - Same as 0x30/0x31.

	// Same size as 0x31.
	u8 data[313]; // attachment data.
} attachment_input_report_t;

I'm seeing about 28000.

Yeah, dunno why it takes 14ms for event and 14ms for the data. That issue doesn't exist for reports that are pushed constantly like 0x30/0x31/etc.

EDIT: btw you won't get something valid if you don't configure attachement.

relevant subcmds:
0x28	attachment write
0x29	attachment read
0x2A	attachment enable
0x2B	attachment info

0x58	ext dev write
0x59	ext dev info
0x5A	ext dev polling enable
0x5B	ext dev polling disable
0x5C	ext dev input format config
0x5D	??
0x60	internal attachment write
0x61	??
0x62	internal attachment Enable
0x63	internal attachment polling enable
0x64	internal attachment polling disable

subcmd with data acks:
0xA8 (0x28)	attachment data
0xA9 (0x29)	attachment info
0xAA (0x2A)	internal attachment read
0xD8 (0x58)	ext dev read
0xD9 (0x59)	ext dev info

[arpruss]

Do you happen to know what registers control sensitivity for clustering and pointing modes?

[CTCaer]

I can't remember at all. And I can't find my notes but only what I pushed in github.
https://github.com/CTCaer/Nintendo_Switch_Reverse_Engineering/blob/ir-nfc/mcu_ir_registers_map.md

https://github.com/CTCaer/Nintendo_Switch_Reverse_Engineering/blob/ir-nfc/mcu_ir_nfc_notes.md
https://github.com/CTCaer/Nintendo_Switch_Reverse_Engineering/blob/ir-nfc/mcu_ir_notes.md

You can eventually find them out by trial and error.
Just add a debug textbox where you can send custom cfg, while it's live.

[arpruss]

Register 0x121 seems to control some sort of a threshold for pointing mode. 0=most sensitive, 7=least sensitive.

[CTCaer]

I'm assuming that's page 1, register 0x21.
Make that clear and continue to use that format (0x121).

[arpruss]

I'm having trouble consistently setting registers. This is a problem running both your c++ code and my python code: often I just can't get the IR LEDs to turn off, or change any other page 00 registers. Would you have any thoughts?

[40476]

Try on another computer, or another bluetooth adapter? I have heard this can be a problem with some devices.

…

On Nov 25, 2024, at 9:45 AM, arpruss ***@***.***> wrote: I'm having trouble consistently setting registers. This is a problem running both your c++ code and my python code: often I just can't get the IR LEDs to turn off, or change any other page 00 registers. Would you have any thoughts? — Reply to this email directly, view it on GitHub <#99 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APEIFYIIEGXVTYED5OKQHAL2CMZYDAVCNFSM6AAAAABQJDPYFOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJYGIYTQMJRGY>. You are receiving this because you are subscribed to this thread.

[CTCaer]

If you are using these tiny usb adapters, you need to be close because they have a very bad signal.
I learnt that first hand with the hd rumble music player. Mine starts losing packets at half meter and a hand in between.
On a laptop or bt with normal antenna you should not have that issue.

You should actually check for ack reply and that it's correct and if not resend the command.

[arpruss]

I tried sending the set register command many times. And I have no problem reading the registers before and after the set register command.

Maybe it's a battery charge issue. I am now recharging the joycon.

[CTCaer]

I never had an issue send a subcmd and not be read, other than signal issues or congestion.

Also if you are in an input report mode where the controller automatically sends the report every 15ms, you need to send a subcmd after you receive the input report.

Also have in mind that some IR registers can't be changed live. At least that's what I remember.

[arpruss]

How do I check for the ack of the set registers command?

[arpruss]

Well, that was dumb of me. I had the endianness wrong on the register numbers. Works fine now.

[arpruss]

Alas it turns out that the joycon IR camera isn't sensitive enough to pick up the four IR leds that I have around my TV from the sofa. :-( They worked great with the wii remote.

But I have now expanded the joycon-python library to include support for the IR pointing, clustering and image modes: https://github.com/arpruss/joycon-python

[CTCaer]

Well, that was dumb of me. I had the endianness wrong on the register numbers. Works fine now.

You should still check for the ack. All subcmds have a 0x21 input report reply which its data is the the ack byte (0x80 | subcmd reply data, normally matches subcmd) and subcmd (as is).

Alas it turns out that the joycon IR camera isn't sensitive enough to pick up the four IR leds that I have around my TV from the sofa. :-( They worked great with the wii remote.

Are you sure? I remember trying it with the wii bar and it was picking the 4 leds from 5-7m like they were flashlights.
Although have in mind that I used a dolphin bar which might emit more 850nm light. (Wii is 940nm).
You should play with the filter register.

[arpruss]

I can't remember what wavelength the LEDs I installed were, but I think I chose them to match the Wii. So wavelength might be the issue.

My Wii bar's LEDs do show up from the sofa in the joycon camera. Normally, to get exact pointing one needs four LEDs not on a straight line, so I have custom LEDs above and below the TV for lightgun games (and I have custom code for pointing). But maybe the Wii bar plus the accelerometer in the joycon will be good enough if I use this algorithm: https://arxiv.org/pdf/2410.17997 . The Wii remote's accelerometer wasn't precise for this purpose, but the joycon is more precise.