Accessing PV Camera Intrinsics/Extrinsics in MRTK3 (Unity + HoloLens 2)

Hi,

We’re currently working on our bachelor’s thesis using the HoloLens 2. The goal of our project is to determine where and for how long the user is looking at certain objects in the environment. To achieve this, we stream video from the PV camera to a Python client that runs YOLOv5 for object detection on each frame. We then compare the user's gaze coordinates with the detected bounding boxes to determine if the user is looking at an object.

We initially got a basic version working using Camera.main, but we quickly noticed that the gaze point was always offset in the received video. We realized we need to use the PV camera instead and obtain its intrinsics and extrinsics to get accurate alignment.

We tried creating a plugin to access Windows.Media.Capture, but we're getting the following error when building:

error CS1705: Assembly 'CameraPlugin' with identity 'CameraPlugin, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' uses 'System.Runtime, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' which has a higher version than referenced assembly 'System.Runtime' with identity 'System.Runtime, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'

Environment:

• Unity 2022.3.6f1
• Visual Studio 2022
• MRTK3
• HoloLens 2

We would really appreciate guidance on:

What is the recommended way to retrieve the PV camera intrinsics and extrinsics in Unity using MRTK3/OpenXR?

How can we resolve the System.Runtime version conflict when trying to access Windows.Media.Capture in a plugin?

If you need more information just ask.

Thank you very much in advance!

Best regards, Linus and Victor