AR kit: How do we access live frame data from camera?

Expo SDK
1

From the Expo AR Blog here:

I saw this:

ExpoTHREE.createARBackgroundTexture(arSession, renderer) is an expo-three utility that returns a THREE.Texture that live-updates to display the current AR background frame.

And I am wondering how do we access the live-updated frames? Can we somehow read the data from the .image property of Texture?

2

The ‘frames’ are actually never inspected in JavaScript, all that is available is an OpenGL texture ID that is constantly updated by the native layer underneath to the latest camera capture. It references the capturedImageTexture property of the arSession here https://github.com/expo/expo-three/blob/9508d0cc16a676c850539f9b5357d8eabe0bcea2/lib/ExpoTHREE.js#L68-L74 as returned in the end of https://github.com/expo/expo/blob/7ae7cbe43fc3ccba495da4d14ce420d1994d8511/ios/Exponent/Versioned/Optional/ARKit/EXGLARSessionManager.m#L31-L142 after constructing the necessary OpenGL camera texture cache. This is updated every frame by https://github.com/expo/expo/blob/7ae7cbe43fc3ccba495da4d14ce420d1994d8511/ios/Exponent/Versioned/Optional/ARKit/EXGLARSessionManager.m#L290-L367 – note that there is no data transfer to JavaScript.

What are you trying to do with a potential .image property in particular? What format would you expect this property to be in? Also it would have new data every frame – would you expect a push or poll API?

3

Hi, thank you for fast reply. I want to obtain the frame data so I can do some basic computer vision tasks or machine learning on it.

Given that, OpenCV will probably be used in most common cases, so 8-bit input image could be a desirable output format:
E.g. Canny Edge Detection: OpenCV: Canny Edge Detection

I guess a push API would be nice, the event of every frame update will call a listener with the frame data passed as a parameter would be ideal.

4

You could render the texture and use gl.readPixels(...) to read into a TypedArray and run OpenCV.js (are you planning on using an Emscripten build?) on that. Also you could use requestAnimationFrame(...) to schedule your own polls for now – that would give you the fastest refresh rate possible in JavaScript.

5

Thank you nikki, I don’t think I am using Emscripten, as I am doing a regular expo project (I am not sure).

And would you happen to know if there is an expo implementation that would allow us to achieve the functionality of Three’s CubeTexture?

https://threejs.org/docs/#api/textures/CubeTexture

Three’s CubeTexture doesn’t work right now as the image loading need some web APIs, e.g. HTMLImageElement, document, etc.

I notice that expo has loadCubeTextureAsync, and I tried to use what it returns like a regular texture but it doesn’t work:

const skybox = {
	nx: require('./nx.jpg'),
	ny: require('./ny.jpg'),
	nz: require('./nz.jpg'),
	px: require('./px.jpg'),
	py: require('./py.jpg'),
	pz: require('./pz.jpg')
}
const coinTexture = await loadCubeTextureAsync({
  assetForDirection: ({ direction }) => skybox[direction],
})
const material = new THREE.MeshBasicMaterial( { map: coinTexture } );
const geometry = new THREE.CylinderGeometry(0.1, 0.1, 0.015, 100);
const cube = new THREE.Mesh(geometry, material);
cube.position.z = -0.4;
scene.add(cube);

Basically I just need different materials for different sides of a geometry, thank you very much!

6

HTMLImageElement isn’t built-in to JavaScript, but it is built-into the DOM libraries conventionally available in web JavaScript contexts. React Native’s JavaScript context is a pure JavaScript context without the DOM, while it does have ‘polyfills’ for many common browser libraries such as requestAnimationFrame(...).

7

Thank you Nikki, so does Expo plan to implement something like Three’s CubeTexture?

https://threejs.org/docs/#api/textures/CubeTexture

It would be really helpful for us to build cube like objects with sides using different materials, thank you! I can’t find a way to use CubeTexture directly because it depends on the DOM.

8

expo-three has a built in function to help with loading cube textures! Here is a snack: https://snack.expo.io/@bacon/expo-three-loadcubetextureasync

9

Hi bacon, thank you. As I mentioned in an ealier post (2 posts back), I have tried this. But somehow it doesn’t work, my code snippet is also up there. Would really appreciate any help! Thanks.

1 Like
10

Gotcha, sorry about that :sweat_smile: Easy solve!
Just use an array of materials! I did the same thing in a game Pillar Valley: https://github.com/EvanBacon/Expo-Pillar-Valley/blob/6ddb09521dbb8f0e04376146861d7414bbadf949/Game/Game.js#L141-L149

const materials = []

const mesh = new THREE.Mesh(
  new THREE.CubeGeometry(1, 1, 1),
  materials,
);
1 Like
11

Thank you so much! Works for CylinderGeometry as well, perfect!

1 Like
12

Yay! !Just remember to clone materials as much as possible :smiley: don’t wanna run out of memory :confused:

Edit: Read my later comment :sweat_smile:

13

Awesome, sorry just a small question: what is the difference between using .clone() and just use the same variable multiple times? Thanks!

14

On second thought, just use the single instance. Basically when you clone the material you create separate references which allows you to use separate manipulation patterns. That’s what I was doing in my example, but this is actually less efficient than using the single instance.
If you find the materials are acting weird and not appearing then you will want to clone. Try and avoid that if you can :sweat_smile:

1 Like
15

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.