Using ExpoThree/ARKit for line drawing -- getting position?

My team and I were exploring using Expo and ARKit to create a basic 3D line drawing app and are unable to find a way to have line drawing map in a realistic “ARKit-y” way and stay fixed on the screen.

We’re aiming for something that works like this simple native iOS app: https://github.com/lapfelix/ARKit-line-drawing

Is there any way to accomplish something like this?

Current code below uses `camera.getMapDirection()’ directional vector and builds a new “point” of the line (a sphere) based on the last point. This allows us to loosely draw on the screen, but the cursor does not stay fixed in the center of the screen like we would expect given the Swift code we looked at in the link above:

import React from 'react'
import { Dimensions, StyleSheet, Text, View, PanResponder } from 'react-native'
import * as THREE from 'three'
import ExpoTHREE from 'expo-three'
console.disableYellowBox = true

export default class App extends React.Component {

  constructor (props) {
    super(props)
    this.lastSphere = {x: 0, y: 0, z: -1}
  }

  componentWillMount () {
    this.panResponder = PanResponder.create({
      onStartShouldSetPanResponder: (evt, gestureState) => true,
      onStartShouldSetPanResponderCapture: (evt, gestureState) => true,
      onMoveShouldSetPanResponder: (evt, gestureState) => true,
      onMoveShouldSetPanResponderCapture: (evt, gestureState) => true,
      onPanResponderGrant: this._handleOnPanResponderGrant.bind(this),
      onPanResponderMove: this._handleOnPanResponderMove.bind(this),
      onPanResponderRelease: this._handlePanResponderRelease.bind(this)
    })
  }

  componentDidMount() {
    this._subscribeGyro();
    this._subscribeAcc();
  }

  componentWillUnmount() {
    this._unsubscribeGyro();
    this._unsubscribeAcc();
  }

  render() {
    return (
      <Expo.GLView
        {...this.panResponder.panHandlers}
        ref={(ref) => this._glView = ref}
        style={{ flex: 1 }}
        onContextCreate={this._onGLContextCreate}
      />
    )
  }

  // React Native Touch event responders
  _handleOnPanResponderGrant = (evt, gestureState) => {
    this.touching = true
  }

  _handleOnPanResponderMove (evt, gestureState) {
    console.log("moving...")
  }

  _handlePanResponderRelease = (evt, gestureState) => {
    console.log("done...")
    this.touching = false
  }

  // AR/WebGL functions
  _onGLContextCreate = async (gl) => {
    const arSession = await this._glView.startARSessionAsync()
    const scene = new THREE.Scene()
    const camera = ExpoTHREE.createARCamera(
      arSession,
      gl.drawingBufferWidth,
      gl.drawingBufferHeight,
      0.01,
      1000
    )
    const renderer = ExpoTHREE.createRenderer({ gl })
    renderer.setSize(gl.drawingBufferWidth, gl.drawingBufferHeight)

    scene.background = ExpoTHREE.createARBackgroundTexture(arSession, renderer)

    const pointerGeometry = new THREE.SphereGeometry(0.02, 22, 22)
    const pointerMaterial = new THREE.MeshBasicMaterial({ color: 0xffffff })
    const pointerSphere = new THREE.Mesh(pointerGeometry, pointerMaterial)
    scene.position.set(0,0,-0.4)
    scene.add(pointerSphere)

    const animate = () => {
      requestAnimationFrame(animate)

      let xFromScreen,
          yFromScreen,
          zFromScreen

      const worldDir = camera.getWorldDirection()
      xFromScreen = worldDir.x + this.lastSphere.x * 0.5
      yFromScreen = worldDir.y + this.lastSphere.y * 0.5
      zFromScreen = worldDir.z + this.lastSphere.z * 0.2

      this.lastSphere = {x: xFromScreen, y: yFromScreen, z: zFromScreen}

      if (this.touching) {
        const geometry = new THREE.SphereGeometry(0.01, 22, 22)
        const material = new THREE.MeshBasicMaterial({ color: 0xff00ff })
        const sphere = new THREE.Mesh(geometry, material)

        sphere.position.set(xFromScreen, yFromScreen, zFromScreen-0.4)
        scene.add(sphere)
      }

      pointerSphere.position.set(worldDir.x + this.lastSphere.x * 0.5, worldDir.y + this.lastSphere.y * 0.5, worldDir.z + this.lastSphere.z * 0.5)

      renderer.render(scene, camera)
      gl.endFrameEXP()
    }
    animate()
  }
}

Why not just position it at the world position of the camera? That would be .getWorldPosition(). The ‘world direction’ is a … direction. Not a position. Which is what you want to initialize the position of the new sphere to.

Thank you! I’m very new to Three.js and their documentation for the camera only lists getWorldDirection() so I didn’t know I could grab position: three.js docs

Using camera.getWorldPosition() definitely improves accuracy over storing the last value and adding a direction vector to it, but if we want to reposition the cursor in the center of the screen as you rotate the camera we’ll still have to add in gyroscope data somehow.

Made a snack of this working with getWorldPosition():
https://snack.expo.io/Sk0N3VpSz

2 Likes

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.