23 Feb /Programmatic ARKit Positioning

Posted by Ryan Stone

Currently iOS does not support markers or targets in a similar fashion to Vuforia (at the moment of this writing iOS 11.3 has a similar feature but it’s not released). So what is the best way to drop an animation on a target? Assuming you’re dealing with a rectangular object we can use the Vision framework’s VNDetectRectanglesRequest

  // viewController 
  let pixelBuffer      = self.sceneView.session.currentFrame?.capturedImage 
  let ciImage          = CIImage(cvImageBuffer: pixelBuffer!) 
  let handler          = VNImageRequestHandler(ciImage: ciImage) 
  let rectService      = RectangleDetectionService(sceneView: self.sceneView)
  let rectangleRequest = VNDetectRectanglesRequest(completionHandler: rectService.handleRectangles) 
  do { 
    try handler.perform([rectangleRequest]) 
  } catch { 

Above we’ve transformed the pixel buffer being received from ARKit (sceneView) into a ciImage that we then pass to the Vision Framework’s request handler. Then we create a new instance of our custom , then defined the completion handler once a rectangle is detected.

// RectangleDetectionService (rectService)

func handleRectangles(request: VNRequest, error: Error?) {
  guard let observations = request.results as? [VNRectangleObservation] else {
  // get the highest confidence rectangle observation
  let highConfidenceObservation = observations.max { a, b in a.confidence < b.confidence }
   guard let highestConfidenceObservation = highConfidenceObservation else {
   // Calculates the position of the 4 corners
   let points = [highConfidenceObservation?.topLeft,
   // Calculates center point
   let center         = CGPoint(x: (highConfidenceObservation?.boundingBox.midX)!,
                                y: (highConfidenceObservation?.boundingBox.midY)!)

   // create a hit test simulating a user touch at the point within the sceneview
   let hitTestResults = self.sceneView.hitTest(center, types: [.existingPlaneUsingExtent, .featurePoint])
   guard let result = hitTestResults.first else {
     print("Rectangle detection error")
    // now that we've calculated a center point and created a hit test in the arView
    anchor = ARAnchor(transform: result.worldTransform)
    self.sceneView.session.add(anchor: anchor)

You should now have an ARAnchor at the center point of the detected rectangle

TAGS:, ,