Skip to content

ARKit custom Feature Points drawing

As you run an ARKit session on your iOS device ARKit allows you to actually see the raw feature points it detects per frame. This can be very useful for certain types of applications where you want the user to get a real feel of how well the session is able to understand the features found in the surrounding physical world. The more features it finds the better the world tracking. You can leverage this to provide realtime feedback and let the user know how to possibly improve the session’s quality.

The easiest way to visualize the session’s raw feature points is to simply set the ARKit “showFeaturePoints”  on the ARSCNView debugOptions array:

A while back I needed to add some custom logic and highlight the feature point visualization differently than what the built in debug option does. What if you want to simply change the color of the visualized featured points? What if you want to show feature points only if the ARFrame has enough of them (> N)? What if you want to actually show point that fall within a certain 3D bounding box only? Etc..

Whatever the reason you might have, implementing your custom feature point drawing code can be quite useful.

Each ARFrame encapsulates the state of everything being tracked for a given moment in time. One of the things we can get from the ARFrame is “rawFeaturePoints”. The rawFeaturePoints property on ARFrame is of type “ARPointCloud”.

An ARPointCloud is simply a collection of points in the world coordinate space of the AR session. Them being in the world coordinate space of the AR session turn out to be quite handy. That means we can simply leverage these points to build our on geometry to then assign to a node we can place in the scene to then add our custom look and logic.

An ARPointCloud has a get property called points:

So what are the steps we can follow to draw our custom feature points?

  1. access the ARFrame
  2. find the feature points from the ARFrame (if any)
  3. leverage our custom feature points SCNNode
  4. prepare our custom feature point geometry
  5. update the SCNNode geometry with the one we just created
  6. add the feature points node to the scene’s root node

We can get the ARFrame simply by leveraging the didUpdate frame call found in the ARSessionDelegate:

At this point we can access the frame’s rawFeaturePoints and then use its “points” property to create our custom geometry:

once we have the geometry we simply set it to be the node’s geometry and then make sure our featurePointsNode is added to the scene’s root node:

GEOMETRY

Let’s take a look at the code needed to create our geometry in the function called “pointCloudGeometry”:

The function above will give you a valid SCNNode geometry that will allow you to draw feature points however you want them to show up. Let’s take a look at the logic. We first prepare the point data. Each point is of type float3, the data is going to be the points array [float3] and the bytes count is going to be the byte size of a float3 (16) times the number of points.

SCNGeometrySource

The data is simply the pointData we prepared. Semantic to be used is vertex. Each point passed in represents a vertex in the geometry. Vector count is the number of points. Each vector has 3 components [from a memory point of view a float3 is made up of 16 bytes. XYZ [4+4+4 = 12], the 16 is due to the 12 + 4 (bytes of padding, aligned).

So now how do we connect the vertices found in the geometry source? We use the SCNGeometryElement, we set the point size to a very small value in meters and then min and max point radius to the same value to make it so the points show up at a constant size in screen space (10). That’s actually quite a nice thing to have, without it, we would have to compute the projection distance and use that if we wanted to make the points look constant in size. Having them constant is nice because we would not see features that are further away in the frostrum.

The rest is pretty self explanatory. We simply instantiate the SCNGeometry to assign to our dedicated featurePointsNode.

Be First to Comment

Leave a Reply

Your email address will not be published. Required fields are marked *