Nick Mathews

Getting Started with Google’s ARCore in Kotlin

January 3, 2020

Here at Stable Kernel, we’ve written several blogs on the impact and use of augmented reality (AR). In short, AR is rendering digital objects or images through a device in such a way that they appear to actually exist in the real world. One popular AR app that took the world by surprise is PokemonGo. Because rendering 3D models to appear in the real world can be difficult, Google and Apple both released SDKs (ARCore and ARKit) that make implementation easy. Today, I’d like to focus on Google’s ARCore in Kotlin and share some quick tips on getting started with your own AR app. Just like we are big fans of AR applications, we are also big fans of GIFs. So this tutorial will guide you through setting up a simple demo that leverages your phone’s camera to display a looped GIF in the real world.

Sceneform and Augmented Images

With ARCore you can use OpenGL yourself to manually render your virtual objects, but that’s a much bigger topic than this blog post can hold. Luckily, Google released an SDK built on top of ARCore, called Sceneform, that greatly simplifies rendering AR objects. Sceneform provides a high-level API for telling ARCore where you want your object rendered and what 3D model to render.

Google also provides a feature in ARCore called Augmented Images that lets you register images you want to detect and ARCore will find instances of that image in the real world and let you render something on top of the image. This is what we’ll use to find the GIF images and render the actual GIF on top of it.

Google has an example of using Sceneform and Augmented Images, but it is all written in Java and renders a 3D object on the image, not a flat “image” like we want. To show the image, we’ll have to use what is called a ViewRenderable. A ViewRenderable is a special type of renderable object that lets you use Android Views (such as Button and ImageView) within your rendered space.

Getting Started

This demo app will be starting as just a blank application. You can read more here about the requirements and packages required for using ARCore such as minSdkVersion of 24, the necessary permissions, whether the AR feature is optional or required for your app, etc. The Augmented Images guide is fairly thorough and straightforward on how to use and set up an AugmentedImageDatabase so that won’t be covered here.

Using Google’s sample Augmented Images app as a guide, we can now start to make our app a reality. The sample Java code doesn’t do anything too complicated, so converting it to Kotlin is fairly simple. You can see the demo app here for the finished version. The following is a breakdown of the different classes used in the demo app.

The MainActivity sets up the fragment and contains a map of detected Augmented Images to the Node that holds the image’s position and how to render it. The activity also contains the code for checking if any new images have been detected in the onUpdateFrame.

The AugmentedImageFragment class is used to configure the ARCore Session as well as set up the AugmentedImageDatabase. This demo uses the programmatic approach of building the database at runtime since we’re only looking for a couple images. I would recommend pre-generating your database in a real-world app.

Finally, the AugmentedImageNode is an object that is created whenever an Augmented Image is detected. It holds the image’s location in the real world and handles telling ARCore how and what to render over the image. Since this demo shows a real GIF over top of a static image, it creates a ViewRenderable using a simple ImageView and uses an image loading library (Glide in this case) to show the GIF.

Sizing the GIF

By default, ARCore and Sceneform use a scale of 250dp in the view being approximately 1 meter in the real world. So if the View you render is 250dp wide, it will appear in the augmented reality world as ~1 meter. Obviously, this won’t work too well for large GIFs or small real-world images. The rendered object will be far too large. We need to change the scale that ARCore uses to render the GIF so it covers the detected image as closely as possible. I tested several ways of doing this, and ended up using this approach:

val imageWidth = 400
val viewWidth = (imageWidth / image.extentX).toInt()
gifView?.sizer = DpToMetersViewSizer(viewWidth)

In the above code, imageWidth is the size of the GIF image. Since all of the GIFs I used came from GIPHY, they all ended up being 400 pixels wide. ARCore makes a rough estimation for how wide the detected image is in meters and that value is in image.extentX. Essentially the above block is figuring out the correct scale of pixels-to-meters if the detected image is one size and the source GIF is another size.

ARCore uses DpToMetersViewSizer as a way to override the default scale so now the rendered GIF almost exactly covers the detected image and appears to bring the image to life.

In Conclusion

Now you have a basic app that can render images on top of other images. Adding other images to detect is as simple as adding the image as an asset, adding the asset to the database, and telling the node what GIF to load. Even with Sceneform helping out, there is still a good bit of boilerplate that I didn’t include in this blog, but once the framework is up and running, adding other virtual objects or images to detect is fairly simple. You can find the full example on our Github.

Happy coding!

Published January 3, 2020
Nick Mathews

Software Engineer at Stable Kernel

Leave a Reply

Your email address will not be published. Required fields are marked *