Lenslist x Lightship Templates Challenge by MPAR

Keanu Christmas

Your AR creation doesn’t need to be super complicated with a whole multiplayer experience to be able to use one of the Lightship templates effectively – it can just be finding a novel, different, curious use for the technique. Have fun with it! MPAR’s first hackathon entry was for a Keanu Reeves themed hackathon at the AR House. Our entry was “Keanu Christmas” and it transforms any tree into a Christmas tree by placing baubles with Keanu’s face on them!

Niantic’s Lightship ARDK is able to understand the world around the player and recognise which parts of the camera view are sky, which are trees, which is the ground and so on. Each of these potential areas for detection are the “segmentations”, so this feature is called Semantic Segmentation.

“Keanu Christmas” only uses 2D textures for its effect but we were pretty happy with how it came out, and we were able to put it together in a weekend.

Achieving something similar with the new Lightship Templates

Contextual Awareness

If you’d like to achieve effects targeting trees, ground, buildings, sky… etc using Lightship’s new Templates, you could use the Semantic Segmentation Template which lets you overlay textures into each Semantic Segmentation area.

Keep in mind that this template can’t place 3D objects into those areas, only a 2D texture, but as you can see in our “Keanu Christmas” app, 2D textures can still be pretty powerful! Sticking with just 2D can also reduce potential lag in your AR application. It is much less for the app to process than 3D objects but can still give a nice subtle effect.

If you do want to use 3D objects within those segmentation areas, Lightship has their Semantic Masking Template which can do just that.

One of the best parts of the Lightship Templates is that they come with mock environments to test out your scene — these can save you hours of time testing your project when it comes to Semantic Segmentation! When building “Keanu Christmas” (and “Weathergy” mentioned next!), we manually uploaded each test to our phone to try out each feature. Using Lightship’s “Virtual Studio” and their mock environments, you can see your app running in a simulated indoor or outdoor environment from within Unity itself on your computer. No uploading to your phone required! This lets you easily see how it looks on trees/sky/ground… etc. Don’t forget about mock environments when building with Segmentation!

Weathergy

Weathergy is a proof of concept AR game which uses the power of Niantic’s Lightship ARDK to control the weather, or at least augment it! We wanted to try to push the boundaries of what we’d tried before and use a range of Lightship’s capabilities together in one AR game. It was lucky enough to win Niantic’s Winter Challenge in the “Understanding” category! We created our own floating elements called “weathergy” which have two variants — “Summer” Weathergy and “Winter” Weathergy. If you tap a “Summer” Weathergy, you bring the environment closer to feeling like Summer. If you tap a “Winter” Weathergy, you bring it closer to Winter. You can then play multiplayer — one person is Team Summer and one is Team Winter.

You’ll recognise one of the main techniques we used, as we mentioned it in the “Keanu Christmas” project. In order to simulate the weather, we used contextual awareness — in particular, Semantic Segmentation. So for Weathergy, we place clouds in the sky when Winter is in the lead and flowers in the trees when Summer is leading. It was curious to see what it’s like to turn effects on and off, especially switching which segmentation you are targeting (e.g. from the sky to the trees). There’s definitely a lot more potential that could be explored here.

It also uses Niantic Lightship’s Real-Time Meshing feature which lets you interact with the real world. In particular, you can generate objects onto real world surfaces like tables, floors, walls… etc. Each weathergy that the players try to tap is generated from a real world surface before it floats around — giving each one a real world location.

Finally, it also uses Lightship’s shared AR to sync the location of each weather for each player, as well as each player’s location in the real world.

Achieving something similar with the new Lightship Templates

Contextual Awareness – Real-time Meshing

Lightship’s Meshing is available in their Real-Time Meshing Template, as well as the Occlusion Templates in the Contextual Awareness category. Just like with Semantic Segmentation in the “Keanu Christmas” example, Niantic provides a mock environment that can help you test Real-Time Meshing concepts in their templates. It can be very useful and save a lot of time when it comes to developing your concept initially as you can test as you go in a reliable way. Once you know your general concept is working in Lightship’s “Virtual Studio”, then it’s much more reliably going to work in real world scenarios — of course, make sure you also test in a range of real world locations and conditions to see if it works as you’d expect!

Contextual Awareness – Semantic Segmentation

To achieve a similar effect using Lightship’s new Templates, you could use the Semantic Segmentation Template which lets you overlay textures into each semantic segmentation area. They’ve even got it working with multiple segmentation areas at once which is very useful, so you could change how the sky, ground and buildings all look in your Augmented Reality scene. This is incredibly useful as back before Templates, it was quite tricky to implement multiple segmentation areas at once.

If you’re looking to include this in your project, we’d recommend including the Template from the start as it’s much easier to test Segmentation in a fresh project — however if you are putting this into an existing app, it’s definitely still possible to do so.

Shared AR

Our shared AR creation used a lot of concepts from an earlier example app Niantic provided called “AR Voyage”. It was an incredible showcase of how to do a range of things using Lightship (and you can find it on the App Store!). However, it was challenging to get sample scripts and concepts from it. Lightship’s new Templates make this much easier! You can work out the basics of shared AR with their Shared Object Interaction Template.

GeoGlitched

GeoGlitched is a game that uses Niantic’s Lightship to generate objects throughout your world that have appeared from a randomly selected location — street signs, flags, newspaper headlines, bills of local currency and more. The player must work out which random location these items were from using each object generated in the real world as a clue.

It uses a whole bunch of open APIs across the web to generate each location’s objects, so the game is always different with new clues for each location!

The goal is to help people explore their own world walking around their town with friends, while also getting to learn about places around the world at the same time. It won first place in Niantic’s “Summer of AR Challenge” and is currently in an alpha test phase where people can register to test it out at https://geoglitched.com. We’re hoping for a bigger public release later this year!

The item generation is done via Lightship’s Real-Time Meshing, which also allows for their built-in occlusion, so generated objects can appear in front of and behind real world objects. This is quite effective as it helps make the items blend in more with the real world.

Achieving something similar with the new Lightship Templates

Contextual Awareness – Real-Time Meshing

Just like with the Weathergy project, the same general Template can be applied to generate items similar to GeoGlitched. Lightship’s meshing is available in their Real-Time Meshing Template. If you’d like to take it further, like we do with GeoGlitched, you could add features to that Template using other tutorials from Niantic. The new Lightship Templates do provide a whole lot of potential on their own though, so you might not even need to depending on your idea.

Meshing is best used at the very start of your project, however if you have an existing concept that already uses Unity’s physics, it’s not hard to bring in Lightship and its Real-Time Meshing as that’ll work just like virtual environments you’ve built in Unity before… but in the real world!

If you’d like to keep up with what we’re building over at MPAR, we’ve got our new website up and running and we’ve recently got ourselves set up on all the typical social media places as MPAROnline (Instagram, Snapchat, Twitter, YouTube, Twitch… all those!). We’ve got a range of other Lightship powered projects in the works — some of which you can get a sneak peek at on our website. We’re looking to build Augmented Reality projects that help with social good and bettering our future, and can’t wait to share a bunch with you soon!

– Meri and Pat from MPAR

Thanks, guys! 🧡 We’re so grateful for your tips, experience, creative processes, projects and guides – sharing knowledge with the rest of the AR Community is priceless!

Follow Meri and Pat from MPAR on their social media and learn from the best. 👾💪

We hope this article kicked-off your imagination and will help you with working on Lightship Templates – can’t wait to see how you’ll make use of them for our Challenge! ⚡️

Final words from us: Take your chance and you won’t regret it! We’re not expecting you to become a pro in a day, but with Templates and guides from Niantic and the Community, even your first Lightship project can be a winning one! 🏆