See the Rhythm with the New AR Music Feature on Instagram


Sound has been proven to play a crucial part when it comes to AR effects. The best stories presenting them are enriched with a fitting song, and a music industry fell in love with social media AR long ago – there are almost 200 promotional effects on Lenslist that were either inspired by a song or were used to create an official music video. And now, this dependence will be even stronger as Instagram AR effects can finally react to the music!

The funcionality has been available to Spark AR Creators since May 26, so we took some time to collect exemplary effects and ask a few Creators about their thoughts on a new feature.

AR effects reacting to music

Music reactive capabilities is yet another of sound features available for Instagram AR effects. Creators can now relate to a natural human need to express emotions with music by creating AR effects that move and play in sync with their favourite songs. For some time now they have been able to be built into an effect by a creator, but now they can also be chosen be the user from Instagram’s music library or even picked up by a device’s microphone during recording!

The Conductor by yarinlidor, Kandinsky in Concert by vi.de.or.bit
 

In the wake of the new Spark AR audio interactivity features I wanted to create something that was telling a story besides being a nice effect. I’m quite aware of the art world and art history and I love to make references to artists and pieces that interest me. Wassily Kandinsky is often credited as a pioneer of the abstract artform, experimenting with audio visualisations in basic shapes and colors. I wanted to create a piece that was retelling his story using the technology of today and bring his visionary work to live.

― Isabelle Udo (vi.de.or.bit)

Audioled by malf.visuals, pARty beARd by rosterizer
 

AR Music is that feature I’ve been waiting for so long. Music is what makes people unite… from all around the world. Seeing artists such as designers and musicians collaborate to create innovative and visually appealing effects is one of the sides that most excites me. Audioled is a filter I created while testing the update. The effect itself is quite simple but watching how camera and images reacts to music is just awesome. Overall I suggest to all creators and users to try this feature becouse it’s pure fun!

― Domenico Amalfitano (malf.visuals)

Old but new feature: the user picks the music

Some could say: it’s not a big deal. AR music was there before, it’s just that creators had to put a song inside an effect beforehand. It was enough for music industry…

Wrong❗

AR effects are all about combining the creator’s and user’s creativity, so giving so much power to the user is a breakthrough, for sure. The key to a great effect is an inclusion of the user into the process of creation. Also, what we have here is an interplay of art and technology. Isabelle Udo told us that she’s very excited about the future of these types of technology mixed with art.

That is why many creators will certainly update their old AR music effects to give their users an opportunity to experience a whole new freedom in mixing visual art of AR with music.

Guys from Pixel Chefs and Nelson shared some thoughts on their music effects created before the newest feature has become available and a future development of AR music effects. We’re rooting for them to update the effects presented here as soon as possible!

Earworm by pixelchefs, Synesthesia by nelzster01
 

The idea for this effect was inspired by the idea of a song being stuck in your head, thus the name earworm. This effect was created before the new audio feature came out and all the audio visuals were animated and triggered manually but we are excited to update it using the new feature so that people can use it with their own favorite music 🎵

― Pixel Chefs (pixelchefs)

The idea for the synesthesia effect was born out of my love for music and the possibility and potential that I imagined for AR music effects by me and a few other creators like jerzy. At the time when I was making the effect, the new audio feature didn’t exist yet so I created my own song, samples and patch systems for it, but the fact that spark AR team listened to the community is heart warming and fills me with gratitude. I see it as a first step in advancing a new genre of effects and look forward to how they solve future obstacles such as isolating key instruments within a song and the cool effects the people create with these tools in the future.

― Nelson (nelzster01)

There are also such creators as Josh Beckwith, who have already managed to update their old sound effects for them to react to the music of user’s choice instead of a built-in track.

Thermachromic was my very first effect on Instagram, first published almost a year ago. I hadn’t updated it since then, so I took these new audio reactivity features as a sign and started working on version 2. I worked in a few new concepts and tweaked the overall look to make it look a bit more like actual thermofilm. The updated version lets you switch modes by tapping (variations of inversion and segmentation masking), and the slider controls the number of gradient repetitions. Since the gradient is repeatable and infinite, I thought it would be cool to have the audio energy drive the movement forever forward. The energy is fed into an accumulator and connected to a UV offset for the gradient texture.

― Josh Beckwith (positlabs) from Popul-AR Agency

Thermachromic by positlabs, Sour Note by chrispelk
 

Tips for Creators from Josh Beckwith

This part will not be very useful to marketers (tips for them are below), but it surely will be for Spark AR creators. Many thanks to Josh Beckwith, who, while being very excited about the new feature, pointed out that it has a few pitfalls where developers could get stuck or confused, so he listed them out and gave his solutions:

  • Currently there is no live mic input in the Spark IDE, but you can use a looped sound clip for testing during development. I usually grab something from the audio library because it’s convenient. Hopefully we will get live mic input in Spark soon, since it was meant to be released back in v82 but there were some issues with it, and it’s been disabled. The team is aware of it and they’re working on a fix.
  • When you test on a device, the microphone won’t turn on until the user starts recording. I hope this will save you all from yelling at your phone in the preview state 😊
  • Instructions! I found that most users won’t just happen to discover audio reactivity, so instructions are pretty important. There are a few custom instructions that might help communicate the idea, but there’s nothing very exact at the moment – one is “tap and hold record to start” and another is “say something.” I would also recommend specifically demonstrating the audio reactivity in the effect demo to really get the idea across.

Tips for Marketers

It goes without saying: AR music is so entertaining that it must be concedered by brands when they want to engage their fans. Here are some quick thoughts on using the feature in promotional effects:

  • Add instructions, so even those of your fans who are not familiar with AR will discover the whole potential of AR music effect,
  • use AR music as it was used before: to promote a new song – it still is a great idea for releasing a track,
  • use it in a new way to get user-generated content by challenging them to find the best soundtrack for the visual representation of your AR effect.
  • brainstorm – the new ideas are here for grabbing. See what Josh Beckwith had to say regarding the new possibilities:

    I’ve only just scratched the surface of it – there is still much more I want to explore with the audio analyzer. I’m thinking of experimenting with autotuning voices and maybe a utility for tuning instruments.

Voice Message by enuriru, Electronic Performer by bourgadotin
 

I have been always fascinated by music videos, and it was amazing to see artists using my effects as a free VFX tool to create a visualisation for their music. Now with the audio reaction filters, you can use mic input to create your own unique video.

― Jerzy Pilch (EQlasses)

Thank you for staying with us for so long! We hope that this article helped you grasp the idea of new AR music features. If you have any idea for how they can be put to the use in a new, creative way or have any questions, feel free to write to us!