Crowdpleaser - Red Bull AdrenaLAN 2019

OVERVIEW

Crowdpleaser is an audience visualizer application. Designed and prototyped during Red Bell AdrenaLAN 2019 hackathon. Crowdpleaser uses augmented reality, with APIs to generate a realtime audience visualization. The AR visualization can be broadcasted to a livestream or mobile app.

Platform

Mobile Phone (iOS AR Kit / Google AR) and Desktop

Tools

Unity
iOS AR Kit
Twitch API - TMI
Google Cloud - Natural Language API
Socket I.O.

SCOPE

24 hours

ROLES

David Rajavicknewsaran - Product Manager, Business Development
Anthony Nguyen - AR Developer
Wister Ng - Web App Developer
Jason Das - UX & UI Design, Game User Research

Winner for Best VR/AR/MR Hack
Red Bull AdrenaLAN 2019

The Challenge

When we look back at how we traditionally played video games, we gathered around the television and participate with a friend beside you. Whether it be a sibling, a parent, a guardian or a group of friends, we wanted to share the experience and thrill. It was not just for competition, but for a feeling of comradery, inclusion, and having an audience.

Fast forward to now, we live in a more connected world. With video games, we were accustomed to couch-co-op gaming. With the advent of multiplayer gaming in online infrastructures, online communities give us limitless opponents. Live streaming gamers have access to audiences wanting to view games in progress. Yet with this success, comes a downfall for live streamers.

 

Emotionally disconnected with no physical presence within our personal space. Non-contextual dialogue and removal of sediment from the conversation. We lose out on the emotion and emphasis, as our only method of engaging with the audience is via a chat room. As we can see with chat rooms, it can be hard to follow conversations and statements.

The Solution

Team Crowdpleaser (L-R) Wister Ng, David Rajavicknewsaran, Ryan Li - Event Organizer (OAM), Jason Das, Anthony Nguyen

 

Our team came together to create Crowdpleaser for the Red Bull AndrenaLAN hackathon. Crowdpleaser is our solution to solve what is hampering the gaming experience for live streamers and the audience. Crowdpleaser uses augmented reality (AR) to create a 3D audience, generated from an active chat room. As users in the chat room type their responses, the application uses Google’s Natural Language to deploy application programming interfaces (API) to generate 3D model animations.

Crowdpleaser would then allow for the AR output to be rebroadcasted as a video feed for their live stream, using Open Broadcast Software (OBS). Another option would be to deliver the AR video in a dedicated mobile device app. This would be a second screen experience for the live streamer to engage within real-time.

Research and Process

Extraction

One of the most popular platforms for gamers to live stream their gaming sessions and content is with Twitch. We looked into Twitch’s developer API library. Using Twitch’s API TMI, Internet Relay Chat (IRC), we were able to isolate users in an active chatroom. We then could figure out which user is which, and assign them eventually a 3D avatar.

Analyze

Once we had each user in the chatroom, we would then scrub their chat comments and run them through Google Cloud’s Natural Language API. The goal was to allow Google to process each comment from a user and decide on the sediment given. The solution was to assign emotional sediments from positive, neutral, and negative.     Leveraging the Google Cloud Natural Language API, we run the extracted text through sentiment analysis to understand what our followers are feeling.

Generate

The sentiment is correlated to a 3D character model expressing that animation whether it be joy, sadness, disgust, or another emotion. The 3D character is animated and compiled using Unity 3D, to then be shown in an AR environment.  The goal was to output the AR content into 2 streams. One that could be broadcasted on a live stream using Open Broadcast Software (OBS) onto Twitch. The 2nd stream would be available to the live streamer using a Crowdpleaser iOS app, powered by ARKit.

Twitch chat feed for a livestream of the game Raft.

As we analyzed the Twitch chat feed, we ran into some issues. One of the factors was the speed of the chat varied user to user. For our current server setup using Socket.io, we were unable to catch every phrase to translate the sentiment. Another issue came in the NLP API. We experienced that certain phrases varied from game to game, in terms of what chatroom language and dialect being used. Terminology and shorthand conversations varied from game to game. Often times certain shorthand was paired with custom emoji icons.

We used four pre-made 3D modelled characters as our avatars in Unity 3D. We were able to assign Twitch user IDs to populate our audience. With the sentiments that we were able to translate using the NLP API, we assigned them animation gestures. Animations such as clapping, cheering, jumping, and thumbs down. Ideally, facial animations on the 3D character models, could be used to emote the sentiments, given the time.

Architecture & Prototype

When we started piecing together the workflow for Crowdpleasure, we determined where APIs would be brought into for processing. As the AR content was created, we used Socket.io servers to compile our final AR deliverables for live streaming. For the time we had, we were able to send out to ARKit on an iPhone 11. Ideally, another AR live stream feed would be reposted on a Twitch channel, using broadcast software like OBS.

 

As we were prototyping the AR output for ARKit, we experienced issues with the placement of our virtual audience. Finding an optimal flat surface for AR placement was tough in our work environment. The happy accident though was were able to populate multiple sections of our AR crowd, as seen in the video.

Conclusion

The ever growing cast of characters from Overwatch.
A sports arena crowd doing "The Wave"
Crowdpleaser demonstration and pitch setup

Key Takeaways

Sentiment Expansion – While we were able to translate some of the chat using the NLP API, we determined that to capture more sentiments in the Twitch chat feed would need more custom Natural Language API. Animations such as clapping, cheering, jumping, and thumbs down. Ideally, facial animations on the 3D character models could be used to emote the sentiments.

 

Audience Analytics – We have created a way for the live streamer to see a visual representation of how the audience is engaging with the content they are streaming. There’s an opportunity to keep track of audience participation, feedback, and time. 

 

Character Licensing – There are opportunities for developers to lend out their game characters as AR avatars in Crowdpleaser. They could release custom skins, emotes, and animations for micro-transactions incentives.

 

Stadium Seating – Giving participants in the live chat to be seen more on the live stream could be another avenue. We can explore crowd placement on Crowdpleaser for future iterations of the platform. We many character models populating the screen, the live streamer could offer users to move up the front. By offering a micro-transaction, that live chat user could be featured more onscreen.

Results

Team Crowdpleaser was awarded 1st prize for Best VR/AR/MR Hack during Red Bull AdrenaLAN 2019. As a team, we were happy with what we were able to accomplish. We felt Crowdpleaser could have been a contender as well for Best Streaming Integration Tool as well. As we presented Crowdpleaser to our panel of judges, we were only able to show a mockup video of how in principle Crowdpleaser would function in the AR mobile app environment. If more time was available, we would have liked to showcase a live stream prototype using OBS. Demonstrating AR/VR applications required more time and knowledge of the parameters.