Crowdpleaser - Red Bull AdrenaLAN 2019


Crowdpleaser is an audience visualizer application. Designed and prototyped during Red Bell AdrenaLAN 2019 hackathon. Crowdpleaser uses augmented reality, with APIs to generate a realtime audience visualization. The AR visualization can be broadcasted to a livestream or mobile app.


Mobile Phone (iOS AR Kit / Google AR) and Desktop


iOS AR Kit
Twitch API - TMI
Google Cloud - Natural Language API
Socket I.O.


2 Days


David Rajavicknewsaran - Product Manager
Anthony Nguyen - AR Developer
Wister Ng - Web App Developer
Jason Das - UX & UI Design, Game User Research

Winner for Best VR/AR/MR Hack
Red Bull AdrenaLAN 2019

The Challenge

When we look back at how we traditionally played video games, we gathered around the television and participate with a friend beside you. Whether it be a sibling, a parent, a guardian or a group of friends, we wanted to share the experience and thrill. It was not just for competition, but for a feeling of comradery, inclusion, and having an audience.

Fast forward to now, we live in a more connected world. With video games, we were accustomed to couch-co-op gaming. With the advent of multiplayer gaming in online infrastructures, online communities give us limitless opponents. Live streaming gamers have access to audiences wanting to view games in progress. Yet with this success, comes a downfall for live streamers.

Emotionally disconnected with no physical presence within our personal space. Non-contextual dialogue and removal of sediment from the conversation. We lose out on the emotion and emphasis, as our only method of engaging with the audience is via a chat room. As we can see with chat rooms, it can be hard to follow conversations and statements.

The Solution

Team Crowdpleaser (L-R) Wister Ng, David Rajavicknewsaran, Ryan Li - Event Organizer (OAM), Jason Das, Anthony Nguyen

Our team came together to create Crowdpleaser for the Red Bull AndrenaLAN hackathon. Crowdpleaser is our solution to solve what is hampering the gaming experience for live streamers and the audience. Crowdpleaser uses augmented reality (AR) to create a 3D audience, generated from an active chat room. As users in the chat room type their responses, the application uses Google’s Natural Language (NLP) to deploy application programming interfaces (API) to generate 3D model animations.

Crowdpleaser would then allow for the AR output to be rebroadcasted as a video feed for their live stream, using Open Broadcast Software (OBS). Another option would be to deliver the AR video in a dedicated mobile device app. This would be a second screen experience for the live streamer to engage within real-time.

Research & Process


One of the most popular platforms for gamers to live stream their gaming sessions and content is with Twitch. We looked into Twitch’s developer API library. Using Twitch’s API TMI, Internet Relay Chat (IRC), we were able to isolate users in an active chatroom. We then could figure out which user is which, and assign them eventually a 3D avatar.


Once we had each user in the chatroom, we would then scrub their chat comments and run them through Google Cloud’s Natural Language API. The goal was to allow Google to process each comment from a user and decide on the sediment given. The solution was to assign emotional sediments from positive, neutral, and negative.     Leveraging the Google Cloud Natural Language API, we run the extracted text through sentiment analysis to understand what our followers are feeling.


The sentiment is correlated to a 3D character model expressing that animation whether it be joy, sadness, disgust, or another emotion. The 3D character is animated and compiled using Unity 3D, to then be shown in an AR environment.  The goal was to output the AR content into 2 streams. One that could be broadcasted on a live stream using Open Broadcast Software (OBS) onto Twitch. The 2nd stream would be available to the live streamer using a Crowdpleaser iOS app, powered by ARKit.

Twitch chat feed for a livestream of the game Raft.
As we analyzed the Twitch chat feed, we ran into some issues. One of the factors was the speed of the chat varied user to user. For our current server setup using, we were unable to catch every phrase to translate the sentiment. Another issue came in the NLP API. We experienced that certain phrases varied from game to game, in terms of what chatroom language and dialect being used. Terminology and shorthand conversations varied from game to game. Often times certain shorthand was paired with custom emoji icons.
Assigned Twitch Chat Users to 3D Character Models - Unity AR Work Environment
We used four pre-made 3D modelled characters as our avatars in Unity 3D. We were able to assign Twitch user IDs to populate our audience. With the sentiments that we were able to translate using the NLP API, we assigned them animation gestures. Animations such as clapping, cheering, jumping, and thumbs down. Ideally, facial animations on the 3D character models, could be used to emote the sentiments, given the time.

Architecture & Prototype

When we started piecing together the workflow for Crowdpleasure, we determined where APIs would be brought into for processing. As the AR content was created, we used servers to compile our final AR deliverables for live streaming. For the time we had, we were able to send out to ARKit on an iPhone 11. Ideally, another AR live stream feed would be reposted on a Twitch channel, using broadcast software like OBS.
As we were prototyping the AR output for ARKit, we experienced issues with the placement of our virtual audience. Finding an optimal flat surface for AR placement was tough in our work environment. The happy accident though was were able to populate multiple sections of our AR crowd, as seen in the video.


The ever growing cast of characters from Overwatch.
A sports arena crowd doing "The Wave"
Crowdpleaser demonstration and pitch setup

Key Takeaways

Sentiment Expansion – We were able to translate some of the chat using the NLP API. It was determined that to capture more sentiments in the chat feed, we would need more custom Natural Language API based on the game.

Cues – Such as clapping, cheering, jumping, and thumbs down. Ideally, facial animations on the 3D character models could be used to emote the sentiments.

Audience Analytics – We have created a way for the live streamer to see a visual representation of how the audience is engaging with the content they are streaming. There’s an opportunity to keep track of audience participation, feedback, and time.

Character Licensing – Opportunities for developers to lend out their game characters as AR avatars in Crowdpleaser. They could release custom skins, emotes, and animations for micro-transactions incentives.

Stadium Seating – We could explore crowd placement on Crowdpleaser, for future iterations of the platform. With many character models populating the screen, the live streamer could offer users to move up the front. By offering a micro-transaction (Twitch Bits), that live chat user could be featured more on screen.


  • Team Crowdpleaser was awarded 1st prize for Best VR/AR/MR Hack during Red Bull AdrenaLAN 2019. As a team, we were happy with what we were able to accomplish.

  • We felt Crowdpleaser could have been a contender as well for Best Streaming Integration Tool as well. We should have considered developing for the Twitch Extension platform at the time. 

  • As we presented Crowdpleaser to our panel of judges, we were only able to show a mockup video of how in principle Crowdpleaser would function in the AR mobile app environment. We would have liked to showcase a live stream prototype using OBS integration. 

  • Demonstrating AR/VR requires knowing the environment you are presenting your pitch well. There’s an expectation of wearing or holding an AR/VR device. There’s also to consider spatial design in case the solution requires a flat surface or space for movement.

For more case studies, click on one of the following below.

TED - Elevate Tech Jam 2019
SceneScout - Travel Mobile App
PlayStation Compass - App Feature