NEO NPCs

 NEO NPCS

A Ubisoft R&D project focused on player-facing GenAI, featured at the GDC 2024 and made in partnership with Nvidia and Inworld AI. The focus is on having real time conversations with NPCs and experiment with this new technology in order to create emerging storytelling.

I joined Neo NPCs in May 2023 after working on Mario + Rabbids : Sparks of Hope. This tech-driven project wanted to create GenAI-powered NPCs with gameplay implications to make new ways of interacting with them using the player’s free inputs. The project needed a designer to make the bridge between the two. After familiarizing myself with the technology and specificities of Large Language Models (LLMs) and GenAI models in general, I focused myself on gamifying conversations and designing innovative gameplay loops with this new paradigm.

Specificities

As the only designer of the project for 7 months, I was in charge of articulating free speech into gameplay and had a focus on two different prototypes : an explorative one, and the proof of concept shown at the GDC 2024. Both involved communicating with NPCs using the player’s voice as input.

I quickly realized this work was about finding the sweet spot between three main axis :
-Realistically respect the expectations of a face-to-face conversation between humans (or else the game would feel very alien and uncanny).
-Find compelling gameplay loops (Ubisoft being a game publisher first, we still wanted to do a game and not an augmented chatbot)
-Respect the constraints and opportunities of GenAI (or else the project would loose its purpose). This drove the design process to be very bottom-up.

The final result would be to have NPCs adapting in realtime to the player’s speech and having their relationship or emotions change dynamically, as well as having NPC to NPC conversations and interactions.

The proof of concept

We showcased a demo where the players embodied a character (named Nelson), who joined a rebellious movement called The Resistance in a dystopic near-future city controlled by autoritarian mega-corporations. After being accepted into the Resistance, Nelson would meet another character named Bloom, with whom they would make a team, and the demo featured the first conversation between the two in which the two would learn more about each other.

Design systems

As a designer I had to analyze what a conversation is, in order to take the essence of it and gamify it later on. I therefore worked to formalize many systems that needed to be flexible enough to be coherent with free speech:
-Emotions (we wanted the player’s input to alter the emotional state of the player)
-Relationship (general progression symbolizing the bond between the avatar and the character, and all charges influencing it)
-Subjects of conversation (we wanted that the character would recognize the topics brought by the player, which would then change their emotional state and relationship.
-Conversational actions (detect and orient the character depending on the strategy used by the player : are they caring? intimidating? apologizing? complaining?)
-Scripted intels, bits of information released upon asking the right question.
-Objectives (main and side objectives to structure the discussion and avoid the white page syndrom).

All of this then had to be integrated into a whole economy of emotion charges and progression and needed to be iterated and balanced.

Production implications of working with GenAI

Working with GenAI in mind brought many challenges and we had to invent new ways of working : defining new ways of working between the data science teams and the design team, harnessing the immensity of free speech using classifiers in order to structure our experience, lots of intricate systemic design with a lot of variables that sometimes were not in our hands and needed to be balanced regardless.

Example of articles on the subject:

Ubisoft let me actually speak with its new AI-powered video game NPCs - The Verge
Ubisoft’s Neo NPC prototype is a glimpse at the future of how we’ll interact with games - IGN
Ubisoft unveils generative AI "NEO NPCs", and the spirit of Peter Molyneux's Milo lives on - Eurogamer
Ubisoft debuts NEO NPC AI prototypes | GDC 2024 - Games Industry