Generative AI and VR/AR/Gaming: Complementary Technologies

This may be pretty obvious to many readers, but I thought it would be fun to have a discussion about it: Generative AI should sharply propel forward the development of lifelike Virtual Reality and Augmented Reality environments (and of sub-VR visual environments, such as those for many computer games).

That's particularly clear, I think, for art, but it should be especially true of Large Language Models, which will make it much easier for users to have conversations with non-player characters (or their non-game analogs). This should sharply increase the realism and immersiveness of the world. And for games, LLM hallucinations needn't be much of a problem: If part of the game is that you need to talk to NPCs to get hints, or to organize joint activities, LLMs' tendency to brazenly confabulate will just make things more life-like.

But that's just my speculation. What do you folks think about it? In particular, are there computer games, VR/AR or otherwise, that are already creating LLM-based NPCs, or otherwise using generative AI in an interesting way? Please post your thoughts in the comments.

Comments

[Zarniwoop]

LLMs' tendency to brazenly confabulate will just make things more life-like

If by "life-like" one means "listening to Donald Trump bloviate about election fraud"

[Internet sloth]

gaslighting the enemy into insanity

[Voize]

I predict the porn industry will drive LLM + VR/AR development just like how it accelerated the Internet in the 90s and videotape in the 70s.

[dwb68]

Probably. I am sure there is a market for a life like sex robot.

Cue all the sex workers whining about losing their jobs to silicone.

[Bwaaah]

Silicone is what gave them their jobs. Silicon is what now takes them away.

[gattsuru]

There have been some efforts along these lines already: Mantella is a Skyrim VR mod intended to integrate LLMs into NPC speech, and some less public attempts to tie LLMs into social networks like VRChat as 'assistants'. They have issues -- long processing time, MLgen text-to-speech being pretty lackluster outside of a couple specific projects, difficulty maintaining or understanding the state of the virtual world, minimal or no integration between the NPC's internal state and its body language -- but some of these are likely surmountable.

That said, there's nothing especially tied to the VR here, with things like Herika as a similar non-VR Skyrim mod, or AI Roguelike as a Rogue-like (or... uh, a couple furry fetish visual novels). A number of those listed problems are far less relevant in these contexts: away from VR, awkward body language and minor delays in voice lines are less noticable (if still noticable, hence the long-lasting complaints about Skyrim and Oblivion's original voice lines).

And there's not clear reason that this would necessarily be more correct. You bring the example of NPCs 'lying', but TheToadyOne's been doing that with far more prosaic techniques for over a decade with DwarfFortress, for example.

/That/ said, it's not clear that the depth of NPC interaction is the problem for VR consumers, or even more beneficial than harmful. There's a lot of fun ideas about making a real version of the Young Lady's Illustrated Primer from a Neal Stephenson novel, but even games like Minecraft or Vintage Story don't really have situations that would benefit from an NPC-like that. Instead, we're mostly just proposing a firehose of content, but there's already more content just in AAA-level titles than even the most dedicated players can get through, with some individual games having months or more of play.

Worse, most games as they exist build communities around people facing _similar_ challenges. If the MLgen expands beyond a mere topcoat, even if you could get something consistently interesting and desirable, you risk every player going off into their own schizoid world, where any two people playing 'the same game' can't meaningfully discuss it, because their interactions have gone so far separate.

Maybe non-gaming AR/VR implementations could benefit more (and be closer to that Illustrated Primer), but at least currently there doesn't seem to be massive demand for those, either.

[dwb68]

This already happens esp in multiplayer games like PUBG and Fortnite. Half the players at the lower levels are bots, spawned with some sort of strategy to give newbs confidence.

My 0.02: bots ruin games. Let all the bots play against each other. But, what ruins games more than bots are cheaters and hackers. Where AI could come in handy is to auto-detect all the cheaters and hackers using aimbots, x-ray hacks to see through walls, etc.

[Champion]

It's an interesting topic, but I don't remember anyone mentioning glasses lately, except at the Apple presentation, and then I feel like they didn't exist at all, so I don't even know what to say. But there are still things that will manifest themselves sooner or later, like my tracker https://jooby.eu/rdc-en/electricity-consumption-tracking-en/, if it wasn't for it, I would probably be left without a home, and this is no exaggeration, it saved me, gave me a signal that the voltage had dropped and realized that a tornado would soon be coming, so I went down to the shelter by heart.

[BrianMcDaniel]

I saw this topic and decided to tell you about my experience of cooperation with https://arrible.com/services/custom-game-development/ team. Arrible team worked on my project on game development using VR technologies. To be honest, I was amazed by their technical skills and creative approach to the project. They helped me create an immersive and engaging game world that fully met my expectations. I am very grateful to the Arrible team and am impressed to write this comment.

[jamescrook]

The intersection of generative AI and VR/AR gaming is definitely an exciting space. AI-driven content creation could make open-world games feel truly limitless, while VR and AR provide the immersion to bring those worlds to life. It’ll be interesting to see how these technologies evolve together—especially in areas like NPC interactions and dynamic storytelling. Speaking of interactive experiences, platforms like 1Win https://1winbet.pk/ are also using tech to enhance entertainment, offering sports betting and casino games with innovative features. The future of digital engagement is looking more dynamic than ever!