Monday, July 20th, 2 pm to 5:15 pm
About
Generative AI is rapidly transforming game development, enabling new approaches to content creation, simulation, and player interaction. However, for interactive systems, the central challenge is not generation alone, but control—the ability to steer generative models in real time, align them with player intent, and integrate them into production pipelines.
CTRL-G (Controllable Generative Graphics for Games) brings together experts from artificial intelligence, graphics/VFX, and games to explore how generative systems can be made controllable, responsive, and usable within modern game engines.
The workshop focuses on two key themes:
(1) Controllable world models for games, including player modeling and responsive content
(2) Game engine education in the era of generative AI
By connecting technical advances with production realities and educational needs, CTRL-G aims to define a practical and forward-looking agenda for generative AI in games.
Schedule: Monday July 20th
2pm-2:15pm: Intro — Josef Spjut
2:15pm-2:45pm: Foundations Industry Research — Anjul Patney, NVIDIA
2:50pm-3:20pm: Games Industry Development — Michael Lentine, Epic Games
3:25pm-3:55pm: Academic Research and Education — Arnav Jhala, NC State and Duke University
4:00pm-4:30pm: Visual Perception — Budmonde Duinkharjav, NVIDIA
4:45-5:15: Panel Discussion moderated by Josef Spjut
Motivation and Relevance to SIGGRAPH
SIGGRAPH has historically shaped the evolution of graphics, simulation, and digital content creation. Generative AI represents a comparable shift, altering not only how assets are produced but how interactive worlds are authored and experienced.
Theme1: Controllable World Models for Games
Recent advances in generative AI enable large-scale models of environments, characters, and behaviors. For games, these systems must be controllable and responsive to players.
Player Modeling and Adaptive Systems
• Inferring player intent, skill, and affect
• Systems inspired by architectures such as Lumine [Tan et al. 2025] and Nitrogen [Magne et al. 2026]
• Using player trajectories to guide generation
Responsive Content Generation
• Transitioning from procedural generation to live, adaptive worlds
• Generating environments, narratives, and interactions conditioned on player state
• Balancing designer intent with emergent behavior
Controllability and Constraints
• Conditioning, fine-tuning, and latent-space steering
• Hybrid symbolic–neural approaches
• Ensuring consistency with game logic and physics
Integration with Game Engines
• Embedding generative models into real-time loops
• Managing latency, memory, and determinism
• Tooling for designers to guide and debug generative systems
This theme emphasizes that the core challenge is direction rather
than generation.
Theme2: Game Engine Education in the Context of Generative AI
Generative AI is reshaping how developers learn, design, and build games. As these tools become embedded in engines, educational practices must evolve.
Key Questions.
• What does it mean to “learn a game engine” when content is partially generated?
• How should curricula incorporate generative workflows and model control?
• What foundational knowledge remains essential?
Shifting Skill Boundaries
• Designers as co-creators with AI systems
• Engineers orchestrating pipelines rather than building from scratch
• Artists working alongside generative tools
Pedagogical Approaches
• Integrating generative AI into game development curricula
• Studio-based learning with hybrid AI + engine workflows
• Teaching evaluation, controllability, and failure modes
Tooling and Interfaces
• Prompt-based vs. visual scripting paradigms
• Debugging generative systems within engines
• Human-in-the-loop design workflows
Long-Term Implications
• Risks of over-reliance on opaque systems
• Maintaining core competencies in graphics and simulation
• Preparing students for rapidly evolving ecosystems
Ethical Considerations
CTRL-G integrates ethical analysis directly into technical discussions. Key issues include:
• Authorship and Ownership in AI-assisted creation
• Labor Transformation across game development and VFX
• Player Manipulation in adaptive, personalized systems
• Bias and Representation in generative outputs
Expected Outcomes
Participants will gain:
• A synthesis of state-of-the-art controllable generative techniques
• Insights into real-time integration challenges
• Strategies for evolving curricula and training pipelines
• Acritical understanding of ethical implications
Target Audience
• Game developers and technical artists
• AI and simulation researchers
• Graphics and VFX professionals
• Educators in game design, AI, and computer graphics
Acknowledgments
We acknowledge the help of generative AI in preparing this workshop proposal
Organizers
Josef Spjut, Research Scientist, NVIDIA and Adjunct Professor at Duke University
Arnav Jhala, Associate Professor, North Carolina State University and Adjunct Associate Professor, Duke University