miHoYo's AI model "Gamer" plays Honkai: Star Rail on its own from screen pixels, and the team says it doesn't endorse cheating

miHoYo's Honkai AI R&D Team, with researchers from NUS, Tsinghua and others, unveiled Gamer, an AI model that outputs keyboard and mouse input from the game screen alone, shown progressing Honkai: Star Rail's story and solving puzzles. The team says it doesn't endorse cheating.

miHoYo's AI model "Gamer" plays Honkai: Star Rail on its own from screen pixels, and the team says it doesn't endorse cheating
Cover of the Gamer project page, listing miHoYo, the National University of Singapore, Tsinghua University, the University of Chinese Academy of Sciences and the University of Science and Technology of China, with footage from games the model was tested on. Image: Gamer project page

miHoYo's Honkai AI R&D Team, together with researchers from the National University of Singapore, Tsinghua University, the University of Chinese Academy of Sciences and the University of Science and Technology of China, has unveiled an AI model called Gamer. Instead of reading internal game data, it watches the screen like a human and outputs keyboard and mouse input directly. The project page shows gameplay of it progressing the main story, following NPCs and solving puzzles in Honkai: Star Rail.

How Gamer plays

According to the project page and the arXiv technical report, Gamer takes in five game frames per second and outputs a mouse movement every 200 milliseconds plus six 33-millisecond key states, an action rate of 30 per second. Training has three stages:

  • Continued pre-training: over 10,000 hours of gameplay with aligned frames and keyboard/mouse input, to learn controls that generalize across games.
  • Instruction tuning: teaching the model to follow task instructions and "think" only at key moments like changing route or target, using memory summaries to track completed goals and failed attempts.
  • Reinforcement learning: online practice in game environments, with environment-verified rewards to improve long tasks.

For real-time response, the team also built an inference engine, Streaming-vLLM. On an NVIDIA H200, per-request latency dropped from 1,306 ms to 65.12 ms, about 20 times faster.

Gamer moving through a train scene in Honkai: Star Rail toward a quest objective on its own, with a test build label in the lower left.
Gamer moving through a train scene in Honkai: Star Rail toward a quest objective on its own, with a test build label in the lower left. Image: Gamer project page

What it did in Star Rail

The project page shows Gamer navigating, interacting with scenes and completing quests in Honkai: Star Rail, and taking on five types of in-game puzzles. In another test covering seven puzzle types and 40 tasks the model hadn't seen, the reinforcement-learned Gamer-RL had an average success rate of 49.7%, 10.5 points higher than the instruction-tuned-only version (39.2%). The page also shows tests in Genshin Impact, Zenless Zone Zero, Minecraft and other games.

Could it become a cheat tool?

The most obvious question for this kind of technology is whether it could be used for botting or cheating. The team states on the project page that Gamer is academic research into general agents for interactive environments, with possible uses in game testing, accessibility and human-computer interaction research, and that it doesn't endorse using Gamer for cheating, unauthorized access, exploiting security flaws or anything that harms players or fairness.

On data, the team released model files for its pre-training scale study, with code on GitHub. For data security reasons, though, it isn't releasing any Honkai: Star Rail data, nor models trained or fine-tuned on it.

For a game company researching "AI that plays games", the most direct use is testing and QA. The team hasn't said whether the technology will appear in miHoYo's development pipeline.

Sources