KnowSim is a user simulator that tracks what the user knows and how their understanding evolves turn by turn. Unlike existing simulators, it maintains a knowledge graph of concepts (Information Units), enabling mechanistic metrics—Knowledge Gain, Delivery Calibration, Cognitive Overload—that explain why the same response helps one user but overwhelms another. Validated against 705 human–AI sessions, KnowSim's rankings align with human judgments at 73–74% sign agreement.
The same response can be perfect for one user and overwhelming for another. Human evaluation captures this—but doesn't scale. How do we automatically evaluate whether an assistant calibrates information to each user's knowledge level?
A three-stage pipeline grounded in learning theory for interactive, multi-turn evaluation.
From a question and reference answer, an LLM extracts a directed acyclic graph of Information Units (IUs)—self-contained concepts connected by prerequisite edges. The same graph is initialized with different mastery levels per user type.
Each turn follows four steps:
Conditioned on current knowledge state.
The model under evaluation (external).
Per-IU teaching quality from assistant and engagement from user.
Embeddings well explained: struggling → partial
Sim. search well explained: prerequisite not mastered → blocked
5 IUs in one turn → cognitive load
Computed directly from the knowledge state trajectory—mechanistic, interpretable, and human-aligned.
Total ordinal state advancement summed over all IUs.
Harmonic mean of precision over the IUs the assistant explained and recall over the IUs the user could learn, penalizing redundant, prerequisite-inappropriate, and ultimately unabsorbed delivery.
Average per-turn information load relative to user capacity. Lower is better.
Holistic quality scored by an LLM judge, shared across all simulators.
Pick a question, then switch between user knowledge levels to see how KnowSim adapts the simulated conversation. Compare side-by-side with baseline simulators.
How the simulated user's understanding evolves turn by turn. Each row is an Information Unit; colors show mastery level.
Across all examples, KnowSim is the only simulator that produces actual mathematical errors at the novice level—baseline simulators say "I'm confused" but compute flawlessly. KnowSim novices also engage for 2–3× more turns, while advanced users terminate early after asking extension questions—a pattern consistent with real learners.
KnowSim's rankings align with human preferences at 73–74% sign agreement across two task domains, outperforming three baseline simulators.
Applied to 9 LLMs, KnowSim reveals that the best model shifts by user knowledge level—aptitude–treatment interactions invisible to aggregate leaderboards.
DeepSeek V4 maximizes knowledge gain (KG=15.5) but at high cognitive overload (CO=0.945). Gemini 3.1 Pro ties for best R̅ (3.25) with the highest IQ (8.39) and lowest CO.
Claude Opus 4.7 leads overall (R̅=2.50) with the highest IQ (8.39) and ties for best delivery calibration (DC=0.128).
Gemini 3.1 Pro achieves the best overall rank (R̅=1.75) with the highest KG and DC, best serving knowledgeable users.
@article{lee2026knowsim,
title={KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn},
author={Lee, Yoonjoo and Jin, Hyoungwook and Kim, Tae Soo and Zhang, Shaoyang and Laban, Philippe and Liao, Q. Vera},
year={2026}
}