Chinese Court Says AI Hallucination Does Not lead To Automatic Liability

A user in Hangzhou discovered their AI assistant had fabricated university details during a normal research request. When the AI assistant promised compensation for the error, the user took the step of suing the ai provider for damages. Last month, the Hangzhou Internet Court delivered China's first ruling on AI hallucination liability and sided with the developer.

The decision, analyzed by legal experts at Gowling WLG, establishes a fault-based framework that will likely influence how generative AI approaches safety compliance globally. Rather than treating AI responses as subject to strict liability, the court classified them as services where providers must demonstrate reasonable duty of care through safety filings, user warnings, and technical guardrails.

The ruling draws a clear distinction. According to the court's analysis, providers face strict liability for illegal content: hate speech, misinformation about public figures, prohibited material. But for issues of accuracy users must prove both developer negligence and tangible harm.

The company had filed required safety assessments with regulators and implemented standard guardrails. The court confirmed this met the reasonable duty of care standard, enough to shield the provider from liability despite the fabrication.

The case hinged on the AI's autonomous promise to pay damages. The court ruled that AI systems lack civil subject status under Chinese law, meaning they cannot form binding legal intent or make enforceable promises. The promise, however convincing to the user, carried no legal weight.

The Hangzhou court's approach may face broader tests as the technology diffuses into the broader economy. As the tech extends further into legal research, health, and financial analysis, the stakes of hallucinations may rise enough that real world damages may be too great to reasonably exempt providers.