[Bug]: OpenAIEncoding differs from tiktoken.Encoding in encode_with_unstable (lone surrogates) and is_special_token (ids outside u32)
![[Bug]: OpenAIEncoding differs from tiktoken.Encoding in encode_with_unstable (lone surrogates) and is_special_token (ids outside u32)](https://www.chat-gpts.plus/wp-content/uploads/2026/11/51601-8efde278-768x403.jpg)
该报错通常出现在把 LiteLLM 的 Rust 版 OpenAIEncoding 当作 tiktoken.Encoding 的替代品做行为对比或兼容性测试时,只有两个边界场景不一致: encode_with_unstable 遇到孤立代理字符(lone surrogate),以及 is_speci








