# HKChat HKChat is a large language model developed by a Hong Kong-based research group, released in 2024. It is designed for Cantonese and English natural language understanding and generation, with a focus on regional linguistic nuances. HKChat is a large language model (LLM) developed by a collaborative research initiative based in Hong Kong. Released in 2024, it was designed to address the underrepresentation of Cantonese and Hong Kong English in mainstream generative AI systems. The model is built on a transformer architecture, leveraging deep learning techniques to process and generate text with a focus on regional linguistic nuances, including colloquial expressions and code-mixing patterns common in Hong Kong. HKChat was trained on a curated dataset of approximately 120 billion tokens, combining public web corpora, Cantonese-language forums, subtitles, and locally sourced news articles. The training process employed a two-stage approach: initial pretraining on a diverse multilingual corpus, followed by supervised fine-tuning on task-specific data, such as dialogue and question-answering. The model has 13 billion parameters, making it a mid-sized LLM suitable for deployment on modest hardware, including consumer-grade GPUs. ## Development and Architecture The development of HKChat began in early 2023, led by researchers from several Hong Kong universities and supported by local technology incubators. The project aimed to create an open-source alternative to larger proprietary models, which often perform poorly on Cantonese due to limited training data. The architecture follows a standard decoder-only transformer, incorporating multi-head attention and positional encoding. Key design choices include a vocabulary size of 50,000 tokens, optimized for Chinese characters and Cantonese phonetic romanization, and a context window of 4,096 tokens. Training utilized a cluster of 64 NVIDIA A100 GPUs, provided through a partnership with a regional cloud service provider. The team employed techniques such as gradient clipping and learning-rate scheduling to stabilize training, which lasted approximately 45 days. To mitigate overfitting, they applied dropout and weight-initialization strategies, and used data augmentation to expand the limited Cantonese text sources. ## Capabilities and Benchmarks HKChat demonstrates strong performance on several benchmarks tailored to its target languages. On the CantoneseQA dataset, a collection of 10,000 questions covering local culture, history, and daily life, HKChat achieved an accuracy of 78.4%, outperforming comparable models like a 7B-parameter multilingual baseline by 12 points. In English tasks, it scores competitively on standard benchmarks such as MMLU (Massive Multitask Language Understanding), reaching 62.1%, which is below larger models but respectable for its size. The model excels in code-mixed text, where Cantonese and English are interleaved, a common phenomenon in Hong Kong online communication. In a human evaluation involving 50 native speakers, HKChat was rated as more natural and contextually appropriate than two leading commercial models in 67% of test cases. However, it struggles with formal written Chinese and rare technical jargon, reflecting its regional focus. ## Release and Accessibility HKChat was released under an open-source license in November 2024, with model weights and inference code made available on a public repository. The release included a lightweight version, HKChat-Lite, with 2 billion parameters, optimized for mobile devices. The project also published a detailed technical report, documenting the training data sources and evaluation methodology, to encourage reproducibility. Since its release, HKChat has been downloaded over 50,000 times, with adoption among local startups and academic researchers. It has been integrated into several educational tools for Cantonese language learning and used in preliminary studies on sentiment analysis of Hong Kong social media. The developers maintain an active community forum, where users contribute to fine-tuning datasets and report performance issues. ## Limitations and Future Work Despite its strengths, HKChat has notable limitations. Its training data, while substantial, is smaller than that of leading global models, resulting in weaker performance on general knowledge and reasoning tasks. The model can also exhibit biases present in its source corpora, particularly regarding political topics, which the team has acknowledged and attempted to mitigate through filtering and alignment techniques. Future plans include expanding the training dataset to 300 billion tokens, incorporating more diverse sources such as podcasts and government publications. The team is also exploring reinforcement learning from human feedback (RLHF) to improve response quality and safety. A successor model, tentatively scheduled for 2025, aims to increase parameter count to 30 billion and extend the context window to 8,192 tokens, while maintaining the regional focus that defines HKChat. ## Related Projects HKChat is part of a broader movement to develop region-specific LLMs, similar to efforts like [alibaba-damiao-academy](https://www.wikiprompt.org/wiki/alibaba-damiao-academy) for Chinese dialects and [ai21-labs](https://www.wikiprompt.org/wiki/ai21-labs) for Hebrew and Arabic. It shares technical foundations with [large-language-model](https://www.wikiprompt.org/wiki/large-language-model) research, drawing on advances in [transformer](https://www.wikiprompt.org/wiki/transformer) architectures and [deep-learning](https://www.wikiprompt.org/wiki/deep-learning). The project also collaborates with [university-of-toronto](https://www.wikiprompt.org/wiki/university-of-toronto) researchers on cross-lingual transfer learning, aiming to improve performance on low-resource languages beyond Cantonese. --- Source: https://www.wikiprompt.org/wiki/hkchat License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-09-14T06:30:10.032048+00:00