Initial Model and Data Planning

Motivation

The DzoSEM project began as an attempt to design a custom semantic embedding model for Dzongkha to power cross-lingual search with English queries.

The first plan intended to pretrain a Dzongkha representation as a multi-stage modeling problem. It proposed a SentencePiece tokenizer, masked-language-model pretraining with approximately 20 percent masking, contrastive-learning stages for several categories of representation failure, and a multiple-negative ranking loss for retrieval .

Synthetic Data Exploration

The central obstacle with training a model from scratch quickly became data scale. A useful cross-lingual embedding model would need broad exposure to world knowledge and to corresponding Dzongkha expressions.

Because the available corpus was expected to be too small, synthetic data generation was explored. Claude, Gemini, ChatGPT, NVIDIA-hosted models, Kimi, DeepSeek, and other candidates were tested. Several open or smaller systems produced Tibetan instead of Dzongkha, which made them unsuitable for the intended corpus.

Two early principles were established: training examples needed to be long passages rather than isolated sentences, and the desired data scale was extremely large, initially imagined in the range of billions of tokens.

Manual Generation Limits

A manual generation workflow was then tested. The proposed workflow generated four blocks of roughly 2,500 tokens and combined them into one bilingual “page pair” of about 10,000 tokens. A target of 100,000 page pairs would have yielded approximately one billion tokens. Although this clarified the desired dataset format—English and Dzongkha columns, broad topic diversity, varied writing styles, and avoidance of duplicate topics—it also showed that manual generation was impractical.

Corpora, Translation Models, and Existing Resources

Translation System Survey

The project then shifted from abstract pipeline design to a search for usable Dzongkha data and translation infrastructure. A fine-tuned NLLB-based Dzongkha–English model from kinleyrabgay was located, and several translation systems were compared, including Google Translate, GovTech Bhutan’s Dzongkha Machine Translation System , NLLB variants, TranslateGemma, IBM models, and other multilingual systems .

Google Translate was judged too expensive at scale. Some NLLB outputs preserved meaning but sounded unnatural, and long paragraphs often caused structural collapse. TranslateGemma appeared to list Dzongkha but failed practical tests. Many general systems either failed outright or returned Tibetan.

These tests led to a more pragmatic view of translation quality. Exact grammar mattered less for embedding pretraining than broad topic coverage and semantic alignment, but severe structural failure remained unacceptable. A masked-diffusion-language-model translation approach was also considered, but it was set aside because it would require more compute, more data, and experimental infrastructure.

Synthetic quality evaluation remained a concern. Some generated Dzongkha articles appeared strong, yet their assessment often depended on Google Translate or Gemini, and those systems did not always agree with one another. This made independent evaluation necessary before synthetic data could be trusted.

Available Corpus Search

A broad search of GitHub, Hugging Face, and related repositories followed. The search found Hugging Face FineTranslations material, MADLAD-400 Dzongkha data, BPEmb and FastText-related Dzongkha vectors, Dzongkha Linux translation files, translation-model repositories, news collections, NTREX document-retrieval resources, and bordIRlines .

The strongest estimate was roughly 19,000 long parallel items from one source plus about 20,000 from MADLAD-related data, or approximately 39,000–40,000 candidate pairs in total.

This was useful but still too small and too narrow to train a strong general-purpose contextual embedding model from scratch.

Embedding Model Benchmarks

Existing embedding models were then tested or reviewed on the Sowa Rigpa traditional-medicine corpus, specifically the Materia Medica on High Altitude Medicinal Plants in Bhutan published by the Faculty of Traditional Medicine with support from the Bhutan Foundation . The candidates included LaBSE, EmbeddingGemma 300M, Qwen3 Embedding 4B and 8B, F2LLM 4B and 8B, and Jina Embeddings v5 variants .

General text performed better than the Materia Medica book . The plant text was difficult because it contained unusual terminology, philosophical or indirect phrasing, multiple expressions for the same concept, and dense co-occurrence of related medical terms.

SPLADE, custom SPLADE distillation, improved chunking, and tokenizer work were considered. More aggressive chunking was rejected because each plant section was already a logical unit, while SPLADE raised cross-lingual and tokenizer-dependence concerns .

Cross-Lingual Bridge Attempts

The project also investigated possible bridges through bordIRlines, NTREX, multilingual document-retrieval evaluations, and three-language translation paths. Translation through an intermediate language was rejected as likely to compound errors. ImageBind-style emergent alignment was discussed as an analogy, but the conclusion was that chained translation manifolds would accumulate multiplicative error rather than solve the underlying Dzongkha representation problem .

Outreach and Corpus Prospects

An outreach effort was made to a GovTech-associated researcher . The request asked about downloadable GovTech translation-model weights, possible corpus access, and collaboration around a reported 700,000-aligned-item corpus. The message explained that existing embedding architectures had been stress-tested without success and linked the request to DzoSEM and prior contact with the Bhutan Foundation. The researcher claimed to not have access.

Diagnosing the Retrieval Problem

Single-Vector Bottleneck

The project reached its clearest technical diagnosis in the retrieval phase. The first limitation was architectural: each long plant passage was being reduced to a single vector. That vector necessarily averaged many concepts, so no embedding model could perfectly preserve every individual detail from a dense passage using one fixed-size representation.

Distributional Limits

A deeper corpus-distribution explanation followed. DzoSEM, as originally conceived, appeared blocked because SONAR and NLLB already provided strong cross-lingual representations, and a newly trained model would not automatically outperform them .

The medical-book failures were partly artifacts of single-vector reduction, and translating to English before embedding was often a simpler practical solution.

The strongest diagnosis was that SONAR and Jina could already separate broad topics, but the medical corpus failed because the passages were densely packed with related terminology. In a low-resource setting, concepts such as pregnancy and herbs repeatedly appeared together, so distributional learning placed them too close in semantic space.

English models avoid this problem because they have seen enormous and diverse corpora in which such concepts also occur independently. Adding unrelated documents would help separate broad genres, but it would not teach fine-grained distinctions inside a tightly clustered medical domain.

The real solution to train a standard embedding model from scratch would require a Dzongkha equivalent of a large educational corpus such as Cosmopedia, not merely a small collection of translated documents or movie transcripts .

Film and Documentary Corpus Idea

A movie and documentary corpus was considered as a possible data source. The idea was to extract text from approximately 30–35 hours of Bhutanese films and documentaries using Dzongkha ASR and, where available, English subtitles.

The idea faced several problems: most films lacked English subtitles, copyright and licensing were uncertain, ASR alone would not provide English alignment, and the amount of text would still be too small and insufficiently diverse for a general embedding model.

Repositioning DzoSEM Around Existing Models

Application-Centered Framing

DzoSEM was repositioned from a project that would necessarily train a new base model into an application built around existing open-source semantic models. It was confirmed that the earlier SONAR results had been obtained directly through NLLB/SONAR rather than through a fine-tuned DzoSEM model. The product framing changed accordingly: DzoSEM could become a public Dzongkha semantic-search application, even if the underlying representation came from existing models.

SONAR, Jina, and Deployment Direction

SONAR and Jina were compared. SONAR was preferred because it was open source, carried fewer restrictions, and better supported the framing of a Dzongkha semantic embedding application. The planned product direction included tests across multiple corpora, a search interface, possible ClickHouse vector search, and deployment infrastructure if the system were ever used nationally or by a partner organization .

Local Model and Database Setup

The next phase focused on local models, databases, and retrieval implementation. One idea was to use Tibetan semantic-search models for Sowa Rigpa texts and other Tibetan-based Dzongkha materials. Local model-loading scripts were tested in Colab, but memory limitations appeared quickly. LM Studio, MongoDB Compass, and Python were used to test the quantized Gemma Mitra model but was only partially successful and too slow for use.

OCR as Document Ingestion

OCR also became part of the application story. Mistral OCR 4 was stress-tested on standard Dzongkha documents, crowded pages, cropped images, blurry images, fragmented print, Tibetan-looking samples, and multi-page PDF collections. Clear document pages worked well, while blurry images, crowded layouts, and non-document images were weaker. Crowded layouts often required cropping .

Local Implementation and Retrieval Pipeline

Translation-Based Retrieval

A retrieval pipeline using LLM-based translation from Dzongkha to English and English embedding models for search was tested. Successful English queries involved common terms such as lung disease, fever, and inflammation using the model paraphrase-multilingual-mpnet-base-v2 on the Materia Medica book that had previously failed .

Integration Architecture

The OCR, translation, and embedding stage was simplified into a practical architecture for the document-ingestion step. This was planned to be implemented as a n8n workflow connecting the OCR, LLM, Embedding, and Database stages .