In Brief
This article presents a pressing issue: language models There's an observable increase in parameters and model complexity, but the vocabulary size remains unchanged.
Researchers began training a novel model by incorporating 1 million vocabulary tokens in an unconventional manner.
The researchers were eager to explore the enhancements they could achieve with such a dramatic rise in the number of tokens.
The issue raised by the article The piece named “XLM-V: Addressing Vocabulary Constraints in Multilingual Masked Language Models” uncovers the problem that as the parameters and depth of language models expand, their vocabulary sizes do not grow alongside. For example, the mT5 model boasts 13 billion parameters but is limited to a vocabulary of just 250,000 words, which caters to over a hundred languages, resulting in roughly 2,500 unique tokens per language—a rather inadequate figure.
What steps do the writers take? They initiate the training of a new model utilizing 1 million tokens from the vocabulary in an innovative fashion. Building on the previous XLM-R version, this evolution will culminate in XLM-V. The authors were driven to discover the extent of improvement possible through such a substantial token increase.
| Related article: Predictions indicate that costs associated with training AI models could escalate from $100 million to $500 million by 2030. |
What innovations does XLM-V introduce that XLM-R lacked?
The Improving Multilingual Models The Language-Clustered Vocabularies technique involves creating lexical representation vectors for each language. For every language in the pool, a binary vector is created where each element corresponds to a specific word within that language’s domain. A ‘1’ indicates the presence of the word in that language’s dictionary (refer to the attached graphic for visualization). By employing the negative logarithmic probability of each lexeme’s occurrence, the authors enhance referencing methods.
- Subsequently, these vectors are aggregated. Additionally, a sentencepiece model is trained within each cluster to prevent vocabulary overlap among lexically distinct languages.
- The ALP evaluates how well a dictionary can encapsulate a given language.
- Creating dictionaries follows a step-by-step algorithm which initiates with a large dictionary size, gradually reducing it until the token count falls below a predetermined threshold. ULM Highlights of the Top 120+ AI-Generated Content of 2023: Including images, music, and videos.
Read more about AI:
Disclaimer
In line with the Trust Project guidelines Damir serves as the leader of the team, acting as both product manager and editor at Metaverse Post. He specializes in areas such as AI/ML, AGI, LLMs, and Web3, attracting a substantial readership of over a million monthly visitors. Boasting a decade of expertise in SEO and digital marketing, Damir is frequently featured in reputable publications including Mashable, Wired, Cointelegraph, The New Yorker, Inside.com, Entrepreneur, and BeInCrypto. He traverses the UAE, Turkey, Russia, and the CIS as a digital nomad. With a degree in physics, Damir applies his analytical skills to navigate the rapidly evolving online landscape successfully.
