Generative AI for Creating Multilingual Taxonomies

Taxonomies enhance the capability to identify and retrieve desired content.  Automated translation (machine translation) expands the scope of access to information to content in other languages. Applying automated translation to the contented identified using taxonomies thus enables people to find content on the subjects they want and then have it translated if needed. To retrieve such multilingual content, however, a taxonomy must also be multilingual. Concepts in the taxonomy must have names/labels in different languages: the language of the user and the languages of the content so that it can be tagged to the content.

Creating multilingual taxonomies

Fortunately, the data model upon which most large taxonomies are being built and managed supports multilingual concepts. The data model standard SKOS (Simple Knowledge Organization System) supports bilingual and multilingual taxonomies, because it models “concepts,” not “terms,” which has been referred to as “things, not strings,” and concepts can have any number of labels (strings of text) to describe them: a preferred label in each language that is displayed and any number of alternative (variant) labels in any languages, which are not displayed but match user searches and text strings for tagging.

The fact that multilingual taxonomies are simple to support technologically does not mean that they have been easy to create with translations. Whether translation is done by human translators or by machine translation software, translating a taxonomy is not as easy as translating narrative text. Both human translators and automated translation tools, (using rule-based, statistical, or neural network AI methods) look at complete sentences and not just isolated words. Words may have different meanings depending on their use in a sentence, and the context of a sentence may call for a different synonym in the translation target language.

Taxonomies with their hierarchical relationships provide context for the meaning of their concepts, but this is not the same kind of sentence-based context that machine translation programs utilize. Human translators can look at the context of the taxonomy hierarchy, but it’s a much slower task than translating sentences (which can also use machine-assisted translation tools) and requires an understanding of taxonomies.  Relying on machine translation or human translators who don’t understand the purpose or subject area of the taxonomy can lead to errors in translating taxonomies.

LLMs for generating multilingual taxonomies

Now, AI, in the form of LLMs, has become available to help generate taxonomies (which still require human review). LLMs don’t just generate terms, but they generate the correct hierarchical relationships to other concepts, multiple labels (as synonyms) for the same concept, and even definitions for concepts.

LLMs go beyond traditional machine translation and take the existing hierarchy into account when generating a translated label for a taxonomy concept. LLMs also follow specific instructions (or “prompts”) regarding the purpose and nature of the taxonomy. LLMs can also be instructed to focus on the meaning of the concept rather than on a literal translation of the preferred labels and each of the alternative labels. While the preferred labels for a concept in different languages are close translations, the alternative labels are not translations of each other but rather refer back to the concept. Even the number of alternative labels for a concept will vary by language. Finally, definitions are generated based on the concept and not as translations of existing definitions.

I explained this in a little more detail in my prior blog post Generative AI and Taxonomies for Finding Information

LLMs for generating new taxonomies in different languages

LLMs can be trained on, access, and generate text in different languages. This means that LLMs can generate entire taxonomies or parts of taxonomies in different languages from the start (with appropriate human-created prompts), without having to translate an entire existing taxonomy into another language.

Although a monolingual taxonomy does not have the same benefits of retrieving content in multiple languages as does a multilingual taxonomy, sometimes a monolingual taxonomy is desired. If a taxonomist is not available to create an entire taxonomy in the language, and a taxonomy on the subject already exists in another language, then translating a taxonomy has been a method to create a monolingual taxonomy in another language. As previously explained, translation of a taxonomy is not ideal and is prone to errors if not done by a translator-taxonomist. Using LLMs with the involvement of a subject matter expert (not necessarily a taxonomist) will generate a better taxonomy than a translation.

When using AI to build a taxonomy, it’s still best to have the top level developed manually to serve the specific use case, to create certain branches manually that are specific to an organization, and to generate incrementally those parts of the taxonomy built with AI, with the subject matter expert approving/disapproving suggestions for concepts and alternative labels at various stages. What is significant is that a person who is a combined taxonomist/linguist/subject matter expert is not needed to create a taxonomy in each language. In this way, taxonomy creation becomes easier to do globally. 

LLMs and multilingual generation in taxonomy management software

Managing a taxonomy in taxonomy management software has many benefits, especially when managing multilingual concepts. Now taxonomy management software is beginning to include support for LLMs to generate parts or all of a taxonomy, and thus support is integrated into the taxonomy creation and editing workflow. One tool, Graph Modeling from Graphwise, has now (last month) added the feature taxonomy generation in different languages to its Taxonomy Builder generative AI component. You can generate a monolingual taxonomy in any configured project language, and you can generate additional language versions of a taxonomy (creating a multilingual taxonomy) by generating concept preferred labels, alternative labels, and definitions, through LLM generations, not as direct translations. This essentially eliminates the need to translate taxonomies.