Creating a tokenization algorithm based on the knowledge base for the Uzbek language*

Abstract

Currently, the correct selection of tokens from incoming information is one of the important issues in such areas as machine translation, information retrieval, information extraction from text, and information security. Algorithms for extracting tokens from texts are called tokenization. In this study, a tokenization algorithm has been developed that works on the basis of a knowledge base to extract lexemes from a text.