Development of a system for identifying text borrowings based on combined methods and models for searching for incomplete duplicates, taking into account the Kazakh language

Task code, stage

Name of the works under the Agreement and the main stages of its implementation

Result

1

Analysis of existing scientific developments and application software that allows finding near-duplicates in text electronic documents in three languages ​​(Kazakh, English and Russian)

A review of popular software libraries and platforms for searching for partial duplicates in texts was conducted, including the following tools:

• Apache Lucene / Elasticsearch – a full-text search system with similarity search capabilities (fuzzy search).
• FuzzyWuzzy – a Python library for string similarity matching using the Levenshtein library.
• SimString – a library for quickly searching for strings similar to a given string.
• TextRazor – an API for text analysis and finding similar fragments with support for many languages.

As part of the analysis, a detailed study of these software solutions aimed at finding partial duplicates in texts was conducted, with a special focus on their application to processing the Kazakh language. Each of them offers different approaches to comparing and analyzing text data, but support for the Kazakh language, especially with its complex morphology and syntax, varies greatly between the tools. The analysis showed that additional settings are required to work correctly with the Kazakh language, especially in systems that support only Latin languages ​​by default.

Testing on Kazakh texts revealed that the most successful solution for searching for partial duplicates in this language is Elasticsearch with additional settings and the implementation of a morphological analyzer for the Kazakh language. FuzzyWuzzy turned out to be less effective in processing texts with non-Latin characters, and TextRazor does not support the Kazakh language at all to the required extent, which makes it unsuitable for use in this project.

As a result, in order to successfully search for partial duplicates in texts in the Kazakh language, it is planned to study Elasticsearch with settings for processing morphology and tokenization of Kazakh words in the future. This software solution provides the necessary accuracy and flexibility for working with large volumes of data and the integration of specialized algorithms and tools for working with the Kazakh language.

An analysis of monitoring and verification technologies was conducted using standard search algorithms that provide a fairly fast and effective search for complete or partial duplicates, and also guarantee correct text processing. It was found that such systems do not allow checking images and mathematical formulas. In this case, problems arise that cannot be solved using standard procedures. Additional special difficulties arise in the process of analyzing mathematical formulas in compared texts in order to identify similarities in them. In the text, formulas can be given as a picture or as a graphic object created using one of the formula editors. A formula editor is a computer program designed to create and edit mathematical and other formulas. Formula editors are based on the following technologies:

• use of a special markup language, such as TeX, MathML in the LaTex editor, Math for the OpenOffice editor;
• creation of formulas using a graphical interface: KFormula, MathType, MathCastmula, WIRIS Editor, MathCast;
• built-in components: Math Expression Editor Light;
• symbolic calculations: Mathematica.

Comparison of formulas using templates. Searching by formulas is much more complicated than searching by text (for example, formulas x2 and a2 are identical by template, but differ by variable names). Comparison of formulas using converters.

It has been established that a promising direction for automated analysis of mathematical formulas is the creation of converters of formulas from different formats (TeX, Equation, MathType) into the canonical XML/MathML format

2

Analysis of methods for concealing borrowings in documents that allow changing the structure of the document content without changing its content. Identify structural changes that content components may be subject to

3

Models and methods for identifying near-duplicates in the content of electronic documents in the text part of documents and images, on the basis of which it is possible to develop an information system for detecting borrowings. The developed methods should ensure the detection of near-duplicates with significant modifications of documents

1. Classification of models and methods for detecting partial duplicates

1.1. N-gram-based methods.

1.2. Models based on vector representation of text.

1.3. Shingling (splitting text into substrings of fixed length).

1.4. Deep learning-based methods.

Among the methods considered, methods using N-grams, shingles, and deep learning models that provide a high level of accuracy when comparing documents are highlighted as promising for further development.

2. An analysis is conducted and a vision for creating a hybrid method for detecting partial duplicates in tables is presented. It is assumed that the method will allow identifying similarities in text and numeric data of tables separately, and then generalize the results obtained. For text data, sequences of words in canonical form are formed, from which bit sequences are constructed based on the locally sensitive hashing method. In this case, the similarity is calculated based on the Hamming distance with a given threshold value. Identification of similarity between numerical data of tables is implemented based on the method of nearest neighbors with specified metric distances. The method allows identifying incomplete duplicates present in the input table data in comparison with a set of tables selected from scientific publications and diploma and dissertation works.