Price: 2500
Number of applications: 1
05.01.26 (inclusive)
Discussed individually
MVP
ICT tasks
Robotics
Intelligent control systems
Software/ IS
In existing text document analysis systems, the main bottleneck is the stage of reading and preprocessing data: • documents are fully loaded into memory before indexing begins; • Reading and indexing are performed sequentially; • the processing speed directly depends on the file size; • massive checks increase the load on the file system and database; • Traditional approaches do not take into account the features of I/O subsystems and multithreading. As a result, even with the availability of computing resources, the system loses performance, and the user's waiting time for the result increases.
The implementation of the project will allow: • reduce the time needed to read and pre-process documents by 40-60%; • Ensure that indexing starts before the file is fully uploaded.; • increase the overall throughput of the system during mass inspections; • reduce the load on the disk subsystem and RAM; • provide linear scalability of indexing with increasing data volume; • create a universal algorithm applicable in educational, scientific and government systems.
Asylbekov Ulan
Purpose and description of task (project)
The aim of the project is to develop and implement an algorithm for accelerated reading of text data and parallel streaming indexing, which significantly reduces document processing time during mass inspections and an increase in the volume of stored data. The project aims to optimize the following steps: • reading files of various formats (DOCX, PDF, TXT, RTF); • converting documents into a unified text representation; • parallel indexing of text blocks without waiting for the document to be fully loaded; • Minimizing disk I/O operations when working with large data bodies. The algorithm involves the use of chunk-based processing, asynchronous read streams, and adaptive allocation of indexing tasks.
Whatsapp
