Price: 0
Number of applications: 2
31.12.25 (inclusive)
tenge
Idea
ICT tasks
Media sphere
Other technological solutions
Software/ IS
Currently, data is collected from web sources using disparate tools and scripts that: - they scale poorly with an increasing number of sources and volume of data; - unstable when working with dynamic websites (JavaScript, authorization, interactive elements); - they do not provide unified control of task completion, repetition, prioritization, and error handling; - they mix technical data (logs, statuses, errors) and meaningful data, which complicates quality analysis and reproducibility; - they do not have centralized management of data parsing and cleaning schemes; - lead to the loss of intermediate or "rejected" data that could be useful for reanalysis or debugging; - create a high load on the final databases when writing data directly without intermediate layers. As a result, operational risks increase, data reliability decreases, system maintenance becomes more complicated, and the introduction of new sources slows down.
As a result of the implementation of the system, it is expected: - Stable and reproducible data collection from a wide range of sources, including dynamic and secure sites. - Clear separation of responsibility and data; - Centralized management of tasks and schemas via ControlPanel with support for versioning and re-execution. - Increased reliability and fault tolerance through the use of RabbitMQ, retry mechanisms and DLQ. - Saving all intermediate and discarded data to trash_swamp for later analysis, auditing and debugging. - Acceleration of the connection of new sources by generating and versioning parsing schemes, including using a neural network module. - Increased operational transparency: metrics, logs, error statistics, and data quality are available for monitoring and analysis.
Drugakov, Alexander Anatolyevich
Purpose and description of task (project)
A system for collecting data from web sources with imitation of user actions in the browser, task management via RabbitMQ, accumulation of technical and meaningful data in Data Lake (Apache Hudi and Delta Lake), as well as subsequent data cleaning and writing to PostgreSQL Targets: • Stable data collection from dynamic sites (JS, authorization, clicks, scrolling). • Separation of technical and substantive data for quality control and subsequent analytics. • A unified task system with prioritization, repetition, DLQ, and performance monitoring. • Manage parsing schemes centrally via the ControlPanel module. • Flow: collection -> raw data -> cleanup -> normalized PostgreSQL entry. • Archiving of everything superfluous/discarded in S3/MinIO (trash_swamp) for rare in-depth analyses.
Whatsapp
