09.09.25 (inclusive)

The possibility of employment in the company

Finished product

ICT tasks

Media sphere

Internet of Things

Software/ IS,

Streaming log analytics

Functional requirements Collection and normalization Receiving logs from 3 sources (HTTP Ingest + Kafka Topic + local files for backfill). Bringing timestamp to a single UTC format, correct handling of delays/late events (late events window configurable). Unification of fields (service, env, level, traceId, message, meta{}). Enrichment Connect events with the service registry (service → owner, oncall, criticality). Geolocation of IP → country/region (you can use a local geo‑db file). Incident detection There are at least two methods: (a) threshold rules (for example, ERROR rate > X% in Y minutes), (b) statistical anomaly (z‑score or EWM variance) in the number of events/latency. Generating an "incident" with attributes (id, initialtime, service, metric, evidence, status: open/resolved). Search and storage Indexing logs (full text + filters by fields). Elasticsearch/OpenSearch/SQLite FTS/Whoosh are acceptable — please explain the choice. Retention policy (hot/cold storage): hot 3 days, cold 30 days (emulate). API and UI (minimal) REST/gRPC for: (a) ingest (for test), (b) search queries, (c) incident list, (d) incident closure. The simplest search web page (search bar, service/env/level filters, pagination, viewing event details). Reliability Idempotent reception (dedup by traceId+ts or hash(message)). Peak processing/backpressure: queues/buffer, retrays with exponential delay. Non-functional requirements Performance: ≥ 10,000 events/sec on a regular machine (justify measurements and simplifications). Delay from reception to search: P95 < 2 seconds at 5k evt/s. Reliability: do not lose events when restarting a single worker (demonstration via a local persistent buffer). Observability: metrics (ingest rate, lag, error rate), health endpoint, structured logs of the service itself. Restrictions and introductory Language and stack: [REPLACE BY PHOTO] (default: Go/Java/Python + Kafka + any search engine). Deployment: Docker Compose or Kubernetes (minikube/kind). CI — [REPLACE BY PHOTO]. Without external paid services. Test data Generate a simulator of 3 services: checkout, catalog, auth (different ERROR/latency levels). Add noise and delayed events (up to 90 seconds). Attach 1-2 log files (JSON/CEF), HTTP ingest sample. What to provide (Deliverables) Architectural diagram (data flow diagram, queues, indexing, components). A repository with code, Dockerfile, docker‑compose/k8s manifests. A README with startup steps, parameters, and performance profiling. A set of tests (unit + load tests/event generation script). A short technical justification for choosing the search storage, the delay window, and deduplication. Evaluation criteria (category) Architecture and scalability — 30% The quality of the code and tests is 20% Observability and operation — 15% Correctness of incident detection — 20% Search UX and API Design — 10% Documentation and reproducibility — 5% Bonus tasks (optional) Correlation of events by traceId/spanId (simplified tracing). Incremental snapshots "hot→cold" with a cheap format (Parquet on a local/cloud bucket — emulation). Live alerts in Slack/Telegram (webhook emulator).

Architectural diagram (data flow diagram, queues, indexing, components). Repository with code, Dockerfile, docker‑compose/k8s manifests. A README with startup steps, parameters, and performance profiling. A set of tests (unit + load tests/event generation script). A short technical justification for choosing the search storage, delay window, and deduplication.

Dauren Aripov