All countries

Price: 0

Number of applications: 10

Reducing the delay in generating the first response of the language model
Customer

ТОО "Эвотек Центральная Азия"
Decision acceptance deadline

05.08.26 (inclusive)

Form of award

contractual

Product status

MVP

Task type

ICT tasks

Сфера применения

Robotics

Область задачи

Neurotechnology and artificial Intelligence

Type of product

Software/ IS,

Configuration and settings of the language model

Problem description

When using LLM language models in customer service dialog scenarios in a telephone channel, there is a high delay in generating the first response in each specific user conversation session. Given the criticality to delays in real-time voice interaction, this has a negative impact on the overall perception of the person (the interlocutor) of the dialogue with the AI assistant. The delay is related to the formation of the context and caching of the request in the language model. It is possible to insert "stubs" in the time intervals while the LLM response is being generated, but this is not a technological solution to the problem.

Expected effect

Configuration and settings of the language model to reduce the delay in generating the first response.

Full name of responsible person

Torchik V.V.

Purpose and description of task (project)

Search for ways to reduce delays at the first step of the dialogue when using language models. Language models: ChatGPT cloud services, QWEN

Note

Send the solution