Shared architecture of user-facing and programmatic applications of language models

Background. Over the past few years, large language models (LLMs) have evolved rapidly from narrowly specialized dialogue systems into multicomponent computing systems. Developers increasingly need to build hybrid solutions that can hold a dialogue with a human and, at the same time, act as autonomous agents that process structured data through APIs.

Yet, despite the obvious differences in user interfaces and use cases, the underlying architecture of these systems remains poorly documented in the academic literature. As a result, engineering effort is duplicated and computing resources are allocated suboptimally. Understanding the architectural patterns these systems have in common, such as context management mechanisms, sparse-activation strategies, and inference optimization approaches, is essential for building general-purpose frameworks and reducing the total cost of ownership of models.

Objective. This study aims to comprehensively analyze and systematize the architectural choices used in building both user-facing and programmatic applications of language models and to justify general principles for the unified design of such systems.

Materials and methods. The study draws on current open LLM architectures, including dense and sparse transformer networks, as well as on the technical documentation and source code of leading agent frameworks, orchestration systems, and tools for implementing retrieval-augmented generation (RAG; Gao et al., 2023; Naumenko, 2025).

Published empirical findings were also reviewed, together with approaches to evaluating inference performance and model quality in conversational and programmatic scenarios. The study relies on comparative analysis, generalization, and systematization.

Results. Language models trace their origins to classical natural language processing, which relied on statistical methods and rigid linguistic rules. A genuine architectural shift came with the move to recurrent neural networks and long short-term memory (LSTM) networks. LSTMs were designed to mitigate the vanishing gradient problem typical of simple recurrent networks, which made it possible to tackle sequence modeling tasks.

The turning point was the Transformer architecture, introduced in 2017 and built on the self-attention mechanism. Self-attention solved the problem of parallelizing computation and provided a general mathematical framework that later proved equally effective for generating coherent text in dialogue systems and for semantic data analysis in programmatic pipelines (Vaswani et al., 2017).

In the early stage of LLM development, approaches diverged. Encoder-only and encoder–decoder architectures were used mainly for programmatic tasks such as classification, entity extraction, and semantic search, whereas autoregressive decoder-only models focused on generative tasks and became the basis for the first user-facing chatbots (Figure 1).

 

Before 2010

Statistical methods

n-grams, HMMs, rule-based systems

2010–2016

Neural network models

word2vec, GloVe, RNN, LSTM

2017–2019

Transformer architecture

self-attention mechanism

2020–2021

Model scaling

GPT-3, scaling laws

2022–2023

Model alignment

InstructGPT, RLHF, DPO

Since 2023

Agentic systems

RAG, function calling, MoE

Classical approaches

Divergence

Convergence

Sequential text processing statistical and recurrent models

Encoder and encoder–decoder models (BERT, T5) programmatic tasks: classification, search

Autoregressive decoder-only models (GPT) user-facing tasks: text generation, chatbots

General-purpose models shared architecture and infrastructure for user-facing and programmatic applications

 

Figure 1 – Stages in the evolution of large language models [compiled by the author]

 

As models scaled up, it became clear that the general-purpose generative and analytical capabilities of decoder-only transformers could be adapted to almost any computational task. This marked the transition to the era of general-purpose models.

Recent research shows a deep convergence of development methods for user-facing and programmatic applications. Much of this work centers on sparse-activation architectures, in particular the Mixture of Experts (MoE) paradigm (Mu & Lin, 2025; Shazeer et al., 2017). MoE allocates computing resources dynamically, keeping inference fast in interactive user dialogues while also supporting the processing of large enterprise datasets in background programmatic workloads (Table 1).

Table 1 – Key principles of the Mixture of Experts (MoE) paradigm (Mu & Lin, 2025; Shazeer et al., 2017)

Principle	Description
Task decomposition	A complex task is divided into subtasks, each handled by specialized experts
Expert specialization	Each expert is trained to handle a particular type of task or a particular data pattern
Routing (gating)	A routing mechanism determines which expert (or combination of experts) should process each input
Sparse activation	Only a small subset of experts is activated for each input, which improves efficiency
Output combination	The outputs of the active experts are combined (usually by weighting) to produce the final result
Scalability	Allows the number of model parameters to grow without a proportional increase in computational cost
Parallelism	Experts can run in parallel, which speeds up processing
Load balancing	An important consideration is distributing the load evenly across experts so that some are not “overloaded” while others sit “idle”

 

A large body of recent work addresses the evolution of context management and context-window extension. Methods such as Rotary Position Embedding (RoPE; Su et al., 2021) encode token positions and were originally developed to improve the handling of long sequences. Today they are essential for programmatic analysis of multipage documents and complex source code. Distributed attention algorithms such as Ring Attention (Liu et al., 2023) spread attention computation across multiple devices, which is necessary for processing contexts that exceed the memory of a single GPU.

Research on retrieval-augmented generation and vector search deserves separate mention. These techniques began as programmatic mechanisms for improving answer accuracy and have become an integral part of the architecture of modern user-facing agents that can query external knowledge bases in real time (Gao et al., 2023; Lewis et al., 2020; Shmat, 2024). A key direction is the development of agent frameworks and function-calling mechanisms, in which the language model acts not merely as a text generator but as a cognitive core that makes decisions about calling external APIs.

Another active research area is the transfer of alignment methods such as reinforcement learning from human feedback (RLHF; Casper et al., 2023; Ouyang et al., 2022) and direct preference optimization (DPO; Rafailov et al., 2023). RLHF was first applied to train agents in games and simulated environments (Christiano et al., 2017) and was later adapted to reduce toxicity and improve model helpfulness in user-facing chat (Ouyang et al., 2022). Today these techniques are being successfully adapted to make outputs more deterministic and better structured and to minimize hallucinations in programmatic pipelines.

In parallel, inference optimization is being studied intensively, including weight quantization, speculative decoding, and KV cache sharing (Sun et al., 2025). These methods make it possible to build hybrid infrastructures in which the same base model serves thousands of concurrent user requests while also running complex programmatic computations (Khrapova, 2026).

Although user-facing and programmatic applications of language models share a common architecture, integrating them into unified ecosystems raises a number of technical challenges that require comprehensive solutions.

Imbalance in memory management requirements. User interfaces must retain long, unstructured dialogue sessions, whereas programmatic applications must process massive volumes of strictly structured data. To address this, systems are moving to paged KV cache memory management, as in PagedAttention (Kwon et al., 2023). Such mechanisms split the KV cache into fixed-size blocks, reduce fragmentation, and allow memory to be reused efficiently, optimizing resource consumption for both streaming generation in chat and batch processing of large datasets in backend systems.

Conflict between creativity and determinism. User-facing scenarios call for varied responses, whereas programmatic tasks require strict determinism: a model that hallucinates while calling an API or generating an SQL query causes cascading failures in the pipeline. This problem is mitigated through constrained decoding, strict conformance to grammar schemas, and agentic self-reflection loops in which the model acts as a critic, checking its own output against the tools it has been given before the output is finally executed.

Tension between latency and throughput. Interactive user-facing streaming needs ultra-low latency, while background programmatic processing of thousands of concurrent requests needs maximum throughput, and meeting both requirements remains a serious challenge. This gap is bridged by continuous (dynamic) batching, speculative decoding with draft models, and sparse-activation architectures that route tokens only to the experts relevant to a given subtask.

Alignment conflicts. Strict safety and toxicity-filtering algorithms designed to protect end users often lead to over-refusal, where the model unjustifiably declines legitimate but complex programmatic requests submitted through the API. To remove this barrier, separate alignment is used: the base model is fine-tuned with DPO (Rafailov et al., 2023) on specialized tool-calling datasets. Multilevel routing systems are also introduced, in which a lightweight classifier model identifies the context of a request and sends it either to the “safe” user-facing pathway or to the “extended” programmatic one.

Lack of unified benchmarks. A methodological problem is that there are still no benchmarks that can assess both the coherence of user dialogue and the accuracy of programmatic execution. The way forward lies in developing agent-oriented evaluation frameworks in which a model’s performance is measured by how successfully it completes multistep trajectories in simulated environments with real tools.

Conclusion. The analysis shows that dividing language models into “user-facing” and “programmatic” ones is largely an applied distinction that operates at the level of the user interface and orchestration. At the backend level, both paradigms rely on the same technology stack: transformer blocks, self-attention mechanisms, MoE routing, and optimized inference engines with dynamic batching.

Further progress in the field lies not in building isolated models for each task but in designing modular systems that keep a single infrastructure base and switch between the roles of “conversational partner” and “computational agent” by changing system prompts, routing schemes, and external tool-calling mechanisms. Such an architecture allocates computing resources dynamically: simple requests follow lightweight routes, while complex multistep tasks pass through chains of specialized experts. This balances response speed, accuracy of results, and the cost-efficiency of infrastructure use.

Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., Wang, T., Marks, S., Segerie, C.-R., Carroll, M., Peng, A., Christoffersen, P., Damani, M., Slocum, S., Anwar, U., . . . Hadfield-Menell, D. (2023). Open problems and fundamental limitations of reinforcement learning from human feedback (arXiv:2307.15217). arXiv. https://doi.org/10.48550/arXiv.2307.15217

Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D. (2017). Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 4302–4310.

Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023). Retrieval-augmented generation for large language models: A survey (arXiv:2312.10997). arXiv. https://doi.org/10.48550/arXiv.2312.10997

Khrapova, A. O. (2026). Sovremennyye podkhody k upravleniyu i obespecheniyu nadezhnosti GPU-klasternykh sistem v usloviyakh intensivnykh nagruzok na baze iskusstvennogo intellekta [Current approaches to managing and ensuring the reliability of GPU cluster systems under intensive AI workloads]. In Aktualnyye voprosy nauki 2026: Sbornik statey XII Mezhdunarodnoy nauchno-prakticheskoy konferentsii (pp. 50–53). Nauka i Prosveshcheniye.

Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (pp. 611–626). Association for Computing Machinery. https://doi.org/10.1145/3600006.3613165

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33, 9459–9474.

Liu, H., Zaharia, M., & Abbeel, P. (2023). Ring attention with blockwise transformers for near-infinite context (arXiv:2310.01889). arXiv. https://doi.org/10.48550/arXiv.2310.01889

Mu, S., & Lin, S. (2025). A comprehensive survey of mixture-of-experts: Algorithms, theory, and applications (arXiv:2503.07137). arXiv. https://doi.org/10.48550/arXiv.2503.07137

Naumenko, A. O. (2025). Tekhnologiya RAG (Retrieval-Augmented Generation) kak innovatsionnyy podkhod v LLM [RAG (retrieval-augmented generation) technology as an innovative approach in LLMs]. Vestnik nauki, 5(8), 280–289.

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730–27744.

Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 53728–53741.

Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer (arXiv:1701.06538). arXiv. https://doi.org/10.48550/arXiv.1701.06538

Shmat, A. V. (2024). Primeneniye bolshikh yazykovykh modeley i tekhnologii Retrieval-Augmented Generation dlya korporativnykh assistentov [Applying large language models and retrieval-augmented generation technology to corporate assistants]. Izvestiya Tulskogo gosudarstvennogo universiteta. Tekhnicheskiye nauki, (10), 720–726. https://doi.org/10.24412/2071-6168-2024-10-720-721

Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021). RoFormer: Enhanced transformer with rotary position embedding (arXiv:2104.09864). arXiv. https://doi.org/10.48550/arXiv.2104.09864

Sun, S., Yu, J., & Zhou, J. (2025). Optimizing attention for efficient LLM inference: A review. In 2025 8th World Conference on Computing and Communication Technologies (WCCCT) (pp. 482–491). IEEE.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998–6008.