Back to AI
// ai-gatewaysystem overview · september 2026

How AI Gateway works

A walkthrough for a non-technical reader: what the system does, what it is made of, and where the money goes. Every section also says how close this is to talking to a real person and what is still missing, plus a collapsible «how it works technically» block: which tools are used and why.
In three sentencesIt is a platform where a company or an individual sets up their own AI-powered Telegram bot without writing a single line of code: top up the balance, give the bot a personality and rules — and it starts answering users.There are two parts inside: the gateway — the cash desk and dispatcher, it accepts requests, calls the AI provider, meters and charges the money, and shows the owner the reports; and the bot — the one that holds the conversation, remembers context, and decides what to show the person.Everything described below is running in production. Section 08 honestly lists what the system cannot do yet.
// contents
01Who is in the system
02The path of a single message
03Money
04The bot's memory
05What the bot owner does
06Data and privacy
07What happens when things break
08Limits and plans
09The stack in one list
01// roles

Who is in the system

Three roles with different interests and different access. Almost every decision in the system is explained by the fact that there are three of these roles, not one.
The platform owner — us. We keep the service running, pay the AI provider, and earn the difference between what we pay per request and what we charge the client.The bot owner (the client) — a company or a person who set up a bot with us. They top up the balance in advance, configure the bot's personality and rules, and see their spend and their users. There are many clients, and they know nothing about each other.The bot user — whoever writes to the bot in Telegram. The platform is completely invisible to them: they are simply talking to a bot.
bot ownerpays up front, configuresAI providerexternal, paidbot userssee only Telegramplatformgateway (billing) + botstops up balance →← spend reportswe pay for every replyquestion ↓ / reply ↑
Money flows one way — from the client to us and onward to the AI provider; the user pays nothing in this chain and knows nothing about it.
What this buys
  • The client onboards themselves: create a bot, top up, configure — no developer involvement needed.
  • We risk no money: the client pays up front and cannot spend more than they deposited.
  • One service serves all clients at once, yet each of them sees only their own — a new client costs us neither a separate server nor a separate setup.
Human-like
To the user the bot is like a company employee: it represents the owner, speaks in their voice and by their rules, while the «accounting» happens somewhere off screen.
What is missing
A real employee has one identity across all channels: you email them, then call them — it is the same person, and they remember both conversations. Our bot knows a person only inside one bot and one chat. The same person in a different chat is a different interlocutor to it.
02// message-path

The path of a single message

What happens between «the person hit send» and «a reply appeared in the chat» — and what happens after the reply.
From the outside it looks like an ordinary exchange of messages. Inside there are six steps, and half of them exist so that the conversation makes sense and the money is metered honestly.
1 · a person writes to the botan ordinary Telegram message2 · the bot recalls who it talks torecent messages + summary + facts3 · the gateway freezes moneyan upper estimate, like a hotel hold4 · the model writes the replytext streams in, word by word5 · the reply appears in the chatit grows as you watch, no waiting6 · after the replyexact settlement · facts saved in backgrounda human does the same:before answering, they recallwhat the conversation was aboutmoney is counted before the reply,not after — otherwise ten requestscould put you in the reda person starts readingwhile the speaker is still talkingwhat a human doesn't do consciously:they remember by themselves, hereit is a separate paid operation
Steps 2 and 6 are what happens by itself in a human: recalling and remembering. In the system these are separate actions, and each one costs money.
What this buys
  • The reply starts appearing almost immediately — nobody stares at an empty screen for several seconds and walks away.
  • The conversation does not fall apart: the bot answers with the previous messages in mind, not as if it were seeing the person for the first time.
  • Every reply is metered: the spend is tied to a specific user, bot, and model.
Human-like
An interlocutor who first recalls the context and then answers, and who speaks as they go instead of dropping a wall of text after a pause. And after the conversation it «makes a note» — like a person who writes down after a meeting that the client moved to another city.
What is missing
The bot answers only when addressed: it will not write first, will not remind, will not return to yesterday's topic. It cannot step away and find something out — no looking at a calendar, no opening a document, no phone call. All it can do is speak based on what it already knows.And it never takes a pause: for a human «let me think about it until tomorrow» is a normal move, for the bot the answer is always now and always on the first attempt.
03// money

Money

The most technical part of the system is built so that the client cannot go negative and we cannot charge more than we should.
The AI provider charges for the volume of text: how much went into the model and how much it wrote back. The exact amount is known only after the reply — while the decision to serve or not has to be made before it. Hence the reservation mechanics.How it works. Before the request the system freezes an upper estimate — the money is not charged yet, but it is already unavailable. After the reply the actual amount is computed, charged, and the difference is returned. If the request fails, the reservation is released in full and the client pays nothing at all.The second important point is what is being charged for. The reply to the user is not the only expense. There are three more internal ones: compressing a long history, remembering facts, and semantic search. All of them are labelled, and the report shows how much each kind costs.
client balancetopped up in advancereservationan upper estimatemodel replyvolume knownsettlementactual costthe difference returns at onceerror or refusal — the reservation is released in full, the client pays nothingwhat gets charged — four different reasons:reply to the uservisible in the chathistory compressionkeeps the price downremembering factsin the backgroundsemantic searchto recall what's neededthe user sees only one of the four — the owner's report splits them all
The reservation covers the gap between «we started answering» and «we learned the price»; highlighted is what eventually becomes the client's expense.
What this buys
  • The client physically cannot go negative: the ban is wired into the database itself, not only into the code.
  • We do not overcharge: the client is billed the actual cost, not an estimate with a safety margin.
  • The same request cannot be charged twice — every request carries its own unique key.
  • The owner sees the structure of the spend and can influence it: for example, pick a cheaper model for the internal summarization.
Human-like
Like working with a contractor who bills for actual work: first you agree on a ceiling, then you count the time actually spent. And, as with a human, part of the work is invisible preparation that is also paid for.
What is missing
A real contractor matches effort to the task: they will not spend an hour on a trifle and will say «this is expensive, let's do it simpler». The bot answers «hi» with exactly the same diligence as a complex analysis, and will never warn that a question is going to be expensive. Judging that is left to the owner — from the report, after the fact.
04// memory

The bot's memory

Three different memory layers with different lifetimes. They are exactly what makes it a conversation rather than a set of unrelated answers.
The model has no memory as such: everything it is supposed to know is retold to it from scratch every time. So memory is our job, and it is built in three layers.
  • The current conversation. The latest messages — what makes a dialogue a dialogue. Kept while the conversation is alive.
  • The summary. When the exchange gets long, its older part is compressed into a short digest. Otherwise every next reply would get more expensive, and eventually the conversation simply would not fit.
  • Facts about the person. The stable things: their name, where they live, what they do, how they asked to be spoken to. These survive the «start over» command and outlive the conversation itself.
The bot writes the facts down itself, in the background, after replying — not after every message, but with a delay, so as not to pay for it on every single turn.
the current conversationlatest messages · lives while you writesummary of the older partcompressed automatically · lives with the chatfacts about the personname, city, preferences · survive «start over»what the bot holdsin mind while answeringall three layers at once,on every message«start over» — the conversation is wiped, the facts remain«forget me» — everything is wiped · 30 days of silence deletes the chat
Every reply is assembled from three layers; their lifetimes differ, and the user controls them with two simple commands.
What this buys
  • A long exchange does not keep getting more expensive and does not hit a ceiling — that is the summary's job.
  • A person does not have to introduce themselves again every time: the bot remembers who they are and how to talk to them.
  • The owner does not pay for remembering after every single turn — the background pass is debounced and rate-limited.
  • Storage is time-bounded: whatever is not used is deleted by itself.
Human-like
We too remember a conversation in full only briefly, then only the gist, while about the people close to us we keep a stable set of facts in mind. The three layers follow exactly that principle, and the result is recognizable: the bot greets you like an acquaintance and does not re-ask what it has already been told.
What is missing
A human forgets by importance, the bot by timer: a month of silence and the conversation is deleted in full, even if there was something valuable in it.It cannot check its memory out loud: a real interlocutor would ask «you moved, didn't you — am I mixing it up?», while the bot silently uses what it wrote down — and if it wrote it down inaccurately, it will be confidently wrong.And its forgetting is coarse: «forget what I said about work» is beyond it — only wiping everything at once.
05// owner

What the bot owner does

A control panel where the client decides everything themselves without coming to us. That is exactly the difference between «a service» and «a development engagement».
Configuring the bot
  • Personality and rules. Ready-made «personas» (tone and manner) plus custom ones, up to five per client. The persona decides whether the bot talks like support, like a consultant, or like a friend.
  • Greeting and the top-up contact. The first thing the user sees, and where to go if the money runs out.
  • Model. Which model answers and which cheaper one does the internal compression. Changed in one click and applied without restarting the bot.
  • The memory instruction. The text by which the bot decides what is worth remembering about a person. Editable by the owner as well.
  • Behaviour in groups. React only when addressed or read the whole conversation; separately — whether to remember what is written in the group when the bot is not being addressed. The latter is off by default, because it costs money.
Observation and control
  • The log. One feed: who wrote, what kind of call it was, which model, how much it cost. With filters by type and by a specific person or group.
  • Users and groups. A list with names (not numeric identifiers), balance, history, and a «wipe context» button.
  • Blocking. A problematic user or group can be switched off. The bot then simply stays silent — it does not say «you are blocked», because arguing with a bot is pointless and irritating.
  • Balance. Top-ups are done by us; credits and charges are visible in the history.
What this buys
  • The client changes the bot's behaviour themselves and immediately — no queue of «file a ticket, we will deploy it».
  • The spend is transparent: you see not only «how much» but «on what» — on replies to people or on internal operations.
  • We spend no time servicing each client, which is why there can be many of them.
Human-like
This is onboarding an employee: you explain the tone, hand over the rules, say what is important to remember. From there they work on their own and you read the reports.
What is missing
Feedback. A real employee will say «this instruction contradicts itself» or «people keep asking me things I don't know». The bot silently does as written, and you learn about problems in the instruction only from bad answers.And it does not learn on the fly: the only way to correct it is to edit the instruction — it will not remember yesterday's mistake by itself and will repeat it tomorrow.
06// privacy

Data and privacy

People's conversations are the most sensitive thing in the system. The approach is simple: store less and delete sooner.
  • Retention period. A conversation with no activity for 30 days is physically deleted, together with its summary. This happens by itself, via a background sweep.
  • The right to be forgotten. The «forget» command really deletes the data rather than marking it hidden. For groups the owner does the same with a button.
  • An honest warning. On first launch the bot states plainly what it stores and for how long. The text of the promise and the retention setting can only be changed together.
  • Provider training is forbidden. Every request carries a flag prohibiting the use of the data for training models.
  • Client separation. One client cannot see another client's data — on an attempt the system answers «does not exist», so that one cannot even learn that somebody else's bot exists.
What we deliberately gave upEncrypting the conversations in the database. It sounds solid but protects poorly: the bot has to decrypt the history for every reply, which means the key always sits next to the data. It would be the appearance of protection at the price of complexity. The honest path is short retention, physical deletion, and restricted database access.
What this buys
  • The volume of accumulated personal data has an upper bound and does not grow forever.
  • «Delete my data» has a ready answer and a working button behind it, not a development ticket.
  • The promise made to the user and the system's behaviour match — this is verified on every change to the settings.
Human-like
You tell it something — it remembers; you ask it to forget — it forgets. Over time the details fade and only the general impression stays. That is a familiar and understandable model of a relationship.
What is missing
Any sense of «this is between us». A human will tell from the tone and the circumstances that what was said is not to be repeated. The bot draws no such distinction: everything said is just data, and in a group a fact heard from one participant is available in conversations with everyone.The flip side of the same trait: in a group's shared memory any participant can say anything about another, and the bot will take it as fact. A human distrusts the source — the bot does not, yet.
07// failures

What happens when things break

Everything breaks: the AI provider, the network, our own update. The question is not whether a failure will happen, but who will notice it and what it will cost.
the AI provider is downor returned an errorthe person closed the chatmid-replythe update is brokenthe new version didn't come upthe reservation is releasedthe client pays nothingthe error becomes plain textthe reply is already generatedand already paid to the providersettlement is carried throughthe rollout stopsat the first mismatchprod stays on the old version«provider unavailable,try again later»notices nothingthe money adds upnotices nothingthe bot keeps workingsystem's responsewhat happened
The bottom row is what all the rest is for: in two cases out of three the user does not see the failure at all, and in the third they get a clear explanation instead of a technical error.
The middle case deserves a separate explanation, it is not obvious. When a person closes the chat mid-reply, the reply is already written and we have already paid the provider for it. If processing is aborted at that moment, the client is not charged — and such a hole can be exploited deliberately. That is why the settlement step is protected and carried through to the end, even if there is no longer anyone listening.The third case is about updates: the system checks not only «the service responds» but «does the new version respond». The old one can answer «alive» too, and the nastiest failure is the one where everything is green while the changes never actually arrived.
What this buys
  • Provider errors do not turn into losses: the client does not pay for a failed request.
  • You cannot get free replies by dropping the connection.
  • A failed update does not take production down: it simply stays on the previous working version.
  • The user gets human wording instead of error codes, and different wording at that — «wait a minute» and «you are out of money» are different situations calling for different actions.
Human-like
An employee can be unavailable too, and good service accounts for that: instead of silence you get a clear explanation and an invitation to come back later.
What is missing
A human who could not answer right away will come back to the conversation themselves: «sorry, I was busy, here is my answer». The bot does not come back — an interrupted conversation stays interrupted until the person writes again. It does not remember that it still owes a reply.
08// roadmap

Limits and plans

A summary of what the system can and cannot do compared to a live interlocutor — and in what order the gap is planned to be closed.
ability
human
bot today
Hold a coherent conversation
yes
yes
Remember the person between conversations
yes
yes
Answer in its own manner and by the rules
yes
yes
Tell which participant said what
yes
partly
Ask when unsure about its memory
yes
no
Go and look it up: document, calendar, database
yes
no
Take an action, not just answer
yes
no
Write first, remind, return to a topic
yes
no
Understand voice, photos, documents
yes
no
Tell that «this is confidential»
yes
no
Learn from remarks without rewriting the instruction
yes
no
The main gap is visible in the table at once: the bot speaks but does not act. It cannot look anything up, go anywhere, or do anything — only reason out loud on the basis of what it has already been told. Everything else on the list follows from that same limitation.
in progress now
Actions and toolsThe bot gets the right to consult a reference, to search, to invoke the operation it needs — and to decide itself when that is required. A separate task inside it is money: one message from a person will map to several model calls, and the whole chain has to be reserved, not a single step.
next
Working with the client's documentsThe owner uploads their own materials — instructions, a price list, a policy — and the bot answers from them, not only from the conversation. The semantic-search mechanics for this are already built and running on the bot's memory.
next
Dashboard and alertsRight now any specific case can be investigated down to the details, but there is no early warning of «it got slower» or «the spend doubled». Metrics and alerts are needed.
as it grows
Rate limiting and a move to push updatesProtection against a single overly active user, and dropping constant Telegram polling in favour of incoming updates — less latency, less wasted traffic.
deliberately not
Heavy infrastructureMessage queues and business-process engines solve problems that do not exist at the current scale. Every such service is one more thing somebody has to keep running around the clock.
What already looks human today
The bot recognizes a returning person, holds the thread of a long conversation, speaks in the given manner, types in front of your eyes, and does not fall apart when something breaks. For most support and consulting scenarios that is enough.
What sets it apart from an employee
It waits for a question and does nothing on its own initiative; it knows exactly what it was told in this chat; and it will never say «I am not sure, let me check». Until those three things are closed, this is a very good interlocutor — but not yet a colleague.
09// stack

The stack in one list

What is used in the project — for those reading the document to learn the toolset rather than the mechanics.
Language and servicesPython 3.12, fully asynchronous. Two services: the gateway on FastAPI, the Telegram bot on aiogram 3. A monorepo with a shared deploy.
DataPostgreSQL 16 (Supabase), accessed via asyncpg, pgvector for semantic search. Schema and migrations by hand, no ORM: all SQL is written explicitly.
LLM layerLangGraph (the dialogue graph plus state persistence), LangChain (the model client), langmem (rolling summary and long-term memory), trustcall + Pydantic (structured output with validation).
ModelsOpenRouter as the single access point: switching models is a setting, not code. Separate models for replies, for the internal compression, and for embeddings.
BillingA custom Reserve & Settle on Postgres transactions: holds, idempotency by a unique key, two-level balances, a spend log labelled by call type.
InterfaceAn admin panel on Jinja2 + HTMX — no separate frontend project. Sessions, CSRF, invitations, separation of the superadmin from the clients.
ObservabilityLangSmith (EU instance): the full chain of every model call, including the background memory passes. Metrics and alerts are planned.
Testspytest + testcontainers (a real Postgres per run), respx for intercepting HTTP, canary tests on third-party library behaviour, a live bot run in Telegram after the rollout.
OperationsDocker + docker compose, GitLab CI (lint → secret scan → deploy), automatic migrations before the code, a healthcheck with rollback, verification of the commit stamp in the image.
The document describes the state of the system as of September 2026.