mirror of https://github.com/yandex/odyssey.git
167 lines
7.0 KiB
Markdown
167 lines
7.0 KiB
Markdown
|
|
### Odissey architecture and internals
|
|
|
|
Odissey heavily depends on two libraries, which were originally created during its
|
|
development: Machinarium and Shapito.
|
|
|
|
#### Machinarium
|
|
|
|
Machinarium extensively used for organization of multi-thread processing, cooperative multi-tasking
|
|
and networking IO. All Odissey threads are run in context of machinarium `machines` -
|
|
pthreads with coroutine schedulers placed on top of `epoll(7)` event loop.
|
|
|
|
Odissey does not directly use or create multi-tasking primitives such as OS threads and mutexes.
|
|
All synchronization is done using message passing and transparently handled by machinarium.
|
|
|
|
Repository: [github/machinarium](https://github.yandex-team.ru/pmwkaa/machinarium)
|
|
|
|
#### Shapito
|
|
|
|
Shapito provides resizable buffers (streams) and methods for constructing, reading and validating
|
|
PostgreSQL protocol requests. By design, all PostgreSQL specific details should be provided by
|
|
Shapito library.
|
|
|
|
Repository: [github/shapito](https://github.yandex-team.ru/pmwkaa/shapito).
|
|
|
|
#### Core components
|
|
|
|
```
|
|
main()
|
|
.----------.
|
|
| instance |
|
|
thread '----------'
|
|
.--------. .-------------.
|
|
| pooler | | worker_pool |
|
|
'--------' '-------------'
|
|
.--------. .---------. .---------. .---------.
|
|
| router | | servers | | worker0 | ... | workerN |
|
|
'--------' '---------' '---------' '---------'
|
|
.---------. .------. thread thread
|
|
| console | | cron |
|
|
'---------' '------'
|
|
```
|
|
|
|
#### Instance
|
|
|
|
Application entry point.
|
|
|
|
Handle initialization. Read configuration file, prepare loggers.
|
|
Run pooler and worker\_pool threads.
|
|
|
|
[sources/instance.h](sources/instance.h), [sources/instance.c](sources/instance.c)
|
|
|
|
#### Pooler
|
|
|
|
Start router, cron and console subsystems.
|
|
|
|
Create listen server one for each resolved address. Each listen server runs inside own coroutine.
|
|
Server coroutine mostly waits on `machine_accept()`.
|
|
|
|
On incoming connection, new client context is created and notification message is sent to next
|
|
worker using `workerpool_feed()`. Client IO context is detached from pooler `epoll(7)` context.
|
|
|
|
Handle signals using `machine_signal_wait()`. On `SIGHUP`: do versional config reload, add new databases
|
|
and obsolete old ones. On `SIGINT`, `SIGTERM`: call `exit(3)`. Other threads are blocked from receiving signals.
|
|
|
|
[sources/pooler.h](sources/pooler.h), [sources/pooler.c](sources/pooler.c)
|
|
|
|
#### Router
|
|
|
|
Handle client registration and routing requests. Do client-to-server attachment and detachment.
|
|
Ensure connection limits and client pool queueing. Handle implicit `Cancel` client request, since access
|
|
to server pool is required to match a client key.
|
|
|
|
Router works in request-reply manner: client (from worker thread) sends a request message to
|
|
router and waits for reply. Could be a potential hot spot (not an issue at the moment).
|
|
|
|
[sources/router.h](sources/router.h), [sources/router.c](sources/router.c)
|
|
|
|
#### Cron
|
|
|
|
Do periodic service tasks, like idle server connection expiration and
|
|
database scheme obsoletion.
|
|
|
|
[sources/cron.h](sources/cron.h), [sources/cron.c](sources/cron.c)
|
|
|
|
#### Worker and worker pool
|
|
|
|
Worker thread (machinarium machine) waits on incoming connection notification queue. On new connection event,
|
|
create new frontend coroutine and handle client (frontend) lifecycle. Each worker thread can host
|
|
thousands of client coroutines.
|
|
|
|
Worker pool is responsible for maintaining a thread pool of workers. Threads are machinarium machines,
|
|
created using `machine_create()`.
|
|
|
|
[sources/worker.h](sources/worker.h), [sources/worker.c](sources/worker.c),
|
|
[sources/worker_pool.h](sources/worker_pool.h), [sources/worker_pool.c](sources/worker_pool.c)
|
|
|
|
#### Client (frontend) lifecycle
|
|
|
|
Whole client logic is driven by a single `od_frontend()` function, which is a coroutine entry point.
|
|
There are 6 distinguishable stages in client lifecycle.
|
|
|
|
[sources/frontend.h](sources/frontend.h), [sources/frontend.c](sources/frontend.c)
|
|
|
|
#### 1. Startup
|
|
|
|
Read initial client request. This can be `SSLRequest`, `CancelRequest` or `StartupMessage`.
|
|
Handle SSL/TLS handshake.
|
|
|
|
#### 2. Process Cancel request
|
|
|
|
In case of `CancelRequest`, call Router to handle it. Disconnect client right away.
|
|
|
|
#### 3. Route client
|
|
|
|
Call router. Use `Database` and `User` to match client configuration route. Router assigns
|
|
matched route to a client. Each route object has a reference counter.
|
|
All routes are periodically garbage-collected.
|
|
|
|
#### 4. Authenticate client
|
|
|
|
Write client an authentication request `AuthenticationMD5Password` or `AuthenticationCleartextPassword` and
|
|
wait for reply to compare passwords. In case of success send `AuthenticationOk`.
|
|
|
|
#### 5. Process client requests
|
|
|
|
Depending on selected route storage type, do `local` (console) or `remote` (remote PostgreSQL server) processing.
|
|
|
|
Following remote processing logic repeats until client sends `Terminate`,
|
|
client or server disconnects during the process:
|
|
|
|
* Read client request. Handle `Terminate`.
|
|
* If client has no server attached, call Router to assign server from the server pool. New server connection registered and
|
|
initiated by the client coroutine (worker thread). Maybe discard previous server settings and configure it using client parameters.
|
|
* Send client request to the server.
|
|
* Wait for server reply.
|
|
* Send reply to client.
|
|
* In case of `Transactional` pooling: if transaction completes, call Router to detach server from the client.
|
|
* Repeat.
|
|
|
|
#### 6. Cleanup
|
|
|
|
If server is not Ready (query still in-progress), initiate automatic `Cancel` procedure. If server is Ready and left in active transaction,
|
|
initiate automatic `Rollback`. Return server back to server pool or disconnect.
|
|
|
|
Free client context.
|
|
|
|
#### Client error codes
|
|
|
|
In the most scenarios PostgreSQL error messages `ErrorResponce` are copied to a client as-is. Yet, there are some
|
|
cases, when Odissey has to provide its own error message and SQLCode to client.
|
|
|
|
Function `od_frontend_error()` is used for formatting and sending error message to client.
|
|
|
|
| SQLCode | PostgreSQL Code Name | Stage |
|
|
| ------- | -------------------- | ----- |
|
|
| **08P01** | `PROTOCOL_VIOLATION` | Startup, TLS handshake, authentication |
|
|
| **0A000** | `FEATURE_NOT_SUPPORTED` | TLS handshake |
|
|
| **28000** | `INVALID_AUTHORIZATION_SPECIFICATION` | Authentication |
|
|
| **28P01** | `INVALID_PASSWORD` | Authentication |
|
|
| **58000** | `SYSTEM_ERROR` | Routing, System specific |
|
|
| **3D000** | `UNDEFINED_DATABASE` | Routing |
|
|
| **53300** | `TOO_MANY_CONNECTIONS` | Routing |
|
|
| **08006** | `CONNECTION_FAILURE` | Server-side error during connection or IO |
|
|
|
|
PostgreSQL specific error codes can be found in `src/backend/errocodes.txt`.
|