2017-07-17 10:32:14 +00:00
|
|
|
|
2017-07-17 12:12:00 +00:00
|
|
|
### Odissey architecture and internals
|
2017-07-17 10:32:14 +00:00
|
|
|
|
2017-07-17 10:38:28 +00:00
|
|
|
Odissey heavily depends on two libraries, which were originally created during its
|
|
|
|
development: Machinarium and Shapito.
|
2017-07-17 10:32:14 +00:00
|
|
|
|
2017-07-18 14:49:17 +00:00
|
|
|
#### Machinarium
|
2017-07-17 10:32:14 +00:00
|
|
|
|
2017-07-18 10:11:59 +00:00
|
|
|
Machinarium extensively used for organization of multi-thread processing, cooperative multi-tasking
|
|
|
|
and networking IO. All Odissey threads are run in context of machinarium `machines` -
|
2017-07-25 13:31:26 +00:00
|
|
|
pthreads with coroutine schedulers placed on top of `epoll(7)` event loop.
|
2017-07-17 10:32:14 +00:00
|
|
|
|
2017-07-18 10:11:59 +00:00
|
|
|
Odissey does not directly use or create multi-tasking primitives such as OS threads and mutexes.
|
2017-07-18 10:30:23 +00:00
|
|
|
All synchronization is done using message passing and transparently handled by machinarium.
|
2017-07-17 10:32:14 +00:00
|
|
|
|
2017-07-17 10:38:28 +00:00
|
|
|
Repository: [github/machinarium](https://github.yandex-team.ru/pmwkaa/machinarium)
|
|
|
|
|
2017-07-18 14:49:17 +00:00
|
|
|
#### Shapito
|
2017-07-17 10:32:14 +00:00
|
|
|
|
|
|
|
Shapito provides resizable buffers (streams) and methods for constructing, reading and validating
|
2017-07-18 10:11:59 +00:00
|
|
|
PostgreSQL protocol requests. By design, all PostgreSQL specific details should be provided by
|
|
|
|
Shapito library.
|
2017-07-17 10:38:28 +00:00
|
|
|
|
|
|
|
Repository: [github/shapito](https://github.yandex-team.ru/pmwkaa/shapito).
|
|
|
|
|
2017-07-18 13:35:31 +00:00
|
|
|
#### Core components
|
|
|
|
|
2017-07-19 09:47:41 +00:00
|
|
|
```
|
2017-07-18 15:03:29 +00:00
|
|
|
main()
|
|
|
|
.----------.
|
|
|
|
| instance |
|
2017-07-19 09:47:41 +00:00
|
|
|
thread '----------'
|
|
|
|
.--------. .------------.
|
|
|
|
| pooler | | relay_pool |
|
|
|
|
'--------' '------------'
|
|
|
|
.--------. .---------. .--------. .--------.
|
|
|
|
| router | | servers | | relay0 | ... | relayN |
|
|
|
|
'--------' '---------' '--------' '--------'
|
|
|
|
.---------. .----------. thread thread
|
|
|
|
| console | | periodic |
|
|
|
|
'---------' '----------'
|
|
|
|
```
|
2017-07-18 13:35:31 +00:00
|
|
|
|
2017-07-18 14:49:17 +00:00
|
|
|
#### Instance
|
2017-07-18 14:06:53 +00:00
|
|
|
|
2017-07-18 14:49:17 +00:00
|
|
|
Application entry point.
|
2017-07-18 14:27:40 +00:00
|
|
|
|
2017-07-18 14:49:17 +00:00
|
|
|
Handle initialization. Read configuration file, prepare loggers.
|
2017-07-18 14:06:53 +00:00
|
|
|
Run pooler and relay\_pool threads.
|
|
|
|
|
2017-07-18 14:27:40 +00:00
|
|
|
[sources/instance.h](sources/instance.h), [sources/instance.c](sources/instance.c)
|
2017-07-18 14:06:53 +00:00
|
|
|
|
2017-07-18 14:49:17 +00:00
|
|
|
#### Pooler
|
2017-07-18 14:06:53 +00:00
|
|
|
|
|
|
|
Start router, periodic and console subsystems.
|
|
|
|
|
2017-07-18 14:27:40 +00:00
|
|
|
Create listen server one for each resolved address. Each listen server runs inside own coroutine.
|
2017-07-18 14:49:17 +00:00
|
|
|
Server coroutine mostly waits on `machine_accept()`.
|
|
|
|
|
|
|
|
On incoming connection, new client context is created and notification message is sent to next
|
2017-07-25 13:31:26 +00:00
|
|
|
relay worker using `relaypool_feed()`. Client IO context is detached from pooler `epoll(7)` context.
|
2017-07-18 14:06:53 +00:00
|
|
|
|
2017-07-19 11:29:23 +00:00
|
|
|
Handle signals using `machine_signal_wait()`. On `SIGHUP`: do versional config reload, add new databases
|
|
|
|
and obsolete old ones. On `SIGINT`: call `exit(3)`. Other threads are blocked from receiving signals.
|
2017-07-19 10:09:04 +00:00
|
|
|
|
2017-07-18 14:27:40 +00:00
|
|
|
[sources/pooler.h](sources/pooler.h), [sources/pooler.c](sources/pooler.c)
|
2017-07-18 14:06:53 +00:00
|
|
|
|
2017-07-18 14:49:17 +00:00
|
|
|
#### Router
|
2017-07-18 14:06:53 +00:00
|
|
|
|
2017-07-19 09:47:41 +00:00
|
|
|
Handle client registration and routing requests. Do client-to-server attachment and detachment.
|
|
|
|
Ensure connection limits and client pool queueing. Handle implicit `Cancel` client request, since access
|
2017-07-18 14:49:17 +00:00
|
|
|
to server pool is required to match a client key.
|
2017-07-18 14:27:40 +00:00
|
|
|
|
2017-07-18 15:03:29 +00:00
|
|
|
Router works in request-reply manner: client (from relay thread) sends a request message to
|
|
|
|
router and waits for reply. Could be a potential hot spot (not an issue at the moment).
|
2017-07-18 14:06:53 +00:00
|
|
|
|
2017-07-18 14:27:40 +00:00
|
|
|
[sources/router.h](sources/router.h), [sources/router.c](sources/router.c)
|
2017-07-18 14:06:53 +00:00
|
|
|
|
2017-07-18 14:49:17 +00:00
|
|
|
#### Periodic
|
2017-07-18 14:06:53 +00:00
|
|
|
|
2017-07-19 09:47:41 +00:00
|
|
|
Do periodic service tasks, like idle server connection expiration and
|
2017-07-18 14:27:40 +00:00
|
|
|
database scheme obsoletion.
|
2017-07-18 14:06:53 +00:00
|
|
|
|
2017-07-18 14:27:40 +00:00
|
|
|
[sources/periodic.h](sources/periodic.h), [sources/periodic.c](sources/periodic.c)
|
2017-07-18 14:06:53 +00:00
|
|
|
|
2017-07-18 14:49:17 +00:00
|
|
|
#### Relay and Relay pool
|
|
|
|
|
|
|
|
Relay machine (thread) waits on incoming connection notification queue. On new connection event,
|
|
|
|
create new frontend coroutine and handle client (frontend) lifecycle. Each relay thread can host
|
|
|
|
thousands of client coroutines.
|
2017-07-18 14:06:53 +00:00
|
|
|
|
2017-07-18 14:49:17 +00:00
|
|
|
Relay pool is responsible for maintaining a worker thread pool. Threads are machinarium machines,
|
|
|
|
created using `machine_create()`.
|
2017-07-18 14:06:53 +00:00
|
|
|
|
2017-07-18 14:27:40 +00:00
|
|
|
[sources/relay.h](sources/relay.h), [sources/relay.c](sources/relay.c),
|
|
|
|
[sources/relay_pool.h](sources/relay_pool.h), [sources/relay_pool.c](sources/relay_pool.c)
|
2017-07-25 13:31:26 +00:00
|
|
|
|
2017-07-25 13:43:00 +00:00
|
|
|
#### Client (frontend) lifecycle
|
2017-07-25 13:31:26 +00:00
|
|
|
|
2017-07-25 13:43:00 +00:00
|
|
|
Whole client logic is driven by a single `od_frontend()` function, which is a coroutine entry point.
|
|
|
|
There are 6 distinguishable stages in client lifecycle.
|
2017-07-25 13:31:26 +00:00
|
|
|
|
2017-07-25 14:06:58 +00:00
|
|
|
[sources/frontend.h](sources/frontend.h), [sources/frontend.c](sources/frontend.c)
|
2017-07-25 14:02:05 +00:00
|
|
|
|
|
|
|
#### 1. Startup
|
2017-07-25 13:31:26 +00:00
|
|
|
|
2017-07-25 13:43:00 +00:00
|
|
|
Read initial client request. This can be `SSLRequest`, `CancelRequest` or `StartupMessage`.
|
2017-07-25 14:02:05 +00:00
|
|
|
Handle SSL/TLS handshake.
|
2017-07-25 13:43:00 +00:00
|
|
|
|
2017-07-25 14:02:05 +00:00
|
|
|
#### 2. Process Cancel request
|
2017-07-25 13:43:00 +00:00
|
|
|
|
2017-07-25 14:02:05 +00:00
|
|
|
In case of `CancelRequest`, call Router to handle it. Disconnect client right away.
|
2017-07-25 13:43:00 +00:00
|
|
|
|
2017-07-25 14:02:05 +00:00
|
|
|
#### 3. Route client
|
2017-07-25 13:43:00 +00:00
|
|
|
|
2017-07-25 14:02:05 +00:00
|
|
|
Call router. Use `Database` and `User` to match client configuration route. Router assigns
|
2017-07-25 13:43:00 +00:00
|
|
|
matched route to a client. Each route object has a reference counter.
|
|
|
|
All routes are periodically garbage-collected.
|
|
|
|
|
2017-07-25 14:02:05 +00:00
|
|
|
#### 4. Authenticate client
|
|
|
|
|
|
|
|
Write client an authentication request `AuthenticationMD5Password` or `AuthenticationCleartextPassword` and
|
|
|
|
wait for reply to compare passwords. In case of success send `AuthenticationOk`.
|
|
|
|
|
|
|
|
#### 5. Process client requests
|
|
|
|
|
2017-07-25 14:06:58 +00:00
|
|
|
Depending on selected route storage type, do `local` (console) or `remote` (remote PostgreSQL server) processing.
|
2017-07-25 14:02:05 +00:00
|
|
|
|
2017-07-25 14:33:02 +00:00
|
|
|
Following remote processing logic repeats until client sends `Terminate`,
|
|
|
|
client or server disconnects during the process:
|
2017-07-25 14:06:58 +00:00
|
|
|
|
|
|
|
* Read client request. Handle `Terminate`.
|
2017-07-25 14:33:02 +00:00
|
|
|
* If client has no server attached, call Router to assign server from the server pool. New server connection registered and
|
|
|
|
initiated by the client coroutine (relay thread). Maybe discard previous server settings and configure it using client parameters.
|
2017-07-25 14:06:58 +00:00
|
|
|
* Send client request to the server.
|
|
|
|
* Wait for server reply.
|
|
|
|
* Send reply to client.
|
|
|
|
* In case of `Transactional` pooling: if transaction completes, call Router to detach server from the client.
|
2017-07-25 14:33:02 +00:00
|
|
|
* Repeat.
|
2017-07-25 14:02:05 +00:00
|
|
|
|
|
|
|
#### 6. Cleanup
|
2017-07-25 14:33:02 +00:00
|
|
|
|
|
|
|
If server is not Ready (query still in-progress), initiate automatic `Cancel` procedure. If server is Ready and left in active transaction,
|
|
|
|
initiate automatic `Rollback`. Return server back to server pool or disconnect.
|
|
|
|
|
|
|
|
Free client context.
|