async_postgres/pg_replication

Search:
Group by:

PostgreSQL Logical Replication support.

Provides types and procedures for consuming a logical replication stream via the PostgreSQL streaming replication protocol. The streaming API is plugin-agnostic (raw WAL bytes are delivered to a callback). A built-in decoder for the pgoutput logical decoding plugin is included.

Quick start

let conn = await connectReplication("postgresql://user:pass@host/db")
defer: await conn.close()
let slot = await conn.createReplicationSlot("my_slot", "pgoutput", temporary = true)
await conn.startReplication("my_slot", slot.consistentPoint,
    options = {"proto_version": "1", "publication_names": "my_pub"},
    callback = myCallback)

Types

BeginMessage = object
  finalLsn*: Lsn             ## LSN of the commit record
  commitTime*: int64         ## Commit timestamp (microseconds since PG epoch)
  xid*: int32                ## Transaction ID
Transaction begin.
CommitMessage = object
  flags*: byte
  commitLsn*: Lsn
  endLsn*: Lsn
  commitTime*: int64
Transaction commit.
DeleteMessage = object
  relationId*: int32
  keyKind*: char ## 'K' if oldTuple holds only the replica identity key,
                 ## 'O' if it holds the full old row (REPLICA IDENTITY FULL).
  oldTuple*: seq[TupleField]
Row deletion.
InsertMessage = object
  relationId*: int32
  newTuple*: seq[TupleField]
Row insertion.
LogicalMessage = object
  flags*: byte               ## Bit 0: transactional
  lsn*: Lsn
  prefix*: string
  content*: seq[byte]
Generic logical decoding message (via pg_logical_emit_message).
Lsn = distinct uint64
LSN (Log Sequence Number) PostgreSQL Log Sequence Number. Displayed as "X/Y" where X and Y are hex-encoded upper and lower 32-bit halves.
OriginMessage = object
  originLsn*: Lsn
  originName*: string
Replication origin.
PgOutputMessage = object
  case kind*: PgOutputMessageKind
  of pomkBegin:
    begin*: BeginMessage
  of pomkCommit:
    commit*: CommitMessage
  of pomkOrigin:
    origin*: OriginMessage
  of pomkRelation:
    relation*: RelationInfo
  of pomkType:
    typeMsg*: TypeMessage
  of pomkInsert:
    insert*: InsertMessage
  of pomkUpdate:
    update*: UpdateMessage
  of pomkDelete:
    delete*: DeleteMessage
  of pomkTruncate:
    truncate*: TruncateMessage
  of pomkMessage:
    message*: LogicalMessage
A decoded pgoutput plugin message.
PgOutputMessageKind = enum
  pomkBegin, pomkCommit, pomkOrigin, pomkRelation, pomkType, pomkInsert,
  pomkUpdate, pomkDelete, pomkTruncate, pomkMessage
Message types within the pgoutput logical decoding plugin.
PrimaryKeepalive = object
  walEnd*: Lsn ## The walsender's sent position. On a logical stream it counts as
               ## received, so once every earlier message is processed it may be passed
               ## to ``confirmFlushed`` (``startReplication(autoConfirm = true)`` does
               ## this outside a transaction). The auto-reply to a keepalive with
               ## ``replyRequested`` is sent before the callback runs, so that
               ## confirmation goes out with the next status update (the next requested
               ## reply, ``statusInterval``, or ``stopReplication``).
  sendTime*: int64           ## Server send time (microseconds since PG epoch)
  replyRequested*: bool      ## Whether the server wants an immediate status reply
Keepalive message from the server.
RelationCache = Table[int32, RelationInfo]
Cache of relation metadata received during replication. The server sends a Relation message before the first DML for each table in a transaction; clients must cache them.
RelationColumn = object
  flags*: byte               ## Bit 0: part of replica identity key
  name*: string
  typeOid*: int32
  typeMod*: int32
A single column in a relation definition.
RelationInfo = object
  relationId*: int32
  namespace*: string         ## Schema name
  name*: string              ## Table name
  replicaIdentity*: char     ## 'd' (default), 'n' (nothing), 'f' (full), 'i' (index)
  columns*: seq[RelationColumn]
Relation (table) metadata sent by pgoutput before DML events.
ReplicationCallback = proc (msg: ReplicationMessage): Future[void] {....gcsafe.}
Callback invoked for each replication message during streaming.
ReplicationMessage = object
  case kind*: ReplicationMessageKind
  of rmkXLogData:
    xlogData*: XLogData
  of rmkPrimaryKeepalive:
    keepalive*: PrimaryKeepalive
A single message received during replication streaming.
ReplicationMessageKind = enum
  rmkXLogData, rmkPrimaryKeepalive
Replication message types (decoded from CopyData during streaming)
ReplicationMode = enum
  rmDatabase, rmPhysical
Replication mode selected at connection time. rmDatabase sends replication=database (logical replication + ability to run SQL on the chosen database). rmPhysical sends replication=true (physical replication; no SQL on user databases).
ReplicationSlotInfo = object
  slotName*: string
  consistentPoint*: Lsn      ## confirmed_flush_lsn (logical) or restart_lsn (physical)
  snapshotName*: string      ## Snapshot name (only available at CREATE time)
  outputPlugin*: string      ## Output plugin (only available at CREATE time)
  slotType*: string ## Slot type as reported by READ_REPLICATION_SLOT ("physical").
                    ## Empty for CREATE_REPLICATION_SLOT results, which do not return it.
  restartTli*: int64         ## Timeline ID associated with restart_lsn.
                             ## Only populated by READ_REPLICATION_SLOT; 0 when NULL or not applicable.
Information about a replication slot.
SystemInfo = object
  systemId*: string
  timeline*: int32
  xLogPos*: Lsn
  dbName*: string
Result of IDENTIFY_SYSTEM command.
TimelineHistory = object
  filename*: string          ## Timeline history file name (e.g. "00000002.history").
  content*: seq[byte]        ## Raw history file content.
Result of TIMELINE_HISTORY command.
TruncateMessage = object
  options*: byte             ## Bit 0: CASCADE, bit 1: RESTART IDENTITY
  relationIds*: seq[int32]
Table truncation.
TupleDataKind = enum
  tdkNull = 110,            ## NULL value
  tdkText = 116,            ## Text-formatted value
  tdkBinary = 98,           ## Binary-formatted value (protocol_version >= 2)
  tdkUnchanged = 117         ## TOAST value unchanged
Kind of a single field value in a pgoutput tuple.
TupleField = object
  kind*: TupleDataKind
  data*: seq[byte]           ## Empty for null/unchanged
A single field value in a pgoutput tuple.
TypeMessage = object
  typeId*: int32
  namespace*: string
  name*: string
Custom type definition.
UpdateMessage = object
  relationId*: int32
  keyKind*: char ## 'K' if oldTuple holds only the replica identity key,
                 ## 'O' if it holds the full old row (REPLICA IDENTITY FULL),
                 ## '\0' when no old tuple is present.
  oldTuple*: seq[TupleField]
  newTuple*: seq[TupleField]
Row update.
XLogData = object
  startLsn*: Lsn ## Start LSN of the WAL data in this message. On a logical stream it is
                 ## the decoded record's LSN, or ``InvalidLsn`` for a write that is not the
                 ## last for its change (e.g. pgoutput Relation and Type messages).
  walEnd*: Lsn ## Informational. On a physical stream it is the end of WAL the server
               ## can send, which may be ahead of this message's data: acknowledge
               ## ``receivedEndLsn`` (``startLsn + data.len``), never ``walEnd``. On a
               ## logical stream it equals ``startLsn``; confirm ``CommitMessage.endLsn``
               ## or ``PrimaryKeepalive.walEnd`` instead. With any output plugin, the
               ## ``startLsn`` of a transaction's commit message is its end LSN.
  sendTime*: int64           ## Server send time (microseconds since PG epoch)
  data*: seq[byte]           ## Raw WAL data (plugin-dependent format)
WAL data payload from the server.

Consts

InvalidLsn = 0'u
Sentinel value representing an invalid or unset LSN.

Procs

proc `$`(lsn: Lsn): string {....raises: [], tags: [], forbids: [].}
Format an LSN as "X/Y" hex string.
proc `<`(a, b: Lsn): bool {.borrow, ...raises: [], tags: [], forbids: [].}
proc `<=`(a, b: Lsn): bool {.borrow, ...raises: [], tags: [], forbids: [].}
proc `==`(a, b: Lsn): bool {.borrow, ...raises: [], tags: [], forbids: [].}
proc confirmedFlushLsn(conn: PgConnection): Lsn {.inline, ...raises: [], tags: [],
    forbids: [].}
Confirmed flush LSN for current stream, or InvalidLsn outside stream.
proc confirmFlushed(conn: PgConnection; lsn: Lsn): bool {.
    ...raises: [PgError, PgStateError], tags: [], forbids: [].}
Confirm WAL up to lsn as durable. Clamped to received WAL, monotonic. Returns true if advanced. Must be in csReplicating. Received WAL is the highest receivedEndLsn (physical) or XLogData.startLsn / PrimaryKeepalive.walEnd (logical). Returns false without advancing once stopReplication's final status is encoded: the position could no longer be reported. That waits for a running callback to return, so a callback may still confirm after calling stopReplication.
proc connectReplication(config: ConnConfig; mode: ReplicationMode = rmDatabase): Future[
    PgConnection] {....raises: [Exception, ValueError, PgConfigError,
                             PgConnectionError, CatchableError],
                    tags: [RootEffect, WriteIOEffect, TimeEffect], forbids: [].}
Connect with replication param. rmPhysical allows only replication cmds.
proc connectReplication(dsn: string; mode: ReplicationMode = rmDatabase): Future[
    PgConnection] {....raises: [PgConfigError, Exception, ValueError,
                             PgConnectionError, CatchableError], tags: [
    ReadIOEffect, RootEffect, WriteIOEffect, TimeEffect], forbids: [].}
DSN-string variant of connectReplication. See the ConnConfig overload for the meaning of mode.
proc createReplicationSlot(conn: PgConnection; slotName: string;
                           plugin: string = "pgoutput"; temporary: bool = false;
                           timeout: async_backend.Duration = ZeroDuration): Future[
    ReplicationSlotInfo] {....stackTrace: false, raises: [Exception, ValueError,
    CatchableError, PgConnectionError, PgTypeError, PgError, PgProtocolError],
                           tags: [RootEffect, TimeEffect, WriteIOEffect],
                           forbids: [].}

Create a logical replication slot. Returns slot info including the consistent point LSN.

On timeout, the connection is retired (csClosed) unless the wire had settled (asyncdispatch always retires: the timed-out op stays on the socket).

proc currentPgTimestamp(): int64 {....raises: [], tags: [TimeEffect], forbids: [].}
Current time as microseconds since the PostgreSQL epoch (2000-01-01 UTC).
proc decodePgOutput(msg: XLogData): PgOutputMessage {....raises: [PgProtocolError],
    tags: [], forbids: [].}
Convenience: decode the pgoutput message from an XLogData's data field.
proc dropReplicationSlot(conn: PgConnection; slotName: string;
                         wait: bool = false;
                         timeout: async_backend.Duration = ZeroDuration): Future[
    void] {....stackTrace: false, raises: [Exception, ValueError, CatchableError],
            tags: [RootEffect, TimeEffect, WriteIOEffect], forbids: [].}

Drop a replication slot.

On timeout, the connection is retired (csClosed) unless the wire had settled (asyncdispatch always retires: the timed-out op stays on the socket).

proc hasOldTuple(msg: UpdateMessage): bool {.inline, ...raises: [], tags: [],
    forbids: [].}
True if the update carries an old tuple (replica identity key or full row).
proc identifySystem(conn: PgConnection;
                    timeout: async_backend.Duration = ZeroDuration): Future[
    SystemInfo] {....stackTrace: false, raises: [Exception, ValueError,
    CatchableError, PgConnectionError, PgTypeError, PgError, PgProtocolError],
                  tags: [RootEffect, TimeEffect, WriteIOEffect], forbids: [].}

Execute IDENTIFY_SYSTEM and return system identification info.

On timeout, the connection is retired (csClosed) unless the wire had settled (asyncdispatch always retires: the timed-out op stays on the socket).

proc parseLsn(s: string): Lsn {....raises: [PgTypeError], tags: [], forbids: [].}
Parse an LSN from "X/Y" hex string. Converts a malformed value (wrong shape, non-hex halves, or a half wider than 32 bits) into PgTypeError so callers stay under the except PgError contract, mirroring parseTimelineId.
proc parsePgOutputMessage(data: openArray[byte]): PgOutputMessage {.
    ...raises: [PgProtocolError], tags: [], forbids: [].}
Decode a pgoutput logical decoding message from raw WAL bytes.
proc parseReplicationMessage(copyData: sink seq[byte]): ReplicationMessage {.
    ...raises: [PgProtocolError], tags: [], forbids: [].}
Parse a CopyData payload into a ReplicationMessage. Takes ownership of copyData so the XLogData path can reuse the incoming buffer for xlogData.data instead of slicing into a fresh allocation.
proc parseTimelineId(s: string): int32 {....raises: [PgTypeError], tags: [],
    forbids: [].}
Parse timeline id from an IDENTIFY_SYSTEM row (text format). Non-numeric/out-of-range values raise PgTypeError, not RangeDefect.
proc readReplicationSlot(conn: PgConnection; slotName: string;
                         timeout: async_backend.Duration = ZeroDuration): Future[
    ReplicationSlotInfo] {....stackTrace: false, raises: [Exception, ValueError,
    CatchableError, PgConnectionError, PgTypeError, PgError, PgProtocolError],
                           tags: [RootEffect, TimeEffect, WriteIOEffect],
                           forbids: [].}

Read information about an existing physical replication slot.

Only physical slots are supported: the server rejects a logical slot with PgQueryError ("cannot use READ_REPLICATION_SLOT with logical replication slot") and reports a nonexistent slot as PgConnectionError. consistentPoint carries restart_lsn (InvalidLsn when the slot never reserved WAL) and restartTli its timeline (0 when NULL).

On timeout, the connection is retired (csClosed) unless the wire had settled (asyncdispatch always retires: the timed-out op stays on the socket).

proc receivedEndLsn(msg: XLogData): Lsn {....raises: [PgProtocolError], tags: [],
    forbids: [].}

End LSN of the WAL data actually contained in this message (startLsn + len(data)). On a physical stream, use this when acknowledging received data via sendStandbyStatus; do not use walEnd, which may point past data this message does not carry.

Physical replication only: logical data is plugin output, not WAL bytes, so this may point past commits not yet sent. Confirm logical progress with CommitMessage.endLsn or PrimaryKeepalive.walEnd.

proc sendCopyData(conn: PgConnection; data: openArray[byte]): Future[void] {.
    ...raises: [PgError, PgStateError, PgTypeError, PgProtocolError, Exception],
    tags: [RootEffect, TimeEffect], forbids: [].}
Send CopyData during csReplicating. Raises PgStateError (not replicating) / PgConnectionError (connection lost) / PgTypeError synchronously before first suspension. data is encoded into the frame there too, so the caller's buffer need not outlive the returned Future. A hand-built Standby Status Update is recorded like sendStandbyStatus. Writes go out in call order with the library's own. After stopReplication (or the reply to a server-initiated stop) the frame still goes out ahead of the stop's final status while that is queued; once it is encoded the frame is silently dropped, as the walsender reads nothing after the client's CopyDone. Cancelling the returned Future only stops the wait: the frame is still written. A failed write closes the connection.
proc sendStandbyStatus(conn: PgConnection; receiveLsn: Lsn;
                       flushLsn: Lsn = InvalidLsn; applyLsn: Lsn = InvalidLsn;
                       replyRequested: bool = false): Future[void] {.
    ...stackTrace: false, raises: [Exception, PgError, PgStateError, ValueError],
    tags: [RootEffect, TimeEffect], forbids: [].}

Send Standby Status Update. InvalidLsn defaults up to receiveLsn. Raises PgStateError unless the connection is csReplicating, or PgConnectionError when the connection was lost. After stopReplication the update still goes out, ahead of the stop's final status, while that is queued; once it is encoded the update is silently dropped, as the walsender reads nothing after the client's CopyDone. Report before stopping to be sure. Cancelling only stops the wait: the update is still written. A failed write closes the connection.

Values are sent verbatim, unlike confirmFlushed. On a logical stream pass CommitMessage.endLsn or PrimaryKeepalive.walEnd, never receivedEndLsn: a flush past an unsent commit makes the server skip it. The auto-reply, periodic status and stopReplication never report less than the last update sent here: they send the confirmFlushed position or these, whichever is higher.

proc startPhysicalReplication(conn: PgConnection; startLsn: Lsn;
                              slotName: string = ""; timeline: int32 = 0;
                              autoKeepaliveReply: bool = true; statusInterval: async_backend.Duration = ZeroDuration;
                              callback: ReplicationCallback): Future[void] {.
    ...stackTrace: false, raises: [Exception, ValueError, PgStateError,
                                PgConnectionError, PgTypeError, PgProtocolError,
                                CancelledError, CatchableError, PgQueryError,
                                AsyncTimeoutError, PgError, PgUnavailableError],
    tags: [RootEffect, TimeEffect], forbids: [].}

Physical replication streaming. Callback per message, raw WAL in XLogData. Raises PgStateError (closed by close(), or busy) / PgConnectionError (lost) unless csReady. Error handling matches startReplication: a callback exception or any other mid-stream failure poisons the connection (marked closed) and propagates, so reconnect and resume from the last LSN tracked. A ReadyForQuery without CopyDone retires the connection and raises PgUnavailableError too, with no cause named: a physical walsender does reset its streaming flags, so no ordinary stream ends that way.

slotName = "" streams without a slot. Non-zero timeline is sent as TIMELINE n (negative raises ValueError). statusInterval behaves as on startReplication.

On a timeline switch the server may send a result set describing the next timeline between CopyDone and ReadyForQuery; this proc drains and discards it — re-issue IDENTIFY_SYSTEM if you need that information.

Synchronous standbys: the flush LSN governs how much WAL the primary may recycle, so a standby in synchronous_standby_names that relies on the auto-reply must call confirmFlushed (or reply manually) or the primary's COMMITs block waiting on a flush position that never advances.

proc startReplication(conn: PgConnection; slotName: string;
                      startLsn: Lsn = InvalidLsn;
                      options: seq[(string, string)] = @[];
                      autoKeepaliveReply: bool = true;
                      statusInterval: async_backend.Duration = ZeroDuration;
                      callback: ReplicationCallback; autoConfirm: bool = false): Future[
    void] {....stackTrace: false, raises: [Exception, ValueError, PgStateError,
                                        PgConnectionError, CatchableError,
                                        PgTypeError, PgError, PgProtocolError,
                                        CancelledError, PgQueryError,
                                        AsyncTimeoutError, PgUnavailableError],
            tags: [RootEffect, TimeEffect, WriteIOEffect], forbids: [].}

Begin logical replication. Callback invoked per message. Use confirmFlushed for flush tracking; or set autoKeepaliveReply=false and use sendStandbyStatus manually.

autoConfirm (pgoutput only) confirms progress for you: a transaction's CommitMessage.endLsn once the callback has returned for its Commit, and a keepalive's walEnd outside a transaction (reported right away), so the slot also advances while pgoutput skips unpublished transactions. The callback returning therefore means "processed durably"; a callback raising on a transaction's messages confirms nothing of it. A keepalive carries no data, so its position is confirmed before its callback runs.

Nothing is confirmed once stopReplication's final status is encoded. That waits for a running callback to return, so a stop from a Commit's own callback still reports that transaction; messages after it are not confirmed.

To resume after an error, reconnect and pass InvalidLsn: the server restarts from the slot's confirmed_flush_lsn, re-sending anything not yet reported. With autoConfirm, the slot's confirmed_flush_lsn is looked up first and the stream starts from it or startLsn, whichever is later (the server would not start earlier either), so keepalives seen while the server re-reads WAL from restart_lsn never report a position below it. Without it, no lookup is made and startLsn is sent as given. A Commit's position is confirmed at once but reaches the server with the next status: a keepalive's, statusInterval's or the stop's. Set statusInterval on a stream that may stay busy for long, as its walsender sends no idle keepalive meanwhile.

A connection can stream logical replication only once: its walsender does not reset the streaming flags, so the next START_REPLICATION ... LOGICAL on it ends at once (BUG #18754). The library retires the connection (csClosed) and raises PgUnavailableError — accepted by isTransientError — so reconnect and resume with InvalidLsn.

With autoConfirm, requires publication_names in options and autoKeepaliveReply (ValueError otherwise). Without it, omitting publication_names is left to the server, which rejects a logical stream without one (pgoutput); the option checks above still apply.

Returns on server CopyDone or connection close, or raises PgUnavailableError as above. To stop from the client side, call stopReplication from the callback (or a concurrent task).

Errors poison connection. Track LSN for resume. A failing auto-reply propagates too, and the callback is not invoked for that keepalive. Option values are passed unquoted and single-quoted when building the command (keys stay identifier-validated). An empty value means a flag-only option (binary rather than binary ''). Raises PgStateError (closed by close(), or busy) / PgConnectionError (lost) unless csReady, and ValueError for a proto_version other than 1 in options (the value must be the unquoted string "1", an empty one included): the bundled pgoutput decoder supports v1 only. Any value already wrapped in quotes raises ValueError too, so the verbatim-options spelling cannot silently name a publication 'my_pub' or send a thrice-quoted binary flag the plugin rejects mid-stream. publication_names alone may be wrapped in double quotes: it is an identifier list, where "MyPub" keeps the case (unquoted, MyPub means mypub). An empty publication_names and a value containing a NUL byte are rejected the same way. publication_names without an explicit proto_version adds proto_version '1' to the generated command, so a server-side default bump cannot outrun that decoder.

statusInterval (ZeroDuration = off) sends a proactive Standby Status Update at least that often — receive = highest received, flush/apply = confirmFlushed (never below a position the caller itself reported via sendStandbyStatus) — so the slot advances on a server that never requests a reply (wal_sender_timeout = 0). Honoured only with autoKeepaliveReply; under asyncdispatch it fires only while messages are flowing, so a fully idle stream sends nothing until the next message.

Synchronous standbys: the auto-reply reports receive and flush/apply separately, so a consumer in synchronous_standby_names that never calls confirmFlushed keeps wal_sender_timeout reset via the receive field yet never advances flush — the primary's COMMITs then block indefinitely.

proc stopReplication(conn: PgConnection): Future[void] {....stackTrace: false,
    raises: [Exception, PgError, PgStateError, ValueError],
    tags: [RootEffect, TimeEffect], forbids: [].}

Terminate replication. Flushes confirmed position before CopyDone, never below the caller's last reported status (sendStandbyStatus). Raises PgStateError unless the connection is csReplicating, or PgConnectionError when the connection was lost. The final status and CopyDone are queued behind every earlier replication write; a call made once they are queued waits for that same CopyDone. Cancelling only stops the wait: the stop still goes out. A failed write closes the connection.

While the stream's callback runs, the final status waits for it to return, so a position it confirms after this call (autoConfirm's Commit included) is still reported. A call made meanwhile, from the callback or elsewhere, returns once the stop is queued; startReplication reports a failed write. To stop from another task, await startReplication rather than calling close() right after, or the stop may never go out.

proc timelineHistory(conn: PgConnection; timeline: int32;
                     timeout: async_backend.Duration = ZeroDuration): Future[
    TimelineHistory] {....stackTrace: false, raises: [Exception, ValueError,
    CatchableError, PgConnectionError, PgTypeError, PgError, PgProtocolError],
                       tags: [RootEffect, TimeEffect, WriteIOEffect],
                       forbids: [].}
Execute TIMELINE_HISTORY. Raises ValueError if timeline <=0. On timeout, the connection is retired (csClosed) unless the wire had settled (asyncdispatch always retires: the timed-out op stays on the socket).
proc toString(field: TupleField): string {....raises: [PgProtocolError], tags: [],
    forbids: [].}
Convert a TupleField's data to a string by copying the bytes.

Templates

template makeReplicationCallback(body: untyped): ReplicationCallback

Create a ReplicationCallback that works with both asyncdispatch and chronos. Inside body, the current message is available as msg: ReplicationMessage.

Kept module-local: routing this through a shared template with an untyped/typedesc param for the parameter type trips asyncdispatch's {.async.} macro ("cannot use symbol of kind 'func' as a 'param'").

template toInt64(lsn: Lsn): int64
Get the LSN as int64 (for wire protocol encoding).
template toUInt64(lsn: Lsn): uint64
Get the raw uint64 value of an LSN.