Skip to content

Encryption architecture

This section explains how open_pg_tde implements encryption inside PostgreSQL. It covers the key hierarchy, supported encryption algorithms, encryption workflow, and the core changes that make it possible to encrypt tables, WAL, and other database files.

Together, these components form the foundation of data-at-rest encryption in open_pg_tde.

Two-key hierarchy

open_pg_tde uses two kinds of keys for encryption:

  1. Internal keys to encrypt the data. They are stored in PostgreSQL’s data directory under $PGDATA/open_pg_tde
  2. Principal keys, which encrypt internal keys, are stored externally in a Key Management System (KMS) using the key provider API

open_pg_tde uses one principal key per database. Every internal key for the given database is encrypted using this principal key.

Internal keys are used for specific database files: each file with a different Object Identifier (OID) has a different internal key.

Example:

A table with 4 indexes will have at least 5 internal keys, one for the table and one for each index. Additional associated relations, such as sequences or a TOAST table, also have their own keys.

Encryption algorithm

open_pg_tde currently uses the following encryption algorithms:

  • AES-128-XTS (default) or AES-256-XTS for encrypting database files, selected by open_pg_tde.data_cipher; encrypted with internal keys
  • AES-128-CTR, AES-256-CTR for WAL encryption; encrypted with internal keys
  • AES-128-GCM, AES-256-GCM for encrypting internal keys; encrypted with the principal key

Temporary files produced by query execution, such as external sorts and hash joins that exceed work_mem and spill to disk, are encrypted with AES-128-XTS when the core encrypt_temp_files GUC is on. The key is generated per boot, held only in memory, and never written to disk, so temporary files cannot be recovered once the server stops.

Pluggable cipher providers

The ciphers used for data files and WAL are supplied through an internal cipher provider registry (src/encryption/cipher_provider.{c,h}). Each entry, a TdeCipher, bundles:

  • a stable numeric id that is persisted with every internal key (in the key map and the WAL key file), so a key always decrypts with the algorithm it was created with;
  • the block-mode operations used to encrypt/decrypt data pages;
  • the keystream operation used for the WAL/stream XOR path.

The built-in entries (aes-128, aes-256) wrap the OpenSSL-backed primitives, so the registry is a dispatch layer rather than a second implementation. Encryption and decryption look up the cipher by its persisted id, which decouples the algorithm from the key length: two ciphers that share a key length can coexist.

Which cipher a new table’s data files use is selected by the open_pg_tde.data_cipher GUC. To add a new algorithm, register another TdeCipher in TdeCipherRegistryInit() with a new id and expose it as a open_pg_tde.data_cipher option. No changes to the encryption call sites or the on-disk format are required.

Encryption workflow

You can use open_pg_tde to encrypt entire databases or only selected tables.

To support this without metadata changes, encrypted tables are labeled with the tde_heap access method marker.

The tde_heap access method is functionally identical to the heap access method. This allows open_pg_tde to distinguish between encrypted (tde_heap) and non-encrypted (heap) tables.

The initial decision about encryption is made using the postgres event trigger mechanism:

  • When the tde_heap clause is used for CREATE TABLE or ALTER TABLE statements, then the newly created data files are marked as encrypted
  • After this, the file operations encrypt or decrypt the data

Subsequent decisions are done using a slightly modified Storage Manager (SMGR) API:

  • When a database file is re-created with a different ID as a result of a TRUNCATE or a VACUUM FULL command, the newly created file inherits the encryption information
  • The file is then either encrypted or left unencrypted based on that inheritance

WAL encryption functionality

You can control WAL encryption globally via the open_pg_tde.wal_encrypt GUC variable, which requires a server restart.

WAL keys also contain the LSN of the first WAL write after key creation. This allows open_pg_tde to know which WAL ranges are encrypted or not and with which key.

Note

See the Configure WAL encryption chapter for more information.

The setting only controls writes so that only WAL writes are encrypted when WAL encryption is enabled. This means that WAL files can contain both encrypted and unencrypted data, depending on what the status of this variable was when writing the data.

open_pg_tde keeps track of the encryption status of WAL records using internal keys. When the server is restarted it writes a new internal key if WAL encryption is enabled, or if it is disabled and was previously enabled it writes a dummy key signaling that WAL encryption ended.

With this information the WAL reader code can decide if a specific WAL record has to be decrypted or not and which key it should use to decrypt it.

Encrypting other access methods

Currently open_pg_tde only encrypts heap tables and other files such as indexes, TOAST tables, sequences that are related to the heap tables.

Indexes include any kind of index that goes through the SMGR API, not just the built-in indexes in PostgreSQL.

Other table access methods that use the SMGR API could also be encrypted. This requires adding a marker access method and extending the event triggers, using the same approach as with heap tables.

Storage Manager (SMGR) API

open_pg_tde relies on a slightly modified version of the SMGR API. These modifications include:

  • Making the API generally extensible, where extensions can inject custom code into the storage manager
  • Adding tracking information for files. When a new file is created for an existing relation, references to the existing file are also passed to the SMGR functions

With these modifications, open_pg_tde implements an additional layer on top of the normal Magnetic Disk SMGR API: if the related table is encrypted, open_pg_tde encrypts a file before writing it to the disk and, similarly, decrypts it after reading when needed.

WAL encryption

WAL encryption is implemented through a separate, server-wide mechanism that extends PostgreSQL WAL-related APIs. Similar to the SMGR integration, this relies on additional extensions to PostgreSQL core APIs to support encryption of WAL records.

For configuration details, see Configure WAL encryption.