Cl-Postgres Reference Manual

Representation of a database connection. Contains login information in order to be able to automatically re-establish a connection when it is somehow closed.

(defclass database-connection ()
  ((host :initarg :host :reader connection-host)
   (port :initarg :port :reader connection-port)
   (database :initarg :db :reader connection-db)
   (user :initarg :user :reader connection-user)
   (password :initarg :password :reader connection-password)
   (use-ssl :initarg :ssl :reader connection-use-ssl)
   (use-binary :initarg :binary :accessor connection-use-binary :initform nil)
   (service :initarg :service :accessor connection-service)
   (application-name :initarg :application-name :accessor connection-application-name)
   (socket :initarg :socket :accessor connection-socket)
   (meta :initform nil)
   (available :initform t :accessor connection-available)
   (parameters :accessor connection-parameters)
   (timestamp-format :accessor connection-timestamp-format)))

→ database-connection

Create and open a connection for the specified server, database, and user. use-ssl may be :no, :try, :require, :yes, or :full

• :try means if the server supports it
• :require means use provided ssl certificate with no verification
• :yes means verify that the server cert is issued by a trusted CA, but does not verify the server hostname
• :full means expect a CA-signed cert for the supplied hostname and verify the server hostname

If you set it to anything other than :no be sure to also load the CL+SSL library. When it is anything but :no, you must have the CL+SSL package loaded to initiate the connection.

When using ssl, you can set the exported variables *ssl-certificate-file*, *ssl-key-file* and *ssl-root-ca-file* to provide client key, certificate files and root ca files. They can be either NIL, for no file, or a pathname.

On SBCL and Clozure CL, the value :unix may be passed for host, in order to connect using a Unix domain socket instead of a TCP socket.

Service defaults to "postgres".

Application-name defaults to a blank string. If provided, Postgresl can use it to track applications with multiple connections.

Use-binary defaults to nil. This relates to how Postmodern passes the parameters to Postgresql prepared queries. If you want to have cl-postgres pass integer, float or boolean parameters to Postgresql in binary format, then set this optional parameter to t.

If a query to Postgresql does not have a table column which would allow Postgresql to determine the correct datatype and you do not specify differently, Postgresql will treat the parameters passed with a prepared query as text. The default cl-postgres connection will have results like the following examples:

(defparameter *connect-spec* '("database_name" "user_name" "user_password" "localhost" 5432 :no "postgres" "some_app_name" nil))
(defmacro connect-now (&body body)
  `(let ((connection (apply 'open-database *connect-spec*)))
     (unwind-protect (progn ,@body)
       (close-database connection))))

(connect-now
 (prepare-query connection "test4" "select $1")
 (exec-prepared connection "test4" '(42) 'list-row-reader))
(("42"))

(connect-now
 (prepare-query connection "test4" "select $1")
 (exec-prepared connection "test4" '(T) 'list-row-reader))
(("true"))

(connect-now
 (prepare-query connection "test4" "select $1")
 (exec-prepared connection "test4" '(nil) 'list-row-reader))
(("false"))

Still in the default setting, you can specify parameter type as so:

(connect-now
  (prepare-query connection "test1" "select $1::integer")
  (exec-prepared connection "test1" '(42) 'list-row-reader))
((42))

(connect-now
  (prepare-query connection "test1" "select $1::float")
  (exec-prepared connection "test1" '(42.53) 'list-row-reader))
((42.53d0))

(connect-now
  (prepare-query connection "test1" "select $1::boolean")
  (exec-prepared connection "test1" '(T) 'list-row-reader))
((T))

(connect-now
  (prepare-query connection "test1" "select $1::boolean")
  (exec-prepared connection "test1" '(nil) 'list-row-reader))
((NIL))

Setting the use-binary slot in the database connection object to t (the last optional parameter) will allow you to pass parameters as binary BUT Postgresql insists on knowing the type of parameters you will be passing in a prepared query. If you do not pass a list of sample parameters, Postgresql will treat all parameters passed in the exec-prepared function required to be text and will throw an error, typically insufficient data left in message. Unlike Postmodern, cl-postgres does not keep a copy of prepared statements in the database connection object. Thus any error checking would require making a second call to Postgresql to find out what it has in the way of prepared statement parameters will lose any efficiency which would otherwise be gained by passing parameters in binary form.

    (setf *connect-spec* '("database_name" "user-name" "user-password" "localhost" 5432 :no "postgres" "some-app-name" t))

  (connect-now
   (prepare-query connection "test6" "select $1, $2, $3" '(1 nil 1.0))
   (exec-prepared connection "test6" '(42 T 127.89) 'list-row-reader))
((42 T 127.89))

The default for cl-postgres/Postmodern is to continue to pass parameters to Postgresql as text (not in binary format) in order to avoid breaking existing user code. If you want to pass parameters to Postgresql in binary format and want to set that up when you are making the database connection, the following examples may help. We continue the difference in the signatures (cl-postgres using optional parameters and postmodern using keyword parameters) because of the expected downstream breakage if we shifted cl-postgres:open-database to using keyword parameters.

Close a database connection. It is advisable to call this on connections when you are done with them. Otherwise the open socket will stick around until it is garbage collected, and no one will tell the database server that we are done with it.

Re-establish a database connection for a previously closed connection object. (Calling this on a connection that is still open is harmless.)

→ boolean

Returns a boolean indicating whether the given connection is currently connected.

→ hash-table

This method provides access to a hash table that is associated with the current database connection, and is used to store information about the prepared statements that have been parsed for this connection.

→ hash-table

This method returns a mapping (string to string) containing all the configuration parameters for the connection.

Directory where the Unix domain socket for PostgreSQL be found.

On SBCL, when using the :unix keyword as host argument when creating a connection, this variable determines the directory in which CL-Postgres will look for the socket file.

When using SSL (see open-database), these can be used to provide client key, certificate files and root ca files. They can be either NIL, for no file, or a pathname.

When establishing an SSL connection, Postmodern will check to see if unexpected extra data was received prior to the connection being encrypted. Unexpected extra data may indicate an attempted man-in-the-middle attack. By default, this variable is set to :error. You can set the response to be a simple warning (by setting this to :warn) or you can set this to :ignore.

How many times do we try to connect again. Borrowed from pgloader

How many seconds to wait before trying to connect again. Borrowed from pgloader

This function blocks until asynchronous notification is received on the connection. Retrun the channel string, the payload and notifying pid as multiple values. The PostgreSQL LISTEN command must be used to enable listening for notifications.

This function returns the version of the connected postgresql instance as a string.

Takes a postgresql version number which should be a string with the major and minor versions separated by a period e.g. '12.2' or '9.6.17'. Checks against the connection understanding of the running postgresql version and returns t if the running version is the requested version or newer.

→ result

Sends the given query to the given connection, and interprets the results (if there are any) with the given row-reader. If the database returns information about the amount of rows affected, this is returned as a second value.

Example:

(exec-query connection "select 1" 'list-row-reader)
'((1))

(exec-query connection "select name from employees where id=3" 'list-row-reader)

Parse and plan the given query, and store it under the given name. Note that prepared statements are per-connection, so they can only be executed through the same connection that prepared them. Also note that while the Postmodern package will also stored the prepared query in the connection-meta slot of the connection, but cl-postgres prepare-query does not. If the name is an empty string, Postgresql will not store it as a reusable query.

If parameters are not passed, Postgresql will assume the parameters will be text. In order to pass integer, float or boolean parameters as binary even when the database-connection is set to use binary parameters, you need to pass a list of parameters with the same type as you will be using when you call (exec-prepared).

The following example shows preparing and executing a query that will accept a boolean parameter:

  (prepare-query connection "test-bool" "select $1" '(t))
  (exec-prepared connection "test-bool" '(nil) 'list-row-reader)

'((nil))

See also the discussion under open database with respect to the use-binary parameter.

→ result

Execute the prepared statement by the given name. Parameters should be given as a list. Each value in this list should be of a type that to-sql-string has been specialised on. (Byte arrays will be passed in their binary form, without being put through to-sql-string.) The result of the executing the statement, if any, is interpreted by the given row reader, and returned. Again, the number or affected rows is optionally returned as a second value.

The following example shows preparing and executing a query that will accept an integer and a float in that order:

  (prepare-query connection "test-prepl" "select $1" '(10 7.4))
  (exec-prepared connection "test-prep1" '(12 4.2) 'list-row-reader)

'((nil))

Close the prepared query given by name by closing the session connection. Note: This is not the same as keeping the connection open and sending Postgresql query to deallocate the named prepared query. That can be done with the postmodern package's function:

(drop-prepared-statement (name &key (location :both) (database *database*)
                                       (remove-function t))

→ (values string needs-escaping)

Convert a Lisp value to its textual unescaped SQL representation. Returns a second value indicating whether this value should be escaped if it is to be put directly into a query. Generally any string is going to be designated to be escaped.

As of cl-postgres version 1.33.7 the lisp values handled are:

• string
• vector
• cons (must be a proper list and cannot be an empty list) (added in v. 1.33.7)
• array
• integer
• float (single or double except for clisp which does not have a double float)
• ratio
• t
• nil
• :null

Anything else will throw an error that this value cannot be converted to a sql literal.

You can define to-sql-string methods for your own datatypes if you want to be able to pass them to exec-prepared. When a non-NIL second value is returned, this may be T to indicate that the first value should simply be escaped as a string, or a second string providing a type prefix for the value. (This is used by S-SQL.)

Given a ratio, a stream and a digital-length-limit, if *silently-truncate-ratios* is true, will return a potentially truncated ratio. If false and the digital-length-limit is reached, it will throw an error noting the loss of precision and offering to continue or reset *silently-truncate-ratios* to true. Code contributed by Attila Lendvai.

When debugging, it can be helpful to inspect the queries that are being sent to the database. Set this variable to an output stream value (*standard-output*, for example) to have CL-postgres log every query it makes.

When profiling or debugging, the *query-log* may not give enough information, or reparsing its output may not be feasible. This variable may be set to a designator of function taking two arguments. This function will be then called after every query, and receive query string and internal time units (as in (CL:GET-INTERNAL-REAL-TIME)) spent in query as its arguments.

Default value of this variable is 'LOG-QUERY, which takes care of *QUERY-LOG* processing. If you provide custom query callback and wish to keep *QUERY-LOG* functionality, you will have to call LOG-QUERY from your callback function

This function is default value of *QUERY-CALLBACK* and logs queries to *QUERY-LOG* if it is not NIL.

→ readtable

Copies a given readtable.

→ readtable

Returns the default readtable, containing only the readers defined by CL-postgres itself.

Define a new reader for a given type. table defaults to *sql-readtable*. The reader function should take a single argument, a string, and transform that into some kind of equivalent Lisp value. When binary-p is true, the reader function is supposed to directly read the binary representation of the value. In most cases this is not recommended, but if you want to use it: provide a function that takes a binary input stream and an integer (the size of the value, in bytes), and reads the value from that stream. Note that reading less or more bytes than the given size will horribly break your connection.

Since there is no widely recognised standard way of representing dates and times in Common Lisp, and reading these from string representation is clunky and slow, this function provides a way to easily plug in binary readers for the date, time, timestamp, and interval types. It should be given functions with the following signatures:

• :date (days)

Where days is the amount of days since January 1st, 2000.

• :timestamp (useconds)

Timestamps have a microsecond resolution. Again, the zero point is the start of the year 2000, UTC.

• :timestamp-with-timezone

Like :timestamp, but for values of the 'timestamp with time zone' type (which PostgreSQL internally stores exactly the same as regular timestamps).

• :time (useconds)

Refers to a time of day, counting from midnight.

• :interval (months days useconds)

An interval is represented as several separate components. The reason that days and microseconds are separated is that you might want to take leap seconds into account.

→ function

Creates a row-reader, using the given name for the variable. Inside the body this variable refers to a vector of field descriptions. On top of that, two local functions are bound, next-row and next-field. The first will start reading the next row in the result, and returns a boolean indicating whether there is another row. The second will read and return one field, and should be passed the corresponding field description from the fields argument as a parameter.

A row reader should take care to iterate over all the rows in a result, and within each row iterate over all the fields. This means it should contain an outer loop that calls next-row, and every time next-row returns T it should iterate over the fields vector and call next-field for every field.

The definition of list-row-reader should give you an idea what a row reader looks like:

(row-reader (fields)
  (loop :while (next-row)
        :collect (loop :for field :across fields
                       :collect (next-field field))))

Obviously, row readers should not do things with the database connection like, say, close it or start a new query, since it still reading out the results from the current query.

The defun-like variant of row-reader: creates a row reader and gives it a top-level function name.

→ string

This can be used to get information about the fields read by a row reader. Given a field description, it returns the name the database associated with this column.

→ oid

This extracts the PostgreSQL OID associated with this column. You can, if you really want to, query the pg_types table to find out more about the types denoted by OIDs.

→ list

A row reader that builds a list of lists from the query results.

→ alist

A row reader that returns a list of alists, which associate column names with values.

A row reader that completely ignores the result of a query.

Opens a table stream into which rows can be written one at a time using db-write-row. db is either a connection object or a list of arguments that could be passed to open-database. table is the name of an existing table into which this writer will write rows. If you don't have data for all columns, use columns to indicate those that you do.

Closes a bulk writer opened by open-db-writer. Will close the associated database connection when it was created for this copier, or abort is true.

Writes row-data into the table and columns referenced by the writer. row-data is a list of Lisp objects, one for each column included when opening the writer. Arrays (the elements of which must all be the same type) will be serialized into their PostgreSQL representation before being written into the DB.

→ string

Scans string. If any character should be mapped to nothing, it eliminates that character. If any character is not printable ascii, it returns nil. If every character remaining after eliminations is printable ascii, it returns the printable-ascii string. It then calls (uax-15:normalize str form) to normalize the string based on the provided unicode form, defaulting to :nfkc.

→ string

Reads a string and removes any character that should be mapped to nothing per RFC 3454 and RFC 4013.

→ string

Reads a string and converts any character which should be mapped to a space per RFC 3454 and RFC 4013 to a space.

→ boolean

Returns t if every character in the string is printable ascii.

This is the condition type that will be used to signal virtually all database-related errors \(though in some cases socket errors may be raised when a connection fails on the IP level). For errors that you may want to catch by type, the cl-postgres-error package defines a bucket of subtypes used for specific errors. See the cl-postgres/package.lisp file for a list.

→ string

The primary human-readable error message. This should be accurate but terse (typically one line). Always present.

→ string

Detail: an optional secondary error message carrying more detail about the problem. Might run to multiple lines or NIL if none is available.

→ string

Code: the Postgresql SQLSTATE code for the error (see the Postgresql Manual Appendix A for their meaning). Not localizable. Always present.

→ string

The query that led to this error, or NIL if no query was involved.

→ condition

The condition that caused this error, or NIL when it was not caused by another condition.

→ string

For integrity-violation error, given a database-error for an integrity violation, will attempt to extract and return the constraint name (or nil if no constraint was found).

→ string

For various errors, returns the name provided by the error message (or nil if no such name was found.)

Subtype of database-error. An error of this type (or one of its subclasses) is signaled when a query is attempted with a connection object that is no longer connected, or a database connection becomes invalid during a query. Always provides a :reconnect restart, which will cause the library to make an attempt to restore the connection and re-try the query.

The following shows an example use of this feature, a way to ensure that the first connection error causes a reconnect attempt, while others pass through as normal. A variation on this theme could continue trying to reconnect, with successively longer pauses.

(defun call-with-single-reconnect (fun)
  (let ((reconnected nil))
    (handler-bind
        ((database-connection-error
          (lambda (err)
            (when (not reconnected)
              (setf reconnected t)
              (invoke-restart :reconnect)))))
      (funcall fun))))

The condition that is signalled when a notification message is received from the PostgreSQL server. This is a WARNING condition which is caught by the WAIT-FOR-NOTIFICATION function that implements synchronous waiting for notifications.

→ string

The channel string of this notification.

→ string

The payload of this notification.

→ integer

The process ID of the process that sent the notification.