Littoral

CI

Littoral is a Common Lisp port of Smalltalk’s Seaside web framework. You build an application from stateful components instead of routes and templates:

It runs on SBCL and serves through Clack (Hunchentoot by default). Dependencies are managed with ocicl.

Version 0.5.1 fixes the endpoints of an application at /, which answered 404, and serves OpenAPI descriptions of endpoints. Version 0.5.0 added many widgets (breadcrumbs, editing in place, menus, drawers, a command palette, virtual data grids, filter builders, event calendars and more charts and field kinds), presence, comment threads, notifications and an activity feed over server push, and freed Hunchentoot’s threads from open push pages. It gave every HTML element a macro, written through a nickname (h:div). Version 0.4.0 documented the API with an example for nearly every export, each run by the tests, extended the tutorial to testing, a database, signing in and translation, fixed about fifteen bugs found while checking the docs, and made clicks 45–70% faster. Version 0.3.0 put the admin and /config behind sign-in, added metrics for jobs, mail, sign-ins and the database, switched SQLite to WAL mode, and sent mail through the smtp library, with DKIM and pooled connections. Version 0.2.0 added a database with migrations, users and signing in, an admin, mail, durable jobs and an audit log. See the changelog.

(asdf:load-system :littoral)
(defpackage #:hello (:use #:cl #:littoral) (:local-nicknames (#:h #:littoral.html)))
(in-package #:hello)

(defclass counter (component)
  ((count :initform 0 :accessor count-of)))

(defmethod states ((self counter)) (list self))   ; backtrack COUNT

(defmethod render ((self counter))
  (h:h1 () (h:text (count-of self)))
  (h:anchor (:callback (lambda () (incf (count-of self)))) "++")
  (h:anchor (:callback (lambda () (decf (count-of self)))) "--"))

(register-application "/counter" 'counter)
(start :port 8080)

Getting started

You need SBCL and ocicl.

git clone https://github.com/lispnik/littoral.git
cd littoral
ocicl install      # fetches everything in ocicl.csv into ./ocicl
make test          # FiveAM suite, in-process, no sockets
make run           # examples on http://127.0.0.1:8080/examples
make tracker       # the issue tracker on http://127.0.0.1:8080/tracker

A new application

make new NAME=bookshop DIR=~/src/bookshop
cd ~/src/bookshop
make test          # its own FiveAM tests, through the fake browser
make run           # http://127.0.0.1:8080/, in development mode
make build         # bin/bookshop, a standalone executable

The project has a root component (a counter, a confirmation dialog and a form), tests that click through it, a Makefile that finds this checkout, a README and a license.

TEMPLATE=app starts something bigger:

From a REPL: (asdf:load-system :littoral/generator), then (littoral.generator:make-project "bookshop" :directory "~/src/bookshop/" :author "…" :license "MIT" :port 8080).

Other targets:

make fuzz          # random and mutated input, many cases on a random seed (SEED=… repeats one)
make test-chrome   # the examples in headless Chrome, driven from Lisp
make e2e           # littoral.js in headless Chrome (needs Node 22+)
make a11y          # WCAG checks (axe-core) over every example page
make bench         # request and snapshot timings
make build         # bin/littoral-demo, every example in one executable
make docker        # the same as the image littoral-demo
make docs          # regenerate docs/API.md from the docstrings
make clean-check   # build a fresh clone against nothing but its ocicl.csv
make tour          # the narrated video tour, recorded into tour/ (Chrome, Node, ffmpeg)
make lint          # ocicl lint

Documentation is online at https://lispnik.github.io/littoral/. New to littoral? Start with the tutorial, which builds a reading list one idea at a time, then tests it, keeps it in a database, signs readers in and translates it. The full API reference is docs/API.md.

/examples is a guide to the examples. Three of them are complete small applications:

Tracker (make tracker, then /tracker) is a complete issue tracker built to try littoral on something real. It has:

It lives in apps/tracker, in its own system, littoral/tracker.

The smaller examples include counter, multi-counter, login, todo, guess (a task), topics (bookmarkable URLs), report, ajax, upload and progress (a background job reporting through server push). The configuration application is at /config.

Concepts

Seaside Littoral
renderContentOn: html (defmethod render ((self c)) …)
html div class: 'x'; with: […] (h:div (:class "x") …)
html anchor callback: […]; with: 'go' (h:anchor (:callback (lambda () …)) "go")
html textInput on: #name of: self (h:text-input (:value … :callback (lambda (v) …)))
html render: child (render-component child)
children, states children, states generics
self call: c / self answer: x (call self c) / (answer c x)
show:onAnswer: (show self c :on-answer fn)
WATask>>go (define-flow my-task (self) …)
inform: confirm: request: chooseFrom: inform confirm request-input choose-from
decorations add-decoration, form-decoration, validate-with, …
updateRoot:, style, script update-root, style, script
jQuery load html: :on-click (ajax :callback … :update component)
updateUrl:, initialRequest: update-url, initial-request, request-extra-path
html fileUpload callback: (h:file-input (:callback (lambda (file) …)))
WABatchedList, WATableReport batched-list, report + column

HTML

The tag and brush macros are in the littoral.html package, which has a macro for every element in the HTML standard, each under the element’s own name: h:time, h:map, h:search, h:var, h:data, h:option and the rest. Several of those names are CL’s or are common in programs, so don’t :use the package; give it a local nickname, as above, and write h: before each tag:

(defpackage #:my-app
  (:use #:cl #:littoral)
  (:local-nicknames (#:h #:littoral.html)))

A tag’s first argument is its attribute plist, and you can omit it. Literal strings in a tag body are written as escaped text. To write any other value, call h:text (escaped) or h:raw (unescaped). (h:tag name …) writes an element whose name is computed, or one outside HTML, such as SVG’s.

The brushes are h:anchor, h:form, h:text-input, h:password-input, h:number-input, h:hidden-input, h:text-area, h:checkbox, h:select-list, h:radio-group, h:file-input, h:submit-button, h:cancel-button and h:button. Links, fields and the submit and cancel buttons take a :callback; an h:button submits nothing, so give it an AJAX :on-click (see “AJAX”). When a form is submitted, the field callbacks run first, in render order, and the button’s action runs after them. The plain h:a, h:input, h:select, h:textarea and h:option write those elements with no callback, for forms that post elsewhere.

An h:file-input needs its form to be (h:form (:multipart t) …). Its callback receives an h:uploaded-file, with h:file-name, h:file-content-type and h:file-contents (octets), and is not called when no file was chosen.

An h:cancel-button runs its callback and nothing else: what was typed into the form is not applied. (h:form (:default-action thunk) …) runs the thunk when the form is submitted without a button.

Rendering must not change state. Calling call, show, answer or home while a page renders signals render-phase-error; do it in a callback instead.

The page head

A component’s update-root method adds to the page’s <head>. Besides style, script and add-stylesheet, it can add meta tags and links, for link previews and icons:

(defmethod update-root ((self shop) root)
  (call-next-method)                                  ; keeps STYLE and SCRIPT
  (add-head-meta root "description" "Fresh sushi, delivered.")
  (add-head-meta root "og:image" "https://shop.example/preview.png")
  (add-head-link root "icon" "/static/icon.svg" :type "image/svg+xml"))

Open Graph keys (og:…, article:…) are written as property=, the rest as name=. A later add-head-meta for the same key replaces the earlier one, so a child component can override its parent. Values are escaped.

Bookmarkable URLs

Every page URL starts with the application’s path. Each visible component’s update-url method can then add to it:

(defmethod update-url ((self topics) url)
  (when (current-topic self) (add-to-path url (current-topic self)))
  (when (zoomed-p self) (add-parameter url "big")))

That gives URLs like /examples/topics/tasks?big&_s=…&_k=…. When someone opens the bookmark without _s and _k, a new session starts, and the root’s initial-request reads the URL back with request-extra-path and request-parameter (or request-parameter-p).

Widgets

A report is a table. Clicking a column heading sorts by that column, and clicking it again reverses the order. Cells can be rendered with your own functions, and :batch-size pages the rows. A batched-list pages any list: render its batch yourself, then render-component the list to get its page links.

(make-instance 'report
  :rows (lambda () (all-orders))
  :batch-size 20
  :columns (list (column "Order" #'order-id)
                 (column "Total" #'order-total :class "number")
                 (column "" nil :sortable nil
                         :render (lambda (row value)
                                   (declare (ignore value))
                                   (h:anchor (:callback (lambda () (open-order row))) "open")))))

Wizards

make-wizard shows a description’s fields a few at a time:

(call self (make-wizard (make-instance 'subscriber) :title "Sign up"
                        :steps '(("Account" name email) ("Plan" plan seats) ("About you" about))))

It’s an ordinary component, so a flow can call it. /examples/wizard signs someone up in three steps.

More widgets

(defmethod breadcrumb-label ((self order-page)) (format nil "Order ~D" (order-id (page-order self))))
(make-instance 'breadcrumb-trail :of orders)            ; render it above ORDERS
(make-in-place-editor order 'note :on-save #'littoral.db:db-save)

(make-instance 'dropdown-menu :label "Actions"
               :items (list (cons "Rename" (lambda () (rename self))) :separator (cons "Help" "/help")))
(define-description outing
  ((tags :type :tags :choices '("hiking" "food")) (meet :type :time) (dates :type :date-range)))

(make-instance 'lazy-panel :content (lambda () (make-instance 'sales-chart :rows (slow-query))))
(make-instance 'diff-view :old (revision-text before) :new (revision-text after))
(make-instance 'command-palette
               :commands (list (palette-command "New invoice" (lambda () (call-editor self)) :group "Invoices")
                               (palette-command "Help" "/help" :keywords '("docs" "manual"))))

(define-description party
  ((guests :type :stepper :min 1 :max 30) (volume :type :range :step 5) (rating :type :rating)
   (helpers :type :transfer :choices '("Ada" "Grace" "Alan"))
   (password :type :new-password :min-strength 2)))
(with-popover ("Shipping") (h:p () "Free over €50."))
(make-instance 'split-pane :first message-list :second message-view :position 35)
(make-instance 'data-grid :description 'shipment :rows shipments :virtual 12 :conditions t)
(pie-chart '(("Email" . 41) ("Chat" . 23) ("Phone" . 9)) :title "Support requests" :donut t)

They’re in the widgets example. Its Breadcrumbs tab has the breadcrumbs and the in-place editor, in a pantry to browse; the Forms and Inputs tabs have the field kinds, the Menus tab the menus, accordions and disclosures, the Layout tab a popover, a tooltip, a drawer and a split pane, the Big table tab ten thousand rows in a virtual grid with conditions, the Load more tab every symbol of Common Lisp a batch at a time, the Charts tab every kind of chart, the Calendar tab a week of events to drag about, and the Tools tab a lazy panel and a diff view. Its heading has a command palette over the tabs and the other examples. The report example’s chart is a lazy panel, and the wiki’s history shows each revision’s changes. /examples/monitoring charts the server’s own metrics live.

Request cycle

Littoral follows Seaside’s request cycle. A request that names callbacks goes through these steps:

  1. Restore the snapshot of the page the request came from.
  2. Run the callbacks.
  3. Snapshot the result as a new page (_k).
  4. Redirect to that new page.

A request that names no callbacks renders the page. Pages are snapshots of the visible components’ decorations, plus the slots of every object named by their states methods. Because of this, both going back and acting on an old page work the way they do in Seaside.

Dialogs over the page, and toasts

call and show put a component in place of another. show-modal puts one over the whole page instead:

(show-modal (make-instance 'confirm-dialog :message "Delete it?")
            :title "Delete" :on-answer (lambda (yes) (when yes (delete-it))))

(toast "Saved." :kind :success) shows a short message at the corner of the page, announced politely to screen readers. It appears on the next page or AJAX response, or straight away on pages with a push stream. A background job can call (toast … :session session).

Tasks and continuations

Common Lisp has no first-class continuations. define-flow therefore rewrites the flow body in continuation-passing style with cl-cont, and each call suspends the flow until the called component answers. The rewrite has some limits:

Outside a flow, call shows the component and returns immediately. Use show with :on-answer there.

Isolation

After an irreversible step, such as placing an order, the user shouldn’t be able to go back into the pages that led to it and do it again. Seaside’s isolate: prevents this; littoral splits it in two:

(let ((isolation (begin-isolation)))
  … the checkout's calls …
  (place-order …)
  (end-isolation isolation))     ; the checkout pages are gone

Going back to one of those pages then shows the session as it is now. littoral.js also reloads any page Chrome restores from its back/forward cache, because Chrome does this even for no-store pages.

AJAX

To make a component re-renderable in place, mix updatable into its class. Then give an element an :on-click, :on-change or :on-input attribute whose value is (ajax :callback thunk :update component). To re-render on a timer instead, give an element :periodical (periodical seconds :update component). static/littoral.js is small and has no dependencies; it posts the request and swaps the HTML.

ajax takes more options:

(h:button (:on-click (ajax :value "this.dataset.size"
                         :callback (lambda (size) (price-for size))
                         :on-complete "document.getElementById('price').textContent = value"))
  "Price it")

After every update, littoral.js fires a littoral:updated event on document.

Server push

Components can be re-rendered on open pages by the server, without polling (this is Seaside’s Comet, built on server-sent events). A component lists the channels it listens to in its subscriptions method. publish, called from any thread, re-renders every visible subscriber on every open page:

(defvar *news* (make-channel "news"))
(defmethod subscriptions ((self headlines)) (list *news*))   ; headlines is updatable
… (publish *news*) …

(notify component session) re-renders one component for one session. Use it from a background job, and capture *session* in the callback that starts the job. Wrap changes made from another thread in (with-session (session) …) so they hold the session’s lock. A page opens its event stream only if something on it subscribes. Each open page holds one connection, but not a thread: on Hunchentoot, one thread serves every open stream, and on Woo the event loops do. The chat and progress examples use this.

Over a WebSocket

Load littoral/websocket and register the application with :websockets t. Pages that listen for pushes then open one WebSocket instead of an event stream, and their AJAX requests travel over it too, saving a request per click:

(asdf:load-system :littoral/websocket)
(register-application "/chat" 'chat :websockets t)

Nothing else changes: callbacks, publish and notify behave as before. Rendering for pushes happens on a small pool of worker threads, never in the thread that called publish. If the socket can’t open (a proxy that refuses the upgrade, say), the page falls back to server-sent events and plain requests. On Woo, where littoral/woo already serves push from the event loops, pages keep server-sent events. The chat example uses this.

A socket opens only from a page of the same site: its Origin must match the host, or be in littoral.websocket:*allowed-origins*. Sockets count against the event-stream limits. Messages are capped at *max-message-size* (1 MB), malformed messages are ignored, and in deployment mode a failing request answers “The request failed.” rather than the error.

Who’s here

A presence is a place pages can be in, such as one wiki page, and find-presence gives the one for a key. A presence-list shows who has a page in its presence open: “3 people here”, then each person by name with what they’re doing, you first, and the guests counted. Showing a presence list puts its page in the presence, and it redraws by server push as people come and go:

(make-instance 'presence-list :presence (find-presence "chat room"))

(defmethod presences ((self page-editor)) (list (find-presence (page-key self))))
(defmethod presence-activity ((self page-editor)) "editing")

The chat lists who’s in the room by nickname, and each wiki page shows who’s reading it and who’s editing it.

Comment threads

A live-comment-thread shows the comments on something, such as a wiki page or an order, and a box for adding one:

(make-instance 'live-comment-thread :thread (lambda () (format nil "wiki:~A" (page-name self)))
                                    :markdown t :ask-name t)

The comments use time-ago, which writes a time as “3 minutes ago” and has littoral.js keep the words up to date in the page’s language while it stays open. The notification bell and the activity feed use it too. Each wiki page has a comment thread.

Files people upload

littoral/storage keeps uploads on disk or in S3, and gives descriptions file fields:

(setf littoral.storage:*storage*
      (littoral.storage:make-disk-storage #p"/var/lib/app/files/"))   ; served at /files/
;; or (littoral.storage:make-s3-storage :bucket "app-files" :region "eu-west-1"
;;                                      :access-key … :secret-key …)

(define-description photo
  ((caption :required t)
   (picture :type :image :required t :max-size 8000000)    ; with a thumbnail
   (attachment :type :file)))

The gallery example (/examples/gallery, system littoral/storage-demo) uploads photos with an optional attachment.

JSON endpoints

An application can answer JSON beside its pages, for mobile apps, scripts and webhooks. These endpoints belong to the application registered at /notes, so they answer /notes/api/notes/…. find-note and create-note stand for your own code, and the second needs littoral/auth:

(define-endpoint "/notes" :get "/api/notes/:id" (id)
  (let ((note (find-note id)))
    (unless note (endpoint-error 404 "No note ~A." id))
    (list :id (note-id note) :title (note-title note))))

(define-endpoint "/notes" :post "/api/notes" ()
  (let ((user (littoral.auth:require-endpoint-user)))        ; 401 without one
    (values (create-note user (gethash "title" (endpoint-body))) 201)))

OpenAPI. serve-openapi answers GET /openapi.json with an OpenAPI 3.1 description of an application’s endpoints, which Swagger UI, Postman and client generators read. It’s made from what the endpoints say about themselves: a docstring’s first sentence is the summary, and options before the body give the tags, responses, request body and query parameters. Path parameters come from the pattern, and every endpoint is described as answering errors as {"error": message}. With littoral/auth loaded, the description says requests may carry a bearer token.

(define-endpoint "/notes" :get "/api/notes" ()
  "The notes, newest first."
  :tags ("notes") :query (("q" "Words to find")) :responses ((200 "The notes") (401 "Not signed in"))
  (find-notes (request-parameter "q")))

(serve-openapi "/notes" :title "Notes" :version "1" :license "MIT")

find-notes stands for your own code, and request-parameter reads the query. openapi-description gives the description as a hash table, to change or save.

/examples/api calls its own endpoints from the page. The members example answers GET /examples/members/api/me and makes API tokens.

Background jobs

submit-job runs work on a pool of threads (*job-workers*, 4) and keeps the page up to date while it does. This runs in a callback. resize-photo stands for the work, and the job-view that watch-job returns must be rendered as a child:

(let ((job (submit-job (lambda ()
                         (dotimes (i 10)
                           (resize-photo i)
                           (job-progress (/ (1+ i) 10) "Resizing…")))
                       :name "Resize photos" :attempts 3 :backoff 2)))
  (watch-job job (make-instance 'job-view :job job)))

Jobs live in memory, so a restart forgets them; for work that must not be lost, use durable jobs (below). The progress example runs one job that succeeds and one that fails once and is retried.

Durable jobs

littoral/jobs keeps jobs in the database, in a job_queue table, so they survive restarts and any process sharing the database can run them. Here digest-mail is a mail of your own (see “Sending mail”) and user one of littoral/auth’s users:

(littoral.jobs:define-job (send-digest :attempts 5 :retry-seconds 60) (user-id)
  (littoral.mail:queue-mail (digest-mail (littoral.auth:find-user-by-id user-id))))

(littoral.jobs:define-job (tidy-up :every (littoral.jobs:daily-at 3)) ()   ; every night at three
  (littoral.jobs:purge-durable-jobs))

(littoral.db:connect-database :sqlite3 :database-name "app.db")
(littoral.jobs:create-job-tables)
(littoral.jobs:start-job-runner :threads 2)         ; on the database just connected

(littoral.jobs:enqueue-job (list 'send-digest (littoral.auth:user-id user)) :in 600)

/examples/jobs (system littoral/members-demo) has a report that shows its progress in a durable-job-view, a job that fails twice and then succeeds, one that always fails, and a job that sends mail ten seconds later through the mail demo’s outbox. Generated apps start a runner and tidy up every night.

Sending mail

littoral/mail writes MIME messages, sends them over SMTP, makes them from templates, and keeps them in an outbox in the database until they’ve gone. It needs a database connected (see “Keeping data in a database”). order-email, order-number and tracking-url stand for your own code:

(setf littoral.mail:*mailer* (littoral.mail:smtp-mailer-from-url "smtp://app:secret@smtp.example.org:587")
      littoral.mail:*mail-from* "Bookshop <hello@example.org>")

(littoral.mail:define-mail order-shipped (order)
  :to (order-email order)
  :subject (translate "Your order ~A is on its way" (order-number order))
  :html ((h:p () "It left the warehouse today.")
         (h:p () (h:anchor (:href (tracking-url order)) "Track it"))))

(littoral.mail:create-mail-tables)
(littoral.mail:start-mail-delivery)   ; a job runner sends the outbox
;; …and in an action:
(littoral.mail:queue-mail (order-shipped order))

Live redefinition

In development mode, recompile a component’s render method in Emacs (C-c C-c) and every open page showing that component redraws itself within a couple of seconds, with its state intact. A watcher notices when methods of render, style, script, update-root, children or render-decoration are redefined. Pages showing an instance of that class then reload, and because each page’s URL names its saved state, reloading keeps that state. Pages with a push stream hear about it through the stream; the others poll a cheap endpoint, so live reloading holds no connection (and on Hunchentoot no thread) open.

For changes the watcher can’t see, such as a helper function or a stylesheet, call (reload-pages). Set *live-reload* to nil to turn it off. Deployment-mode pages are never affected.

Browser code in Lisp (Parenscript)

littoral/parenscript lets a component’s browser behaviour be written in Parenscript, next to its render method:

(asdf:load-system :littoral/parenscript)
(defpackage #:my-app
  (:use #:cl #:littoral #:parenscript #:littoral.parenscript)
  (:local-nicknames (#:h #:littoral.html))
  (:shadowing-import-from #:littoral #:call))         ; Parenscript exports CALL too
(in-package #:my-app)

(defclass buttons (updatable component) ())

(defmethod render ((self buttons))
  (h:button (:on-click (in-browser (ps (chain this class-list (toggle "on")))))
    "Toggle")                                        ; runs in the browser only
  (let ((shout (client-callback (lambda (text) (string-upcase text)) :update self)))
    (h:button (:on-click (in-browser (ps (chain littoral (call (lisp shout) "hi")
                                              (then (lambda (answer) (alert answer)))))))
      "Ask the server")))

(define-script buttons (self)
  (chain console (log "ready")))

Development tools

Applications in :development mode, the default, end each page with a toolbar:

Each halo has three buttons:

When Emacs is connected through SLIME/Swank, the code view also has an edit button that opens that definition in Emacs. Set *source-editor* to a function of a pathname and a character position to use another editor.

The toolbar reports the last action’s callback time, snapshot time and size, and the render time. Profile adds a table of every component’s inclusive render time, indented by nesting. The session browser shows each session’s age, idle time, pages kept, the objects in its newest snapshot, and the snapshot entries held across all its pages.

/examples/devtools is a page to try them on.

The debugger. When a callback or a render signals an error in a development-mode application, the page becomes a debugger:

An AJAX or WebSocket request that fails shows the same page in place of the one you were on. Deployment mode shows a plain error page with no detail. Set *debug-errors* to t to have errors land in the Lisp debugger instead, restarts and all.

/config lists the registered applications. From there you can add, remove and configure them and browse their sessions. The configurable settings are title, root class, mode, session timeout, pages kept, session limit, cookie sessions, the expiry notice, stylesheets, scripts and basic-auth credentials. Use (configure-admin :user "u" :password "p") to put it behind HTTP basic auth, or, with littoral/auth loaded, (littoral.auth:configure-signed-in-admin :role :admin) to let users with a role sign in to it.

Configuration

(register-application "/shop" 'shop-root
  :title "Shop"
  :mode :deployment          ; no toolbar or halos
  :session-timeout 1800      ; seconds idle
  :max-continuations 50      ; pages the back button can reach
  :cookie-sessions t         ; session key in a cookie, not the URL
  :stylesheets '("/static/shop.css")
  :credentials '("user" . "password")
  :max-sessions 10000        ; the least recently used are evicted
  :expired-notice 'session-expired-notice   ; say so when a session has gone
  :error-handler (lambda (condition)        ; a component or HTML string
                   (make-instance 'oops :condition condition)))

start also runs a thread that reaps idle sessions every minute (start-reaper, stop-reaper, reap-all-sessions).

Every response carries Referrer-Policy: same-origin, so session keys in URLs never leak to other sites, plus X-Content-Type-Options: nosniff and X-Frame-Options: SAMEORIGIN. Session cookies get Secure when the request came over HTTPS, either directly or through a proxy that sets X-Forwarded-Proto: https, and HTTPS responses carry Strict-Transport-Security: max-age=31536000 (*hsts-max-age*, NIL to leave it out), so browsers that have seen the site over HTTPS never try plain HTTP.

To keep configuration across restarts, start with a file:

(start :port 8080 :configuration-file "littoral.conf")

Any applications saved in the file are configured first, and every change made in /config is written back to it. The file is plain Lisp, one plist per application, and is created with mode 600 because it can hold basic-auth credentials. Classes are stored by name, so an application whose system isn’t loaded is skipped with a warning. Only the settings /config edits, and the languages, are saved: error handlers, the :around-… hooks, :websockets, :content-security-policy and :max-request-size aren’t. From code, use save-configuration, load-configuration, and configure-application; the last changes an application’s settings without dropping its sessions.

(make-lack-app) returns a plain Lack application you can mount into a larger Lack/Clack stack. Lack’s :mount middleware strips the prefix without setting :script-name, so pass the prefix yourself: (:mount "/apps" (make-lack-app :prefix "/apps")). A :script-name set by the server or proxy is honoured as well. Set *debug-errors* to enter the debugger on errors instead of rendering an error page.

Translations

Write strings in English and pass them through translate. A language’s catalogue gives the translations, and a string it lacks shows as written:

(define-translations "fr"
  ("Add a contact" "Ajouter un contact")
  ("Remove ~A?" "Supprimer ~A ?")
  ("~D contact" "~D contact" "~D contacts"))        ; one form per plural form

(defmethod render ((self contacts-app))
  (h:h1 () (translate "Contacts"))                    ; written as text, like a literal
  (h:p () (translate-plural (length (contacts self)) "~D contact" "~D contacts"))
  (h:anchor (:callback …) (translate "Add a contact")))

(register-application "/contacts" 'contacts-app :languages '("fr" "de"))

Each session has a language. When a session starts, it takes the first of the application’s :language (by default “en”) and :languages that the browser’s Accept-Language asks for. set-language changes it from a callback, and the language-chooser component links to each language, named in its own words. The page’s lang attribute follows the session’s language.

The contacts example is translated into French and German.

Keeping data in a database

littoral/db stores described objects in SQL through cl-dbi. It’s tested with SQLite and PostgreSQL. A table is a description plus a name:

(defclass contact (littoral.db:persistent)      ; adds id and version
  ((name :initarg :name :initform nil) (email :initarg :email :initform nil)))
(define-description contact ((name :required t) (email :type :email)))
(littoral.db:define-table contact)

(littoral.db:connect-database :sqlite3 :database-name "app.db")
(littoral.db:create-table 'contact)
(littoral.db:db-save (make-instance 'contact :name "Ada" :email "ada@example.org"))
(littoral.db:db-select 'contact :where "name LIKE ?" :params '("A%") :order-by "name" :limit 20)

Schema migrations

create-table is fine for trying things out. Once a database holds data you care about, keep its schema in numbered SQL files and let migrate apply the new ones:

migrations/001-contacts.sql
migrations/002-contacts-get-a-phone.sql
migrations/003-search.sqlite.sql          ; 003 for SQLite…
migrations/003-search.postgres.sql        ; …and for PostgreSQL

(littoral.db:define-migrations :my-app (asdf:system-relative-pathname :my-app "migrations/"))
(littoral.db:migrate :my-app)              ; applies 002 and 003 if they're new; returns what it applied
(littoral.db:migration-status :my-app)     ; each file, and when it was applied

Generated applications keep their schema this way: make migrate, or bin/app migrate on a server, and starting the app migrates too.

Full-text search

Give define-table the fields to index, and db-search finds objects by their words, the best matches first. For an article class with title and body fields:

(littoral.db:define-table article :search (title body))
(littoral.db:db-search 'article "macro writ")      ; words starting so, all of them
(littoral.db:highlight-matches (article-body a) "macro writ")   ; HTML, the words <mark>ed

/examples/search (system littoral/search-demo) searches the documentation of every documented symbol in COMMON-LISP, as you type.

A generated admin

littoral/admin builds a whole administration interface from descriptions and tables. Given classes with tables (define-table), here project and issue:

(littoral.admin:register-admin "/admin" '(project issue)
  :database '(:sqlite3 :database-name "app.db")
  :role :admin)                     ; users with the admin role sign in

For each class it gives:

A command palette in its header (Ctrl-K) goes to any table, or straight to a new record in one. Each request’s changes run in one transaction. :database gives the admin its own database (through the :around-request hook).

Beside the tables it has two pages of its own:

Who may use it:

Users and signing in

littoral/auth adds users, signing in, roles, permissions and password reset, all kept in the database through littoral/db (SQLite or PostgreSQL). The members example (/examples/members, system littoral/members-demo) shows all of it:

/examples/accounts is the same page signing in against an existing table.

(littoral.db:connect-database :sqlite3 :database-name "app.db")
(littoral.auth:create-auth-tables)
(littoral.auth:add-user "ada" "ada@example.org" "a long passphrase" :roles '(:admin))

(defclass app (littoral.auth:auth-root component) ())        ; answers /reset links
(defclass reports (littoral.auth:restricted component) ())   ; signed-in users only…
(defmethod littoral.auth:required-role ((self reports)) :admin)   ; …with the admin role

Your own users table

Users come from *user-store*. By default that’s Littoral’s users table. To sign in against a table you already have, describe its columns:

(defparameter *accounts*
  (littoral.auth:make-sql-user-store
   :table "accounts" :id "account_id" :name "login" :email "email_address"
   :password "pw_hash" :active "enabled"                ; :active is optional
   :roles-query "SELECT r.name FROM roles r JOIN account_roles ar ON ar.role_id = r.id
                 WHERE ar.account_id = ?"               ; or :roles-column "roles"
   :writable t))                                        ; password changes write pw_hash

(littoral.auth:create-auth-tables :users nil)           ; only Littoral's own tables
(register-application "/app" 'app :around-request (littoral.auth:using-user-store *accounts*))

Roles and permissions

Roles and permissions are kept in two tables, keyed by name, so they work with any store:

(littoral.auth:grant-role user :editor)
(littoral.auth:grant-permission :editor :edit-posts)
(littoral.auth:has-permission-p :edit-posts)       ; for the current user
(littoral.auth:require-permission :edit-posts)     ; in a callback: 403 otherwise
(defmethod littoral.auth:required-permission ((self post-editor)) :edit-posts)

The audit log

littoral/audit keeps an audit_log table recording who did what, and when. create-auth-tables creates it, and from then on these are recorded with no further code:

Applications record their own events:

(littoral.audit:audit "invoice.sent" :subject "invoice:42" :subject-label "Invoice 42"
                                     :details '(:to "ada@example.org"))
(littoral.audit:audit-events :action "sign-in" :since (- (get-universal-time) 86400))

An activity-feed shows what’s happened lately, from the log: who did what to what, and how long ago, newest first, a batch at a time with a button for older events. Events recorded in this process reach open pages as they happen, through *audit-channel*, published once their transaction commits. :action, :actor and :subject narrow it. The words come from *activity-phrases* (signed in, created, changed and so on, translated), or from :describe, a function of an event. Like audit-log-view, it shows whatever it’s given, so put it where those who see it may know these things.

The members demo shows ada the recent activity and the log, under Admins.

Notifications

littoral/notifications keeps messages for users until they read them, in a notifications table. A toast is gone once seen; a notification waits, on whatever page and in whatever browser the user comes back with, across restarts.

(littoral.notifications:create-notification-tables)
(littoral.notifications:notify-user user "Your export is ready." :url "/exports/7")
(make-instance 'littoral.notifications:notification-bell)   ; in the page's header

In the members demo, the bell is beside the signed-in user’s name: ada making bob an editor notifies him, and “Send me a notification in three seconds” sends one from a durable job.

PostgreSQL

littoral/db and littoral/auth are tested on SQLite and on PostgreSQL. Connect with (littoral.db:connect-database :postgres :database-name "app" :host "localhost" :username "app" :password "…"); the driver is pure Lisp. make test-postgres LITTORAL_TEST_DATABASE=postgres://user:password@host:5432/db runs the database tests against one, as CI does.

Testing your application

littoral/test is the fake browser littoral’s own test suite uses. It calls your application’s Lack handler directly, with no sockets and no real browser, so tests are fast and need nothing installed. It keeps cookies, follows redirects, and reads links, fields and buttons out of the HTML. It posts forms (multipart too), AJAX requests and event streams the way a browser running littoral.js would. It works with any test framework; with FiveAM:

(defpackage #:my-app/tests (:use #:cl #:fiveam #:littoral #:littoral.test))
(in-package #:my-app/tests)

(test adding-to-the-list
  (with-fresh-applications (("/todo" 'my-app:todo-list :mode :deployment))
    (let ((b (make-instance 'browser)))
      (visit b "/todo")
      (fill-in b "new-title" "Buy milk")      ; by the field's DOM id
      (press b "Add")                         ; by the button's label
      (is (has-text-p b "Buy milk"))
      (let ((before (browser-url b)))
        (click b "remove")                    ; by the link's text
        (back-to b before)                    ; the back button
        (is (has-text-p b "Buy milk"))))))

select-option chooses from a drop-down and select-options several items from a list box. with-fresh-applications gives a test its own registry of applications. ajax-request and ajax-specs exercise AJAX; open-stream, sink-text and wait-for exercise server push. The API reference lists everything.

In a real browser

The fake browser doesn’t run JavaScript. For drag and drop, the back button with Chrome’s caches, or passkeys, littoral/browser-test drives headless Chrome from Lisp over the DevTools protocol, so the tests stay in FiveAM next to the rest:

(defpackage #:my-app/chrome-tests (:use #:cl #:fiveam #:littoral.browser-test))
(in-package #:my-app/chrome-tests)

(test passkeys
  (with-chrome (page :base "http://localhost:8080")   ; a fresh profile each time
    (add-virtual-authenticator page)                  ; a passkey device that always says yes
    (visit page "/examples/members")
    (click-text page "Sign in")
    (type-into page "#sign-in-name" "ada")            ; real key events, by CSS selector
    (type-into page "#sign-in-password" "correct horse battery")
    (click-text page "Sign in" :among "button")
    (click-text page "Add a passkey")
    (wait-until page "document.body.innerText.includes('Passkey 1')")
    (is (search "Passkey 1" (text page)))))

Accessibility

make a11y runs axe-core’s WCAG 2 A and AA rules over every example page, and over the states reached by acting on them, in headless Chrome. CI runs it on every push. What littoral does for you:

Security

Deployment and scale

The deployment guide covers the whole process:

Performance. These figures were measured over real HTTP against bench/load-server.lisp, using Hunchentoot on an Apple-silicon Mac with SBCL 2.6.8. make load measures the clicks, the memory per session, and 100 and 300 push streams (bench/load.mjs). The ab rows are ApacheBench against the same server, and the push row is bench/load.mjs streams with STREAMS=600, with start’s default of 100 threads:

Measurement Result
clicks (action, redirect, page), 32 users ~8,400 /s on the counter, ~7,800 /s in the store; p99 ~13 ms
rendering a session’s page (ab, 32 connections) ~10,900 requests/s
static files (ab) ~15,000 requests/s
memory per session, back-button pages included ~10 KB (counter), ~25 KB (store)
open push streams 600 at once: 600 of 600 updates delivered, 22 threads, 121 MB for the whole server

make bench measures the same cycle in-process, without the network: about 60 µs per click on the counter.

Against Seaside on Pharo 12, measured the same way (see “Littoral and Seaside compared” below):

Requests in one session are serialised by its lock, and requests in different sessions run in parallel. The test suite runs concurrent sessions, many requests against one page, and concurrent AJAX.

Threads. Hunchentoot gives every connection its own thread while it makes requests, and start allows 100 by default (:max-threads). Idle keep-alive connections occupy a thread too. Open pages with server push don’t: once a page’s event stream has begun, its connection leaves Hunchentoot’s thread for littoral’s push hub. One thread watches every stream, and up to *push-render-threads* (4) more render and write their updates. On Linux, 10,000 open push pages took the server from 4 threads to 5, and their sessions and streams added 40 MB to its heap (76 MB to 116 MB); every page received its update. That was measured before the render threads were added, which add up to four more threads while pages are open.

A closed page’s stream ends within about one keep-alive interval (*keepalive-seconds*, 5 s by default).

Hunchentoot and Woo compared. An optional system, littoral/woo, runs littoral on Woo, an event-loop server built on libev. It is faster at ordinary requests, and it serves server push from its event loops: each open push page is two libev watchers. bench/compare.sh hunchentoot and bench/compare.sh woo run the same measurements against each, on an Apple-silicon Mac (8 cores) with SBCL 2.6.8. Woo ran with 4 event-loop workers.

  Hunchentoot Woo Woo vs Hunchentoot
static files, keep-alive (ab, 64 connections) 10,400 req/s 29,000 req/s 2.8×
page render, keep-alive 8,000 req/s 18,000 req/s 2.2×
page render, a new connection per request 9,700 req/s 19,300 req/s 2.0×
counter clicks, 64 users from 4 clients 6,700 /s 10,200 /s 1.5×
store clicks, same setup 5,200 /s 6,500 /s 1.2×
p99 click latency, 64 users 27 ms 22 ms 20% lower
one user clicking 3,070 /s 3,940 /s 1.3×
memory per session (counter / store) 10 KB / 25 KB 8.5 KB / 24 KB about the same
idle server memory 90 MB 87 MB about the same
600 open push pages: threads 22 7  
600 open push pages: memory 121 MB 99 MB about the same
600 open push pages: updates delivered 600 of 600 600 of 600  
most push pages tried about 1,000 on macOS (the select(2) limit); 10,000 on Linux 6,000 on macOS, at about 100 MB  

To run on Woo, install the libev C library (brew install libev, or your system’s package), then:

(asdf:load-system :littoral/woo)
(littoral:start :port 8080 :server :woo :workers 4)

CI runs the browser checks on both servers.

Littoral and Seaside compared. bench/compare.sh seaside runs the same measurements against Seaside on Pharo 12, built by bench/seaside/setup.sh:

Seaside runs with Zinc’s limits raised: 1,000 connections, a backlog of 128, and Pharo’s socket semaphore table at its maximum. Without that, it refuses connections at 64 concurrent users.

A Pharo image runs Smalltalk on one OS thread, so the first column of littoral is Woo with a single event loop, which is the per-core comparison. The setup was an Apple M3 with 8 cores, SBCL 2.6.8, Seaside master (0c52094) on Pharo 12.0, and the median of six runs over two days:

  Seaside littoral, Woo, 1 loop littoral, Hunchentoot littoral, Woo, 4 loops
static file, keep-alive (ab, 64 connections) 540 req/s 11,800 req/s 10,400 req/s 29,000 req/s
page render, keep-alive 7,600 req/s 14,200 req/s 8,000 req/s 18,000 req/s
page render, a new connection per request 7,000 req/s 14,000 req/s 9,700 req/s 19,300 req/s
counter clicks, 64 users from 4 clients 2,770 /s 7,000 /s 6,700 /s 10,200 /s
store clicks, same setup 1,750 /s 4,760 /s 5,200 /s 6,500 /s
p99 click latency, 64 users 83 ms 20 ms 27 ms 22 ms
one user clicking (rate, median latency) 2,330 /s, 0.4 ms 4,120 /s, 0.2 ms 3,070 /s, 0.3 ms 3,940 /s, 0.2 ms
memory per session (counter / store) 20 KB / 55 KB 8.5 KB / 24 KB 10 KB / 25 KB 8.5 KB / 24 KB
idle server memory 92 MB 72 MB 90 MB 87 MB

State that the back button sees. Snapshots copy the slots of the objects states names, but share the slots’ values. A list changed in place therefore changes in every snapshot. Either replace such values ((setf (items self) (append …))) or name the object as (deep object) in states. A deep entry copies the object’s lists, vectors, strings and hash tables. Other instances are still shared, and cycles are kept. In development mode, Littoral notices such a value changed in place by a page’s actions: it warns (state-changed-in-place, once per slot) and the toolbar names the slot. A slot changed in place on purpose, such as a cache, can be listed by shared-state-slots.

Several processes. Sessions live in the memory of the process that made them. To run several processes behind one site, give each one an instance id; it prefixes every session key:

(start :port 8081 :instance-id "a")   ; session keys look like a.Xq3…

Then route by that prefix. With nginx, for example:

map $arg__s $littoral_backend { ~^a\. 127.0.0.1:8081; ~^b\. 127.0.0.1:8082; default 127.0.0.1:8081; }
location / { proxy_pass http://$littoral_backend; proxy_buffering off; }

With cookie sessions, route on the cookie instead. Turning proxy_buffering off, or sending X-Accel-Buffering: no (littoral does this on its event streams), keeps server push working.

Health, metrics and logs.

Restarts. Sessions don’t survive a restart. A session is a graph of live objects, the closures its pages’ callbacks hold, and cl-cont continuations for flows in progress. None of these can be serialised, which is also true of Seaside outside image-based persistence. Configuration does survive (see :configuration-file). Keep data that must last in your own store, not in components. Use :expired-notice to tell people when their session is gone.

A deploy needn’t restart: run two instances and drain the old one (drain), which goes on serving its sessions while the new version takes everyone else. The deployment guide has the unit files, a script and proxy configurations.

What a restart costs is small. A page whose session is gone starts a new one when it’s next loaded. Its AJAX requests, event stream, WebSocket and live-redefinition checks are told to reload the page rather than being given a new session they can’t use. Sign-ins are kept in the database, so an auth-root application signs the user straight back in.

License

MIT