Setting up EventStoreDB with Lets Encrypt certificates


I want to set up EventStoreDB (eventstore-oss=20.6.1-2) on a cloud instance to play with, and I want it to be secured. It took some time to get the certs sorted out, but I’m documenting it here now.

I’m using Cloudflare to provide DNS. I want the instance to run at prod.esdb.<example.org>.

Step 1: Generate the certificates

I’m using certonly with the Cloudflare DNS provider. I followed these instructions.

sudo certbot certonly \
  --dns-cloudflare \
  --dns-cloudflare-credentials ~/.config/certbot/cloudflare.ini \
  -d prod.esdb.<example.org> -d '*.prod.esdb.<example.org>'

Now we get a few files in /etc/letsencrypt/live/<example.org>/: cert.pem, chain.pem, fullchain.pem, privkey.pem, and README. Of these, we only need cert.pem and privkey.pem.

Step 2: Copying, converting and specifying the certificates

Place cert.pem and privkey.pem in /etc/eventstore/certs, convert the private key, and tighten the permissions:

# mkdir /etc/eventstore/certs
# cd /etc/eventstore/certs
# cat > cert.pem
...
# cat > privkey.pem
...
# openssl rsa -in privkey.pem -out privkey.key
# chown eventstore:evenstore *
# chmod 600 *

If you don’t don’t do the last step to convert the private key, you’ll see errors like the following in journalctl when you systemctl start eventstore:

 [50108, 1,10:24:37.714,INF] TLS is enabled on at least one TCP/HTTP interface - a certificate is required to run EventStoreDB.
 [50108, 1,10:24:37.719,FTL] Host terminated unexpectedly.
 System.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.
    at System.Convert.FromBase64_ComputeResultLength(Char* inputPtr, Int32 inputLength)
    at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)
    at System.Convert.FromBase64String(String s)
    at EventStore.Core.CertificateLoader.FromFile(String certificatePath, String privateKeyPath, String password) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Core/CertificateLoader.cs:line 37
    at EventStore.Core.VNodeBuilder.WithServerCertificateFromFile(String certificatePath, String privateKeyPath, String password) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Core/VNodeBuilder.cs:line 705
    at EventStore.ClusterNode.ClusterVNodeHostedService.BuildNode(ClusterNodeOptions options, Func`1 loadConfigFunc) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.ClusterNode/ClusterVNodeHostedService.cs:line 163
    at EventStore.ClusterNode.ClusterVNodeHostedService.Create(ClusterNodeOptions opts) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.ClusterNode/ClusterVNodeHostedService.cs:line 144
    at EventStore.Core.EventStoreHostedService`1..ctor(String[] args) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Core/EventStoreHostedService.cs:line 45
    at EventStore.ClusterNode.ClusterVNodeHostedService..ctor(String[] args) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.ClusterNode/ClusterVNodeHostedService.cs:line 35
    at EventStore.ClusterNode.Program.Main(String[] args) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.ClusterNode/Program.cs:line 22
eventstore.service: Main process exited, code=exited, status=1/FAILURE
eventstore.service: Failed with result 'exit-code'.

Use an eventstore.conf similar to this:

---
# Paths
Db: /var/lib/eventstore
Index: /var/lib/eventstore/index
Log: /var/log/eventstore

# Certificates configuration
CertificateFile: /etc/eventstore/certs/cert.pem
CertificatePrivateKeyFile: /etc/eventstore/certs/privkey.key
TrustedRootCertificatesPath: /etc/ssl/certs
CertificateReservedNodeCommonName: prod.esdb.<example.org>

# Network configuration for GCP
# The machine is in a VPC but reachable from prod.esdb.<example.org>
# Static GCP VPC internal IP - set in GCP console
IntIp: 10.142.0.2
ExtIp: 10.142.0.2
IntHostAdvertiseAs: prod.esdb.<example.org>
ExtHostAdvertiseAs: prod.esdb.<example.org>

HttpPort: 2113
IntTcpPort: 1112
EnableExternalTcp: false
EnableAtomPubOverHTTP: true

# Projections configuration
RunProjections: All

Step 3: Starting EventStoreDB

$ sudo systemctl start eventstore

Then check up on it after 10 seconds or so:

$ sudo systemctl status eventstore

Since AtomPub is enabled, we can access the machine remotely on https://prod.esdb.<example.org>:2113/.

Log in with admin:changeit and change the default passwords!

Step 4: Cycling the certificates

certbot will set up a cron or systemd timer to replace or regenerate the certificates. You could automate providing them to EventStoreDB, but I’m just using a manual script for now. To automate this, you could just put it in root's crontab, but remember that it will restart ESDB automatically. Or you could use a certbot hook.

#!/bin/bash
cd /etc/eventstore/certs
sudo cp /etc/letsencrypt/live/prod.esdb.<example.org>/cert.pem /etc/letsencrypt/live/prod.esdb.<example.org>/privkey.pem .
openssl rsa -in privkey.pem -out privkey.key
chown eventstore:evenstore *
chmod 600 *
systemctl restart eventstore

Step 5: Connection with the Node.js client

import {
  EventData,
  EventStoreConnection,
  writeEventsToStream,
} from '@eventstore/db-client'

const connection = EventStoreConnection.builder()
  .secure()
  .defaultCredentials({
    username: 'user-you-create',
    password: 'password',
  })
  .singleNodeConnection('prod.esdb.<example.org>:2113')

;(async () => {
  await writeEventsToStream('test')
    .send(
      EventData.json('Test', {
        hello: 'world',
      }).build()
    )
    .execute(connection)
    .then((x) => console.log('success', x))
    .catch((err) => console.log('error', err))
})()

Compiling and running this, you should see an event in the test stream at https://prod.esdb.<example.org>:2113/web/index.html#/streams/test/1.