For the ICT officer

Administrator roadmap

Installing GrantLine, keeping it running, and getting it back when something breaks. Nobody needs this page to use the platform. Applicants, reviewers and grants staff want the user guide.

On this page

Start

What you are running

One Docker Compose stack on one server. The application, the database and the documents all stay on that machine, and nothing calls into the Ministry’s internal network. The full detail, every setting included, is in docs/deploy/DEPLOYMENT-DOCKER.md in the handover pack. This page is the order to do things in.

Where: One Ubuntu server, one Docker Compose stack, in /opt/grantline.
Coming in: 443 and 80, on a DMZ install only. 22 from the admin network.
Going out: The mail relay. Nothing else.
Updates: Signed files copied onto the server. Nothing is downloaded.

Roadmap

Eight steps

Steps 1 to 4 happen once. Steps 5 to 8 are the job from then on.

Before installation

Agree with the supplier

The supplier builds the release with some of these answers inside it. Changing one later means a new release, so settle them first.

1.1
Choose the public hostname and get its DNS record created.
A name, never an IP address. You type it once in the set-up wizard, and it is locked after that.
1.2
Decide: closed network or DMZ.
Closed means only the Ministry network reaches the server. DMZ means applicants reach it from the internet, and INTERNAL_NETWORKSkeeps the staff pages to the Ministry’s own address ranges.
1.3
Choose the certificate.
Caddy's own certificate authority, a certificate the Ministry issues, or Let's Encrypt. Let's Encrypt works only when ports 80 and 443 are open to the internet.
1.4
Get the mail relay details from the Ministry's IT. Five of them: host, port, user, password, sender address.
Without them the system runs and sends nothing. To an applicant that looks the same as losing their application.
1.5
Choose where backups go, and check this server can reach it.
A disk or share other than the system disk. Each backup holds a copy of .env, so protect it like the keys.
1.6
Prepare the server: Ubuntu 22.04 or 24.04, Docker Engine 24 or later with the Compose plugin.
8 vCPU, 16 GB of memory, 100 GB SSD at the least. The database will not start on fewer than 6 CPUs.
1.7
Set the clock and time zone, with NTP on.
Deadlines, reminders and the audit log all use it. A server on the wrong zone closes rounds at the wrong hour.
```
sudo timedatectl set-timezone America/St_Lucia
```
1.8
Set the firewall.
DMZ: open 443 and 80 inbound. Closed: nothing inbound from the internet. Either way, 22 from the admin network only, and outbound only to the mail relay.

Install

Day one, about 15 minutes

The supplier hands over three files: the release, its checksum, and .env. The .env comes separately, the way the Ministry moves passwords. The server needs no internet.

2.1

Check the copy is complete, and unpack it into /opt/grantline.

Keep that folder. The services that start GrantLine with the server expect it.

sha256sum -c grantline-<version>.tar.sha256sudo mkdir -p /opt/grantlinesudo tar -xf grantline-<version>.tar -C /opt/grantline --strip-components=1cd /opt/grantline && sha256sum -c SHA256SUMS

2.2
Put the carried .env in place.
With a Ministry certificate, also put cert.pem and key.pem in docker/certs/.
```
sudo cp <carried>/.env /opt/grantline/.env && sudo chmod 600 /opt/grantline/.env
```
2.3
Run the installer.
It checks .env matches the release, loads the images, creates the database, starts everything and tests it.
```
sudo ./install-offline.sh
```
2.4
Compare the release key ID it prints with the one the supplier gave you in writing.
A forged release would carry its own key file. The printed ID is the real check.
2.5
Answer its questions: the mail relay, starting with the server, and the first administrator.
Say yes to starting with the server. Write the administrator's password down when it is shown; it is not shown again. Press Enter at the relay question to add it later.
2.6
With Caddy's own certificate: install its root certificate on staff computers.
The installer saves it as docker/certs/caddy-root.crt. Without it every browser shows a warning.

Check it works

Before anyone is told

Each check catches a fault that otherwise shows up weeks later, in front of an applicant.

3.1
Sign in and open Settings → System. Every check should be green.
Mail stays amber until a relay is set. Each amber or red row has a How to fix link to this page.
3.2
Press Send me a test email on Settings → System → Mail. Then sign up as an applicant with a spare address and wait for the confirmation.
Two different services send those two emails, each with its own copy of the relay settings. This is the step most often missed.
3.3
Upload a document to a test application, sign out, sign back in and open it.
Proves documents are kept on their volume, not inside a container that goes on the next restart.
3.4
Run the release check.
It tests health, the scheduler, storage, sign-in and the version, and changes nothing. It ends with All checks passed.
```
sudo ./verify-release.sh <version>
```
3.5
DMZ only: check the split from both sides.
From a phone on mobile data, /portal loads and /dashboard is not found. From a staff desk, both load.
3.6
Restart the server and check everything comes back by itself.
Settings → System should be green again within a few minutes, without anyone touching it.
```
sudo reboot
```

Hand over to the grants team

First week

Your last set-up tasks. After this the system is theirs to run and yours to keep running.

4.1
Store the first administrator's password in the Ministry's credential store.
If it is lost, the bootstrap command under When something is wrong sets a new one.
4.2
Invite the grants staff under Settings → Members & roles, each with their post.
The post decides what a person can see. Panel members get the Panel Member post. Anyone without a post waits as a Viewer.
4.3
Set the organisation name and logo under Settings → Branding.
They appear in the staff console and on the applicant portal.
4.4
Show the team Help, and the tour on each page.
Most questions you would otherwise get are answered by the tour on the page the person is stuck on.
4.5
Agree who is called, and by whom, when the system is down.

Routine

Ongoing

Settings → System checks the server every 15 minutes and emails the administrators when a check changes. Green means there is nothing to do.

5.1
When an alert arrives, open Settings → System and follow the How to fix link.
Each link lands on its entry in System checks below.
5.2
Monthly: apply operating system security updates and restart in a quiet hour.
GrantLine starts with the server. Confirm Settings → System is green afterwards.
5.3
Quarterly: review who has an account, with the grants team.
People change jobs. Leavers who keep an account are the most likely audit finding.
5.4
Quarterly: rehearse a restore, and time it. Step 6 has the command.

Backups and restores

Quarterly drill

The backup service runs every 24 hours and keeps 30 days. Once a week it restores the newest backup into a scratch database to prove it works.

6.1
Each run writes three files: the database, the documents, and a copy of .env.
db-<stamp>.sql.gz, files-<stamp>.tar.gz and env-<stamp>.env. A restore needs all three from the same run.
6.2
Treat the backup folder like .env itself.
The env file in each backup holds every key in clear text. A restricted share, encrypted at rest.
6.3
Copy backups off this server on a schedule.
If the backup folder is on this machine, the backups are lost with it.
6.4
Rehearse a restore without touching the live system.
It restores into a throwaway container, prints row counts, then deletes it. Time it: that is the number you want during an incident.
```
sudo ./verify-restore.sh /path/to/backups/db-<stamp>.sql.gz
```
6.5
To put a backup back on the live system, run restore-backup.sh with its stamp.
With no stamp it lists the backups. It asks you to type RESTORE, takes the site down, restores the database and documents, and starts it again. Anything recorded after that backup is lost.
```
sudo ./restore-backup.sh <stamp>
```
6.6
Keep the release file of the running version.
A replacement server is built from it and the backup. Without internet there is no other way to rebuild.

Updates

Per release

A release arrives as two files. The server checks its signature before it touches anything. Never apply one on the morning a round closes.

7.1
Copy both files into the updates folder on the server.
On local disk, not a network share.
```
cp grantline-<version>.tar grantline-<version>.tar.sha256 /opt/grantline/updates/incoming/
```
7.2
Wait for Settings → System → Updates to show it as ready, then read the notes and any blockers.
Usually within a minute. A blocker names what is missing, often a new line for .env. Nothing can be applied until it is cleared.
7.3
Press Apply in a quiet hour.
A backup is taken first. Visitors see an updating page for about 3 to 10 minutes. If the new release fails its checks, the previous one and its database come back by themselves.

7.4

Read the result. If it failed, send the log to the supplier.

journalctl -u grantline-update-applyls /opt/grantline/updates/archive/

7.5
Optional: let the server apply releases by itself in a set window.
Off by default. Set UPDATE_AUTO_APPLY=window and a window such as UPDATE_WINDOW="Sun 02:00-04:00" in .env.
7.6
Change the hostname on the server, never in Settings.
Point the new name at the server in DNS first. The command checks the certificate, writes .env and restarts what uses the address. No new release is needed.
```
sudo ./install-offline.sh --address new.name.gov.lc
```

When something is wrong

Incidents

Work down this list. The first four cover most of what happens.

8.1
Open Settings → System, if it loads. Each red row links to its fix.
8.2
Is any container restarting?
A restart loop is almost always a wrong password in .env.
```
docker compose ps
```
8.3
Read the first error in the log, not the last.
A container in a loop prints the same failure over and over. The useful line is at the start of the cycle.
```
docker compose logs --tail=100 <service>
```
8.4
Is the disk full?
```
df -h
```
8.5
Is the application reaching the database?
Run it on the server. 503 means the application is up and the database is not.
```
curl -fsS https://<hostname>/api/health
```
8.6
Nothing is running after a restart: check the start-up service.
```
systemctl status grantline
```
8.7
The administrator's password is lost: set a new one.
Running it again only resets that account's password. If another administrator can still sign in and mail works, Send reset link under Settings → Members is quicker.
```
docker compose run --rm bootstrap --email '<email>' --name '<name>' --password '<new password>'
```
8.8
If you escalate, send what you did, the exact error, docker compose ps, and the relevant logs.
Never send the .env file or any key from it. No support request needs one.

Reference

When a System check is amber or red

Settings → System runs these checks every 15 minutes. Green needs nothing. Each amber or red row there links to its entry here.

Database Scheduler Event queue Mail relay Notification email Backups Restore test Disk space Document storage HTTPS certificate Data sovereignty Updates

Database

Technician.

What it means: The application cannot reach its database, or the check itself failed.
What to do: On the server, run docker compose ps and check that db is running and healthy. Then read docker compose logs --tail=100 db.

Scheduler

Technician.

What it means: Timed jobs have not run when they should: reminders, digests, scheduled reports and these checks.
What to do: Open the Scheduler row on Settings → System to see which jobs are late. On the server, run docker compose ps inngest and docker compose logs --tail=100 inngest app. INNGEST_EVENT_KEY and INNGEST_SIGNING_KEY must be set in .env.

Event queue

Technician first, then an administrator.

What it means: Events are waiting to be delivered, so the notifications and emails behind them have not gone out. The usual cause is a stopped scheduler.
What to do: Fix the Scheduler row first. If the message says events gave up after repeated failures, open Settings → Automations to see which ones.

Mail relay

Technician sets it. An administrator tests it.

What it means: No mail relay is set, or the relay does not answer. Applicants get no receipts, reminders or letters, and password resets do not arrive. While no relay is set, the server holds the emails it would have sent, up to 30 days, and the row says how many are waiting. A password reset is held only for the hour its link works, then dropped. Scheduled reports are not held. On an evaluation copy amber is expected: it sends no email.
What to do: Get the relay's host, port, user, password and sender address from the Ministry's IT. Settings → System → Mail shows the .env lines to fill in. After editing .env, run docker compose up -d, then press Check connection and Send me a test email. Held emails go within 15 minutes after that.

Notification email

Administrator.

What it means: Notification emails are late, or could not be delivered after a week of retries.
What to do: Check the Mail relay row first. If mail is green, check the recipients' email addresses for typos.

Backups

Technician.

What it means: The last backup failed, is late, or the application cannot see the backup folder.
What to do: Run docker compose logs --tail=100 backup. Check that BACKUP_PATH in .env points at a mounted disk or share with free space. If the folder is not mounted, run docker compose up -d.

Restore test

Technician.

What it means: Once a week the backup service restores the newest backup into a scratch database. That test failed or has not run lately, so nobody knows whether the backups can be restored.
What to do: Read docker compose logs backup for the restore test's error. Then run a restore drill by hand: sudo ./verify-restore.sh with the newest db-*.sql.gz from the backup folder.

Disk space

Technician.

What it means: A disk is 80% full or more. At 95% the database stops accepting changes.
What to do: Run df -h on the server. Free space or add storage. Backups older than BACKUP_RETENTION_DAYS are deleted on their own.

Document storage

Technician.

What it means: The application has no storage settings, so nobody can upload files.
What to do: Check that SITE_URL and SERVICE_ROLE_KEY are set in .env (SERVICE_ROLE_KEY is copied from the packaging machine), then run docker compose up -d. Run docker compose ps storage to check the storage service is running.

HTTPS certificate

Technician, with the Ministry's IT for a Ministry certificate.

What it means: The HTTPS certificate expires within 30 days, or already has. Browsers will refuse the site once it expires.
What to do: With a Ministry-issued certificate: get a renewed one, replace cert.pem and key.pem in docker/certs, and run docker compose restart caddy. With tls internal or Let's Encrypt, Caddy renews it itself; read docker compose logs caddy to see why it has not.

Data sovereignty

Technician.

What it means: A setting would send data off this server: hosted mail (RESEND_API_KEY), an AI key (AI_API_KEY), or a scheduler outside this stack. On an evaluation copy, amber means its AI model is reached outside it, which is allowed there.
What to do: Do what the message says: remove RESEND_API_KEY, remove AI_API_KEY and AI_BASE_URL, or set INNGEST_BASE_URL to http://inngest:8288. Then run docker compose up -d. The full statement is docs/deploy/DATA-SOVEREIGNTY.md in the handover pack.

Updates

An administrator applies a release. The technician does the rest.

What it means: A release is waiting to be applied, an update failed, or the update agent has stopped reporting.
What to do: A release waiting: read its notes on Settings → System and press Apply in a quiet hour. A failed update: the previous release is running again; send the agent log to the supplier. The agent silent: run systemctl status grantline-update-tick.timer. The folder not mounted: run sudo bin/update-agent.sh init.

Reference

The containers

What each service does, so a red one tells you what has stopped working.

caddy: HTTPS, certificates and routing. The only container that uses the server's ports.
app: The application: the applicant portal and the staff console.
db: PostgreSQL. All application data.
auth: Accounts, signing in and out, sessions, and the codes inside password reset links. Sends sign-up confirmations through the relay. The app sends the reset emails.
storage: Uploaded application and reporting documents, on disk.
inngest: The scheduler behind reminders and escalations. It also runs the System checks, and has no route off the server.
backup: Every 24 hours, a copy of the database, the documents and .env. Once a week it proves the newest copy restores.
autoheal: Restarts a container whose health check stays red.
migrate: Runs first on every start: applies schema changes and security policies, checks them, then stops. The application starts only if it succeeds.
bootstrap: Run by hand only: creates the first administrator, or sets a new password for that account.

Reference

The commands you will use

Run them from /opt/grantline.

What is running, and what is healthy.
```
docker compose ps
```
The last 100 lines of a service's log. Swap app for any service.
```
docker compose logs --tail=100 app
```
Start everything, or apply a change to .env. Runs migrate first, every time.
```
docker compose up -d
```
Restart one service without touching the rest.
```
docker compose restart app
```
Is the system working, database included. Run it on the server.
```
curl -fsS https://<hostname>/api/health
```
Did the stack start with the server?
```
systemctl status grantline
```
Tests health, scheduler, storage, sign-in and version. Changes nothing.
```
sudo ./verify-release.sh
```
Lists the backups. Add a stamp to restore one.
```
sudo ./restore-backup.sh
```
The log of the last update.
```
journalctl -u grantline-update-apply
```
A database shell, for when it is really needed.
```
docker compose exec db psql -U postgres
```

Reference

Four things that are easy to get wrong

The hostname is locked after set-up. Applicants' browsers use the address typed in the set-up wizard. Only install-offline.sh --address changes it, on the server. Point DNS at the server before you change it: a wrong address still works inside the Ministry and fails for every applicant, which internal testing misses.
Three keys come from the supplier. Each of JWT_SECRET, ANON_KEY, SERVICE_ROLE_KEY in .env must be the one the release was built with. Never generate new ones on the server. The installer refuses a .env that does not match.
Schema changes run before the application. docker compose up -d runs the migrate service first, every time, and the application starts only when it succeeds. It also applies the row-level security policies, which the migrations do not carry.
The application uses a restricted database role. That is what keeps each organisation's data apart. Do not point the application at the owner account to fix a permissions error; fix the grant.