Effortless HashiCorp Vault unsealing - a step-by-step guide using systemd services – Sysadmin - Linux administration, security and hardening

Introduction

Vault by HashiCorp requires unsealing after every restart to ensure the security of the secrets it stores. This tutorial will guide you through automating the unseal process using a systemd service on a Linux system.

Prerequisites

Vault installed and configured on your system
Access to unseal keys
Basic knowledge of systemd and bash scripting
gpg installed for encryption

How to install gpg

Debian/Ubuntu

1
2
3


sudo apt update
sudo apt -y install gnupg
gpg --version

CentOS/RHEL

1
2
3


sudo yum update
sudo yum -y install gnupg
gpg --version

Fedora

1
2
3


sudo dnf update
sudo dnf -y install gnupg
gpg --version

Arch Linux

1
2
3


sudo pacman -Syu
sudo pacman -S gnupg
gpg --version

OpenSUSE

1
2
3


sudo zypper refresh
sudo zypper install gpg2
gpg --version

Step 1: Preparation

Log in via SSH: Connect to your server as a standard user and switch to root.
1

sudo -i
Start a new bash session: Execute a new bash shell and disable history.

Explanation: For the highest security, run the command in a new shell session where history is disabled, and ensure no sensitive information is stored.

1
2


bash
set +o history

3.Create a file to store the GPG passphrase: Ensure the file is accessible only to the root user.

1

echo "your-passphrase" > /root/.gpg_passphrase

4.Set the permissions to make it readable only by the root user.

1

chmod 400 /root/.gpg_passphrase

Step 2: Encrypt the Unseal Keys

Create an encrypted file to store your unseal keys:

1
2


echo -e "your-unseal-key-1\nyour-unseal-key-2\nyour-unseal-key-3" | gpg --batch --yes --passphrase-file /root/.gpg_passphrase --symmetric --cipher-algo AES256 -o /root/.vault_unseal_keys.gpg
chmod 400 /root/.vault_unseal_keys.gpg

Clear the bash history and exit the temporary session:
1 2

history -c exit

Step 3: Create the Unseal Script

Create a script to unseal Vault that securely fetches the unseal keys. Save the following script to /usr/local/bin/unseal_vault.sh.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51


#!/bin/bash

export VAULT_ADDR='https://<vault IP address>:8200'

# Create log file if it doesn't exist
LOGFILE=/var/log/unseal_vault.log
if [ ! -f "$LOGFILE" ]; then
    touch "$LOGFILE"
    chown vault:vault "$LOGFILE"
else
    echo "$LOGFILE exists"
fi

# Log the start time
echo "Starting unseal at $(date)" >> $LOGFILE

# Wait for Vault to be ready
while ! vault status 2>&1 | grep -q "Sealed.*true"; do
  echo "Waiting for Vault to be sealed and ready..." >> $LOGFILE
  sleep 5
done

echo "Vault is sealed and ready at $(date)" >> $LOGFILE

# Load the GPG passphrase
GPG_PASSPHRASE=$(cat /root/.gpg_passphrase)

# Decrypt the unseal keys
UNSEAL_KEYS=$(gpg --quiet --batch --yes --decrypt --passphrase "$GPG_PASSPHRASE" /root/.vault_unseal_keys.gpg)
if [ $? -ne 0 ]; then
  echo "Failed to decrypt unseal keys at $(date)" >> $LOGFILE
  exit 1
fi

echo "Unseal keys decrypted successfully at $(date)" >> $LOGFILE

# Convert decrypted keys to an array
UNSEAL_KEYS_ARRAY=($(echo "$UNSEAL_KEYS"))

# Unseal Vault
for key in "${UNSEAL_KEYS_ARRAY[@]}"; do
# commented out because I do not want to debug it anymore
  vault operator unseal "$key" # >> $LOGFILE 2>&1
  #if [ $? -ne 0 ]; then
  #  echo "Failed to unseal with key $key at $(date)" >> $LOGFILE
  #  exit 1
  #fi
  #echo "Successfully used unseal key $key at $(date)" >> $LOGFILE
done

echo "Vault unsealed successfully at $(date)" >> $LOGFILE

Make the script executable:

1

chmod 500 /usr/local/bin/unseal_vault.sh

Step 4: Modify Vault Service

Modify the /etc/systemd/system/vault.service file to include a dependency on the vault-unseal.service.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


[Unit]
Description=HashiCorp Vault
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
Requires=vault-unseal.service

[Service]
User=vault
Group=vault
EnvironmentFile=/etc/vault.d/vault.env
ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target

Step 5: Create Vault Unseal Service

Create a new service file at /etc/systemd/system/vault-unseal.service with the following content:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


[Unit]
Description=Unseal Vault
After=vault.service
Requires=vault.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/unseal_vault.sh
Environment=VAULT_ADDR=https://<vault IP address>:8200
Environment=DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus

[Install]
WantedBy=multi-user.target

Step 6: Create vault.env file

1
2
3
4


cat << 'EOF' > /etc/vault.d/vault.env
VAULT_ADDR=https://<vault IP address>:8200
DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus
EOF

Step 7: export variables to .bashrc

1
2
3
4


BASHRC_PATH="$HOME/.bashrc"
echo "export VAULT_ADDR='https://<vault IP address>:8200'" >> $BASHRC_PATH
echo "export DBUS_SESSION_BUS_ADDRESS=\$XDG_RUNTIME_DIR/bus" >> $BASHRC_PATH
source $BASHRC_PATH

With these modifications, the vault-unseal.service will be considered part of the vault.service process. Restarting vault.service will now also trigger the vault-unseal.service.

Step 6: Reload Systemd and Start Services

Reload systemd to apply the changes and start the services:

1
2


systemctl daemon-reload
systemctl restart vault.service

Step 8: Enable both services during boot

1
2


systemctl enable vault-unseal.service
systemctl enable vault.service

Explanation: What happens during the boot process

When your system boots up, the following sequence happens:

vault.service starts: This is the main service for Vault. It will start according to its configuration.
vault-unseal.service starts: This service is configured to start after vault.service because of the After=vault.service directive. This means that vault-unseal.service will not start until vault.service has fully started.

The vault-unseal.service depends on vault.service, and it will only execute the unseal script after the Vault service is running.

Behavior on Manual Restart

Manual restart of `vault.service`

When you manually restart vault.service using the command:

1

systemctl restart vault.service

Here’s what happens:

vault.service stops: The Vault service stops and then starts again.
vault-unseal.service automatically restarts: The service vault-unseal.service automatically restarts just because vault.service was restarted. The vault-unseal.service is set to run after vault.service during the boot process.

Ensuring Unseal After Restart

To ensure that the vault-unseal.service runs every time vault.service is restarted, run the below command:

1

tail -f /var/log/unseal_vault.log

In another SSH session, restart the vault.service:

1

systemctl restart vault.service

Then, check the status of both services to ensure they are working as expected:

1
2


systemctl status vault.service
systemctl status vault-unseal.service

Summary

This configuration ensures a secure method of unsealing Vault by encrypting the unseal keys with GPG and securely storing the passphrase. The script fetches the passphrase and decrypts the keys at runtime, enhancing the security of your setup.

By following this guide, you ensure that sensitive unseal keys are not exposed in plaintext, and access to the passphrase is restricted to the root user, providing an additional layer of security.

The vault-unseal.service will run both during the boot process and any manual restarts of vault.service, keeping Vault automatically unsealed and operational.

Walkthrough video

Bash script that automates the whole process

1

vim vault.sh

Put the below content:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151


#!/bin/bash

echo "This quick installer script requires root privileges."
echo "Checking..."
if [[ $(/usr/bin/id -u) -ne 0 ]]; 
then
    echo "Not running as root"
    exit 0
else
        echo "Installation continues"
fi

SUDO=
if [ "$UID" != "0" ]; then
        if [ -e /usr/bin/sudo -o -e /bin/sudo ]; then
                SUDO=sudo
        else
                echo "*** This quick installer script requires root privileges."
                exit 0
        fi
fi

# Step 1: Create passphrase file
echo "<your passphrase>" > /root/.gpg_passphrase
chmod 400 /root/.gpg_passphrase

# Step 2: Encrypt unseal keys
echo -e "your-unseal-key-1\nyour-unseal-key-2\nyour-unseal-key-3" | gpg --batch --yes --passphrase-file /root/.gpg_passphrase --symmetric --cipher-algo AES256 -o /root/.vault_unseal_keys.gpg

chmod 400 /root/.vault_unseal_keys.gpg

# Step 3: Clear bash history and exit the temporary session
history -c

# Step 4: Create the unseal script
cat << 'EOF' > /usr/local/bin/unseal_vault.sh
#!/bin/bash

export VAULT_ADDR='https://<vault IP address>:8200'

# Create log file if it doesn't exist
LOGFILE=/var/log/unseal_vault.log
if [ ! -f "$LOGFILE" ]; then
    touch "$LOGFILE"
    chown vault:vault "$LOGFILE"
else
    echo "$LOGFILE exists"
fi

# Log the start time
echo "Starting unseal at $(date)" >> $LOGFILE

# Wait for Vault to be ready
while ! vault status 2>&1 | grep -q "Sealed.*true"; do
  echo "Waiting for Vault to be sealed and ready..." >> $LOGFILE
  sleep 5
done

echo "Vault is sealed and ready at $(date)" >> $LOGFILE

# Load the GPG passphrase
GPG_PASSPHRASE=$(cat /root/.gpg_passphrase)

# Decrypt the unseal keys
UNSEAL_KEYS=$(gpg --quiet --batch --yes --decrypt --passphrase "$GPG_PASSPHRASE" /root/.vault_unseal_keys.gpg)
if [ $? -ne 0 ]; then
  echo "Failed to decrypt unseal keys at $(date)" >> $LOGFILE
  exit 1
fi

echo "Unseal keys decrypted successfully at $(date)" >> $LOGFILE

# Convert decrypted keys to an array
UNSEAL_KEYS_ARRAY=($(echo "$UNSEAL_KEYS"))

# Unseal Vault
for key in "${UNSEAL_KEYS_ARRAY[@]}"; do
# commented out because I do not want to debug it anymore
  vault operator unseal "$key" # >> $LOGFILE 2>&1
  #if [ $? -ne 0 ]; then
  #  echo "Failed to unseal with key $key at $(date)" >> $LOGFILE
  #  exit 1
  #fi
  #echo "Successfully used unseal key $key at $(date)" >> $LOGFILE
done

echo "Vault unsealed successfully at $(date)" >> $LOGFILE
EOF

chmod 500 /usr/local/bin/unseal_vault.sh

# Step 5: Modify vault.service
cat << 'EOF' > /etc/systemd/system/vault.service
[Unit]
Description=HashiCorp Vault
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
Requires=vault-unseal.service

[Service]
User=vault
Group=vault
EnvironmentFile=/etc/vault.d/vault.env
ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target
EOF

# Step 6: Create vault-unseal.service
cat << 'EOF' > /etc/systemd/system/vault-unseal.service
[Unit]
Description=Unseal Vault
After=vault.service
Requires=vault.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/unseal_vault.sh
Environment=VAULT_ADDR=https://<vault IP address>:8200
Environment=DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus

[Install]
WantedBy=multi-user.target
EOF

# Step 7: Create vault.env file
cat << 'EOF' > /etc/vault.d/vault.env
VAULT_ADDR=https://<vault IP address>:8200
DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus
EOF

# Step 8: Reload systemd and start services
systemctl daemon-reload
systemctl enable vault-unseal.service
systemctl enable vault.service
systemctl restart vault.service

# Step 9: export variables to .bashrc
BASHRC_PATH="$HOME/.bashrc"
echo "export VAULT_ADDR='https://<vault IP address>:8200'" >> $BASHRC_PATH
echo "export DBUS_SESSION_BUS_ADDRESS=\$XDG_RUNTIME_DIR/bus" >> $BASHRC_PATH
source $BASHRC_PATH

Make the bash script executable:

1

chmod +x vault.sh

Run the script:

1

sudo ./vault.sh

Bash script that reverts all changes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56


#!/bin/bash

echo "This quick installer script requires root privileges."
echo "Checking..."
if [[ $(/usr/bin/id -u) -ne 0 ]]; 
then
    echo "Not running as root"
    exit 0
else
        echo "Installation continues"
fi

SUDO=
if [ "$UID" != "0" ]; then
        if [ -e /usr/bin/sudo -o -e /bin/sudo ]; then
                SUDO=sudo
        else
                echo "*** This quick installer script requires root privileges."
                exit 0
        fi
fi
rm -f /var/log/unseal_vault.log
rm -f /root/.gpg_passphrase
rm -f /root/.vault_unseal_keys.gpg
rm -f /usr/local/bin/unseal_vault.sh
rm -f /etc/systemd/system/vault-unseal.service
rm -f /etc/vault.d/vault.env
cat << 'EOF' > /etc/systemd/system/vault.service
[Unit]
Description=HashiCorp Vault
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target

[Service]
User=vault
Group=vault
ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl restart vault

BASHRC_PATH="$HOME/.bashrc"
sed -i "/export VAULT_ADDR='https:\/\/<vault IP address>:8200'/d" $BASHRC_PATH
sed -i "/export DBUS_SESSION_BUS_ADDRESS=\$XDG_RUNTIME_DIR\/bus/d" $BASHRC_PATH
source $BASHRC_PATH