Trying to secure the drbd-network of proxmox cluster

Hello everybody,

am trying to secure the drbd. i have used this guide:

security-best-practices-for-linstor-software-defined-storage-clusters

i stucked on the point Securing the LINSTOR Controller’s REST API

i can not get into the drbd_storage on proxmox since i configured the https API. I found the following in the logs:

:~# journalctl -u pvestatd -n 50 --no-pager

API Return-Code: 302. Message: Could not query all size infos for res groups, because:
                                         <html>
                                         <head><title>Document moved</title></head>
                                         <body><h1>Document moved</h1>
                                         This document has moved <a href="https://172.16.222.254:3371/v1/queries/resource-groups/query-all-size-info">here</a>.<p>
                                         </body>
                                         </html>

my storage.cfg file is looking like this:

drbd: drbd_storage
        content rootdir,images
        controller 172.16.222.254 #Virtual_IP
        resourcegroup pve-rg

By the way i did not deactive the http api. In this case the connections will be redirected to https.

my goal is to secure my DRBD-Network api, drbd-replication and the communication between the nodes over ssl

i am glad to hear your advice. :slight_smile:

Best Regards
Youzersef

I am also trying to figure this out as well. I have a vip and certs setup but I can’t figure out how to get proxmox to use the ssl connection

I was able to figure this out by adding the following cert info to storage.cfg

drbd: linstor
    resourcegroup pve-rg

    content rootdir,images

    controller 172.17.0.10

    apicrt /etc/linstor/ssl/linstor_controller_crt.pem

    apikey /etc/linstor/ssl/linstor_controller_prv.pem

    apica  /etc/linstor/ssl/ca_crt.pem
1 Like

i will test it today :smiley: thank you for sharing your solution

I got stuck on this as well, did not seem to find the information I wanted/needed…

For anyone struggling:
CREATE WORK DIRECTORIES

mkdir -p linstor-ctrl/ssl pve01 pve02 pbs01

GENERATE CONTROLLER KEYSTORE

keytool -keyalg rsa
-keysize 2048
-genkey -keystore linstor-ctrl/ssl/keystore.jks
-storepass linstor
-keypass linstor
-alias linstor-ctrl
-dname “CN=linstor-ctrl”
-ext SAN=dns:linstor-ctrl,ip:172.16.20.50
-validity 3650

GENERATE LINSTOR-CONTROLLER CA

keytool -exportcert
-alias linstor-ctrl
-keystore linstor-ctrl/ssl/keystore.jks
-storepass linstor
-rfc
-file linstor-ctrl/controller-ca.crt

GENERATE SATELLITE KEYSTORES
PVE01 KEYSTORE

keytool -keyalg rsa
-keysize 2048
-genkey -keystore pve01/keystore.jks
-storepass linstor
-keypass linstor
-alias pve01
-dname “CN=pve01”
-ext SAN=dns:pve01,ip:172.16.20.10
-validity 3650

PVE02 KEYSTORE

keytool -keyalg rsa
-keysize 2048
-genkey -keystore pve02/keystore.jks
-storepass linstor
-keypass linstor
-alias pve02
-dname “CN=pve02”
-ext SAN=dns:pve02,ip:172.16.20.20
-validity 3650

PBS01 KEYSTORE

keytool -keyalg rsa
-keysize 2048
-genkey -keystore pbs01/keystore.jks
-storepass linstor
-keypass linstor
-alias pbs01
-dname “CN=pbs01”
-ext SAN=dns:pbs01,ip:172.16.20.30
-validity 3650

BUILD THE LINSTOR-CONTROLLER TRUSTSTORE

keytool -importkeystore -srcstorepass linstor -deststorepass linstor -keypass linstor -srckeystore pve01/keystore.jks -destkeystore linstor-ctrl/ssl/truststore.jks

keytool -importkeystore -srcstorepass linstor -deststorepass linstor -keypass linstor -srckeystore pve02/keystore.jks -destkeystore linstor-ctrl/ssl/truststore.jks

keytool -importkeystore -srcstorepass linstor -deststorepass linstor -keypass linstor -srckeystore pbs01/keystore.jks -destkeystore linstor-ctrl/ssl/truststore.jks

CREATE SATELLITE TRUSTSTORES
PVE01 TRUSTSTORE

keytool -importkeystore -srcstorepass linstor -deststorepass linstor -keypass linstor -srckeystore linstor-ctrl/ssl/keystore.jks -destkeystore pve01/certificates.jks

PVE02 TRUSTSTORE

keytool -importkeystore -srcstorepass linstor -deststorepass linstor -keypass linstor -srckeystore linstor-ctrl/ssl/keystore.jks -destkeystore pve02/certificates.jks

PBS01 TRUSTSTORE

keytool -importkeystore -srcstorepass linstor -deststorepass linstor -keypass linstor -srckeystore linstor-ctrl/ssl/keystore.jks -destkeystore pbs01/certificates.jks

COPY KEYSTORES AND TRUSTSTORES
CREATE THE SSL DIRECTORY FOR THE LINSTOR CONTROLLER

mkdir -p /etc/linstor/ssl
ssh -t root@pve02 “mkdir -p /etc/linstor/ssl”

COPY LINSTOR-CONTROLLER KEYSTORE AND TRUSTSTORE

cp linstor-ctrl/ssl/keystore.jks /etc/linstor/ssl/keystore.jks
cp linstor-ctrl/ssl/truststore.jks /etc/linstor/ssl/certificates.jks

COPY LINSTOR-CONTROLLER KEYSTORE AND TRUSTSTORE TO PVE02

scp linstor-ctrl/ssl/keystore.jks root@pve02:/etc/linstor/ssl/keystore.jks
scp linstor-ctrl/ssl/truststore.jks root@pve02:/etc/linstor/ssl/certificates.jks

COPY PVE01 SATELLITE KEYSTORE AND TRUSTSTORE

cp pve01/keystore.jks /etc/linstor/keystore.jks
cp pve01/certificates.jks /etc/linstor/certificates.jks

COPY PVE02 SATELLITE KEYSTORE AND TRUSTSTORE

scp pve02/keystore.jks root@pve02:/etc/linstor/keystore.jks
scp pve02/certificates.jks root@pve02:/etc/linstor/certificates.jks

COPY PBS01 SATELLITE KEYSTORE AND TRUSTSTORE

scp pbs01/keystore.jks root@pbs01:/etc/linstor/keystore.jks
scp pbs01/certificates.jks root@pbs01:/etc/linstor/certificates.jks

ENABLE HTTPS COMMUNICATION

sed -i ‘0,/[https]/s/[https]/#[https]/’ /etc/linstor/linstor.toml

cat << EOF | sudo tee -a /etc/linstor/linstor.toml

[https]
enabled = true
port = 3371
listen_addr = “0.0.0.0”
keystore = “/etc/linstor/ssl/keystore.jks”
keystore_password = “linstor”

ONLY USE TRUSTSTORE WHEN YOU WANT RESTRICTED CLIENT ACCESS!

truststore = “/etc/linstor/ssl/certificates.jks”

truststore_password = “linstor”

EOF

scp /etc/linstor/linstor.toml root@pve02:/etc/linstor/linstor.toml
systemctl restart linstor-controller.service // LINSTOR-GUI IS NOW AVAILABLE AT HTTPS://<YOUR-VIRTUAL-IP:3371

CONVERT KEYSTORES TO PEM CLIENT CERTIFICATES
EXPORT THE PVE01 SATELLITE KEYSTORE TO A PKCS#12 FILE

keytool -importkeystore -srckeystore pve01/keystore.jks -destkeystore pve01/client.p12 -storepass linstor -keypass linstor -srcalias pve01 -srcstoretype jks -deststoretype pkcs12

CONVERT TO PEM WITH A PASSWORD

openssl pkcs12 -in pve01/client.p12 -out pve01/client_with_pw.pem

REMOVE THE PASSWORD, EXTRACT THE KEY AND APPEND THE CERTIFICATE

openssl rsa -in pve01/client_with_pw.pem -out pve01/client.pem
openssl x509 -in pve01/client_with_pw.pem >> pve01/client.pem

EXPORT THE PVE02 SATELLITE KEYSTORE TO A PKCS#12 FILE

keytool -importkeystore -srckeystore pve02/keystore.jks -destkeystore pve02/client.p12 -storepass linstor -keypass linstor -srcalias pve02 -srcstoretype jks -deststoretype pkcs12

CONVERT TO PEM WITH A PASSWORD

openssl pkcs12 -in pve02/client.p12 -out pve02/client_with_pw.pem

REMOVE THE PASSWORD, EXTRACT THE KEY AND APPEND THE CERTIFICATE

openssl rsa -in pve02/client_with_pw.pem -out pve02/client.pem
openssl x509 -in pve02/client_with_pw.pem >> pve02/client.pem

EXPORT THE PBS01 SATELLITE KEYSTORE TO A PKCS#12 FILE

keytool -importkeystore -srckeystore pbs01/keystore.jks -destkeystore pbs01/client.p12 -storepass linstor -keypass linstor -srcalias pbs01 -srcstoretype jks -deststoretype pkcs12

CONVERT TO PEM WITH A PASSWORD

openssl pkcs12 -in pbs01/client.p12 -out pbs01/client_with_pw.pem

REMOVE THE PASSWORD, EXTRACT THE KEY AND APPEND THE CERTIFICATE

openssl rsa -in pbs01/client_with_pw.pem -out pbs01/client.pem
openssl x509 -in pbs01/client_with_pw.pem >> pbs01/client.pem

DEPLOY SATELLITE CERTIFICATES
PVE01

cp pve01/client.pem /etc/linstor/client.pem

PVE02

scp pve02/client.pem root@pve02:/etc/linstor/client.pem

PBS01

scp pbs01/client.pem root@pbs01:/etc/linstor/client.pem

UPDATE LINSTOR-CLIENT CONFIGURATION
PVE01

rm -rf /etc/linstor/linstor-client.conf

cat << EOF > /etc/linstor/linstor-client.conf
[global]
controllers = linstor+ssl://172.16.20.50:3371
certfile = /etc/linstor/client.pem
EOF

COPY LINSTOR-CLIENT CONFIGURATION TO PVE02

ssh -t root@pve02 “rm -rf /etc/linstor/linstor-client.conf”
scp /etc/linstor/linstor-client.conf root@pve02:/etc/linstor/linstor-client.conf

COPY LINSTOR-CLIENT CONFIGURATION TO PBS01

ssh -t root@pbs01 “rm -rf /etc/linstor/linstor-client.conf”
scp /etc/linstor/linstor-client.conf root@pbs01:/etc/linstor/linstor-client.conf

CONFIGURE SATELLITE NETWORK COMMUNICATION
PVE01

cat << EOF > /etc/linstor/linstor_satellite.toml
[netcom]
type=“ssl”
port=3367
server_certificate=“/etc/linstor/keystore.jks”
trusted_certificates=“/etc/linstor/certificates.jks”
key_password=“linstor”
keystore_password=“linstor”
truststore_password=“linstor”
ssl_protocol=“TLSv1.2”
EOF

COPY TO PVE02

scp /etc/linstor/linstor_satellite.toml root@pve02:/etc/linstor/linstor_satellite.toml

COPY TO PBS01

scp /etc/linstor/linstor_satellite.toml root@pbs01:/etc/linstor/linstor_satellite.toml

RESTART SATELLITE SERVICE ON ALL NODES

systemctl restart linstor-satellite.service

ssh -t root@pve02 “systemctl restart linstor-satellite.service”
ssh -t root@pbs01 “systemctl restart linstor-satellite.service”

SWITCH SATELLITE COMMUNICATION TO SSL
PVE01

linstor node interface modify --communication-type ssl -p 3367 pve01 default

PVE02

linstor node interface modify --communication-type ssl -p 3367 pve02 default

PBS01

linstor node interface modify --communication-type ssl -p 3367 pbs01 default

DISABLE HTTP COMMUNICATION

sed -i ‘0,/[http]/s/[http]/#[http]/’ /etc/linstor/linstor.toml

systemctl restart linstor-controller.service
systemctl restart linstor-satellite.service

ssh -t root@pve02 “systemctl restart linstor-satellite.service”
ssh -t root@pbs01 “systemctl restart linstor-satellite.service”

GENERATE CLIENT CERTIFICATES FOR PROXMOX REST API
PVE01

openssl pkcs12 -in pve01/client.p12 -clcerts -nokeys -out pve01/client.crt
openssl pkcs12 -in pve01/client.p12 -nocerts -nodes -out pve01/client.key

cp pve01/client.crt /etc/linstor/client.crt
cp pve01/client.key /etc/linstor/client.key

PVE02

openssl pkcs12 -in pve02/client.p12 -clcerts -nokeys -out pve02/client.crt
openssl pkcs12 -in pve02/client.p12 -nocerts -nodes -out pve02/client.key

scp pve02/client.crt root@pve02:/etc/linstor/client.crt
scp pve02/client.key root@pve02:/etc/linstor/client.key

COPY LINSTOR-CONTROLLER CA

cp linstor-ctrl/controller-ca.crt /etc/linstor/controller-ca.crt
scp linstor-ctrl/controller-ca.crt root@pve02:/etc/linstor/controller-ca.crt

EDIT DRBD STORAGE CONFIGURATION

nano /etc/pve/storage.cfg

apicrt /etc/linstor/client.crt
apikey /etc/linstor/client.key
apica /etc/linstor/controller-ca.crt

EXAMPLE

drbd: drbdpool
content images
controller 172.16.20.50
resourcegroup pve-rg

# exactsize yes # Use exactsize parameter only temporarly to migrate running VMs

apicrt /etc/linstor/client.crt
apikey /etc/linstor/client.key
apica /etc/linstor/controller-ca.crt

I hope this makes some sense.
My config is a 2-node PVE Cluster setup with 1 PBS node acting a quorom (qdevice) for PVE and a diskless/tie-breaker (quorum) for LINSTOR/DRBD
The linstor-controller (172.16.20.50) is HA and can move between PVE01 and PVE02.

I’m also continously improving so if someone knows a better way, please share :slight_smile:

A quick note: This does not encrypt the DRBD replication data, that needs another method.