Set up Pacemaker on Red Hat Enterprise Linux in Azure

Using SBD with Azure Shared Disk

By using Azure Shared Disks, you can mount the same disk across all virtual machines that are part of the cluster. You can host your SBD device on that shared disk without extra infrastructure requirements.

Benefits

• Provides a native Azure shared block device option for SBD without requiring additional resources.
• Virtual machines attach the managed disk directly, reducing dependency on additional network considerations.

Important considerations

• You can use an Azure shared disk with the Premium SSD SKU as an SBD device.
• Review the List of Supported Operating Systems.
• SBD devices that use an Azure premium shared disk support locally redundant storage (LRS) and zone-redundant storage (ZRS).
• Depending on the type of your deployment, choose the appropriate redundant storage for an Azure shared disk as your SBD device.
  • An SBD device that uses LRS for Azure premium shared disk (skuName - Premium_LRS) supports only deployments in an availability set.
  • An SBD device that uses ZRS for an Azure premium shared disk (skuName - Premium_ZRS) is recommended for deployments in availability zones.
• The Azure shared disk that you use for SBD devices doesn't need to be large. The maxShares value determines how many cluster nodes can use the shared disk. For example, you can use P1 or P2 disk sizes for your SBD device on a two-node cluster such as SAP ASCS/ERS or SAP HANA scale-up.
  • For clusters with more than two nodes, refer to the documented maxShares for your selected disk.
• Don't attach an Azure shared disk SBD device across different Pacemaker clusters.
• If you use multiple Azure shared disk SBD devices, check on the limit for a maximum number of data disks that can be attached to a VM.
• For more information about limitations for Azure shared disks, carefully review the "Limitations" section of Azure shared disk documentation.

Using SBD with iSCSI targets

This solution requires you to host Internet Small Computer System Interface (iSCSI) targets on at least one additional virtual machine (VM).

Benefits

• These iSCSI host servers can host iSCSI targets for other Pacemaker clusters in the same region as well.
• If you're already using them on-premises, they don't require any changes to how you operate the Pacemaker cluster.

Important considerations

• You must use three iSCSI host servers to have the highest level of resiliency for your cluster.
  • Using only one server introduces a single point of failure that prevents your cluster from fencing if it's down.
  • Pacemaker doesn't allow fencing if you only have two targets and one is down.
• iSCSI target host servers must reside in the same region as your clusters.
• Network routing between your clusters and iSCSI host servers must not traverse any non-redundant network devices (like a Network Virtual Appliance).
  • Maintenance events and other issues with network devices can negatively affect the stability and reliability of the overall cluster configuration.

Using the Azure Fence Agent

By using the Azure Fence Agent, your cluster can fence nodes by directly calling the Azure APIs to restart failed nodes.

Benefits

• No extra resources needed.
• Managed Identities remove any credential maintenance.

Important considerations

• Use Managed Identities for authentication. If you're currently using a service principal, Update the Azure Fence Agent from SPN to MSI.
• The Azure fence agent requires outbound connectivity to public Azure endpoints. For more information along with possible solutions, see Public endpoint connectivity for VMs using standard ILB.
• The monitoring and fencing operations are deserialized. As a result, if there's a longer running monitoring operation and simultaneous fencing event, there's no delay to the cluster failover because the monitoring operation is already running.

Deploy an Azure Shared Disk for SBD

To create and attach an Azure shared disk by using PowerShell, run the following commands. If you want to deploy resources by using the Azure CLI or the Azure portal, see Deploy a ZRS disk.

$ResourceGroup = "<ResourceGroupName>"
$DiskSizeInGB = 4
$DiskName = "<SBDDiskName>"
# Must be Equal to or greater than the Number of Nodes in the Cluster
$ShareNodes = 2 
# Options are "Premium_LRS" or "Premium_ZRS"
$SKUName = "<DiskSKU>"
# VMs to attach the disk to
$vmNames = @("sap-cl1", "sap-cl2")
# Lun to attach the disk to. You should use the same lun on all servers in the cluster.
$lunNumber = <lunNumber>

$diskConfig = New-AzDiskConfig -Location $Location -SkuName $SkuName -CreateOption Empty -DiskSizeGB $DiskSizeInGB -MaxSharesCount $ShareNodes

$dataDisk = New-AzDisk -ResourceGroupName $ResourceGroup -DiskName $DiskName -Disk $diskConfig

# Attach SBD disk to cluster VMs
foreach ($vmName in $vmNames) 
{
  $vm = Get-AzVM -ResourceGroupName $resourceGroup -Name $vmName
  Add-AzVMDataDisk -VM $vm -Name $diskName -CreateOption Attach -ManagedDiskId $dataDisk.Id -Lun $lunNumber
  Update-AzVM -VM $vm -ResourceGroupName $resourceGroup -Verbose
}

Use iSCSI targets

Build iSCSI target host servers

1. Deploy three virtual machines that run on a supported RHEL OS version. The VMs don't need to be large. VM sizes such as Standard_E2s or Standard_D2s are sufficient.

   Note

   You don't need to use RHEL for SAP with HA and Update Services, or RHEL for SAP Apps OS image for the iSCSI target server. You can use a standard RHEL OS image instead. However, the support life cycle varies between different OS product releases.

2. Install the latest updates, and reboot if required.

   sudo dnf -y update
   

3. Install the iSCSI target package.

   sudo dnf install -y targetcli
   

4. Enable and start the iSCSI service.

   sudo systemctl start target
   sudo systemctl enable target
   

5. Open the port in the firewall.

   sudo firewall-cmd --add-port=3260/tcp --permanent
   sudo firewall-cmd --reload
   

Create iSCSI targets

For each cluster, you need to provision an iSCSI disk on each iSCSI host server, and then grant every cluster node access to that disk. In this example, you create disks for two different clusters:

• ascsnw1: The ASCS/ERS cluster for NW1
• hdbnw1: The HANA Database cluster for NW1
• sap-cl1 and sap-cl2: Hostnames for the NW1 ASCS/ERS cluster nodes
• sap-db1 and sap-db2: Hostnames for the NW1 HANA cluster nodes

1. Create the root folder for all SBD devices.

   sudo mkdir /sbd
   

2. Create the SBD device for the first cluster (ascsnw1).

   # Create Storage Object
   sudo targetcli backstores/fileio create sbdascsnw1 /sbd/sbdascsnw1 50M write_back=false
   # Create iSCSI Target
   sudo targetcli iscsi/ create iqn.2006-04.ascsnw1.local:ascsnw1
   # Creates the LUN and attaches it to the iSCSI Target
   sudo targetcli iscsi/iqn.2006-04.ascsnw1.local:ascsnw1/tpg1/luns/ create /backstores/fileio/sbdascsnw1
   # Grant Cluster Node 1 (sap-cl1) access to the iSCSI Target
   sudo targetcli iscsi/iqn.2006-04.ascsnw1.local:ascsnw1/tpg1/acls/ create iqn.2006-04.sap-cl1.local:sap-cl1
   # Grant Cluster Node 2 (sap-cl2) access to the iSCSI Target
   sudo targetcli iscsi/iqn.2006-04.ascsnw1.local:ascsnw1/tpg1/acls/ create iqn.2006-04.sap-cl2.local:sap-cl2
   

3. Create the SBD device for the second cluster (hdbnw1).

   # Create Storage Object
   sudo targetcli backstores/fileio create sbdhdbnw1 /sbd/sbdhdbnw1 50M write_back=false
   # Create iSCSI Target
   sudo targetcli iscsi/ create iqn.2006-04.hdbnw1.local:hdbnw1
   # Creates the LUN and attaches it to the iSCSI Target
   sudo targetcli iscsi/iqn.2006-04.hdbnw1.local:hdbnw1/tpg1/luns/ create /backstores/fileio/sbdhdbnw1
   # Grant Cluster Node 1 (sap-db1) access to the iSCSI Target
   sudo targetcli iscsi/iqn.2006-04.hdbnw1.local:hdbnw1/tpg1/acls/ create iqn.2006-04.sap-db1.local:sap-db1
   # Grant Cluster Node 2 (sap-db2) access to the iSCSI Target
   sudo targetcli iscsi/iqn.2006-04.hdbnw1.local:hdbnw1/tpg1/acls/ create iqn.2006-04.sap-db2.local:sap-db2
   

4. Save the configuration.

   sudo targetcli saveconfig
   

5. Verify the setup.

   sudo targetcli ls
   
   o- / ............................................................................................... [...]
     o- backstores .................................................................................... [...]
     | o- block ........................................................................ [Storage Objects: 0]
     | o- fileio ....................................................................... [Storage Objects: 2]
     | | o- sbdascsnw1 ..................................... [/sbd/sbdascsnw1 (50.0MiB) write-thru activated]
     | | | o- alua ......................................................................... [ALUA Groups: 1]
     | | |   o- default_tg_pt_gp ............................................. [ALUA state: Active/optimized]
     | | o- sbdhdbnw1 ....................................... [/sbd/sbdhdbnw1 (50.0MiB) write-thru activated]
     | |   o- alua ......................................................................... [ALUA Groups: 1]
     | |     o- default_tg_pt_gp ............................................. [ALUA state: Active/optimized]
     | o- pscsi ........................................................................ [Storage Objects: 0]
     | o- ramdisk ...................................................................... [Storage Objects: 0]
     o- iscsi .................................................................................. [Targets: 2]
     | o- iqn.2006-04.hdbnw1.local:hdbnw1 ......................................................... [TPGs: 1]
     | | o- tpg1 ..................................................................... [no-gen-acls, no-auth]
     | |   o- acls ................................................................................ [ACLs: 2]
     | |   | o- iqn.2006-04.sap-db1.local:sap-db1 .......................................... [Mapped LUNs: 1]
     | |   | | o- mapped_lun0 ..................................................... [lun0 fileio/sbdhdb (rw)]
     | |   | o- iqn.2006-04.sap-db2.local:sap-db2 .......................................... [Mapped LUNs: 1]
     | |   |   o- mapped_lun0 ..................................................... [lun0 fileio/sbdhdb (rw)]
     | |   o- luns ................................................................................ [LUNs: 1]
     | |   | o- lun0 ................................. [fileio/sbdhdbnw1 (/sbd/sbdhdbnw1) (default_tg_pt_gp)]
     | |   o- portals .......................................................................... [Portals: 1]
     | |     o- 0.0.0.0:3260 ........................................................................... [OK]
     | o- iqn.2006-04.ascsnw1.local:ascsnw1 ....................................................... [TPGs: 1]
     |   o- tpg1 ..................................................................... [no-gen-acls, no-auth]
     |     o- acls ................................................................................ [ACLs: 2]
     |     | o- iqn.2006-04.sap-cl1.local:sap-cl1 .......................................... [Mapped LUNs: 1]
     |     | | o- mapped_lun0 ................................................. [lun0 fileio/sbdascsnw1 (rw)]
     |     | o- iqn.2006-04.sap-cl2.local:sap-cl2 .......................................... [Mapped LUNs: 1]
     |     |   o- mapped_lun0 ................................................. [lun0 fileio/sbdascsnw1 (rw)]
     |     o- luns ................................................................................ [LUNs: 1]
     |     | o- lun0 ............................... [fileio/sbdascsnw1 (/sbd/sbdascsnw1) (default_tg_pt_gp)]
     |     o- portals .......................................................................... [Portals: 1]
     |       o- 0.0.0.0:3260 ........................................................................... [OK]
     o- loopback ............................................................................... [Targets: 0]
   

Configure Azure Fence Agent

1. Create Identity

   • Managed identity
   • Service principal

   To create a managed identity (MSI), create a system-assigned managed identity for each VM in the cluster. User assigned managed identities aren't supported at this time.

2. Create a custom role.

   Your identity needs permissions through Azure RBAC to perform fencing actions against your VMs. To comply with the Least Privileged Access (LPA) Security Model, create a custom RBAC role.

   Use the following definition for your role, replacing your Subscription ID(s) where needed:

   {
         "Name": "Linux Fence Agent",
         "description": "Allowed to power-off and start virtual machines",
         "assignableScopes": [
                 "/subscriptions/<Subscription 1 ID (GUID)>",
                 "/subscriptions/<Subscription N ID (GUID)>"
         ],
         "actions": [
                 "Microsoft.Compute/*/read",
                 "Microsoft.Compute/virtualMachines/powerOff/action",
                 "Microsoft.Compute/virtualMachines/start/action"
         ],
         "notActions": [],
         "dataActions": [],
         "notDataActions": []
   }
   

3. Assign the custom role to your identities.

   • Managed identity
   • Service principal

   For each VM in your cluster, assign its managed identity to the custom "Linux Fence Agent" role for every VM in the cluster, including itself. For detailed steps, see Assign a managed identity access to a resource by using the Azure portal.

   Important

   Be aware that the assignment and removal of authorization with managed identities can be delayed until effective.

1. [A] Enable the SBD service.

   sudo systemctl enable sbd
   

2. [A] Enable software watchdog.

   echo softdog | sudo tee /etc/modules-load.d/softdog.conf
   sudo modprobe softdog
   

3. [A] Discover the iSCSI device ID.
   1. Determine mount point based on LUN number

      ls -l /dev/disk/azure/scsi1/lun1
      lrwxrwxrwx. 1 root root 12 Apr 16 20:22 /dev/disk/azure/scsi1/lun1 -> ../../../sdb
      

   2. Get the iSCSI device ID from mount.

      ls -l /dev/disk/by-id/scsi-3* | grep -i sdb
      lrwxrwxrwx. 1 root root  9 Apr 16 20:22 /dev/disk/by-id/scsi-360022480055c9f501a24256ea0f87617 -> ../../sdb
      

4. [1] Create the SBD device.

   sudo sbd -d /dev/disk/by-id/scsi-360022480055c9f501a24256ea0f87617 -1 60 -4 120 create
   

5. [1] Add the SBD device to the cluster.

   sudo pcs stonith create sbd fence_sbd devices=/dev/disk/by-id/scsi-360022480055c9f501a24256ea0f87617 op monitor interval=600 timeout=15
   

6. [1] Change SBD configuration settings.

   sudo pcs property set stonith-timeout=210
   sudo pcs property set stonith-enabled=true
   

7. [A] Validate SBD config file.

   sudo vi /etc/sysconfig/sbd
   

   [...]
   SBD_DELAY_START=no
   [...]
   SBD_PACEMAKER=yes
   [...]
   SBD_STARTMODE=always
   [...]
   

1. [A] Enable the required services.

   sudo systemctl enable sbd iscsi iscsid
   

2. [A] Enable software watchdog.

   echo softdog | sudo tee /etc/modules-load.d/softdog.conf
   sudo modprobe softdog
   

3. [1] Update the InitiatorName for node 1.

   sudo vi /etc/iscsi/initiatorname.iscsi
   [...]
   InitiatorName=iqn.2006-04.sap-cl1.local:sap-cl1
   

4. [2] Update the InitiatorName for node 2.

   # Node 2
   sudo vi /etc/iscsi/initiatorname.iscsi
   [...]
   InitiatorName=iqn.2006-04.sap-cl2.local:sap-cl2
   

5. [A] Restart the iSCSI services.

   sudo systemctl restart iscsi iscsid
   

6. [A] Mount the iSCSI targets from all the iSCSI host servers.

   for hostServer in sbd-iscsi1 sbd-iscsi2 sbd-iscsi3; do
      iscsiadm -m discovery --type=st --portal=${hostServer}:3260 
      iscsiadm -m node -T iqn.2006-04.ascsnw1.local:ascsnw1 --login --portal=${hostServer}:3260
      iscsiadm -m node --portal=${hostServer}:3260 -T iqn.2006-04.ascsnw1.local:ascsnw1 --op=update --name=node.startup --value=automatic
   done
   

7. [A] Discover the iSCSI device IDs.
   1. Determine the iSCSI mount points.

      lsscsi
      [0:0:0:0]    disk    Msft     Virtual Disk     1.0   /dev/sda
      [1:0:0:1]    disk    Msft     Virtual Disk     1.0   /dev/sdb
      [2:0:0:0]    disk    LIO-ORG  sbdascsnw1       4.0   /dev/sdc
      [3:0:0:0]    disk    LIO-ORG  sbdascsnw1       4.0   /dev/sdd
      [4:0:0:0]    disk    LIO-ORG  sbdascsnw1       4.0   /dev/sde
      

   2. Get the iSCSI device IDs.

      ls -l /dev/disk/by-id/scsi-3* | grep sd[c,d,e]
      lrwxrwxrwx 1 root root  9 Jul 21 18:02 /dev/disk/by-id/scsi-3600140537cf4c6d604a4ae4b58f1a528 -> ../../sdd
      lrwxrwxrwx 1 root root  9 Jul 21 17:50 /dev/disk/by-id/scsi-360014056e4d07b80e1148ac973330dff -> ../../sdc
      lrwxrwxrwx 1 root root  9 Jul 21 18:04 /dev/disk/by-id/scsi-360014059f135275c24647d49268123e5 -> ../../sde
      

8. [1] Create the SBD devices.

   sudo sbd -d /dev/disk/by-id/scsi-3600140537cf4c6d604a4ae4b58f1a528 -1 60 -4 120 create
   sudo sbd -d /dev/disk/by-id/scsi-360014056e4d07b80e1148ac973330dff -1 60 -4 120 create
   sudo sbd -d /dev/disk/by-id/scsi-360014059f135275c24647d49268123e5 -1 60 -4 120 create
   

9. [1] Add the SBD devices to the cluster.

   sudo pcs stonith create sbd fence_sbd \
      devices=/dev/disk/by-id/scsi-3600140537cf4c6d604a4ae4b58f1a528,/dev/disk/by-id/scsi-360014056e4d07b80e1148ac973330dff,/dev/disk/by-id/scsi-360014059f135275c24647d49268123e5 \
      op monitor interval=600 timeout=120
   

10. [1] Change SBD configuration settings.

    sudo pcs property set stonith-timeout=210
    sudo pcs property set stonith-enabled=true
    

11. [A] Validate SBD config file.

    sudo vi /etc/sysconfig/sbd
    

    [...]
    SBD_DELAY_START=no
    [...]
    SBD_PACEMAKER=yes
    [...]
    SBD_STARTMODE=always
    [...]
    

1. [1] Configure Azure Fence Agent.

   Note

   When using Azure government cloud, you must specify the cloud= option when configuring the Azure Fence Agent. For example, cloud=usgov for the Azure US government cloud.

   • Managed identity
   • Service principal

   sudo pcs stonith create rsc_st_azure fence_azure_arm msi=true \
      resourceGroup="<ResourceGroupName>" subscriptionId="<SubscriptionID>" \
      pcmk_host_map="sap-cl1:<AzureVMNameCL1>;sap-cl2:<AzureVMNameCL2>" \
      power_timeout=240 pcmk_reboot_timeout=900 pcmk_monitor_timeout=120 \
      pcmk_monitor_retries=4 pcmk_action_limit=3 pcmk_delay_max=15 \
      meta failure-timeout=120s op monitor interval=3600 timeout=120
   

2. [1] Configure the cluster for Azure Fence Agent.

   sudo pcs property set stonith-enabled=true
   sudo pcs property set stonith-timeout=900