Configuring the Slurm License Sensor

The Slurm license sensor script regularly queries the license servers and updates the number of available licenses in the Slurm accounting database. It automatically updates the Slurm accounting database with changes to the license, for example, when new licenses are added. Each Slurm accounting database requires one Slurm license sensor.

Edit the file $SCHRODINGER/utilities/slurm-license-sensor:
- CLUSTER_DNS_NAME_REGEX: a regular expression that matches the DNS names of all instances managed by Slurm, but does not match any instance not managed by Slurm. This is used to distinguish between license checkouts from within the Slurm cluster and external resources, for example, a laptop or a workstation.
- SLURM_CLUSTERS: a comma-separated list of all clusters managed by Slurm that report to the same Slurm accounting database
- SLURM_INSTALLDIR: the installation directory of Slurm
- SCHRODINGER_LICENSING_NAMESPACE = "SLM_": Set the prefix to add for each license feature to distinguish whether it is using FlexLM licenses or the Schrodinger Licensing Manager. This needs to match the prefix used in the setup to generate the configuration for license resource attributes (See Step 1 of Configuring Local License Checking). If you are no longer using FlexLM licenses, and transitioned to use Schrodinger Licensing, this can be set to an empty string.
Confirm that the license sensor can query the license server and passes this information to the Slurm accounting database:
```
$SCHRODINGER/utilities/slurm-license-sensor
```
Note: This command is best executed as the same user that updates the Slurm accounting database, for example the slurm user. This command requires that the slurm user account has access to the license server that hosts the licenses for the Schrödinger Suite.
Confirm that the license information is available in Slurm:
```
scontrol show lic
```

The names of the licenses stored in the Slurm accounting database are the same as found in the license file of the Schrödinger Suite. For example, if there is one license server hosting 100 licenses of feature X to two Slurm clusters, alice and bob, modify the SLURM_CLUSTERS setting:

SLURM_CLUSTERS = ["alice", "bob"]

The Slurm license sensor adds the resource information to the Slurm accounting database as follows:

sacctmgr -i add resource name=X count=100 server=schrodinger_alice cluster=alice servertype=flexlm percentallowed=100
sacctmgr -i add resource name=X count=100 server=schrodinger_bob cluster=bob servertype=flexlm percentallowed=100

Each Slurm cluster is listed in the Slurm accounting database in their own “server” namespace, using up to 100% of the available licenses. This avoids the need to assign fixed percentages of permitted license usage to each cluster. For example, if both clusters were to share the same server=schrodinger namespace with percentallowed=50, jobs submitted to cluster alice would only be able to use up to 50% of the licenses, even if there are no jobs running on cluster bob. See the Slurm Licenses Guide for more information.

To keep the information in the Slurm accounting database up to date, the license sensor should run in regular, short intervals. We recommend executing the license sensor every 30 seconds. An easy way to implement this is by using a systemd timer.

Creating a Systemd Timer for the Slurm License Sensor

Create a file /etc/systemd/system/slurm-license-sensor.service:
```
[Unit]
Description=Schrodinger license sensor for Slurm
After=slurmctld.service slurmdbd.service

[Service]
Type=simple
User=slurm
ExecStart=SCHRODINGER/run SCHRODINGER/utilities/slurm-license-sensor
					
[Install]
WantedBy=multiuser.target
```
Replace SCHRODINGER with the absolute path of the Schrödinger Suite installation. If users access the license server via environment variables LM_LICENSE_FILE or SCHROD_LICENSE_FILE, you can add a line:
```
Environment=SCHROD_LICENSE_FILE=PORT@LICENSE-SERVER
```
to the [Service] section of the service file.
Create a file /etc/systemd/system/slurm-license-sensor.timer
```
[Unit]
Description=Schrodinger license sensor for Slurm

[Timer]
OnCalendar=*:*:0/30
AccuracySec=1s

[Install]
WantedBy=timers.target
```
The line OnCalendar=*:*:0/30 specifies that the license sensor script should run every 30 seconds.

Change permissions to make the file executable:

chmod +x $SCHRODINGER/utilities/slurm-license-sensor

Enable and activate the timer:

systemctl enable slurm-license-sensor.timer
systemctl start slurm-license-sensor.timer