Custom jobs are submitted but nothing happens
When a custom job is submitted it remains in the Submitted or Running state but no processor usage is indicated in the "System overview" tab. Also, steps in the job output from "Monitoring" tab do not indicate any progress in execution.
Also, executing the following commands result in a timeout from Slurm srun command:
sudo su -l sen2agri-service srun ls -al
which returns something like:
srun: Required node not available (down, drained or reserved) srun: job 3257 queued and waiting for resources
This usually happens when during high level products processing (L3X, L4X), the root partition (/) became full and no disk space was available on this partition.
In this case, the Slurm does not execute any commands and it should be re-initiated.
sudo -u sen2agri-service scontrol update NodeName=localhost State=RESUME sudo systemctl restart slurmd slurmdbd slurmctld mariadb
The succes of the operation can be checked again by :
sudo su -l sen2agri-service srun ls -al
which should display now the list of files in the current directory.
