[AMPL 24926] AMPL CE running multiple jobs in parallel using SLURM

Google Group Mirror
1

Hello,

For research purposes, I need to execute a solver built upon ASL on thousands of problem instances. The solver used is custom (made of the interaction of CPLEX + custom routines in an iterative fashion). I have access to a high performance computing cluster using SLURM and would like to be able to integrate AMPL with this.

The header of my licence file looks like this (I am omitting the key and my email address):

Temporary AMPL, BARON, CONOPT, CPLEX, GUROBI, KNITRO, LGO,

LINDOGLOBAL, LOQO, MINOS, SNOPT, XPRESS, OCTERACT, COPT & MOSEK

single-machine server license expiring 20240104

licensed to AMPL Community Edition License for

email.address@server.ca:

The problem is, AMPL will complain when running jobs in parallel, apparently because it won’t allow running multiple jobs on the same machine at once. Is there a way to resolve this?

Thank you for your assistance,

Claudio C

2

Hi Claudio,

Thank you for reaching out. The AMPL CE license should allow multiple runs on the same machine. Could you please provide more details on how multiple instances are being invoked on the same machine? The license needs to be activated once and then multiple processes can use the license and will renew the license as needed. If they processes are running inside containers, then the license needs to be activated once in each container when it starts. Could you please send us the amplkey.log file that should be next to ampl.lic?

3

Hi,

Sorry for reviving this post, but I am currently running into the same problem. I’m copying here the sbatch script I use to submit SLURM jobs (with sensitive information omitted) and the last relevant lines of “amplkey.log”. I think the problem is that each compute node has its own “fingerprint” which requires activation at each node, but I am not sure how to do it.

#SBATCH --account=<omitted>
#SBATCH --mail-user=<omitted>
#SBATCH --mail-type=ALL
#SBATCH --job-name=optprob_
#SBATCH --time=05:00:00
#SBATCH --cpus-per-task=2
#SBATCH --mem-per-cpu=3G
#SBATCH --output=log_sched/%x-%j.out
#SBATCH --array=0-89

# Get config filenames from file
readarray -t myArray < case_list.txt

# Load necessary modules
module load StdEnv/2023
module load python/3.11.5
module load flexiblascore/.3.3.1
module load flexiblas/3.3.1
module load metis/5.1.0
module load ipopt/3.14.14

# Load this project's virtual environment
source $HOME/MyProjects/env_dtspn/bin/activate

# Print task ID
echo "Starting task ${myArray[${SLURM_ARRAY_TASK_ID}]} (ID: $SLURM_ARRAY_TASK_ID)"

# Activate AMPL license for current computer node
$HOME/SOLVERS/amplkey activate --uuid <omitted>

# Submit jobs for each configuration file
python -O src/DTSPN_singlephase.py Input_configs/${myArray[${SLURM_ARRAY_TASK_ID}]}

# Deactivate virtual environment
deactivate

And the amplkey.log is:

2025/10/28 17:06:23 Failed GET request to https://ampl.com/amplkey/v0.1/amplkey.config.json: Get "https://ampl.com/amplkey/v0.1/amplkey.config.json": dial tcp 162.159.135.42:443: i/o timeout
2025/10/28 17:06:26 Failed GET request to https://ampl.com/amplkey/v0.1/amplkey.config.json: Get "https://ampl.com/amplkey/v0.1/amplkey.config.json": dial tcp 162.159.135.42:443: i/o timeout
2025/10/28 17:06:27 Reading /lustre06/project/6006742/walton/SOLVERS/ampl.lic
2025/10/28 17:06:27 Reading /lustre06/project/6006742/walton/SOLVERS/ampl.lic
2025/10/28 17:06:27 Config cache has expired. Trying to update.
2025/10/28 17:06:27 Config cache has expired. Trying to update.
2025/10/28 17:06:28 Failed GET request to https://ampl.com/amplkey/v0.1/amplkey.config.json: Get "https://ampl.com/amplkey/v0.1/amplkey.config.json": dial tcp 162.159.135.42:443: i/o timeout
2025/10/28 17:06:28 Failed GET request to https://ampl.com/amplkey/v0.1/amplkey.config.json: Get "https://ampl.com/amplkey/v0.1/amplkey.config.json": dial tcp 162.159.135.42:443: i/o timeout
2025/10/28 17:06:28 Failed GET request to https://ampl.com/amplkey/v0.1/amplkey.config.json: Get "https://ampl.com/amplkey/v0.1/amplkey.config.json": dial tcp 162.159.135.42:443: i/o timeout
2025/10/28 17:06:30 Reading /lustre06/project/6006742/walton/SOLVERS/ampl.lic
2025/10/28 17:06:30 Config cache has expired. Trying to update.
2025/10/28 17:06:31 Failed GET request to https://ampl.com/amplkey/v0.1/amplkey.config.json: Get "https://ampl.com/amplkey/v0.1/amplkey.config.json": dial tcp 162.159.135.42:443: i/o timeout
2025/10/28 17:06:31 Failed GET request to https://ampl.com/amplkey/v0.1/amplkey.config.json: Get "https://ampl.com/amplkey/v0.1/amplkey.config.json": dial tcp 162.159.135.42:443: i/o timeout
2025/10/28 17:06:32 Reading /lustre06/project/6006742/walton/SOLVERS/ampl.lic
2025/10/28 17:06:32 Config cache has expired. Trying to update.
2025/10/28 17:06:33 Failed GET request to https://ampl.com/amplkey/v0.1/amplkey.config.json: Get "https://ampl.com/amplkey/v0.1/amplkey.config.json": dial tcp 162.159.135.42:443: i/o timeout
2025/10/28 17:06:33 Failed GET request to https://ampl.com/amplkey/v0.1/amplkey.config.json: Get "https://ampl.com/amplkey/v0.1/amplkey.config.json": dial tcp 162.159.135.42:443: i/o timeout
2025/10/28 17:06:33 Failed GET request to https://ampl.com/amplkey/v0.1/amplkey.config.json: Get "https://ampl.com/amplkey/v0.1/amplkey.config.json": dial tcp 162.159.135.42:443: i/o timeout
2025/10/28 17:06:33 Failed GET request to https://ampl.com/amplkey/v0.1/amplkey.config.json: Get "https://ampl.com/amplkey/v0.1/amplkey.config.json": dial tcp 162.159.135.42:443: i/o timeout
2025/10/28 17:06:35 Reading /lustre06/project/6006742/walton/SOLVERS/ampl.lic
2025/10/28 17:06:35 Config cache has expired. Trying to update.
2025/10/28 17:06:36 Failed GET request to https://ampl.com/amplkey/v0.1/amplkey.config.json: Get "https://ampl.com/amplkey/v0.1/amplkey.config.json": dial tcp 162.159.135.42:443: i/o timeout
2025/10/28 17:06:36 Failed GET request to https://ampl.com/amplkey/v0.1/amplkey.config.json: Get "https://ampl.com/amplkey/v0.1/amplkey.config.json": dial tcp 162.159.135.42:443: i/o timeout
2025/10/28 17:06:38 Failed GET request to https://ampl.com/amplkey/v0.1/amplkey.config.json: Get "https://ampl.com/amplkey/v0.1/amplkey.config.json": dial tcp 162.159.135.42:443: i/o timeout
2025/10/28 17:06:37 Failed GET request to https://ampl.com/amplkey/v0.1/amplkey.config.json: Get "https://ampl.com/amplkey/v0.1/amplkey.config.json": dial tcp 162.159.135.42:443: i/o timeout

Also, the first lines of one of my log files:

cat log_sched/optprob_-51459114.out 
Starting task optprob-6-0_no_0.cfg (ID: 0)
Error: Failed to request activation. Error: <nil>

Thank you in advance for any help.

Best,
Walton.

ps: I am able to activate my license and run my code in the login node. This problem happens when I try to submit jobs to the computing nodes.

1 Like