Deployed 96ba949 with MkDocs version: 1.6.1

Author: cartalla

debug/index.html

@@ -128,6 +128,18 @@
 
             <li class="nav-item" data-bs-level="1"><a href="#debug" class="nav-link">Debug</a>
               <ul class="nav flex-column">
+            <li class="nav-item" data-bs-level="2"><a href="#config-stack-deploys-but-parallelcluster-stack-doesnt" class="nav-link">Config stack deploys, but ParallelCluster stack doesn't</a>
+              <ul class="nav flex-column">
+              </ul>
+            </li>
+            <li class="nav-item" data-bs-level="2"><a href="#parallelcluster-stack-creation-fails" class="nav-link">ParallelCluster stack creation fails</a>
+              <ul class="nav flex-column">
+            <li class="nav-item" data-bs-level="3"><a href="#headnodewaitcondition-failed-to-create" class="nav-link">HeadNodeWaitCondition failed to create</a>
+              <ul class="nav flex-column">
+              </ul>
+            </li>
+              </ul>
+            </li>
             <li class="nav-item" data-bs-level="2"><a href="#slurm-head-node" class="nav-link">Slurm Head Node</a>
               <ul class="nav flex-column">
               </ul>
@@ -157,6 +169,21 @@
 
 <h1 id="debug">Debug</h1>
 <p>For ParallelCluster and Slurm issues, refer to the official <a href="https://docs.aws.amazon.com/parallelcluster/latest/ug/troubleshooting-v3.html">AWS ParallelCluster Troubleshooting documentation</a>.</p>
+<h2 id="config-stack-deploys-but-parallelcluster-stack-doesnt">Config stack deploys, but ParallelCluster stack doesn't</h2>
+<p>This happens when the lambda function that create the cluster encounters an error.
+This is usually some kind of configuration error that is detected by ParallelCluster.</p>
+<ul>
+<li>Open the CloudWatch console and go the log groups</li>
+<li>Find the log group named /aws/lambda/*-CreateParallelCluster</li>
+<li>Look for the error</li>
+</ul>
+<h2 id="parallelcluster-stack-creation-fails">ParallelCluster stack creation fails</h2>
+<h3 id="headnodewaitcondition-failed-to-create">HeadNodeWaitCondition failed to create</h3>
+<p>If the stack fails with an error like:</p>
+<p><code>The following resoure(s) failed to create
+[HeadNodeWaitCondition2025050101134602]</code></p>
+<p>Connect to the head node and look in <code>/var/log/ansible.log</code> for errors.</p>
+<p>If it shows that it failed waiting for slurmctld to accept requests then check <code>/var/log/slurmctld.log</code> for errors.</p>
 <h2 id="slurm-head-node">Slurm Head Node</h2>
 <p>If slurm commands hang, then it's likely a problem with the Slurm controller.</p>
 <p>Connect to the head node from the EC2 console using SSM Manager or ssh and switch to the root user.</p>

deployment-prerequisites/index.html

@@ -259,11 +259,11 @@ <h2 id="make-sure-using-at-least-python-version-37">Make sure using at least pyt
 Simply install the newer version and then use it to create and activate a virtual environment.</p>
 <pre><code>$ python3 --version
 Python 3.6.8
-$ yum -y install python3.11
-$ python3.11 -m venv ~/.venv-python3.11
-$ source ~/.venv-python3.11/bin/activate
+$ yum -y install python3.12
+$ python3.12 -m venv ~/.venv-python3.12
+$ source ~/.venv-python3.12/bin/activate
 $ python3 --version
-Python 3.11.5
+Python 3.12.8
 </code></pre>
 <h2 id="make-sure-required-packages-are-installed">Make sure required packages are installed</h2>
 <pre><code>cd aws-eda-slurm-cluster
@@ -283,10 +283,10 @@ <h3 id="install-cloud-development-kit-cdk-optional">Install Cloud Development Ki
 <p>The following link documents how to setup for CDK.
 Follow the instructions for Python.</p>
 <p><a href="https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_prerequisites">https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_prerequisites</a></p>
-<p>Note that CDK requires a pretty new version of nodejs which you may have to download from, for example, <a href="https://nodejs.org/dist/v16.13.1/node-v16.13.1-linux-x64.tar.xz">https://nodejs.org/dist/v16.13.1/node-v16.13.1-linux-x64.tar.xz</a></p>
+<p>Note that CDK requires a pretty new version of nodejs which you may have to download from, for example, <a href="https://nodejs.org/dist/v20.19.0/node-v20.19.0-linux-x64.tar.xz">https://nodejs.org/dist/v20.19.0/node-v20.19.0-linux-x64.tar.xz</a></p>
 <pre><code>sudo yum -y install wget
-wget https://nodejs.org/dist/v16.13.1/node-v16.13.1-linux-x64.tar.xz
-tar -xf node-v16.13.1-linux-x64.tar.xz ~
+wget https://nodejs.org/dist/v20.19.0/node-v20.19.0-linux-x64.tar.xz
+tar -xf node-v20.19.0-linux-x64.tar.xz ~
 </code></pre>
 <p>Add the nodjs bin directory to your path.</p>
 <p><a href="https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_install">https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_install</a></p>
@@ -323,6 +323,12 @@ <h2 id="create-slurmdbd-instance">Create Slurmdbd Instance</h2>
 <p><strong>Note</strong>: Before ParallelCluster 3.10.0, the slurmdbd daemon that connects to the data was created on each cluster's head node.
 The recommended Slurm architecture is to have a shared slurmdbd daemon that is used by all of the clusters.
 Starting in version 3.10.0, ParallelCluster supports specifying an external slurmdbd instance when you create a cluster and provide a cloud formation template to create it.</p>
+<p><strong>Note</strong>: The Slurm version used by slurmdbd must be greater than or equal to the version of your clusters.
+If you have already deployed a slurmdbd instance then you will need to create a new slurmdbd
+instance with the latest version of ParallelCluster.
+Also note that Slurm only maintains backwards compatibility for the 2 previous major releases so
+at some point you will need upgrade your clusters to newer versions before you can use the latest version
+of ParallelCluster.</p>
 <p>Follow the directions in this <a href="https://docs.aws.amazon.com/parallelcluster/latest/ug/external-slurmdb-accounting.html#external-slurmdb-accounting-step1">ParallelCluster tutorial to configure slurmdbd</a>.
 This requires that you have already created the slurm database.</p>
 <p>Here are some notes on the required parameters and how to fill them out.</p>

exostellar-infrastructure-optimizer/index.html

@@ -170,6 +170,14 @@
             </li>
             <li class="nav-item" data-bs-level="2"><a href="#debug" class="nav-link">Debug</a>
               <ul class="nav flex-column">
+            <li class="nav-item" data-bs-level="3"><a href="#how-to-connect-to-ems" class="nav-link">How to connect to EMS</a>
+              <ul class="nav flex-column">
+              </ul>
+            </li>
+            <li class="nav-item" data-bs-level="3"><a href="#how-to-connect-to-controller" class="nav-link">How to connect to Controller</a>
+              <ul class="nav flex-column">
+              </ul>
+            </li>
             <li class="nav-item" data-bs-level="3"><a href="#updateheadnode-resource-failed" class="nav-link">UpdateHeadNode resource failed</a>
               <ul class="nav flex-column">
               </ul>
@@ -808,6 +816,19 @@ <h3 id="run-a-test-job-using-slurm">Run a test job using Slurm</h3>
 <pre><code>srun --pty -p xio-
 </code></pre>
 <h2 id="debug">Debug</h2>
+<h3 id="how-to-connect-to-ems">How to connect to EMS</h3>
+<p>Use ssh to connect to the EMS using your EC2 keypair.</p>
+<ul>
+<li><code>ssh-add private-key.pem</code></li>
+<li><code>ssh -A rocky@${EMS_IP_ADDRESS}</code></li>
+</ul>
+<p>You can <a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/agent-install-rocky.html">install the aws-ssm-agent</a> so that you can connect from the EC2 console using SSM.</p>
+<h3 id="how-to-connect-to-controller">How to connect to Controller</h3>
+<ul>
+<li>First ssh to the EMS.</li>
+<li>Get the IP address of the controller from the EC2 console</li>
+<li>As root, ssh to the controller</li>
+</ul>
 <h3 id="updateheadnode-resource-failed">UpdateHeadNode resource failed</h3>
 <p>If the UpdateHeadNode resource fails then it is usually because as task in the ansible script failed.
 Connect to the head node and look for errors in:</p>
@@ -816,17 +837,30 @@ <h3 id="updateheadnode-resource-failed">UpdateHeadNode resource failed</h3>
 <p>When this happens the CloudFormation stack will usually be in UPDATE_ROLLBACK_FAILED status.
 Before you can update it again you will need to complete the rollback.
 Go to Stack Actions, select <code>Continue update rollback</code>, expand <code>Advanced troubleshooting</code>, check the UpdateHeadNode resource, anc click <code>Continue update rollback</code>.</p>
+<p>The problem is usually that there is an XWO controller running that is preventing updates to
+the profile.
+Cancel any XWO jobs and terminate any running workers and controllers and verify that all of the XWO profiles are idle.</p>
 <h3 id="xio-controller-not-starting">XIO Controller not starting</h3>
 <p>On EMS, check that a job is running to create the controller.</p>
 <p><code>squeue</code></p>
 <p>On EMS, check the autoscaling log to see if there are errors starting the instance.</p>
 <p><code>less /var/log/slurm/autoscaling.log</code></p>
-<p>EMS Slurm partions are at:</p>
+<p>EMS Slurm partitions are at:</p>
 <p><code>/xcompute/slurm/bin/partitions.json</code></p>
 <p>They are derived from the partition and pool names.</p>
 <h3 id="worker-instance-not-starting">Worker instance not starting</h3>
 <h3 id="vm-not-starting-on-worker">VM not starting on worker</h3>
-<h3 id="vm-not-starting-slurm-job">VM not starting Slurm job</h3></div>
+<p>Connect to the controller instance and run the following command to get a list of worker instances and VMs.</p>
+<pre><code>xspot ps
+</code></pre>
+<p>Connect to the worker VM using the following command.</p>
+<pre><code>xspot console vm-abcd
+</code></pre>
+<p>This will show the console logs.
+If you configured the root password then you can log in as root to do further debug.</p>
+<h3 id="vm-not-starting-slurm-job">VM not starting Slurm job</h3>
+<p>Connect to the VM as above.</p>
+<p>Check /var/log/slurmd.log for errors.</p></div>
             </div>
         </div>
 

exostellar-workload-optimizer/index.html

@@ -174,6 +174,14 @@
             </li>
             <li class="nav-item" data-bs-level="2"><a href="#debug" class="nav-link">Debug</a>
               <ul class="nav flex-column">
+            <li class="nav-item" data-bs-level="3"><a href="#how-to-connect-to-ems" class="nav-link">How to connect to EMS</a>
+              <ul class="nav flex-column">
+              </ul>
+            </li>
+            <li class="nav-item" data-bs-level="3"><a href="#how-to-connect-to-controller" class="nav-link">How to connect to Controller</a>
+              <ul class="nav flex-column">
+              </ul>
+            </li>
             <li class="nav-item" data-bs-level="3"><a href="#updateheadnode-resource-failed" class="nav-link">UpdateHeadNode resource failed</a>
               <ul class="nav flex-column">
               </ul>
@@ -343,6 +351,19 @@ <h3 id="run-a-test-job-using-slurm">Run a test job using Slurm</h3>
 <pre><code>srun --pty -p xwo-amd-64g-4c hostname
 </code></pre>
 <h2 id="debug">Debug</h2>
+<h3 id="how-to-connect-to-ems">How to connect to EMS</h3>
+<p>Use ssh to connect to the EMS using your EC2 keypair.</p>
+<ul>
+<li><code>ssh-add private-key.pem</code></li>
+<li><code>ssh -A rocky@${EMS_IP_ADDRESS}</code></li>
+</ul>
+<p>You can <a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/agent-install-rocky.html">install the aws-ssm-agent</a> so that you can connect from the EC2 console using SSM.</p>
+<h3 id="how-to-connect-to-controller">How to connect to Controller</h3>
+<ul>
+<li>First ssh to the EMS.</li>
+<li>Get the IP address of the controller from the EC2 console</li>
+<li>As root, ssh to the controller</li>
+</ul>
 <h3 id="updateheadnode-resource-failed">UpdateHeadNode resource failed</h3>
 <p>If the UpdateHeadNode resource fails then it is usually because a task in the ansible script failed.
 Connect to the head node and look for errors in:</p>
@@ -351,6 +372,9 @@ <h3 id="updateheadnode-resource-failed">UpdateHeadNode resource failed</h3>
 <p>When this happens the CloudFormation stack will usually be in UPDATE_ROLLBACK_FAILED status.
 Before you can update it again you will need to complete the rollback.
 Go to Stack Actions, select <code>Continue update rollback</code>, expand <code>Advanced troubleshooting</code>, check the UpdateHeadNode resource, anc click <code>Continue update rollback</code>.</p>
+<p>The problem is usually that there is an XWO controller running that is preventing updates to
+the profile.
+Cancel any XWO jobs and terminate any running workers and controllers and verify that all of the XWO profiles are idle.</p>
 <h3 id="xwo-controller-not-starting">XWO Controller not starting</h3>
 <p>If a controller doesn't start, then the first thing to check is to make sure that the
 <code>/opt/slurm/exostellar/resume_xspot.sh</code> script ran successfully on the head node.</p>
@@ -361,12 +385,22 @@ <h3 id="xwo-controller-not-starting">XWO Controller not starting</h3>
 <p><code>squeue</code></p>
 <p>On EMS, check the autoscaling log to see if there are errors starting the instance.</p>
 <p><code>less /var/log/slurm/autoscaling.log</code></p>
-<p>EMS Slurm partions are at:</p>
+<p>EMS Slurm partitions are at:</p>
 <p><code>/xcompute/slurm/bin/partitions.json</code></p>
 <p>They are derived from the partition and pool names.</p>
 <h3 id="worker-instance-not-starting">Worker instance not starting</h3>
 <h3 id="vm-not-starting-on-worker">VM not starting on worker</h3>
-<h3 id="vm-not-starting-slurm-job">VM not starting Slurm job</h3></div>
+<p>Connect to the controller instance and run the following command to get a list of worker instances and VMs.</p>
+<pre><code>xspot ps
+</code></pre>
+<p>Connect to the worker VM using the following command.</p>
+<pre><code>xspot console vm-abcd
+</code></pre>
+<p>This will show the console logs.
+If you configured the root password then you can log in as root to do further debug.</p>
+<h3 id="vm-not-starting-slurm-job">VM not starting Slurm job</h3>
+<p>Connect to the VM as above.</p>
+<p>Check /var/log/slurmd.log for errors.</p></div>
             </div>
         </div>
 

index.html

@@ -329,5 +329,5 @@ <h4 class="modal-title" id="keyboardModalLabel">Keyboard Shortcuts</h4>
 
 <!--
 MkDocs version : 1.6.1
-Build Date UTC : 2025-04-30 21:30:56.431142+00:00
+Build Date UTC : 2025-05-12 17:13:22.467008+00:00
 -->