Overview

In Ignition version 8.1.13, a feature was added to set the Gateway to automatically capture and record thread dumps during the critical CPU events.

When enabled, Ignition will conduct automatic thread dumps at the specified interval when above the target CPU Usage Threshold. If the threshold is met or exceeded, a timestamped thread dump will be saved to the logs folder. When the number of thread dumps in the logs folder exceeds the Retention Count, old files will be automatically removed.

The default sample rate (60 seconds) can be overridden by adding an additional parameter to the Ignition configuration file.

Additional Parameter:

This method would be the simplest; however if you are not on 8.1.13, or if you need to take a thread dump of an active Client instead of the Gateway, then you can perform one of the following methods instead.

Method 1: Use system.util.threadDump() on a Gateway Tag Change Script that is Triggered by the Value Change of the Gateway’s CPU Tag

You can use the following script on the Gateway CPU Tag's Tag Change script:

import os
import time

#Add an if statement here to check if the CPU is above a specific percentage where 0.00 is 0% and 1.00 is 100%
#if newValue.getValue() > [CPU percentage]:
cpu = str(int(100*newValue.getValue()))
now = system.date.now()
hour = str(system.date.getHour24(now))
minute = str(system.date.getMinute(now))
second = str(system.date.getSecond(now))
date = str(system.date.getDayOfMonth(now))
month = str(system.date.getMonth(now)+1)
if int(month) < 10: 
    month = "0"+month  
if int(minute) < 10:
    minute = "0"+minute
if int(hour) < 10:
    hour = "0"+hour
if int(second) < 10:
    second = "0"+second
if int(date) < 10:
    date = "0"+date
#basePath = "/var/log/ignition/cpu-monitoring/"
basePath = "C:\\Program Files\\Inductive Automation\\Ignition\\logs\\cpu-monitoring\\\\"
filePath = basePath+month+"-"+date+"--"+hour+"h"+minute+"m"+second+"s CPU at "+cpu+".txt"
threads = system.util.threadDump()
system.file.writeFile(filePath,threads)
for relFile in os.listdir(basePath):
    fullPath = basePath+relFile
#   print os.path.getctime(direc+all)*1000
    created = system.date.fromMillis(long(int(os.path.getctime(fullPath)*1000)))
    if system.date.daysBetween(created,system.date.now()) >= 1:
        os.remove(fullPath)

Method 2: Use system.util.threadDump() on a Tag Event Script that is triggered by the Value Change of the Gateway’s CPU Tag and Store the Thread Dumps in a Folder on the Server

You can use the following script on the Gateway CPU Tag's Tag Event script:

import os
import time

#Add an if statement here to check if the CPU is above a specific percentage where 0.00 is 0% and 1.00 is 100%
#if currentValue.value > 1.00:

cpu = str(int(100*currentValue.value))
now = system.date.now()
hour = str(system.date.getHour24(now))
minute = str(system.date.getMinute(now))
second = str(system.date.getSecond(now))
date = str(system.date.getDayOfMonth(now))
month = str(system.date.getMonth(now)+1)
if int(month) < 10: 
    month = "0"+month  
if int(minute) < 10:
    minute = "0"+minute
if int(hour) < 10:
    hour = "0"+hour
if int(second) < 10:
    second = "0"+second
if int(date) < 10:
    date = "0"+date
cpu = str(int(100*currentValue.value))
#basePath = "/var/log/ignition/cpu-monitoring/"
basePath = "C:\\Program Files\\Inductive Automation\\Ignition\\logs\\cpu-monitoring\\\\"
filePath = basePath+month+"-"+date+"--"+hour+"h"+minute+"m"+second+"s CPU at "+cpu+".txt"
threads = system.util.threadDump()
system.file.writeFile(filePath,threads)
for relFile in os.listdir(basePath):
    fullPath = basePath+relFile
#   print os.path.getctime(direc+all)*1000
    created = system.date.fromMillis(long(int(os.path.getctime(fullPath)*1000)))
    if system.date.daysBetween(created,system.date.now()) >= 1:
        os.remove(fullPath)

The main difference between the Tag Event and Gateway Tag Change methods is that Tag Event script method will utilize the gateway-tag-events thread, whereas the Gateway Tag Change script method will have its own dedicated thread.

Method 3: Manually Download the Thread Dump From the Gateway Webpage

The Ignition Gateway webpage has a section that allows users to view snapshots of the most recently active threads on the Gateway. This page is located at Status > Diagnostics: Threads section of the Gateway webpage. From this page, you can download thread dumps as a .txt file.

While taking thread dumps from the Gateway webpage, you should pause live values to reduce impact on the Gateway’s CPU. This helps reduce load on the CPU, since the live values page is regularly taking snapshots of the threads.

You should take at least 3-5 thread dumps about 10-15 seconds apart. The reason you want to do this is to provide enough time for any threads to resolve. The threads with the same stack trace that persist throughout all the thread dumps are suspicious, because their processes existed through the course of the issue, so they would be one of the first things to look for in a thread dump.

Method 4: Save Thread Dumps using the Diagnostics Menu (Designer/Vison Clients)

In Ignition version 8.1.12, a feature was added to allow users to manually save Client threads similar to the Threads page on the Ignition Gateway webpage. Users can navigate to the top menu bar of a Designer/Vision Client (if enabled, Project Properties > User Interface > Uncheck “Hide Menu Bar”) and select Help > Diagnostics. In the Thread Viewer tab, there is an option at the bottom to Save Thread Dump.

Just like in Method 3, you should take at least 3-5 thread dumps taken about 10-15 seconds apart.

Method 5: Use Jstack to Take the Thread Dump

The jstack utility from the Java Development Kit (JDK) is a useful tool for obtaining thread dumps, especially when the Gateway or Client is unresponsive. Otherwise, one of the following methods can be used:

Gateway/Perspective:

Use the Automatic Thread dump setting (8.1.13+)
Use a script that calls system.util.threadDump() to automatically take thread dumps based off the CPU Usage Tag
Manually download thread dumps from the Gateway webpage.

Designer/Vison Client:

Call system.util.threadDump() from within the client
Manually download thread dumps from the Diagnostics menu of the Designer or Vision Client (8.1.12+)

In order to use jstack on an Ignition Gateway or Client, you will need the following:

The JDK installed.
The ability to run the command line (cmd) with admin privileges.
The Process ID (PID) for the Java process running the Ignition Gateway or Client.

Windows:

Step 1: Download and Install the JDK Azul Zulu Java Build: https://www.azul.com/downloads/?package=jdk

Oracle Java Build: https://www.oracle.com/java/technologies/downloads/

Download the JDK for the respective Java version your customer is using. Ignition 8+ utilizes Java Runtime Environment (JRE) 11, so you would download JDK 11.

Step 2: Obtaining the PID

Windows 8, 8.1, and 10:

Open Task Manager (Ctrl + Shift + Esc). Navigate to the Details tab. Find the Java process in the Name column and determine its PID from the PID column.

Windows 7, Vista, XP:

Open Task Manager (Ctrl + Shift + Esc). Navigate to the Processes tab and click View > Select Columns. Select the "PID (Process Identifier)" check box and click OK. In the Processes tab, find the Java process in the Image Name column and determine its PID from the PID column.

You can also get the PID for the Java process by running netstat and looking for the port Ignition listens on:

netstat -anon | find "<PORT>"

Replace <PORT> with the port number Ignition listens on.

Step 3: Run jstack

Open the cmd with admin privileges and run jstack, which is located in the "bin" folder of the JDK:

"C:\\Program Files\\Java\\[jdk version]\\bin\\jstack" <PID> > "C:\\thread-dumps\\thread-dump-1.txt"

If using Azul Zulu Java, replace [jdk version] with the jdk-#, where # is the JDK version (e.g. jdk-11), and <PID> with the PID from Step 2.

If using Oracle Java, replace [jdk version] with jdk1.8.0_###, where ### is the JDK version (e.g. jdk1.8.0_311), and <PID> with the PID from Step 2.

Since a thread dump is a "snapshot" from the time it was taken, it might be useful to grab several thread dumps over a period of time to get a better picture. The command below creates a thread dump every 3 seconds until 5 thread dumps have been generated:

FOR /L %i IN (1, 1, 5) DO ("C:\\Program Files\\Java\\[jdk version]\\bin\\jstack" <PID> > "C:\\thread-dumps\\thread-dump-%i.txt" && timeout 3)

Linux:

Step 1: Download the JDK

You can install OpenJDK-11 onto a Linux machine using the following command:

apt-get install openjdk-11-jdk-headless

Step 2: Obtain the PID

You can use one of the following two methods to get the PID of the Ignition Gateway:

Method 1:

You can use the Linux terminal command "jps" (Java Virtual Machine Process Status Tool) to list out the Java Virtual Machines (JVMs) running on the local system. The process ID that you want should be for the org.tanukisoftware.wrapper.WrapperSimpleApp process. The process ID for the Ignition Gateway in this example is 1359.

$ jps -l
--
14729 jdk.jcmd/sun.tools.jps.Jps
1359 org.tanukisoftware.wrapper.WrapperSimpleApp

Method 2:

You can also use the Linux terminal command "ps" to list out the currently running processes and filter the list by those related to java.

Run the following line:

ps -aux | grep java

With the method, you can list out any processes that utilizes files with the word Java in it. There may be more than one process that is returned from this line, but the one you should look for should have a similar name as Method 1's example (org.tanukisoftware.wrapper.WrapperSimpleApp). In addition, the process should be using the JRE located in /lib/runtime/jre-nix/bin/java of the Ignition install directory. Below is an example output of the Ignition Gateway's process using "ps". The PID of this process is 1359, just like in Method 1.

ps -aux | grep java
--
[UserIndentifier] 1359 0.6 19.4 5906512 1582888 ? Sl May23 85:50 /home/[username]/Desktop/[Ignition install directory]/lib/runtime/jre-nix/bin/java -Ddata.dir=data -Xms1024m -Xmx2 048m -Djava.library.path=lib -classpath lib/wrapper.jar:lib/core/common/annotati ons-13.0.jar:lib/core/common/barcode4j-2.1.jar:lib/core/common/bcpkix-jdk15on-1. 69.jar:lib/core/common/bcprov-jdk15on-1.69.jar:lib/core/common/bcutil-jdk15on-1. 69.jar:lib/core/common/common.jar:lib/core/common/commons-cli-1.4.jar:lib/core/c ommon/commons-codec-1.13.jar:lib/core/common/commons-collections4-4.4.jar:lib/co re/common/commons-compress-1.19.jar:lib/core/common/commons-io-2.2.jar:lib/core/ common/commons-lang3-3.11.jar:lib/core/common/commons-logging-1.2.jar:lib/core/common/commons-math3-3.6.1.jar:lib/core/common/commons-text-1.9.jar:lib/core/comm on/core-java6-3.2.1.4.jar:lib/core/common/cron4j-2.2.5.jar:lib/core/common/curve sapi-1.06.jar:lib/core/common/guava-26.0-jre.jar:lib/core/common/httpclient-4.5. 13.jar:lib/core/common/httpcore-4.4.13.jar:lib/core/common/ia-gson-2.8.5.jar:lib /core/common/java-cup-10k.jar:lib/core/common/jfreechart-min-1.0.0.jar:lib/core/ common/jgrapht-core-1.4.0.jar:lib/core/common/jheaps-0.11.jar:lib/core/common/ji de-common-3.7.4.jar:lib/core/common/joda-time-2.10.2.jar:lib/core/common/jose4j- 0.7.2.jar:lib/core/common/jul-to-slf4j-1.7.26.jar:lib/core/common/jython-ia-2.7. 2.1.jar:lib/core/common/kotlin-stdlib-1.5.31.jar:lib/core/common/kotlin-stdlib-c ommon-1.5.31.jar:lib/core/common/kotlin-stdlib-jdk7-1.5.31.jar:lib/core/common/k otlin-stdlib-jdk8-1.5.31.jar:lib/core/common/kotlinx-coroutines-core-jvm-1.5.2.j ar:lib/core/common/log4j-over-slf4j-1.7.26.jar:lib/core/common/logback-classic-1 .2.3.jar:lib/core/common/logback-core-1.2.3.jar:lib/core/common/metrics-core-4.2 .0.jar:lib/core/common/poi-4.1.1.jar:lib/core/common/poi-ooxml-4.1.1.jar:lib/cor e/common/poi-ooxml-schemas-4.1.1.jar:lib/core/common/rsyntaxtextarea-3.0.3.jar:l ib/core/common/slf4j-api-1.7.26.jar:lib/core/common/xmlbeans-3.1.0.jar:lib/core/ gateway/FastInfoset-1.2.16.jar:lib/core/gateway/TableLayout-20050920.jar:lib/cor e/gateway/commons-fileupload-1.4.jar:lib/core/gateway/gateway-8.1.17.jar:lib/cor e/gateway/gateway-api-8.1.17.jar:lib/core/gateway/hasp-srm-api-1.0.0.jar:lib/cor e/gateway/hsqldb-2.6.0.jar:lib/core/gateway/istack-commons-runtime-3.0.8.jar:lib /core/gateway/jSerialComm-2.6.2.jar:lib/core/gateway/jakarta.activation-1.2.1.ja r:lib/core/gateway/jakarta.activation-api-1.2.1.jar:lib/core/gateway/jakarta.xml .bind-api-2.3.2.jar:lib/core/gateway/javax.mail-1.5.2.jar:lib/core/gateway/javax .mail-api-1.5.2.jar:lib/core/gateway/javax.servlet-api-3.1.0.jar:lib/core/gatewa y/jaxb-runtime-2.3.2.jar:lib/core/gateway/jetty-client-9.4.45.v20220203.jar:lib/ core/gateway/jetty-http-9.4.45.v20220203.jar:lib/core/gateway/jetty-io-9.4.45.v2 0220203.jar:lib/core/gateway/jetty-rewrite-9.4.45.v20220203.jar:lib/core/gateway /jetty-security-9.4.45.v20220203.jar:lib/core/gateway/jetty-server-9.4.45.v20220 203.jar:lib/core/gateway/jetty-servlet-9.4.45.v20220203.jar:lib/core/gateway/jet ty-util-9.4.45.v20220203.jar:lib/core/gateway/jetty-util-ajax-9.4.45.v20220203.j ar:lib/core/gateway/jetty-webapp-9.4.45.v20220203.jar:lib/core/gateway/jetty-xml -9.4.45.v20220203.jar:lib/core/gateway/libignition-x64.so:lib/core/gateway/metro -8.1.17.jar:lib/core/gateway/protobuf-java-3.8.0.jar:lib/core/gateway/simple-orm -8.1.17.jar:lib/core/gateway/sqlite-jdbc-3.23.1.jar:lib/core/gateway/stax2-api-4 .2.1.jar:lib/core/gateway/stax-ex-1.8.1.jar:lib/core/gateway/tape-1.2.3.jar:lib/ core/gateway/timingframework-1.0.jar:lib/core/gateway/txw2-2.3.2.jar:lib/core/ga teway/websocket-api-9.4.45.v20220203.jar:lib/core/gateway/websocket-client-9.4.4 5.v20220203.jar:lib/core/gateway/websocket-common-9.4.45.v20220203.jar:lib/core/ gateway/websocket-server-9.4.45.v20220203.jar:lib/core/gateway/websocket-servlet -9.4.45.v20220203.jar:lib/core/gateway/wicket-core-6.30.0.jar:lib/core/gateway/w icket-datetime-6.30.0.jar:lib/core/gateway/wicket-extensions-6.30.0.jar:lib/core /gateway/wicket-request-6.30.0.jar:lib/core/gateway/wicket-util-6.30.0.jar:lib/c ore/gateway/woodstox-core-6.2.6.jar:lib/core/gateway/xmlsec-2.3.0.jar -Dwrapper. key=K13F7bZrF-2WMrC9NWp_Wmxxu_H0Vhj4 -Dwrapper.port=32000 -Dwrapper.jvm.port.min =31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.disable_console_input=TRUE -Dwrapper.pid=1104 -Dwrapper.version=3.5.42-st -Dwrapper.native_library=wrapper -Dwrapp er.arch=x86 -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 -D wrapper.lang.domain=wrapper -Dwrapper.lang.folder=../lang org.tanukisoftware.wrapper.WrapperSimpleApp com.inductiveautomation.catapult.Catapult

Step 3: Run jstack

With the correct PID and a chosen filename, run the file line to take a thread dump and output it to a file.

jstack -l <PID> > [Filename].out

If you want to take multiple thread dumps and at a given rate, you can use the following code block. Create a file called dump.sh and paste the following code.

if [ $# -ne 3 ]; then
echo This is a file to generate multiple Java thread dumps using Jstack. Requires that you have JDK installed.
echo
echo parameters: $0 process_id repetitions interval
exit1
fi

PID=$1
N=$2
INTERVAL=$3

for ((i=1;i<=$N;i++))
do
d=$(date +%Y%m%d-%H%M%S)
dump="threadDumps/threaddump-$PID-$d.txt"
echo $i of $N: $dump
jstack -l $PID > $dump
sleep $INTERVAL
done

To have the code in the new file take multiple thread dumps, you'll need to run the file as an executable in the terminal and pass in three parameters:

process_id
- process_id is an integer for the PID of the JVM obtained Step 2.
repetitions
- repititions is an integer for the number of times that you want to take a thread dump.
interval
- interval is an integer for the time delay in seconds between each repetition.

You can also include a single letter suffix to the interval number for seconds(s), minutes(m), hours(h), and days(d). The value for interval is simply passed into the Linux "sleep" function, which is called every time after a thread dump is taken, therefore any possible parameters for sleep apply for dump.sh.

If for example, you want to take 5 thread dumps taken 15 seconds apart for the PID in Step 2, you would execute the file with the following parameters: