Introduction

Ignition has the ability to configure alarms on tags that trigger when certain thresholds are met. For example, we may have a Memory Tag with an Integer data type and we want to be notified when the value of the tag is greater than 0. This functionality is applied to the Ignition service when the Gateway has the Alarm Notification Module installed and can be configured on a tag's Tag Editor in the Designer. Furthering the feature-set of alarm notifications, Ignition can take advantage of the Voice Notification Module in order to send phone calls directly to users who should be aware of a given alarm. 

https://docs.inductiveautomation.com/display/DOC81/Alarm+Notification+Module 

https://docs.inductiveautomation.com/display/DOC81/Voice+Notification+Module 

We will now discuss the basic nature of the Voice Notification Module, the technology that drives it, and how to identify a DTMF tone issue, for example, when a phone's keypress is not recognized by Ignition.

Background

The Voice Notification Module uses the Session Initiation Protocol (SIP) to implement Voice Over IP (VOIP) based phone systems. Once the session has been initiated, your specific SIP service decides how to actually transfer the voice data. For example, a popular SIP service provider is Skype, which uses the Real-time Transport Protocol (RTP) for the transfer. 

https://docs.inductiveautomation.com/display/DOC81/Voice+Notification+Profile 

There are also basic telephony appliances that support SIP and can handle calls from Ignition such as the Grandstream HandyTone series. Essentially, Ignition sends the Grandstream device the phone number, script, and voice where the device then routes the call. More information on Grandstreams, their configurations, and troubleshooting can be found here:

https://support.inductiveautomation.com/hc/en-us/articles/360047175492-How-to-configure-the-Grandstream-HT-503-and-HT-813-for-use-with-Ignition-Alarming-voice- 

Once the basic alarm configs, voice notification profiles, and pipelines have been set up in Ignition, an alarm triggers, a phone call will go out to the User or Roster and when someone answers, the automated system will ask them to press a key to proceed or acknowledge an event. Sometimes, after pressing the requested keys, the automatic system fails to proceed. This is where a basic understanding of Wireshark, RFCs, RTP events, and DTMF tones come into play for diagnosing why the keypresses might not be recognized.

Wireshark:

Wireshark is a robust network analysis tool. It allows us to analyze network packets by presenting captured packet data in as much detail as possible. We will use Wireshark to inspect the RTP sections of the packets sent between the Ignition Gateway and the service used.

https://www.wireshark.org/download.html 

https://www.wireshark.org/docs/wsug_html/ 

RFCs:

A Request for Comments (RFC) is a formal document from the Internet Engineering Task Force. They are written to describe methods, behaviors, and standards for a particular technology used on the internet and the number that succeeds RFC is the identifier for that document.  

https://www.ietf.org/standards/rfcs/ 

RTP events:

An RTP event is a transaction that occurs over the Real-time Transport Protocol, which is a common network protocol for audio and streaming. 

https://en.wikipedia.org/wiki/Real-time_Transport_Protocol

RFC 1889 is the RFC that describes the Real-Time Transfer Protocol:

https://www.ietf.org/rfc/rfc1889.txt 

DTMF tones:

DTMF stands for Dual-Tone Multi-Frequency and it is the sound made from any given DTMF signal when you press a key on your phone. The DTMF tone is the actual event/transaction we are interested in analyzing that occurs over the RTP.

https://en.wikipedia.org/wiki/Dual-tone_multi-frequency_signaling

RFC 2833 describes the DTMF payload over RTP:

https://www.ietf.org/rfc/rfc2833.txt 

Troubleshooting

NOTE: If using a Grandstream device, be sure to confirm that all configurations and troubleshooting have been followed using this KBA article. In particular, the FXO Port tab settings for the "Preferred DTMF method" should be set to "RFC 2833" (DTMF payloads). If you have followed this guide accurately and are still experiencing issues, proceed here.

1. The first step in diagnosing an unrecognized keypress when trying to proceed or acknowledge an alarm's voice notification is to confirm that Ignition is actually seeing the keypress. This can be done by noting the time of the keypress and then checking the logs for an entry confirming the press. These log entries can be seen from the Ignition Gateway Web server > Status Tab > Logs page or in the wrapper.log file located in the Ignition install directories /logs folder. 

The specific logger that notes the event is the "CallManager" logger which should display something similar to the following INFO level logger message: 

The timestamp should line up with when the user pressed the key to confirm this is the specific event you are interested in. "dtmf" confirms a DTMF payload and the [1] refers to the specific key that was pressed; in this case, it was the "1" key. If this seems in order with your test and the keypress was not recognized during the voice notification phone call, then it is time to repeat the first step while taking a Wireshark capture.

2. Startup Wireshark, begin a capture on the Network Adaptor where the communication is occurring and follow the same procedure of noting the time of the keypress and confirming the CallManager log entries in Ignition still occur. Once the keypress has been verified and seen in Ignition, stop the Wireshark capture.

3. In Wireshark, we will use the "Display Filter" to parse out the packets we are interested in. Since the transactions are RTP Events, we can use the filter:

rtpevent

to pare down the captured results. https://www.wireshark.org/docs/dfref/r/rtpevent.html

Here is an example of a Wireshark frame/packet containing an RTP Event:

The protocol should be an RTP Event and the Info column should contain the payload "DTMF <number> <value>" where <number> is the word of the keypress and <value> is the number of the keypress. If you do not see any RTP Events, confirm you captured traffic on the appropriate network adaptor. If you are positive that you have the correct network adaptor selected but see no results, then this is evidence that there is a break in the network communication altogether. You should reach out to your Systems/Network Administrator to figure out where along the chain the communications are failing.

4. Once you are able to see the RTP Event packets in Wireshark, selecting a given Frame will show the following sections that can be drilled down into for more details: 

5. Check the RFC 2833 RTP Event properties. The "Volume" and "Event Duration" properties of the "RFC 2833 RTP Event" section might first jump out and for good reason: it is likely that the event Volume is 0 and the Event Duration, in milliseconds, could appear short (such as 160ms pictured above).

The Volume property: denotes the volume of the tone in dB, which typically ranges from 0 to -36 dBm0 so a volume of 0 is the baseline volume for a DTMF tone; it does not mean there is no audible tone.

The Event Duration property: it is important to note that this elapsed time is only a portion of the total time when a DTMF keypress is split over multiple RTP Event packets. This means that the initial event might have a duration of 0 but this should not be confused with the event not occurring at all.

The End of Event property: is a boolean value that will be False while the keypress is held and True when the keypress is released. 

If an RTP event contains a frame with an Event Duration that has some total > 0 and the Volume is within an acceptable range, then these packets should be considered valid and the tone should be recognizable by the destination service or device as an RTP event.

6. Use Wireshark "rptevent" display filter to compare values for each property. When typing in the display filter "rtpevent" you can use dot-notation to drill into the individual event properties:

rtpevent.duration
rtpevent.end_of_event
rtpevent.event_id
rtpevent.reserved
rtpevent.volume

If your keypress is being recognized in Wireshark and the Ignition logs but not making the voice notification proceed or acknowledge the alarm, then it is likely that the End of Event property is not being sent properly. Use the display filter:

rtpevent.end_of_event == True

to filter for frames that show an RTP Event where the End of Event value was equal to True. If no results are returned in the capture, then this is likely the reason why we are seeing the failure.

The Voice Notification Module requires an End of Event value of True to recognize a valid keypress.

Changing the filter from True to False should still show packets where the DTMF tone was found on the network. The End of Event value is sent from the Service provider (like Skype) and they should be contacted if an End of Event value of True is not coming through the line.