Backfill a full Store and Forward disk cache on a second connection

Introduction

Ignition uses a store-and-forward system to reliably and efficiently store data to a database - most notably used with Tag History and Transaction groups. 

A common example for when Store and Forward comes into play would be when the connection to a database server is lost momentarily. Ignition will store the data until the connection is back up, at which point it will forward it to the database.

This system offers three main benefits: data loss prevention, guaranteed ordering, and enhanced performance. 

Background

Store and Forward engines are automatically created for each database connection. 

The default process of the Store and Forward system is as follows:

1. Data is generated (by Tag History / Transaction Group).
2. Data goes into the memory buffer.
3.  If enough time passes (Write Time: default 5s) or if a certain amount of data accumulates in the memory buffer (Write Size: default 25 records) then it is sent to the local (disk) cache.
4. Data is first pulled from the local cache, then the memory buffer.
   • If the data fails to forward it is returned to the buffer or cache.
5. If the data fails to forward too many times then it becomes quarantined.
   • Quarantined data must be manually retired or deleted. 

Understanding the flow of logic, it's easy to see how a prolonged network or database issue can cause the local cache to become large or eventually full. Depending on the size of the cache (Max Records: Default 25,000) this can take some time for the Store and Forward system to work through - possibly creating a bottleneck.

In order to both keep up with current data and forwarding the backed up cache data, it may be necessary to backfill the cache data with a secondary Store and Forward engine. This will allow the primary Store and Forward engine to keep up with all current data forwarding and appropriately continue storing. 

Process

Note: The instructions below are only required for Ignition versions 8.1.20 and prior. 8.1.21 and later provides a user interface method for backfilling existing Store and Forward data to a secondary Store and Forward profile. If you are using 8.1.21 or higher, disregard the steps below and instead refer to the user manual section on Disk Cache Management.

Here, let's assume the Store and Forward engine in question (and matching Database connection name) is DB1.

1. Disable DB1's local disk cache by setting Disk Cache Enabled to False
   • Gateway Webpage > Config > Databases > Store and Forward > DB1 > Edit > Disk Cache Enabled
2. On the Ignition server, navigate to the datacache folder in Ignition's install directory. By default this is located in "C:\Program Files\Inductive Automation\Ignition\data\datacache\"
   
3. Rename the DB1 folder and all its contents to something else. For example DB1_backfill. Sometimes this can take a few seconds for the Store and Forward engine to release the lock on the datacache files - in some rare instances you may need to stop or restart the Ignition service. 
   
4. Re-enable DB1's local disk cache. At this point a new datacache folder should be created for DB1.
5. Create a new connection to the same database server and schema as DB1 and call it DB1_backfill. This is a duplicate connection, so verify all the connection properties are identical by checking the settings in the DB1 database connection.
   • Optional: schedule the data to be forwarded on off hours so as to not potentially impact production. This can be set by enabling the Enable Schedule property and specifying the Schedule Pattern property in DB1_backfill's Store and Forward settings.