Marshal Signal Generator
Marshal offers a solution to accomplish event-driven messaging and information transfer from any system.
Automated transfer of information requires a trigger, i.e. a condition or event that initiates the integration process. These triggers can be separated into two main categories:
Event-driven: something happens, or changes, in a system causing the system to activate the integration to another system. A typical example is a system containing an interface (GUI) that allows users to display and edit information in another, back-end, system.
Time-driven: a system periodically polls another system to see if anything has changed or to check some specific criteria. This is the predominant mechanism when it comes to backup, distribution and synchronization of information.
In theory, event-driven integrations are to be preferred since these occur directly and are more efficient in terms of the amount of information transferred. Event-driven mechanisms, however, requires the triggering system to have control over the integration both in terms of content and technology. For this reason, there’re a lot of cases where an event-driven integration is too costly or simply not feasible.
The key advantage of a time-driven integration is that it depends solely on the system that wants or needs the information. As for the other system, the source system, it just needs to be possible to integrate.
The Marshal Signaller software makes it possible to develop event-driven integrations triggered by changes in a system that, in itself, has no support for raising the signals in question. This is achieved without requiring any changes to the signaling system.
The Marshal Signaller software is an extension to our backend solution for information transfer and export, Marshal Archiver. The events are defined using an export model, designed in Marshal Editor, and a configuration file specifying how to signal the event or change.
Marshal Signaller comes with a signal database to keep track of what state changes, in the source system, that has been signaled and when. This is needed to avoid double signaling of events since, in most cases, there will be no way to determine this directly from the information available in the source system.
The following occurs when a change in the source system is caught by an export model:
- If the event’s already been signaled, nothing occurs.
- If the event is new Marshal Signaller takes one of the following actions:
- Call a web service (http/https) in e.g. Camel, BizTalk or another system.
- Call Marshal Archiver to invoke another integration or export.
Installation
Marshal Signaller is an extension to Marshal Archiver and is intended for installation on a server computer where the Marshal Archiver software is available.
Step 2 - Create a database in PostgreSQL, MySQL or SQL Server and run the corresponding script to create the tables needed: mysql.sql, postgressql.sql or sqlserver.sql. The script files are available in the Marshal Signaller installation folder.
For convenience, you may create an ODBC data source referring to the database.
Define the event
An event is primarily defined through an export model, a Marshal model. This model is created using the Marshal Editor software.
In the example, shown above, applications are fetched from an application system. An application is sent in by an applicant (UserID) and is then registered with a serial number (Number) uniquely identifying the application as a record within the organization.
The objective is to send a message, whenever an application arrives, without having to change the application system itself. For instance, this could be to inform an official on the arrival of a new application or to communicate the record number back to the applicant.
Execution of the model results in exports similar to:
The export needs to include the information required to identify the object being signaled as well as the information that is to be passed in the signal message itself. In this case, Number identifies the application whereas UserID and Title is intended as message content.
If the signal corresponds to a state change – which is not the case in this example – it is often necessary to export two fields. One identifying the object, whose state has changed, and one containing the actual state, see “Define your message” for further details.
If the export model is to run at high frequency the export needs to be efficient. This requires excluding any applications already signaled. This is done by exporting the Date field and avoiding fetching any applications that have arrived prior to last one that’s been signaled. This feedback mechanism is discussed in more detail in the “Define your message” chapter.
Define your message
Besides the export model for retrieving source system data, a configuration file is needed. It controls how Marshal Signaler executes the export model, how it interprets the exports and how the final signal messages are assembled.
The configuration file is an XML file according to:
The XML must start with an event element that points out the signal database as well as defines what element, in the export files, that constitutes an event.
event attributes |
| name | Name of the event. |
| database | Connection string for the signal database. |
| xpath | Xpath expression for selecting the exported element corresponding to the event. See w3schools for an introduction to xpath. |
| prefix | Prefix used in Xpath expressions. This prefix is tied to the namespace used in the export. |
| xmlns | Must be http://Marshal.Signal/ for Marshal Signaller to accept the configuration file. |
There is a set of sub elements to be used inside event elements: filter, identity, state and signal. These are described below.
The filter element is used to limit the export beyond the criteria specified in the export model. The filter element provides a possibility to use a value from the previous event to filter or control an export. The element may contain parameter sub elements used to pass constant values to the export job. All values, constant or not, passed to the Marshal Archiver export job as a result of the filter element are available using the param(x) operator, see “Configure Marshal Archiver”.
filter attributes |
| name | Parameter name. Refer to this name, using the param operator, when configuring the Marshal Archiver job. |
| xpath | Xpath expression used to obtain the event property to use for filtering. The expression is relative to the element obtained by the event xpath. |
| fallback | The value to be used when filtering for the first time, i.e. if filtering on a date this is the starting date. |
The identity element specifies the node (element or attribute) in the event xml that uniquely identifies the event. If the signal refers to a state change, for a certain object, it is possible to provide a state element as well. This element specifies how to obtain the state of the object.
identity and state attributes |
| xpath | Xpath expression, relative to the event element, for selecting the property corresponding to the identity or state of the event object. |
# | Identity | State | Signal? |
| 1 | A1 | X | Yes |
| 2 | A1 | Y | Yes |
| 3 | A1 | X | Yes |
| 4 | A1 | X | No |
| 5 | A2 |
| Yes |
| 6 | A2 |
| No |
A signal is sent only if the state is not the last signaled state for the specific identity. If no pattern is provided, for accessing the state, the signal is sent only if the identity has never been signaled.
The table shows when a signal is sent. In the example there’s a sequence of six events concerning two different objects (A1 and A2).
To define the signal message to be assembled and sent, a signal element is used. Two types of signals are supported: web or Marshal. A call to a web service means that an URL is constructed as specified by the signal element. The URL is then invoked using as an http-get-request. The other option means calling Marshal Archiver to run a job with parameter set according to the signal element.
signal attributes |
| name | Signal name. |
| url | The URL to be invoked per event. Any parameter elements are added to the URL as name-value-pairs. If a URL is specified any job attribute is ignored. |
| job | The Marshal Archive job to be executed per event. Any parameter elements define the parameters passed to, and available in, the job. |
The parameter element is available as a sub element to filter and signal. It is used to define parameters to limit the export or to pass along with the signal.
parameter attributes |
| name | Parameter name. Refer to this name during configuration of the export job in Marshal Archiver. |
| xpath | Xpath expression for retrieving the parameter value. The expression is relative to the element obtained by the event xpath. |
| fallback | The parameter value to be used if the xpath expression does not return a value. |
| value | Constant parameter value |
When used
in a filter element the parameter element must have a constant
value since filters are evaluated prior export, i.e. there is no XML on which
to apply an xpath.
All in all, the example used in this document will result in two calls, one to “http://myServer/signalConsumer.jsp?user=mia.lee@test.com&number=2012%2F16%3A1&title=Application%20for...” and one to “http://myServer/signalConsumer.jsp?user=ahan@test.com&number=2012%2F17%3A1&title=
Application%20for...”.
Configure Marshal Archiver
The last part of setting up a signal solution based on Marshal Signaller is to configure an export job in Marshal Archiver making use of the model and the configuration file.
- Publish the Marshal model using Marshal Repository Admin.
- Start the Marshal Archiver Admin software and create a new job.
- Check the ”enabled” checkbox.
- Lower the log level to errors only since it will run often.
- Add a step to the new Marshal job with the following settings:
- Select the Marshal model (Marshal Structure).
- Assign a temporary export directory. (Export root directory).
- Specify the proper export criteria (Export filter). Possibly using parameters or filter defined in the configuration file.
- Assign the Marshal.Signaller.Scan software as pre-processor (Preprocessor).
- Assign the Marshal.Signaller.Send software as post-processor (Postprocessor).
- Add an extra argument, param(SignalDefinition), as post-processor argument.
- Add a new schedule to the job:
- Specify when the job is to be run for the first time.
- Specify how often the job is to be repeated, e.g. every third minute.
- Add a job parameter with
- key = SignalDefinition
- value = absolute path of signal configuration file.
Troubleshooting
If the signal mechanism does not deliver the expected signal the below will provide some starting points for troubleshooting.
- Is the Marshal Archiver service running on the computer? Check Services under Configuration in the Server Manager console.
- Marshal Archiver will report any error to the Event Viewer.
- Verify that Marshal Archiver loads the export jobs when starting. Both successful and failed loads are reported. If you can’t find the log records in the Event Viewer, restart the Marshal Archiver service.
- After an export ought to have run, check if there’s an error message in the Event Viewer. You may increase the log level so that successful exports are logged as well.
- If there are no error reported, check the signal database instead.
- Check that the event, corresponding to event element of the configuration file, is registered in the t_signal_u_Event table. If the event is registered, check the column c_u_LastScan to see when an export was last processed by the signaling subsystem.
- Check, in table t_signal_u_Signal, if any signal messages been sent. Its columns will provide the following info:
- Signal name
- Final URL invoked.
- When the call was made.
- What the signal consumer responded. Preferrably, 200 OK.
- What identity and state that caused the signal.
- If everything looks fine, but the expected signals are just not generated, you should check the selection criteria provided in the export model, the configuration file and Marshal Archiver job. Try, for instance, to manually run the export from the Marshal Editor and to see if you get an export and what is actually in it.
