How do I instrument cross-tier FTL communication and correlate exit calls in TIBCO BusinessWorks 6.x?
Table of Contents
Background
TIBCO BusinessWorks 6.x integrates with different TIBCO and non-TIBCO systems like databases, TIBCO EMS Server, SOAP Service over HTTP/JMS, and TIBCO messaging frameworks like FTL.
AppDynamics supports JMS, SOAP, JDBC and HTTP backend detection from TIBCO BusinessWorks out-of-the-box. However, non-standard and proprietary backend calls to TIBCO FTL are not detected by the AppDynamics Java Agent out-of-the-box. Backend instrumentation and custom-activity-correlation are needed to capture the end-to-end flow between upstream and downstream tiers in a single Business Transaction.
Below is a solution that we successfully tested and implemented with a customer to instrument cross-tier FTL communication and correlate exit calls in TIBCO BusinessWorks 6.x.
Test Environment and Scenario
For testing purposes, we used:
- TIBCO FTL 5.2.x
- TIBCO BusinessWorks 6.x
- AppDynamics JavaAgent 4.5
We had three TIBCO BusinessWorks Applications: PT, BT and TT. Each JVM used a unique TierName and all the three JVMs reported to the same application in the AppDynamics Controller.
PT is a simple REST Service implemented with TIBCO BusinessWorks 6.x that accepts API requests at http://localhost:8080. It sends an FTL request to BT and receives a FTL reply from BT using FTLRequestReply activity. It then provides the reply back to the end user.
BT uses a FTLSubscriber activity to receive the FTL request from PT, uses FTLRequestReply activity to send another request to TT and receive reply from TT, then replies back to BT using FTLReply activity.
TT uses a FTLSubscriber activity to receive the FTL request from BT, makes an outgoing HTTP call using SendHTTPRequest activity, gets back a response and then replies back to BT using FTLReply activity.
Challenges
- As mentioned above, unlike normal HTTP/JMS/JDBC backends, there is currently no out-of-the-box support in the AppDynamics Java Agent for TIBCO FTL exits.
- Even if the customer was able to get the class/methods for TIBCO FTL exits to define custom exit points, there are two kinds of exit points:
- One type of exit point is implemented in the FTLRequestReply activity that sends a request message on a normal FTL endpoint.
- Another type of exit point is implemented in the FTLReply activity that sends a reply message on an FTL INBOX.
- Even if the customer applied instrumentation on both types of exits as a custom exit, they would need to inject the
singularityheaderin both the FTL request and reply messages in order to preserve the cross-tier correlation. However, since FTL is a non-standard call, they would need to identify the proper place/method to inject/extract thesingularityheaderin the FTL message.
Solution
These 8 steps will guide you through how to instrument cross-tier FTL communication and correlation in TIBCO BusinessWorks 6.x.
- Apply the attached
custom-activity-correlation.xmlfor all three JVMs (open the attached file named “Step 1 - custom-activity-correlation.xml.txt”). There are two producer and two consumer entries here. For this specific customer, we used three FTL activities:
- FTLRequestReply: It has a producer part and a consumer part. The producer part sends a request message on a normal FTL Endpoint and the consumer part receives a reply from an FTL INBOX. In the attached
custom-activity-correlation.xmlfile, the first producer and the second consumer are from this activity. - FTLReply: It is a producer that sends a reply message on an FTL INBOX. In the attached
custom-activity-correlation.xmlfile, the second producer is from this activity. - FTLSubscriber: It is just a consumer that receives requests from normal FTL Endpoint. In the attached
custom-activity-correlation.xmlfile, the first consumer is from this activity.
You are putting thesingularityheaderin both producers and reading thesingularityheaderin both consumers in the respective FTL calls invoked by these activities. If there is no FTLRequestReply and FTLReply activity involved in the BW code (no request-reply scenario) and you just have FTLPublisher and FTLSubscriber activities (only send-receive scenario), you can use thecustom-activity-correlation.xmlin the attached file named “Step 1 - custom-activity-correlation.xml (only send-receive scenario).”
- FTLRequestReply: It has a producer part and a consumer part. The producer part sends a request message on a normal FTL Endpoint and the consumer part receives a reply from an FTL INBOX. In the attached
- Make sure that the FTL message schema in the BW application has a new optional String Type element named
singularityheaderadded to it. You have to do this code change to make sure the application can understand the presence of this new header element in the FTL message and successfully parse this.
Add this new optional element for all the TIBCO BW 6.x nodes that are sending and receiving FTL messages.
NOTE: Make sure that this element is added at the end of your other existing elements in the FTL message schema or the application will fail to parse the FTL message with the
singularityheader.
- TIBCO BusinessWorks 6.x uses Jersey Server and by default, we exclude the above class in our code for async processing. In order to make sure thread-correlation happens properly, we must add the following include to the
JAVA_AGENT_HOME/verx.x.x.x/conf/app-agent-config.xmlin thefork-configsection for all your FTL nodes. (Note: The field scrolls horizontally)<include filter-type="EQUALS" filter-value="org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2"/> - Add the Appagent node property
error-safety-rule-error-threshold=-1for all nodes that are sending/receiving FTL messages.
NOTE: The custom correlation is applied on the nodes about 15 minutes after the application starts. If you start putting load on the application as soon as the nodes start, thesingularityheaderwill not be recognized and it will trigger an error in the Business Transaction. Eventually, the instrumentation will be neutralized and thesingularityheaderwill never be set. You need to add the above node property to make sure that the instrumentation is not neutralized during the first 15 minutes of the application startup when there is nosingularityheaderset in the FTL messages.
- Add Appagent node property
track-async-inside-custom-exit=truefor all nodes sending/receiving FTL messages. This node property is used to track async calls inside the custom exit points. - If SpringBeans instrumentation is enabled for any of your TIBCO BW FTL Nodes (for BTs or SEPs), add the Appagent node property
ignore-exitcall-on-reentry=falsefor all nodes sending/receiving FTL messages. This will ensure that your custom exit calls are not lost due to SpringBeans instrumentation. Otherwise, you can simply disable SpringBeans instrumentation for BTs and SEPs on all such nodes. - If you are making outgoing WebService calls instead of HTTP calls in any of your nodes in your BW application, make sure that WebService type exit points are disabled on all such nodes. We have seen the FTL custom exit calls get lost when used with WebService type exits.
- Start all the three TIBCO BusinessWorks 6.x JVMs with the AppDynamics Java Agent instrumentation and put load on the PT Tier. After around 15 minutes, you will see the complete end-to-end application logic in your AppDynamics Controller dashboard. See Controller dashboard examples, below.
Example Dashboards
Below are examples of Controller dashboards that display the complete end-to-end logic after following the steps above.
Application Dashboard
Node Dashboards
BT Dashboard
BT Snapshot
