HTTP Request parameters of web application extraction

[CommunityUser]

Hi all,

 As part of custom appdynamics extension to monitor a web application, we would like to extract the HTTP Request parameters (all of them) and do some processing on them. We were hoping to do this as an extension, but are lost on how to inspect the HTTPRequest submitted by the user.

We thought HTTP Data Collectors do not fit our use case as we do not wish a few of the HTTP request, rather we need all of them.

Kindly suggest or point us to any sample code you may have.

[CommunityUser]

Any pointers?

[Peter_Holditch]

The APM part of the AppDynamics platform is focussed around collecting information concerning outlyer transactions to enable deep-dive troubleshooting of non-functional issues, and is this not suited to providing you with the parameters of every single request.

The Business iQ part, on the other hand, facilitiates real time business intelligence based on having the business data available for every Business Transaction, so given a use case where you want access to the parameters of each request, Business iQ is the way to go.

Take a look here for more detail.

[CommunityUser]

Thank you very much for the response @pholditch.

Is it possible to extend or customize the Business IQ part ..esp HTTP Request Data Collector?

We would like to HTTP Request Data and selectively send them to the controller - kindly help us on how this can be achieved.

[Peter_Holditch]

I am not sure what you mean by "extend the Business iQ part"...

Once a data collector is configured for analytics, all the data it collects from every transaction is sent to the controller.

From there, you can use ADQL to query it either via the UI, or programatically using the REST API.

Does that enable what you need?  If not, can you elaborate on what you are trying to achieve?

[CommunityUser]

Sorry that I was lost in my exploration that did not revisit this thread.

In more detail, we wish to write some generic validations for our entire suite of applications (not specific to one application) on the input from end users. Data Collectors is what we are looking for - but we cannot configure as there are many applications. So we wish to extend the agent (App Agent or Machine Agent) to inspect the HTTP Request for this purpose.

Looking for guidance on the approach to follow to achieve this:

1. Should we be looking to customize the app agent or machine agent for this?

2. Can we use the SDK API??

https://github.com/qldong/java-sdk/tree/master/application-agent-api

[Peter_Holditch]

What would you want to happen in your application if one of your validations failed?

[CommunityUser]

We would want those to be reported back to the controller and then we could allow configuration of alerts based on the reported custom attributes.

[Peter_Holditch]

This should be possible using our analytics product.  Can you give an example of a concrete use-case you want to implement?  If so, I can try and point out the right set of platform features to use to achieve it.

[CommunityUser]

Ok...here we go.

We have 80 odd applications supporting a legal regulation. As part of the
regulation, we need to run some generic rules against the input submitted
by end users. We have appdynamics already monitoring all these applications
and are hence looking to extend appdynamics agent (machine or application)
to extract the parameters from request and run the rules and report to
controller if there are any issues found.

We can then schedule alert emails based on the events reported.

[Peter_Holditch]

OK, can you give an example of some inputs and a rule?

And what should happen if the rule is violated?  SImply an alert, or should the application behaviour change in case of a violation?

[CommunityUser]

For now we are looking to only send alerts to the audit team though in the
future would not rule out more functionality addition as a reaction.

A simple rule is that any mention of "religion" or "race" as part of the
many input entries of these 80 odd applications need to be alerted as these
are cases of possible favouritism that needs additional audit scrutiny. So
the rule screens for matches of mention of religion or race on all the
inputs coming in the Http request.

[Peter_Holditch]

So presumably, you have a regular expression (referred to as 'regular-expression' below) that you would run on a string field, and when it matches that is indicative of a mention of religion.

On that assumption, if you collected transaction analytics data from all transactions that could contain mention of religion, you could then use ADQL to identify the offending ones.  e.g.

SELECT count * FROM transactions WHERE segments.userData.inputText REGEXP('regular-expression')

You could use this query to create a metric that you could alert on if non zero using a health rule in the usual way.

This would probably be the best way to approach your problem using the AppDynamics platform.

[CommunityUser]

Thank you Peter for your suggestions so far.

Some of the challenges with the afore mentioned solution would be:

1. The rule I had mentioned was one simple rule and there are many others that need to be applied.

2. The approach intends to capture the field into appdynamics schema. With the scale of inputs & applications, this would quickly grow to unmanageable size and with AppDynamics on MySQL, this would start to deteriorate the performance. There are usually multiple fields per form per application that we need to run the rules through. This would work if we could report to the appdynamics controller only if there is negative hit on the rules which is usually rare (maybe less than 5% of the time).

I was hoping that there could be some means of hooking into appdynamics given that request level stats are already being pulled out (like url, User Principal etc). The last route on my mind is to extend the machine agent and start listening into the HTTP ports on all the requests coming in, then apply the rules and report back to controller (very reluctant on this approach as it complicates on SSL, HTTP port listening etc)

[Peter_Holditch]

The event data stored by analytics is not held in MySQL, so you need not worry on that front.

Based on your other comments though, I am just not convinced that trying to implement your rules within AppDynamics is the correct architectural approach to solve this specific problem

Maybe you could apply your rules in code and publish the failure count via a JMX MBean?  You could use AppD to monitor the MBean and alert on the breaches that way?

[CommunityUser]

The reasons for exploring the AppDynamics customization route are:

1. AppDynamics already inspects requests

2. Rolling out to the scale of so many application is easy as they all already have AppdYnamics configured.

3. Due to the impact on each of those 80 odd applications, we wanted to keep them outside of the application code.

4. If we could do it at appdynamics, we can avoid impact on so many applications and that save us regression effort and centralization of code related to this audit rather than distribution of same with each application.

[CommunityUser]

We are trying to selectively send some of the http parameter values back to
controller and have logic around this. Hence would like to customise or
extend the part so that we could achieve this.