Correlating web and app server logs

[sbaytals]

Hey guys!
I have IBM HTTP Servers in front of WebSphere App Servers. Is there a way to correlate data in the logs of both products out of box? Without having developers add something to the log output.
Like HTTP Session ID or similar? WAS is using cookies for session affinity.
Has anyone done this in real life?
Thanks in advance!

[lguinn2]

People do this all the time!! If there is any id that is in common between two sources (HTTP Session ID, a customer id, a transaction number, cookies, etc etc) then Splunk can correlate the data. This is one of the core strengths of Splunk. Of course, if there is nothing in common between the two sources, then you have a problem.

There is no need for the developers to do anything if there is data in common, although there may be a need for the Splunk admin to configure fields. This is called "field extraction" in Splunk; the definition of a field can be changed at will. Unlike a database schema, fields in Splunk are dynamic; they can even be created on the fly if needed.

Splunk has a free downloadable app for WebSphere data, which will create the fields (and a lot of other cool stuff) for you: Splunk App for WebSphere

For the HTTP servers, you may be able to assign the access_combined or access_combined_wcookie sourcetypes to the input source logs. These sourcetypes will define the standard field extractions for Apache logs.

But if your data is in a format that Splunk does not already recognize, you will need to create your own field extractions.
The Interactive Field Extractor can help you create the fields. People in this forum can help as well (although we need to see sanitized snippets of the log files in order to help).

Read more about field extractions here. And, to forestall the next question: you do not want index-time field extractions. So don't bother to even look it up right now. 🙂

[lguinn2]

Okay, that makes sense! So my answer is not an answer at all... Sorry! Feel free to vote it down so that another reader knows that you are looking for different info.

Although I have to believe that this data can be correlated...

Can you post about 10-15 lines from each log for us to look at? I hope that isn't too much to ask.

[sbaytals]

I am using access_combined to parse IHS access logs. But on WAS side there aren't any fields that i can tie to IHS. So i was wondering if there's anything i can enable to have WAS add something to the logs to let me tie them together.I did download the WAS app for Splunk.I did setup a forwarder appliance and i can see the relevant WAS event types and such.Unfortunately there are no transaction IDs or Session IDs that are printed in the logs by default.So i was asking if there are any Splunk customers that have IBM IHS with WAS and can do the correlation. And what do they do it on?
Thanks!