combine two log files to eachother no corresponding fields

[arjangoos]

We have an access_log file with an 500 error and want to relate this to the application log of the application:

access_log:
84.86.179.166 - - 16 0.016 [12/May/2015:15:27:01 +0200] GET /wmo/wicket/bookmarkable/nl.rotterdam.ioo.wmo.web.wizard.pages.OverzichtPage?18 HTTP/1.1" 500 3961 Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko ZrD6VR3+lj9yqADqNbE3RFxE"

application_log:
ERROR | [http-jboss-local/10.9.8.162:8430-14] | 2015-05-12 15:27:01,160 | org.apache.wicket.DefaultExceptionMapper.java | Unexpected error occurred
java.lang.NullPointerException

How can we relate these two files/lines together?

[cruschke_bde]

I recently had to solve the same situation and came up with using the transaction command, so your search would look like this

tag::host="jboss_prod" (source="/usr/share/jbossas/domain/log/servers/sso/saml*" AND loglevel=ERROR) OR
(source="/usr/share/jbossas/domain/log/servers/sso/access*" AND status=500) |
transaction host startswith=loglevel=ERROR endswith=status=500 maxspan=1s 

The ANDs are for pure readability ... Not sure if the field name for your HTTP status codes is status but you might get the point.

So with the search command you filter for either loglevel=ERROR in your Jboss Logfile or for any HTTP status=500 in your access_log, then starting a transaction, make sure that the host value is the same for Exception and HTTP 500 and finally maxspan=1s as the ERROR is followed by the HTTP 500 in less than a second.

[Yasaswy]

Hi.. purely from the info above, looks like the only common element seems to be time. So if that holds good across all records maybe you can just do a join on time. That being said, you can also consider using lookup tables to tie/map the source hosts appropriately.

[arjangoos]

Ok but how can I do that?

[Yasaswy]

Assuming x is your index (and both sources are in the same index) you can try something like:

index=x earliest=-5h@h latest=now source="access_log"|join _time [search index=x earliest=-5h@h latest=now source="application_log"]

Above is just a format... use appropriate fields (maybe sourcetype .... if relevant to your case) in the search.

[arjangoos]

Hi,

I made this search:

tag::host="jboss_prod" source="/usr/share/jbossas/domain/log/servers/sso/saml*" loglevel=ERROR earliest=-5d@d latest=now |join _time [search tag::host="jboss_prod" source="/usr/share/jbossas/domain/log/servers/sso/access*" earliest=-5d@d latest=now]

But no results, am I doing something wrong?

[Yasaswy]

Hmm... not sure. Can be be whole bunch of things 🙂 ... do you have all the required access /are tags defined in app context etc?? but are you seeing results if you run those searches individually?

tag::host="jboss_prod" source="/usr/share/jbossas/domain/log/servers/sso/saml*" loglevel=ERROR earliest=-5d@d latest=now

and

tag::host="jboss_prod" source="/usr/share/jbossas/domain/log/servers/sso/access*" earliest=-5d@d latest=now

If the above are working independently, which is the larger dataset? are the time stamps matching?