We have an access_log file with an 500 error and want to relate this to the application log of the application:
126.96.36.199 - - 16 0.016 [12/May/2015:15:27:01 +0200] GET /wmo/wicket/bookmarkable/nl.rotterdam.ioo.wmo.web.wizard.pages.OverzichtPage?18 HTTP/1.1" 500 3961 Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko ZrD6VR3+lj9yqADqNbE3RFxE"
ERROR | [http-jboss-local/10.9.8.162:8430-14] | 2015-05-12 15:27:01,160 | org.apache.wicket.DefaultExceptionMapper.java | Unexpected error occurred
How can we relate these two files/lines together?
I recently had to solve the same situation and came up with using the
transaction command, so your search would look like this
tag::host="jboss_prod" (source="/usr/share/jbossas/domain/log/servers/sso/saml*" AND loglevel=ERROR) OR
(source="/usr/share/jbossas/domain/log/servers/sso/access*" AND status=500) |
transaction host startswith=loglevel=ERROR endswith=status=500 maxspan=1s
ANDs are for pure readability ... Not sure if the field name for your HTTP status codes is
status but you might get the point.
So with the search command you filter for either loglevel=ERROR in your Jboss Logfile or for any HTTP status=500 in your access_log, then starting a transaction, make sure that the host value is the same for Exception and HTTP 500 and finally maxspan=1s as the ERROR is followed by the HTTP 500 in less than a second.
Hi.. purely from the info above, looks like the only common element seems to be time. So if that holds good across all records maybe you can just do a join on time. That being said, you can also consider using lookup tables to tie/map the source hosts appropriately.
Assuming x is your index (and both sources are in the same index) you can try something like:
index=x earliest=-5h@h latest=now source="access_log"|join _time [search index=x earliest=-5h@h latest=now source="application_log"]
Above is just a format... use appropriate fields (maybe sourcetype .... if relevant to your case) in the search.
I made this search:
tag::host="jbossprod" source="/usr/share/jbossas/domain/log/servers/sso/saml*" loglevel=ERROR earliest=-5d@d latest=now |join _time [search tag::host="jbossprod" source="/usr/share/jbossas/domain/log/servers/sso/access*" earliest=-5d@d latest=now]
But no results, am I doing something wrong?
Hmm... not sure. Can be be whole bunch of things 🙂 ... do you have all the required access /are tags defined in app context etc?? but are you seeing results if you run those searches individually?
tag::host="jboss_prod" source="/usr/share/jbossas/domain/log/servers/sso/saml*" loglevel=ERROR earliest=-5d@d latest=now
tag::host="jboss_prod" source="/usr/share/jbossas/domain/log/servers/sso/access*" earliest=-5d@d latest=now
If the above are working independently, which is the larger dataset? are the time stamps matching?