topic Re: combine two log files to eachother no corresponding fields in Splunk Search

combine two log files to eachother no corresponding fields

arjangoos — Tue, 12 May 2015 13:49:45 GMT

We have an access_log file with an 500 error and want to relate this to the application log of the application:

access_log:
84.86.179.166 - - 16 0.016 [12/May/2015:15:27:01 +0200] GET /wmo/wicket/bookmarkable/nl.rotterdam.ioo.wmo.web.wizard.pages.OverzichtPage?18 HTTP/1.1" 500 3961 Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko ZrD6VR3+lj9yqADqNbE3RFxE"

application_log:
ERROR | [http-jboss-local/10.9.8.162:8430-14] | 2015-05-12 15:27:01,160 | org.apache.wicket.DefaultExceptionMapper.java | Unexpected error occurred
java.lang.NullPointerException

How can we relate these two files/lines together?

Re: combine two log files to eachother no corresponding fields

Yasaswy — Tue, 12 May 2015 14:22:43 GMT

Hi.. purely from the info above, looks like the only common element seems to be time. So if that holds good across all records maybe you can just do a join on time. That being said, you can also consider using lookup tables to tie/map the source hosts appropriately.

Re: combine two log files to eachother no corresponding fields

arjangoos — Tue, 12 May 2015 14:31:00 GMT

Ok but how can I do that?

Re: combine two log files to eachother no corresponding fields

Yasaswy — Tue, 12 May 2015 14:55:02 GMT

Assuming x is your index (and both sources are in the same index) you can try something like:

index=x earliest=-5h@h latest=now source="access_log"|join _time [search index=x earliest=-5h@h latest=now source="application_log"]

Above is just a format... use appropriate fields (maybe sourcetype .... if relevant to your case) in the search.

Re: combine two log files to eachother no corresponding fields

arjangoos — Mon, 28 Sep 2020 19:57:11 GMT

Hi,

I made this search:

tag::host="jboss_prod" source="/usr/share/jbossas/domain/log/servers/sso/saml*" loglevel=ERROR earliest=-5d@d latest=now |join _time [search tag::host="jboss_prod" source="/usr/share/jbossas/domain/log/servers/sso/access*" earliest=-5d@d latest=now]

But no results, am I doing something wrong?

Re: combine two log files to eachother no corresponding fields

Yasaswy — Tue, 12 May 2015 15:53:21 GMT

Hmm... not sure. Can be be whole bunch of things 🙂 ... do you have all the required access /are tags defined in app context etc?? but are you seeing results if you run those searches individually?

tag::host="jboss_prod" source="/usr/share/jbossas/domain/log/servers/sso/saml*" loglevel=ERROR earliest=-5d@d latest=now

and

tag::host="jboss_prod" source="/usr/share/jbossas/domain/log/servers/sso/access*" earliest=-5d@d latest=now

If the above are working independently, which is the larger dataset? are the time stamps matching?

Re: combine two log files to eachother no corresponding fields

cruschke_bde — Wed, 13 May 2015 09:11:27 GMT

I recently had to solve the same situation and came up with using the transaction command, so your search would look like this

tag::host="jboss_prod" (source="/usr/share/jbossas/domain/log/servers/sso/saml*" AND loglevel=ERROR) OR (source="/usr/share/jbossas/domain/log/servers/sso/access*" AND status=500) | transaction host startswith=loglevel=ERROR endswith=status=500 maxspan=1s

The ANDs are for pure readability ... Not sure if the field name for your HTTP status codes is status but you might get the point.

So with the search command you filter for either loglevel=ERROR in your Jboss Logfile or for any HTTP status=500 in your access_log, then starting a transaction, make sure that the host value is the same for Exception and HTTP 500 and finally maxspan=1s as the ERROR is followed by the HTTP 500 in less than a second.