topic Re: Reading same field from multiple log files in Splunk Search

Reading same field from multiple log files

hegdevageesh — Tue, 07 Apr 2020 12:50:40 GMT

I have 2 log files from different sources. Both log files have statements either indicating a "Transaction-Start" or "Transaction-End" . "EPOCH" is a field common in both log files indicating the timestamp of either start or end of a transaction.

Now I want to write a query that fetches EPOCH of "Transaction-Start" from log file 1, call it as, say start and EPOCH of "Transaction-End" from log file 2, call it as, say end. Following this, I want to find the difference between end and start and display only those logs with a difference higher than a threshold, say 10000

What I have tried writing is below :

index=someIndex ENVIRONMENT="someEnv" (source="/log/source1.log" "Transaction-Start" "EPOCH" as start) OR (source="/log/source2.log" "Transaction-End" "EPOCH" as end) 
    | eval difference=end-start 
    | where difference>10000

But this is not working. Looking for help in composing this search in the right way.

Re: Reading same field from multiple log files

richgalloway — Tue, 07 Apr 2020 13:08:10 GMT

One cannot rename fields in a base search (the part before the first |). That's done using eval or rename.
Also, since the start and end times are in separate events, you will not find both 'start' and 'end' together. The starting and ending events must be combined via some common field. Is there a transaction ID or other field that can be used to join the two events?

Re: Reading same field from multiple log files

hegdevageesh — Tue, 07 Apr 2020 13:13:57 GMT

@richgalloway, yes, there is a "TXID" field that is common field in both log files.

Re: Reading same field from multiple log files

richgalloway — Tue, 07 Apr 2020 13:28:19 GMT

Try this. It assumes EPOCH is an integer. If it isn't then you'll need to use strptime to convert it into an integer.

index=someIndex ENVIRONMENT="someEnv" EPOCH=* (source="/log/source1.log" "Transaction-Start") OR (source="/log/source2.log" "Transaction-End") 
| eval start = if(source="/log/source1.log", EPOCH, null()), end = if (source="/log/source2.log", EPOCH, null())
| stats values(*) as * by TXID
| eval difference=end-start 
| where difference>10000