How to display respective entries from two different logs based on a common extracted field value?

[amiivas]

Hi All,
I have two different sources of log and want to display respective entries from each source based on a extracted field value from the first log. For e.g:
Log 1: Jan 6 15:33:13 xxxxx : trans(2735890423)[response][247.116.54.12] gtid(2735890423😞 |Test|service|247.116.54.12|2f4ad7ae-a4f9-324d-8d1a-8d98b414c496|2735890423||||/rest/services|documentId
Note - the field that need to be extracted from this log is value of gtid(2735890423), which is extracted as tranid. (highlighted with bold font)
Log 2: Jan 6 15:33:13 xxxxx : trans(2316097519)[response] gtid(2735890423): |Test|service|transaction type|response||2f4ad7ae-a4f9-324d-8d1a-8d98b414c496|2735890423:2316097519|2018-01-06T15:33:13-08:00|5|86|86|success|200 OK

Requirement is, get the value of the dptranid from log 1 and search the other source log for respective entries. This has to be done dynamically, meaning the entry from log 1 has to be search from a different search param but the query has to be in such a way that it returns entry from both logs.

For eg: as of know we are using: index="log1" /rest/services --> which results in entries from log 1. then we manually select the tran id from the log and then use another search query to get the result from log 2. I want to write a single query for the same purpose.

Thanks.

[mayurr98 · Accepted solution]

hey amiivas,

Try this run anywhere search :

| makeresults 
| eval _raw="Jan 6 15:33:13 xxxxx : trans(2735890423)[response][247.116.54.12] gtid(2735890423): |Test|service|247.116.54.12|2f4ad7ae-a4f9-324d-8d1a-8d98b414c496|2735890423||||/rest/services|documentId" 
| append 
    [| makeresults 
    | eval _raw="Jan 6 15:33:13 xxxxx : trans(2316097519)[response] gtid(2735890423): |Test|service|transaction type|response||2f4ad7ae-a4f9-324d-8d1a-8d98b414c496|2735890423:2316097519|2018-01-06T15:33:13-08:00|5|86|86|success|200 OK"] 
| rex field=_raw "gtid\((?P<trans_id>[^])]+)" 
| stats values(_raw) as raw_event by trans_id

If you want to try with raw logs then assuming that you have two seperate indexes i.e. index1 and index2 for both sources try this: NOTE: if you have only one index then write only index=your_index (source=source1 */rest/services* OR source=source2)

  index=index1  OR index=index2  (source=source1 */rest/services* OR source=source2)
    | rex field=_raw "gtid\((?P<trans_id>[^])]+)" 
    | stats values(_raw) as raw_event by trans_id

Let me know if that helps you!

[amiivas]

Hi mayurr98,

index=index1 OR index=index2 (source=source1 */rest/services* OR source=source2) | rex field=_raw "gtid\((?P<trans_id>[^])]+)" | stats values(_raw) as raw_event by trans_id

I was trying with raw logs and the above query only resulted in entries from Log 1.

But your anywhere search is working as what expected.

Thanks,

[mayurr98]

Okay the problem must be in initial filter search

Can you just run

index=index1  OR index=index2  ((source=source1 */rest/services*) OR source=source2))

See if you are able to get logs from both sources??
If yes then run rex command and see if trans_id is getting extracted from both searches and like this you can troubleshoot!

Let me know if this helps you!

[amiivas]

Thanks it worked.. Awesome thanks. Did some here and there, some extra brackets, some more filtering and it worked.

[martin_mueller]

https://answers.splunk.com/answers/129424/how-to-compare-fields-over-multiple-sourcetypes-without-jo...