Splunk Search

How to count specific field in a mini batch log events?


Each log event has more than 1 transaction because we are logging a mini batch log events. So, for every 2 minutes a bunch of transactions are logged as single event. Below is a sample.

In this case, how can I count a no of transactions like no. of Code and no. of minCode. If I do "timechart span=2m count" it gives each log event (contains multiple trans of mini batch logs) as 1.

Please help me find the count of each transactions.

Sample log event...

2021-05-11 21:36:33,634{"level":"INFO","message":"COMMON_FIELDS - Code:1001 | Status:New | minCode:ABC"} {"level":"INFO","message":"COMMON_FIELDS - Code:1002 | Status:New | minCode:DEF"}{"level":"INFO","message":"COMMON_FIELDS - Code:1003 | Status:Modify | minCode:XYZ"}


2021-05-11 21:38:31,524: {"level":"INFO","message":"COMMON_FIELDS - Code:1011 | Status:New | minCode:RTY"} {"level":"INFO","message":"COMMON_FIELDS - Code:1012 | Status:New | minCode:HJK"}{"level":"INFO","message":"COMMON_FIELDS - Code:1013 | Status:Modify | minCode:VFR"}{"level":"INFO","message":"COMMON_FIELDS - Code:1014 | Status:New | minCode:KLO"}


The result I expect is something like this...

using ==> | timechart span=2m count

_time count
2021-05-11 21:26:00 3
2021-05-11 21:28:00 4


using ==> | timechart span=5m count

_time count
2021-05-11 21:26:00 7


Labels (1)
0 Karma


There are three steps. (OK, four.  Step 0 is to beg your developer to write multiple transactions as conformant JSON array instead of just jam them into one unstructured string.)  First, transform the concatenated JSON into conformant JSON array. (Your developer should have done this without Splunk.  Using rex this way is not robust.)  Secondly, extract the appropriate JSON objects. (A combination of spath and mvexpand.)  Third, extract Code and minCode. (kv aka extract)


| rex mode=sed "s/ {/ [{/ s/} *{/},{/g s/}$/}]/" ``` transform concatenated JSON into array ```
| eval _raw = replace(_raw, "^[^\[]+", "") ``` retain JSON only ```
| spath path={}
| mvexpand {}
| spath input={}
| rename _raw as temp, message AS _raw
| kv pairdelim="|" kvdelim=":"


Your sample data give me

1001New2021-05-11 21:36:33.634INFOABC
1002New2021-05-11 21:36:33.634INFODEF
1003Modify2021-05-11 21:36:33.634INFOXYZ
1011New2021-05-11 21:38:31.524INFORTY
1012New2021-05-11 21:38:31.524INFOHJK
1013Modify2021-05-11 21:38:31.524INFOVFR
1014New2021-05-11 21:38:31.524INFOKLO

Below is data emulation that you can play with and compare with real data


| makeresults
| eval data = mvappend("2021-05-11 21:36:33,634: {\"level\":\"INFO\",\"message\":\"COMMON_FIELDS - Code:1001 | Status:New | minCode:ABC\"} {\"level\":\"INFO\",\"message\":\"COMMON_FIELDS - Code:1002 | Status:New | minCode:DEF\"}{\"level\":\"INFO\",\"message\":\"COMMON_FIELDS - Code:1003 | Status:Modify | minCode:XYZ\"}",
"2021-05-11 21:38:31,524: {\"level\":\"INFO\",\"message\":\"COMMON_FIELDS - Code:1011 | Status:New | minCode:RTY\"} {\"level\":\"INFO\",\"message\":\"COMMON_FIELDS - Code:1012 | Status:New | minCode:HJK\"}{\"level\":\"INFO\",\"message\":\"COMMON_FIELDS - Code:1013 | Status:Modify | minCode:VFR\"}{\"level\":\"INFO\",\"message\":\"COMMON_FIELDS - Code:1014 | Status:New | minCode:KLO\"}")
| mvexpand data
| eval _time = strptime(replace(data, ": .*", ""), "%F %H:%M:%S,%N")
| rename data AS _raw
``` data emulation above ```



Tags (3)
0 Karma


Thanks Yuanliu.

I have requested my developer to write multiple transactions as conformant JSON array. But, it will take sometime.

In the meantime, I'll try with your recommendation and let you know.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In the last month, the Splunk Threat Research Team (STRT) has had 2 releases of new security content via the ...

Announcing the 1st Round Champion’s Tribute Winners of the Great Resilience Quest

We are happy to announce the 20 lucky questers who are selected to be the first round of Champion's Tribute ...

We’ve Got Education Validation!

Are you feeling it? All the career-boosting benefits of up-skilling with Splunk? It’s not just a feeling, it's ...