Splunk Search

How to list all the events on same timestamp and capture only the required line?

Thulasinathan_M
Communicator

Hi Splunk Experts,

I'm trying to list all the events on same timestamp and trying to capture only the required lines. But I'm not getting the expected results, seems like there is no "\n" in the aggregated event eventhough it breaks into new lines. Kindly shred some lights. Thanks in advance!!

 

I've events something like below, after aggregating them by _time:

 

Line1 blablabla
Line2 blablabla
<Interested line1>
<Interested line2>
<Interested line3>
<Ends Here>
Unwanted Line blablabla

 


Query Using:

 

index=xxx 
| reverse 
| stats list(_raw) as raw by _time
| rex field=raw "(?<Events>(\<Interested.*)((\n.*)?)+\<Ends Here\>)"

 


Result for the Above query:

 

<Interested line1>

 

 

Labels (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Thulasinathan_M,

did you tried to invert the two commands?

index=xxx 
| bin span=1h _time
| rex "(?<Events>(\<Interested.*)((\n.*)?)+\<Ends Here\>)"
| stats values(Events) AS Events BY _time

In addition, when you use _time as grouping key, usa always a bin command to group _time values or use timechart command, otherwise you'll have too many results.

Ciao.

Giuseppe

0 Karma

Thulasinathan_M
Communicator

Thanks @gcusello!!
But those are Single Line Events, so I can't perform REX before stats.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Thulasinathan_M,

yes I supposed this, for this reason I hinted to do this!

Could you share some sample of your logs?

Ciao.

Giuseppe

0 Karma

Thulasinathan_M
Communicator

@gcusello Sure, Here is the sample Events, which are all of single line events.

index=xxx
| reverse
| stats list(_raw) as raw by _time
| rex field=raw "(?<Events>(Event Type.*)((\n.*)?)+Event ID:  \d+)"


Events:

2023-08-20 22:10:10.879 Date: 20/08/2023
2023-08-20 22:10:10.879 User: DILE\Administrator
2023-08-20 22:10:10.879 Event Type: Information
2023-08-20 22:10:10.879 Event Source: AdsmClientService
2023-08-20 22:10:10.879 Event Category: None
2023-08-20 22:10:10.879 Event ID: 4101
2023-08-20 22:10:10.879 Computer: MIKEDILE
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Thulasinathan_M,

as I said, you can extract fields before the stats command and then use the xtracted field (or fields) in addition to the entire raw:

 

index=xxx
| rex "(?<Events>(Event Type.*)((\n.*)?)+Event ID:  \d+)"
| stats list(_raw) AS raw values(Event_Type) AS Event_Type BY _time

 

if you want you can extract also the other fields in the same way, always before the stats command.

Ciao.

Giuseppe

0 Karma

yuanliu
SplunkTrust
SplunkTrust

It looks like that your intention is to capture raw events with "Event Type" and "Event ID" in them.  It would have been so much easier if you just describe the actual goal.

You are correct that when you use list command, the resultant field doesn't have newline "\n" in it.  It is simply a multivalued field that Splunk's Statistics tab presents in multiple lines.

I see two different approaches to this problem.  But before that, let me comment that you should approach your developer or aggregator, whoever made these logs into multiple events, and beg, harass, or intimidate them to combine these into a single event for Splunk.  It will not only be better for Splunk, but also for people who may read the log files manually.

The most straightforward approach will be to not bother with regex or "\n".

index=xxx
| reverse
| stats list(_raw) as raw by _time
| eval Events = mvappend(mvfind(raw, "Event Type:"), mvfind(raw, "Event End:"))

Note "Events" here is also multi-valued.  In my opinion, multivalued fields are more useful subsequently.  But if you really want them to be single valued with newline, just insert newline as exemplified in the next method.

If you really, really must go with "\n", just insert it.

index=xxx
| reverse
| stats list(_raw) as raw by _time
| eval raw = mvjoin(raw, "
")
| rex field=raw "(?<Events>(Event Type.*)((\n.*)?)+Event ID:  \d+)"

 

Tags (2)
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...