Splunk Search

Can we setup regex to test on first 300 characters of event?

vishaltaneja070
Motivator

Hello,

I need to check the regex condition only on first 300 characters, if the regex condition available after than then it should ignore the event.

Is it possible?

0 Karma
1 Solution

vishaltaneja070
Motivator

Solved the issue by Using LOOKAHEAD option available in transforms.conf
https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/Transformsconf

[route_to_index]
LOOKAHEAD=300

View solution in original post

0 Karma

vishaltaneja070
Motivator

Solved the issue by Using LOOKAHEAD option available in transforms.conf
https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/Transformsconf

[route_to_index]
LOOKAHEAD=300
0 Karma

DavidHourani
Super Champion

Check the new comment I added to my answer as you might run into some issues with this config.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi vishaltaneja07011993,
I cannot test this but it should run.

my_search
| eval _raw=substr(_raw,1,300)
| rex field=_raw "your_regex"
| ...

Bye.
Giuseppe

0 Karma

vishaltaneja070
Motivator

I need to do it at index time. based on a word on first 300 charters of event we need to change the index. Like debug or error logs.

0 Karma

FrankVl
Ultra Champion

Is this something you want to do at search time, or do you want to use this to do some routing/filtering at index time?

If at search time, you could extract the first 300 characters into a field and then apply your regex to that field.

And is that 300 character really the only way to determine where the regex should look? Is there no recognizable pattern in the event behind which the regex should not apply?

0 Karma

vishaltaneja070
Motivator

Hello @FrankVI

I need to do it at index time. based on a word on first 300 charters of event we need to change the index. Like debug or error logs.

0 Karma

DavidHourani
Super Champion

Hi @vishaltaneja07011993,

If you want to match your regex on the first 300 characters only, you can use a regex like this one and add any character you want to allow :

^[a-z]{0,300}$

This will match only on 0->300 characters.

Cheers,
David

0 Karma

vishaltaneja070
Motivator

Need to do it index time. And we need to match a specific word only

I need to do it at index time. based on a word on first 300 charters of event we need to change the index. Like debug or error logs.

0 Karma

DavidHourani
Super Champion

Have a look here for a good example of index time extractions :
https://answers.splunk.com/answers/728933/can-you-help-me-create-the-regex-for-an-index-time.html
What exactly do you want to extract ? Is there a specific word in the first 300 characters ?

0 Karma

vishaltaneja070
Motivator

Like if there is debug word there in event but in first 300 characters only , then the event should go to debug index ,not the normal one .

0 Karma

DavidHourani
Super Champion

Careful when using lookahead as this might affect line breaking for the events that contain more than 300 characters. Go for something like this instead :
In props.conf

 [mysourcetype]
 TRANSFORMS-newindex = index2debug, OtherIndex

in transforms.conf

 [index2debug]
 DEST_KEY =_MetaData:Index
 REGEX = ^.{0,300}debug
 FORMAT = debugindex

Cheers,
David

vishaltaneja070
Motivator

Hello @DavidHourani

I have applied Lookahead for that particular transforms stanza only. so i dnt think it will affect the line breaking.
Also, It is not working, tried with below stanza

06/04/2019 02:55:06 AM
LogName=abccccccccccc
SourceName=fjdjhfjhjdjhjdhfhfjhjd
EventCode=hdfhdfjd
EventType=fdfdfdfd
Type=Error
ComputerName=dfdfdfdfdf.dfdfd.dfdf
dfdfd    
jajhsjhdjjdhjhjsjhjujjfjhhsjhjhfjjhjjdghjjfhjjdhjjdhjfhhhhdhdhdhhhhdhhhdhhhhhhdg
jdhjdjfhjdhjdfhjdjdfjdhfjdjhfdjfh
kfkkjkfhdfkdfhkfhkfhdf
Severity: debug

Still the above regex is accepting debug log which is at more than 300 characters away.

0 Karma

FrankVl
Ultra Champion

That looks like nicely structured logs (windows logs by the look of it?). Why not simply filter for type=Debug, instead of depending on the 300 char limit?

0 Karma

vishaltaneja070
Motivator

@FrankVl

I have given one example the format changes in different logs, i need some generalised thing which can work for all kind of logs.

0 Karma

DavidHourani
Super Champion

Its actually 300max then the word debug so 305 ^^
And yes, if youve tested your solution and it didnt backfire then please ignore my comment :):)

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...