Splunk Search

How to break events and extract fields from Scripted Input

Path Finder

Here is the sample data

AppPoolName : TestApp
PrivateMemory : 2000
State : Started
Application :
IdentityType : NetworkService
RecyclingTime : 00:00:00
AppPoolName : .NET v4.0
PrivateMemory : 2000
State : Started
Application :
IdentityType : ApplicationPoolIdentity
RecyclingTime : 01:00:00
AppPoolName : .NET v4.0 Classic
PrivateMemory : 2000
State : Started
Application : /DefaultApp1
/DefaultApp2
/DefaultApp3
IdentityType : ApplicationPoolIdentity
RecyclingTime : 01:00:00

What i am trying to achieve is, every name on left side of colon to be as a field name and values on right side as a field value.
Therefore my approach is Spliting the whole event at RecycleTime so we get multiple events and then further extracting fields using Search-Time extraction

I tried various events like MUSTBREAKAFTER, ... _BEFORE ..etc but its not splitting in to multiple event.
here is my code

Props.conf,
[sourcetype]
MAXTIMESTAMPLOOKAHEAD = 150
MUSTBREAKAFTER = RecyclingTime
NOBINARYCHECK = 1
REPORT-1appfield = AppPoolName
REPORT-2app
field = Application
REPORT-3appfield = IdentityType
REPORT-4app
field = PrivateMemory
REPORT-5appfield = RecyclingTime
REPORT-6app
field = State

transforms.conf
[AppPoolName]
REGEX = (?i).? : (?P\w+)
[Application]
REGEX = (?ism)Application : (?P.+)(?=IdentityType)
[IdentityType]
REGEX = (?i)IdentityType : (?P.+)(?=RecyclingTime)
[PrivateMemory]
REGEX = (?i).
? : (?P\d+)
[RecyclingTime]
REGEX = (?i).? : (?P\d+:\d+:\d+)
[State]
REGEX = (?i).
? : (?P\w+)

can you guys suggest, what am i missing in above code to extract events and there fields.

0 Karma
1 Solution

Builder

If you use a script like powershell, just make a loop and print everything as key="value", for example you data you look like:

2014-11-06T04:10:09.000+10:00, AppPoolName="TestApp", PrivateMemory=2000, State="Started",...

If you data is already a hash is very simple to make a loop, if the data is a single string, you might need to split by ":" to load it into a hash table first. As you are already using Scripted Input, a few more lines on your script will save heaps of time on the Splunk side.

View solution in original post

Builder

If you use a script like powershell, just make a loop and print everything as key="value", for example you data you look like:

2014-11-06T04:10:09.000+10:00, AppPoolName="TestApp", PrivateMemory=2000, State="Started",...

If you data is already a hash is very simple to make a loop, if the data is a single string, you might need to split by ":" to load it into a hash table first. As you are already using Scripted Input, a few more lines on your script will save heaps of time on the Splunk side.

View solution in original post

Path Finder

Thanks a ton mussktop, i did managed to get powershell output into splunk, by using key=value method. I have yet more complicating scenarios coming soon, like grasping all appwiz.cpl entries in splunk etc.. Thanks again for you help

0 Karma

SplunkTrust
SplunkTrust

please mark this as answered - thx

0 Karma

Builder

If that's a scripted input, I would suggest to code the script to format the content as key="value". I use a couple of Powershell scripts to collect data from different sources like Sharepoint, AD, EventLog, etc... and I even created a library to output each events in key=value format. Much easier!

0 Karma

Path Finder

Do you mean having a hash tables or arrays.?

0 Karma