topic Re: How to break events and extract fields from Scripted Input in Splunk Search

How to break events and extract fields from Scripted Input

dilipbailwal — Mon, 28 Sep 2020 17:59:43 GMT

Here is the sample data

AppPoolName : TestApp
PrivateMemory : 2000
State : Started
Application :
IdentityType : NetworkService
RecyclingTime : 00:00:00
AppPoolName : .NET v4.0
PrivateMemory : 2000
State : Started
Application :
IdentityType : ApplicationPoolIdentity
RecyclingTime : 01:00:00
AppPoolName : .NET v4.0 Classic
PrivateMemory : 2000
State : Started
Application : /DefaultApp1
/DefaultApp2
/DefaultApp3
IdentityType : ApplicationPoolIdentity
RecyclingTime : 01:00:00

What i am trying to achieve is, every name on left side of colon to be as a field name and values on right side as a field value.
Therefore my approach is Spliting the whole event at RecycleTime so we get multiple events and then further extracting fields using Search-Time extraction

I tried various events like MUST_BREAK_AFTER, ... _BEFORE ..etc but its not splitting in to multiple event.
here is my code

Props.conf,
[sourcetype]
MAX_TIMESTAMP_LOOKAHEAD = 150
MUST_BREAK_AFTER = RecyclingTime
NO_BINARY_CHECK = 1
REPORT-1app_field = AppPoolName
REPORT-2app_field = Application
REPORT-3app_field = IdentityType
REPORT-4app_field = PrivateMemory
REPORT-5app_field = RecyclingTime
REPORT-6app_field = State

transforms.conf
[AppPoolName]
REGEX = (?i).? : (?P\w+)
[Application]
REGEX = (?ism)Application : (?P.+)(?=IdentityType)
[IdentityType]
REGEX = (?i)IdentityType : (?P.+)(?=RecyclingTime)
[PrivateMemory]
REGEX = (?i).? : (?P\d+)
[RecyclingTime]
REGEX = (?i).? : (?P\d+:\d+:\d+)
[State]
REGEX = (?i).? : (?P\w+)

can you guys suggest, what am i missing in above code to extract events and there fields.

Re: How to break events and extract fields from Scripted Input

musskopf — Tue, 04 Nov 2014 22:46:07 GMT

If that's a scripted input, I would suggest to code the script to format the content as key="value". I use a couple of Powershell scripts to collect data from different sources like Sharepoint, AD, EventLog, etc... and I even created a library to output each events in key=value format. Much easier!

Re: How to break events and extract fields from Scripted Input

dilipbailwal — Wed, 05 Nov 2014 09:26:02 GMT

Do you mean having a hash tables or arrays.?

Re: How to break events and extract fields from Scripted Input

musskopf — Wed, 05 Nov 2014 21:24:21 GMT

If you use a script like powershell, just make a loop and print everything as key="value", for example you data you look like:

2014-11-06T04:10:09.000+10:00, AppPoolName="TestApp", PrivateMemory=2000, State="Started",...

If you data is already a hash is very simple to make a loop, if the data is a single string, you might need to split by ":" to load it into a hash table first. As you are already using Scripted Input, a few more lines on your script will save heaps of time on the Splunk side.

Re: How to break events and extract fields from Scripted Input

dilipbailwal — Wed, 12 Nov 2014 10:34:17 GMT

Thanks a ton mussktop, i did managed to get powershell output into splunk, by using key=value method. I have yet more complicating scenarios coming soon, like grasping all appwiz.cpl entries in splunk etc.. Thanks again for you help

Re: How to break events and extract fields from Scripted Input

MuS — Wed, 12 Nov 2014 13:22:24 GMT

please mark this as answered - thx