Getting Data In

How to extract fields from my log when every record in the log has different position of the field?

varunchhabra
New Member

I have a log file that contains time stamped events. The type of action done is defined by the string parameter like : FunctionCalled.
I want to extract the action field and use it. But the problem is that the keywords are not following any pattern. It could be like:

TimeStamp1 FunctionCalled
TimeStamp2 ABC::CDE<>FunctionCalledAgain

In such case, the neither the regex nor delimited pattern helping me to get the field.

On the side note, Is it necessary for the log to be ordered so as to apply generic statistical functions?

Plz help.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi varunchhabra,
if your logs have a limited number of types you can build one extraction for each situation (FunctionCalled1, FunctionCalled2, FunctionCalledn) and then configure a calculated field as a coalesce field or an eval command
| eval FunctionCalled=coalesce(FunctionCalled1, FunctionCalled2,...,FunctionCalledn).

Bye.
Giuseppe

0 Karma

niketn
Legend

How many of above patterns may exist? If there is no way for you to come up with regular expression,

Option 1
You create separate Field Extractions for FunctionalCalled say FunctionalCalled1 and FunctionalCalled2 etc and then use the following in your query:

<YourBaseSearch> FunctionalCalled1="FunctionalCalled" OR FunctionalCalled2="FunctionalCalled"

Option 2
If you do not want to perform field extraction then in base search you can use "FunctionalCalled" and in your subsequent pipe you can use searchmatch to create your own field with eval

| eval FunctionalCalled1=case(searchmatch("FunctionalCalled"),"FunctionalCalled",true(),"Other")

Ideally, interesting fields should be a key value pair, you should also read Logging best practices in order to better utilize Splunk's capabilities: http://dev.splunk.com/view/logging/SP-CAAAFCK

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Automated Threat Analysis: Available in ES Premier

Automated Threat Analysis: Centralize and Accelerate Phishing Investigations in Splunk Enterprise ...

What’s New in Splunk AI: Volume 02

Welcome to the second edition of “What’s New in Splunk AI” where we look at the latest and greatest updates, ...

Best Practices: Splunk auto adjust pipeline queue

When you enable autoAdjustQueue in Splunk, maxSize should be understood as the queue size Splunk starts with ...