Splunk Search

How to extract and apply header information to every log line?

RReichel
Explorer

Hello Splunk Guru's,

The file below contains a header of 7 lines followed by an undetermined number of log lines. I would like for the header to apply to each and every log line. For instance, I would like to be able to search on our Version=6 and find all log lines associated with this version.

Timestamp=2016-01-08T14:29:20
SmartRecorderSN=HL3BC085
Version=6
FirmwareVersion=3.09.14
EventDurationSetpoint=30
BlackoutSetpoint=5
Iteration=322966

TRAT2,HL3BC085201601081429001212ER.SDE,2016-01-08T14:29:01,521,0.0004,0.000000,0.000000,1.0000,-1.5000,1,-0.0016,1.0000
ECR,HL3BC085201601081429001212ER.SDE,9,2016-01-08T14:29:00,00000,1,521,3326464,0,0.000000,0.000000
ECR,HL3BC085201601081429135674CDR.SDE,9,2016-01-08T14:29:13,00000,1,429,3345602,0,0.000000,0.000000
TRC,HL3BC085201601081429135674CDR.SDE,2016-01-08T14:29:13,429,0.000000,0.000000,0,0,30,1,1,0,-1,-1
TRAT2,HL3BC085201601081429291213ER.SDE,2016-01-08T14:29:27,521,0.0004,0.000000,0.000000,1.0000,-1.5000,1,-0.0016,1.0000
ECR,HL3BC085201601081429291213ER.SDE,9,2016-01-08T14:29:27,00000,1,521,3388928,1,0.000000,0.000000
ECR,HL3BC085201601081429435675CDR.SDE,9,2016-01-08T14:29:43,00000,1,429,3357073,0,0.000000,0.000000
TRC,HL3BC085201601081429435675CDR.SDE,2016-01-08T14:29:43,429,0.000000,0.000000,0,0,30,1,1,0,-1,-1
EndTimeStamp=2016-01-08T14:30:02

Kind Regards,
Rob

0 Karma

Richfez
SplunkTrust
SplunkTrust

We need to make your event break properly, meaning that whole group should be a single event. To do this, you would need something like the following in your etc/apps/MyApp/local/props.conf (or etc/system/local/props.conf if you don't have this in an app).

[sourcetype-to-break]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = ^Timestamp=

That should make any new events that get indexed have all entire group be one event. There is more information on breaking events correctly in the docs for configuring event linebreaking.

Obviously, change "sourcetype-to-break" to your sourcetype you have set on that input. You are manually setting a sourcetype on the input, aren't you? 🙂

Now, one thing this might do is mess up the timestamping of events, so check that. I don't THINK this will be a problem but I wanted to mention it just in case. So, if you have problems where events don't come in with the time stamp set to whatever's in the "Timestamp=" line, then be sure to post back because that should be easy to fix.

0 Karma

RReichel
Explorer

https://answers.splunk.com/answers/122078/how-to-handle-metadata-in-file-headers.html

It seems someone else has already asked this question. So in my understanding, to get splunk to process the file the way I want, I need to modify the file before it gets into splunk with the header information appended to each line.

0 Karma

RReichel
Explorer

Hi Rich,

Thank you for your quick response! Your answer created a single event for me containing all of the log lines. This will work for what I initially asked which was to search the events by version.

However, I failed to mention that I would also like to have each log line as it's own event so that I can extract many different fields from it. For instance, take a look at the two lines below.

ECR,HL3BC085201601081429001212ER.SDE,9,2016-01-08T14:29:00,00000,1,521,3326464,0,0.000000,0.000000

ECR,HL3BC085201601081429135674CDR.SDE,9,2016-01-08T14:29:13,00000,1,429,3345602,0,0.000000,0.000000

If these were in the same event, then when I try to extract the second field there would be two different values: HL3BC085201601081429001212ER.SDE and HL3BC085201601081429135674CDR.SDE.

Please pardon my ignorance as I am fairly new to splunk. If it is possible I would like to get that header information to essentially act as a default field such as "source" that applies to each log line (event) in the source file.

RReichel
Explorer

To answer your other questions, I am manually setting a sourcetype on the input and the Timestamp was coming through just fine with your solution.

0 Karma

Richfez
SplunkTrust
SplunkTrust

Right now, EACH of those lines comes through as one event?

I'm thinking we need to fix your event breaking first then deal with any fallout after that, but it depends on your answer to the previous question. 🙂

0 Karma

abhijitp
Path Finder

You are correct. Right each of these lines are interpreted as one event (and hence missing the header info).

0 Karma