topic How to extract and apply header information to every log line? in Splunk Search

How to extract and apply header information to every log line?

RReichel — Sat, 09 Jan 2016 00:30:38 GMT

Hello Splunk Guru's,

The file below contains a header of 7 lines followed by an undetermined number of log lines. I would like for the header to apply to each and every log line. For instance, I would like to be able to search on our Version=6 and find all log lines associated with this version.

Timestamp=2016-01-08T14:29:20
SmartRecorderSN=HL3BC085
Version=6
FirmwareVersion=3.09.14
EventDurationSetpoint=30
BlackoutSetpoint=5
Iteration=322966

TRAT2,HL3BC085201601081429001212ER.SDE,2016-01-08T14:29:01,521,0.0004,0.000000,0.000000,1.0000,-1.5000,1,-0.0016,1.0000
ECR,HL3BC085201601081429001212ER.SDE,9,2016-01-08T14:29:00,00000,1,521,3326464,0,0.000000,0.000000
ECR,HL3BC085201601081429135674CDR.SDE,9,2016-01-08T14:29:13,00000,1,429,3345602,0,0.000000,0.000000
TRC,HL3BC085201601081429135674CDR.SDE,2016-01-08T14:29:13,429,0.000000,0.000000,0,0,30,1,1,0,-1,-1
TRAT2,HL3BC085201601081429291213ER.SDE,2016-01-08T14:29:27,521,0.0004,0.000000,0.000000,1.0000,-1.5000,1,-0.0016,1.0000
ECR,HL3BC085201601081429291213ER.SDE,9,2016-01-08T14:29:27,00000,1,521,3388928,1,0.000000,0.000000
ECR,HL3BC085201601081429435675CDR.SDE,9,2016-01-08T14:29:43,00000,1,429,3357073,0,0.000000,0.000000
TRC,HL3BC085201601081429435675CDR.SDE,2016-01-08T14:29:43,429,0.000000,0.000000,0,0,30,1,1,0,-1,-1
EndTimeStamp=2016-01-08T14:30:02

Kind Regards,
Rob

Re: How to extract and apply header information to every log line?

Richfez — Sat, 09 Jan 2016 00:51:32 GMT

Right now, EACH of those lines comes through as one event?

I'm thinking we need to fix your event breaking first then deal with any fallout after that, but it depends on your answer to the previous question. 🙂

Re: How to extract and apply header information to every log line?

abhijitp — Sat, 09 Jan 2016 00:57:19 GMT

You are correct. Right each of these lines are interpreted as one event (and hence missing the header info).

Re: How to extract and apply header information to every log line?

Richfez — Sat, 09 Jan 2016 02:19:34 GMT

We need to make your event break properly, meaning that whole group should be a single event. To do this, you would need something like the following in your etc/apps/MyApp/local/props.conf (or etc/system/local/props.conf if you don't have this in an app).

[sourcetype-to-break]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = ^Timestamp=

That should make any new events that get indexed have all entire group be one event. There is more information on breaking events correctly in the docs for configuring event linebreaking.

Obviously, change "sourcetype-to-break" to your sourcetype you have set on that input. You are manually setting a sourcetype on the input, aren't you? 🙂

Now, one thing this might do is mess up the timestamping of events, so check that. I don't THINK this will be a problem but I wanted to mention it just in case. So, if you have problems where events don't come in with the time stamp set to whatever's in the "Timestamp=" line, then be sure to post back because that should be easy to fix.

Re: How to extract and apply header information to every log line?

RReichel — Mon, 11 Jan 2016 18:58:45 GMT

Hi Rich,

Thank you for your quick response! Your answer created a single event for me containing all of the log lines. This will work for what I initially asked which was to search the events by version.

However, I failed to mention that I would also like to have each log line as it's own event so that I can extract many different fields from it. For instance, take a look at the two lines below.

ECR,HL3BC085201601081429001212ER.SDE,9,2016-01-08T14:29:00,00000,1,521,3326464,0,0.000000,0.000000

ECR,HL3BC085201601081429135674CDR.SDE,9,2016-01-08T14:29:13,00000,1,429,3345602,0,0.000000,0.000000

If these were in the same event, then when I try to extract the second field there would be two different values: HL3BC085201601081429001212ER.SDE and HL3BC085201601081429135674CDR.SDE.

Please pardon my ignorance as I am fairly new to splunk. If it is possible I would like to get that header information to essentially act as a default field such as "source" that applies to each log line (event) in the source file.

Re: How to extract and apply header information to every log line?

RReichel — Mon, 11 Jan 2016 19:01:09 GMT

To answer your other questions, I am manually setting a sourcetype on the input and the Timestamp was coming through just fine with your solution.

Re: How to extract and apply header information to every log line?

RReichel — Mon, 11 Jan 2016 20:48:46 GMT

https://answers.splunk.com/answers/122078/how-to-handle-metadata-in-file-headers.html

It seems someone else has already asked this question. So in my understanding, to get splunk to process the file the way I want, I need to modify the file before it gets into splunk with the header information appended to each line.