Hello Splunk Guru's,
The file below contains a header of 7 lines followed by an undetermined number of log lines. I would like for the header to apply to each and every log line. For instance, I would like to be able to search on our Version=6 and find all log lines associated with this version.
Timestamp=2016-01-08T14:29:20 SmartRecorderSN=HL3BC085 Version=6 FirmwareVersion=3.09.14 EventDurationSetpoint=30 BlackoutSetpoint=5 Iteration=322966 TRAT2,HL3BC085201601081429001212ER.SDE,2016-01-08T14:29:01,521,0.0004,0.000000,0.000000,1.0000,-1.5000,1,-0.0016,1.0000 ECR,HL3BC085201601081429001212ER.SDE,9,2016-01-08T14:29:00,00000,1,521,3326464,0,0.000000,0.000000 ECR,HL3BC085201601081429135674CDR.SDE,9,2016-01-08T14:29:13,00000,1,429,3345602,0,0.000000,0.000000 TRC,HL3BC085201601081429135674CDR.SDE,2016-01-08T14:29:13,429,0.000000,0.000000,0,0,30,1,1,0,-1,-1 TRAT2,HL3BC085201601081429291213ER.SDE,2016-01-08T14:29:27,521,0.0004,0.000000,0.000000,1.0000,-1.5000,1,-0.0016,1.0000 ECR,HL3BC085201601081429291213ER.SDE,9,2016-01-08T14:29:27,00000,1,521,3388928,1,0.000000,0.000000 ECR,HL3BC085201601081429435675CDR.SDE,9,2016-01-08T14:29:43,00000,1,429,3357073,0,0.000000,0.000000 TRC,HL3BC085201601081429435675CDR.SDE,2016-01-08T14:29:43,429,0.000000,0.000000,0,0,30,1,1,0,-1,-1 EndTimeStamp=2016-01-08T14:30:02
We need to make your event break properly, meaning that whole group should be a single event. To do this, you would need something like the following in your etc/apps/MyApp/local/props.conf (or etc/system/local/props.conf if you don't have this in an app).
[sourcetype-to-break] SHOULD_LINEMERGE = True BREAK_ONLY_BEFORE = ^Timestamp=
That should make any new events that get indexed have all entire group be one event. There is more information on breaking events correctly in the docs for configuring event linebreaking.
Obviously, change "sourcetype-to-break" to your sourcetype you have set on that input. You are manually setting a sourcetype on the input, aren't you? 🙂
Now, one thing this might do is mess up the timestamping of events, so check that. I don't THINK this will be a problem but I wanted to mention it just in case. So, if you have problems where events don't come in with the time stamp set to whatever's in the "Timestamp=" line, then be sure to post back because that should be easy to fix.
It seems someone else has already asked this question. So in my understanding, to get splunk to process the file the way I want, I need to modify the file before it gets into splunk with the header information appended to each line.
Thank you for your quick response! Your answer created a single event for me containing all of the log lines. This will work for what I initially asked which was to search the events by version.
However, I failed to mention that I would also like to have each log line as it's own event so that I can extract many different fields from it. For instance, take a look at the two lines below.
If these were in the same event, then when I try to extract the second field there would be two different values: HL3BC085201601081429001212ER.SDE and HL3BC085201601081429135674CDR.SDE.
Please pardon my ignorance as I am fairly new to splunk. If it is possible I would like to get that header information to essentially act as a default field such as "source" that applies to each log line (event) in the source file.
Right now, EACH of those lines comes through as one event?
I'm thinking we need to fix your event breaking first then deal with any fallout after that, but it depends on your answer to the previous question. 🙂