In a log with multiple date fields in different formats, how to create a custom histogram with the date of my choice?

[chrisboy68]

Hi, I have an unstructured log like below. In the histogram, the events are getting confused and I suspect it's because of the multiple Date fields in different formats. How can I configure a custom histogram to include the event in between the tags and just be one of the dates (of my choice). Thank you!

For some reason I cant embed XML start and end tags in the forum. Updated below.

Chris

tagxml
Jan 12 16:43:37 10.160.82.29
Date: 2015-01-12T16:43:00-0500
Date: 1/12/2015 4:43 PM
Server: My Server
Error Message : My Error Message
tagxml

[lguinn2 · Accepted solution]

For the file clip that you have above, you could start with the following sourcetype definition:

In inputs.conf at the source of the data:

[monitor://pathtothisfile]
sourcetype=mySourceType

There! You just created a sourcetype! In props.conf on the indexer, you define how the sourcetype is parsed:

[mySourceType]
SHOULD_LINEMERGE=true
TRUNCATE=0
BREAK_ONLY_BEFORE = \<tagxmlforbeginningofevent\>
MAX_EVENTS =512
TIME_PREFIX = \<xmltagjustbeforethedateyouwant\>\s*
MAX_TIMESTAMP_LOOKAHEAD=25

I don't know if these settings are exactly right for your file, but it should be pretty close. You can look at the definition of props.conf in the manual for info on each setting. TIME_PREFIX is important because it identifies the character string just before the timestamp that you want. MAX_TIMESTAMP_LOOKAHEAD is needed to say "after you find the prefix, just look at the next 25 characters to find the timestamp." This prevents Splunk from continuing to scan the event looking for other possible timestamps.

[chrisboy68]

Perfect, ok now i figured out where to read and put my regex's in . Thank you,

Chris

[lguinn2]

GACK!! I put SHOULD_LINEMERGE=false in my original answer and it should have been SHOULD_LINEMERGE=true

I fixed the answer!

[Richfez]

While I or someone may have an answer to this specifically, I'd like to know a bit more regarding "the events are getting confused". From looking at this data and reading your post, there may be a "better" way to fix this by cleaning up how the data is interpreted when input. Can you explain a bit better?

[chrisboy68]

I have two date fields named "Date" in one event. Example below

---

Date: 2015-01-13T09:28:00-0500
Date: 1/13/2015 9:28 AM
multi line log entry
stuff
stuff

---

I get one event as "Date: 2015-01-13T09:28:00-0500" and thats it, the rest of the payload of the log is not there. There is another event "Date: 1/13/2015 9:28 AM" that has the payload of the log.

Thanks for your help!

Chris

[tom_frotscher]

Thats exactly what @rich7177 was going to tell you. Splunk recognized two events, while you only want it to be one event. What you can do is to tell splunk which of those two dates is the one you want to specify for your events, and as rich7177 said, this has to be done while indexing. You indexed those events under a certain sourcetype, you have to adjust this sourcetype, so it breaks events at the occurence of the first date. Therefore, you will need to tell splunk the format of your date. If you are not that experienced, this is done best with the preview feature.

[chrisboy68]

Thanks. Can you give me a hint where I would set up the custom format for a specific source type? I can read docs from there.

Thank you,

Chris