topic Re: Multiple Timestamps in Getting Data In

Multiple Timestamps

mawwx3 — Thu, 24 Jun 2010 06:47:24 GMT

My events have two different times in them, one from when the dns server processed them, and then another is added to the beginning of the events from what I assume is splunk. I want splunk to treat the second time as the events timestamp. I have manually assigned the sourcetype for the port as dns_data. Here is part of my props.conf file from $SPLUNK_HOME/etc/sys/local:

props.conf--

[sourcetype::dns_data]
TIME_PREFIX = \w+ \d+ \d\d:\d\d:\d\d foo\.bar\.com
TIME_FORMAT = %b %d %H:%M:%S %Y

Here are some events:

Jun 23 16:29:19 foo.bar.com Wed Jun 23 16:27:15 2010: 123.123.123.123 -> 321.321.321.321: 52826 NOERR 'something.somewhere.com.' A IN (n#5) (x#9)

Jun 23 16:29:19 foo.bar.com Wed Jun 23 16:27:15 2010: 123.123.231.321 -> 25.321.321.31: 1572 NOERR 'something.somewhere.com.' AAAA IN (x#1)

Jun 23 16:29:19 foo.bar.com Wed Jun 23 16:27:15 2010: 213.213.21.231 -> 123.123.123.123: 25373 NOERR 'something.somewhere.com.' A IN (a#1) (n#6) (x#11) ANS 'something.somewhere.com. A IN 21.231.231.21

Any help would be appreciated.

Thanks.

Re: Multiple Timestamps

ziegfried — Fri, 25 Jun 2010 02:41:13 GMT

I guess you should use [dns_data] as your stanza name instead of [sourcetype::dns_data]. The other part of your config looks quite reasonable to me.

Re: Multiple Timestamps

Lowell — Fri, 25 Jun 2010 02:58:06 GMT

I you didn't have a TIME_PREFIX I think you would be ok (because you have %b in your time format, splunk shouldn't match your first date timestamp because it doesn't have a day of the week shown). That said, I using TIME_PREFIX is better.

I see a couple minor but potentially problem-causing issues with your current config:

I don't think the sourcetype:: prefix is allowed, as suggested by Siegfried.
Your TIME_PREFIX doesn't allow for a trailing space between the host name and the date that you want to use.
The TIME_PREFIX doesn't allow for a single-digit day (I'm assuming this is a syslog file, in which case the standard format for July 4 would be "Jul__4 2010" (had to use an underscore to get the formatting to show up on this site.) Your current regex only allows for a single space.
I recommend using a ^ (start of line) in your TIME_PREFIX to let the regex engine do less work.

Here is what I would suggest you put in your props.conf file:

[dns_data]
TIME_PREFIX = ^\w+ .\d \d\d:\d\d:\d\d \S+\s+
TIME_FORMAT = %b %d %H:%M:%S %Y

If you want, you can also use a SEDCMD entry to drop that first date out of there entirely so you consume less bytes in your index. I have some log sources that do this and it makes it easily to review the logs when there aren't two sets of dates for your eyes to scan over.

 SEDCMD-drop_extra_date = s/^\w+ .\d \d\d:\d\d:\d\d (.*)$/\1/

Keep in mind the timestamp extraction happens first, so you don't have to worry about your SEDCMD getting in the way of your TIME_PREFIX.

You could also move the second date (the one your are keeping) to the front if you want to. (To be more consitent with your other syslog mesages.

 SEDCMD-use_2nd_date = s/^\w+ .\d \d\d:\d\d:\d\d (\S+) (\w+ \w+ .\d \d\d:\d\d:\d\d \d{4})(.*)$/\2 \1 \3/

(I haven't tested these SEDCMD expressions, so they may not work right away. They are mostly just to give you and idea of what you can do.)

Still not working huh? Here are a few more things to double check:

Verify that your sourcetype is actually dns_data on your indexed events and not syslog or something like that. (You aren't using any sourcetype renaming transformers are you? If you don't know what I'm talking about then you probably are not.)
Make sure your putting your changes in the correct file. Should be $SPLUNK_HOME/etc/system/local/props.conf (You noted "sys" instead of "system" in your original question, but I'm assuming that was just a shortcut notation which I get, but splunk would not.)

See this link for more props debugging techniques

Re: Multiple Timestamps

mawwx3 — Fri, 25 Jun 2010 22:53:41 GMT

I entered in the suggested code for props.conf and it still does not extract the correct time. I really don't know why it will not take it. It acts like it doesn't even read the [dns_data] stanza. Would there be any other reason why it would not read it or act upon it?

Re: Multiple Timestamps

Lowell — Sat, 26 Jun 2010 02:45:18 GMT

I've added a few more config testing ideas to my answer. Best of luck.

Re: Multiple Timestamps

jrodman — Sat, 26 Jun 2010 05:13:29 GMT

Note that Lowell's TIME_PREFIX skips over the day of the week, and yours doesn't. That's pretty key.