topic Re: How do I recognize a time in epoch seconds? in Getting Data In

How do I recognize a time in epoch seconds?

bselig — Thu, 28 Oct 2010 00:02:53 GMT

Total newbie here.

I have a data file (a few lines here):

1280718483,204.28.227.23:53;5;5.49;13;2183;2183;0;0;0-2103;2-0;3-48;5-32;15-0;*-0;2183;0;0;0;0 1280718543,204.28.227.23:53;5;5.75;6;16;16;0;0;0-16;2-0;3-0;5-0;15-0;*-0;16;0;0;0;0 1280804716,204.28.227.23:53;4;6.74;77;2412;2412;0;0;0-2332;2-0;3-48;5-32;15-0;*-0;2410;2;0;0;0 1280804776,204.28.227.23:53;5;5.57;14;2391;2391;0;0;0-2343;2-0;3-0;5-48;15-0;*-0;2391;0;0;0;0

The actual file has 500+ lines (events?) going back several months.

The first number in each line (e.g.128071848) is the date in seconds since the epoch.

How can I get splunk (using 4.1.5) to recognize this as the date?

The file is called "tns-stats-0.log.0" located in /home/lis/log/lis and I have the following in etc/system/local/props.conf.

[source::.../lis/tns-stats-0.log.0] TIME_FORMAT=%s

which is supposed to, from what I can gather, treat the format as seconds since epoch.

Yet, splunk insists on assigning all of the events the time associated with the file itself.

Someone please tell me what I'm missing here. Based on what I've read in other answers and the splunk docs, this should work.

Re: How do I recognize a time in epoch seconds?

cfrantsen — Thu, 28 Oct 2010 00:33:03 GMT

Try adding TIME_PREFIX=^ and perhaps take a look at MAX_TIMESTAMP_LOOKAHEAD if the rest of the event text might include something that looks like an epoch time.

Re: How do I recognize a time in epoch seconds?

bselig — Thu, 28 Oct 2010 01:52:16 GMT

No luck. I added
TIME_PREFIX=^
and then did
splunk stop
splunk clear eventdata
splunk start
and the data from that file still shows up identified with "timestamp=none" and the time of all the events reading as the file time.

Re: How do I recognize a time in epoch seconds?

ftk — Thu, 28 Oct 2010 02:09:06 GMT

Add the following to props.conf:

TIME_FORMAT=%s
TIME_PREFIX=^

Re: How do I recognize a time in epoch seconds?

bselig — Thu, 28 Oct 2010 02:50:52 GMT

Have tried that in that order and reversed (though maybe the prefix info should be seen first). After I changed props.conf in etc/system/local I stopped splunk, cleaned evendata and restarted splunk.
Still getting the same results where in the events table it shows "_time" as the file time and "timestamp" as 'none'.

Seems like your suggestion is how it should work (that's what I've been trying), but it insists on not behaving that way and I'm at a loss as to where to look to see why. Any insight into splunk logs that might have info about either not finding the data or not interping it right?

Re: How do I recognize a time in epoch seconds?

ftk — Thu, 28 Oct 2010 03:36:47 GMT

Hmm, are we certain that your props.conf line is hitting? With the ... it should hit, but would you mind using the full path to the log file (or using sourcetype instead) and trying this again?

Re: How do I recognize a time in epoch seconds?

bselig — Thu, 28 Oct 2010 03:50:57 GMT

Yeah. The open question I had was if the props.conf file is hitting. I did try the full absolute path to the file to no avail.
One of the things that isn't real clear to a newbie like myself is which of the various props.conf one should be modifying. This is currently in system/local. Will try sourcetype next.

Re: How do I recognize a time in epoch seconds?

bselig — Thu, 28 Oct 2010 03:59:26 GMT

OK - here's what seems to be necessary.

First - in etc/system/local/props.conf put the sourcetype and the format:
[tns-stats]

TIME_FORMAT=%s

Doing it with a source path spec seems to not take.

THEN (and this seems to be the secret sauce), in etc/apps/search/input.conf have

[monitor:///home/lis/log/lis/tns-stats-0.log.0]

sourcetype = tns-stats

and then it assigns the right timestamp to each event line.

Re: How do I recognize a time in epoch seconds?

bselig — Thu, 28 Oct 2010 04:01:44 GMT

CRAP! - what I put gets munged together by this input box - the "[tns-stats]" and "TIME_FORMAT=%s" should be on separate lines. Same for the "[monitor:///home/lis/log/lis/tns-stats-0.log.0]" and "sourcetype = tns-stats" that goes in input.conf.
--- hope that's readable (is there markup for these comment boxes so one can be more informative?)

Re: How do I recognize a time in epoch seconds?

bselig — Thu, 28 Oct 2010 04:05:44 GMT

The REAL answer is that you appear to have to use sourcetype and not just [source::] in props.conf:

[tns-stats] TIME_FORMAT=%s

AND, then you have to define the sourcetype in apps/search/input.conf:

[monitor:///home/lis/log/lis/tns-stats-0.log.0] sourcetype = tns-stats

which seems a bit odd to me since I thought the global spec would be seen before the app level spec, but then what do I know.

Anyway, this now works.

Re: How do I recognize a time in epoch seconds?

bselig — Thu, 28 Oct 2010 04:07:18 GMT

Thanks to ftk for nudging me towards the sourcetype route. I did the "answer my own question" so I could better format a succinct answer fo those that come after.

Re: How do I recognize a time in epoch seconds?

ftk — Thu, 28 Oct 2010 07:14:39 GMT

As for markup in comments, you can use the backtick (above the ~) to escape code.