Getting Data In

How do I recognize a time in epoch seconds?

Explorer

Total newbie here.

I have a data file (a few lines here):

1280718483,204.28.227.23:53;5;5.49;13;2183;2183;0;0;0-2103;2-0;3-48;5-32;15-0;*-0;2183;0;0;0;0
1280718543,204.28.227.23:53;5;5.75;6;16;16;0;0;0-16;2-0;3-0;5-0;15-0;*-0;16;0;0;0;0
1280804716,204.28.227.23:53;4;6.74;77;2412;2412;0;0;0-2332;2-0;3-48;5-32;15-0;*-0;2410;2;0;0;0
1280804776,204.28.227.23:53;5;5.57;14;2391;2391;0;0;0-2343;2-0;3-0;5-48;15-0;*-0;2391;0;0;0;0

The actual file has 500+ lines (events?) going back several months.

The first number in each line (e.g.128071848) is the date in seconds since the epoch.

How can I get splunk (using 4.1.5) to recognize this as the date?

The file is called "tns-stats-0.log.0" located in /home/lis/log/lis and I have the following in etc/system/local/props.conf.

[source::.../lis/tns-stats-0.log.0]
TIME_FORMAT=%s

which is supposed to, from what I can gather, treat the format as seconds since epoch.

Yet, splunk insists on assigning all of the events the time associated with the file itself.

Someone please tell me what I'm missing here. Based on what I've read in other answers and the splunk docs, this should work.

1 Solution

Explorer

The REAL answer is that you appear to have to use sourcetype and not just [source::] in props.conf:

[tns-stats]
TIME_FORMAT=%s

AND, then you have to define the sourcetype in apps/search/input.conf:

[monitor:///home/lis/log/lis/tns-stats-0.log.0]
sourcetype = tns-stats

which seems a bit odd to me since I thought the global spec would be seen before the app level spec, but then what do I know.

Anyway, this now works.

View solution in original post

Explorer

The REAL answer is that you appear to have to use sourcetype and not just [source::] in props.conf:

[tns-stats]
TIME_FORMAT=%s

AND, then you have to define the sourcetype in apps/search/input.conf:

[monitor:///home/lis/log/lis/tns-stats-0.log.0]
sourcetype = tns-stats

which seems a bit odd to me since I thought the global spec would be seen before the app level spec, but then what do I know.

Anyway, this now works.

View solution in original post

Motivator

As for markup in comments, you can use the backtick (above the ~) to escape code.

Explorer

Thanks to ftk for nudging me towards the sourcetype route. I did the "answer my own question" so I could better format a succinct answer fo those that come after.

0 Karma

Motivator

Add the following to props.conf:

TIME_FORMAT=%s
TIME_PREFIX=^

Explorer

CRAP! - what I put gets munged together by this input box - the "[tns-stats]" and "TIME_FORMAT=%s" should be on separate lines. Same for the "[monitor:///home/lis/log/lis/tns-stats-0.log.0]" and "sourcetype = tns-stats" that goes in input.conf.
--- hope that's readable (is there markup for these comment boxes so one can be more informative?)

Explorer

OK - here's what seems to be necessary.

First - in etc/system/local/props.conf put the sourcetype and the format:
[tns-stats]

TIME_FORMAT=%s

Doing it with a source path spec seems to not take.

THEN (and this seems to be the secret sauce), in etc/apps/search/input.conf have

[monitor:///home/lis/log/lis/tns-stats-0.log.0]

sourcetype = tns-stats

and then it assigns the right timestamp to each event line.

0 Karma

Explorer

Yeah. The open question I had was if the props.conf file is hitting. I did try the full absolute path to the file to no avail.
One of the things that isn't real clear to a newbie like myself is which of the various props.conf one should be modifying. This is currently in system/local. Will try sourcetype next.

0 Karma

Motivator

Hmm, are we certain that your props.conf line is hitting? With the ... it should hit, but would you mind using the full path to the log file (or using sourcetype instead) and trying this again?

0 Karma

Explorer

Have tried that in that order and reversed (though maybe the prefix info should be seen first). After I changed props.conf in etc/system/local I stopped splunk, cleaned evendata and restarted splunk.
Still getting the same results where in the events table it shows "_time" as the file time and "timestamp" as 'none'.

Seems like your suggestion is how it should work (that's what I've been trying), but it insists on not behaving that way and I'm at a loss as to where to look to see why. Any insight into splunk logs that might have info about either not finding the data or not interping it right?

0 Karma

Explorer

Try adding TIME_PREFIX=^ and perhaps take a look at MAX_TIMESTAMP_LOOKAHEAD if the rest of the event text might include something that looks like an epoch time.

0 Karma

Explorer

No luck. I added
TIME_PREFIX=^
and then did
splunk stop
splunk clear eventdata
splunk start
and the data from that file still shows up identified with "timestamp=none" and the time of all the events reading as the file time.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!