Is it possible to extract multiple timestamps from individual lines at search time and then use the time picker on them?

[willial]

I have a log file that's made up of timestamped log messages, so there's a _time for the file, but then multiple timestamps for each individual message, as such:

1/1/2015 12:34:56 Log message here
1/1/2015 01:23:45 Other log message here

I'm extracting the timestamps using rex, but I haven't found a way to use them yet. I'd like to use them with the time picker if at all possible.

[willial · Accepted solution]

Here's my workaround.

I've created a dropdown to mimic parts of the time picker, in this fashion:

Last hour, -1h
Last day, -1d
Last month, -1mon

I'm doing this:

convert mktime(time) AS time | where time>relative_time(now(),"$fauxTimePicker$")

The first part converts the human-readable timestamp "time" into epoch. The second part checks to make sure that value is within the range created between now() and the relative time value from my dropdown.

It's not a fully-functional time picker, but it's a fair enough workaround.

[woodcock]

It is great that you are happy with the format of your time fields but it really makes no sense to use them in preference to the _time field. The concept of _time is central to everything that Splunk does and yet you are trying desperately to avoid this core concept. Why are you not using _time and doing things the normal/easy way? In any case, you certainly can use a timepicker and interact with your time fields but you are going to have to normalize them to the same format. I am assuming that your times are "human readable" but you cannot do math on that so you should use strptime to convert them to epoch. Then, using the token from your timepicker, you can do things like this:

... | where myEpochTime > $time_tok.earliest$

[willial]

My file does not get split into multiple events. I am forced to extract the timestamps manually. I don't know why this is the case. I'm certainly not doing this on purpose.

[woodcock]

Well why didn't you say so before? That is what I was trying to get at with my previous questions. What are you using in inputs.conf, props.conf, and transforms.conf? Let's get your linebreaking and timestamping working and you will be much better off than leaving it broken and trying to work with that.

[willial]

That's the point: I'm not doing anything in transforms, my props file is just two lines to keep from truncating large files (from above: TRUNCATE for long lines and MAX_EVENTS for big linecounts), and my inputs just pull in tcp/ssl and set a sourcetype.

[woodcock]

I would still like to see your file contents because this doesn't make sense. Your TCP input should be treating each line as a separate event unless you have reconfigured LINE_BREAKER or have set SHOULD_LINEMERGE to true. What sourcetype are you setting for this input in inputs.conf? Maybe you are using something that is picking up a configuration from elsewhere because you share the same sourcetype value.

[willial]

File contents are tricky, since they're not always standard. Also they're too big to post. Suffice to say, they consist of a bunch of various system output, with one section being dedicated to system logs as I explained above, always following this format:

1/1/2015 12:34:56 <Log header> Log text.

inputs.conf is (plus one stanza seting SSL password, rootCA, serverCert):

[tcp://port#]
connection_host = dns
index = myindex
sourcetype = my-type

[tcp-ssl:sslport#]
index = myindex
sourcetype = my-type

I've masked these slightly. The sourcetype is unique to these inputs.

props.conf is (some of these files are really big -- 100,000+ lines)

[default]
TRUNCATE = 0
MAX_EVENTS = 150000

I'm not setting LINE_BREAKER or SHOULD_LINEMERGE. SHOULD_LINEMERGE defaults to true in the system/default/props.conf, of course.

[woodcock]

Remember that I asked for FULL PATH for each file. It appears that you are editing files in $SPLUNK_HOME/etc/system/default/ which you DEFINITELY should not be doing. You need to copy out your stuff and put it in new files here:

$SPLUNK_HOME/etc/apps/MyApp/default/

Once that is done, you should reinstall Splunk to make sure that the files that you modified are reverted back to factory install (this will not remove anything in MyApp).

Once you have properly limited the scope of your changes to just your app, you should feel free to use the [default] stanza header if you like (but I still think it is poor form). As an additional benefit, we can be absolutely sure that you are using good defaults for the settings that you have not changed (something which we cannot be sure of now if you indeed have made changes to the default files).

[willial]

I've made no indication that I'm editing or have edited files in system/default -- I pointed out that SHOULD_LINEMERGE defaults to true in system/default/props.conf and that since I'm not setting it elsewhere, it's true. Which you indicated was some sort of problem.

I'm editing files in apps/appname/default.

[woodcock]

Splunk should already be breaking each line as a separate event with it's own timestamp. It actually takes much extra work to force Splunk to join lines and use the file for timestamping. What are your existing configurations ( inputs.conf, props.conf, transforms.conf )? What does this command output?

index=* | dedup host source sourcetype | table *

[willial]

inputs.conf: setting one tcp and one ssl input

props.conf: setting TRUNCATE and MAX_EVENTS

transforms.conf: not using anything here

Command outputs about 75 fields.

[woodcock]

For your events, is there a field called _time that is different that time time in the filename (field = source)?

[willial]

In my cases, the timestamp isn't typically in the filename. I see _time, but it's unclear on where it's being extracted from.

All the timestamps that I actually want are extracted neatly with rex, so that I have: timestamp, logfield1, logfield2,logfield3, etc... where all my necessary log data is lined up with a timestamp, but then I find I can't do anything with the timestamp for filtering purposes without building my own faux time picker with form inputs.

[woodcock]

You can do whatever you like with rex but the point that i am trying to make is that almost certainly Splunk is already creating the _time field with an Epoch conversion of the first field in each event which means you can do whatever you like with the timepicker and it is probably already working!

[willial]

If you look back at my initial question, you'll see that Splunk is not splitting the file into multiple events. It's retaining the entire file as a single event. It is not extracting the timestamp for each line, which is why I'm trying to do it manually in the first place.