topic Re: Unable to extract new timestamp field in Splunk Dev

Unable to extract new timestamp field

wuming79 — Sat, 10 Jun 2017 10:05:01 GMT

Hi,

I have a log file with timestamp = time of saving the file thus the timestamp from splunk are all the same. I not want to extract the epoch time example "1495447178314" and make it as my timestamp. Is that possible?

I also tried to extract new field using regular expression \d{13}[^\}\}]* but could not get anything when I applied it. Using https://regex101.com/ to test it out, somehow it looks like there were 2 matches. Can someone help? Maybe is it the way I do the \}\} ?

15199256 [EPS-log-dispatcher-9] INFO 
1.24978294676695149906 - {"Log Header": "{"endpointKeyHash":null,"applicationToken":null,"headerVersion":null,"timestamp":null,"logSchemaVersion":null}", "Event": {"temperature":35,"timeStamp":1495447178314}}

Re: Unable to extract new timestamp field

maciep — Sat, 10 Jun 2017 12:13:49 GMT

I think you should be able to use TIME_PREFIX to tell Splunk where the timestamp is and %s to tell Splunk the format. These setting need to go in props.conf. If the input is coming from a universal forwarder, then the props config should be on your indexer likely.

TIME_PREFIX = "timeStamp":
TIME_FORMAT = %s

Re: Unable to extract new timestamp field

wuming79 — Mon, 12 Jun 2017 13:34:05 GMT

Do I just add to 1st 2 rows of props.conf file below [custom_log]?

Currently my props.conf file is as follows:

[custom_log]
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Application
description = For IoT Demo
disabled = false
maxDist = 75
pulldown_type = true

I tried to upload the data again but it still shows the file saved time instead of the timestamp. Is there other ways such as extracting it as another field?

Re: Unable to extract new timestamp field

wuming79 — Mon, 12 Jun 2017 14:06:14 GMT

I managed to extract my new field but it seems not what I wanted...the "Time" still 6/10/17.....

Re: Unable to extract new timestamp field

maciep — Mon, 12 Jun 2017 14:24:34 GMT

i'm having a little trouble reading your screenshots. But those settings should be in props.conf for the sourcetype of the log you're ingesting. I see you referenced the stanza for custom_log, but what about for the IoT Temperature sourcetype? The timestamp settings need to be applied there.

Re: Unable to extract new timestamp field

wuming79 — Mon, 12 Jun 2017 14:35:50 GMT

Hi,

I managed to make the time format from Epoch to human readable but I can't really get the millisecond out.

Example timeStamp":1495447178314
From Splunk it converted to "5/22/17 5:59:38.000 PM" but from https://www.epochconverter.com/, it is showing
May 22, 2017 5:59:38.314 PM

Reference document: http://docs.splunk.com/Documentation/Splunk/6.0/Data/Configuretimestamprecognition, .%3N should show the milliseconds.

Another weird thing is, I had to use %s%s.%3N to show the time in my time zone. If I use %s.%3N, it will show all my time to be "12/31/99 11:59:59.999 PM". Am I supposed to use just %s.%3N?

Re: Unable to extract new timestamp field

wuming79 — Mon, 12 Jun 2017 15:03:35 GMT

Actually, I didn't know how and where to edit the props.conf ...
I think I used the UI to change the timestamp format and time prefix and saved a new sourcetype? Is this ok?

[http://imgur.com/4Nn9Bf1][1]

Re: Unable to extract new timestamp field

maciep — Mon, 12 Jun 2017 15:30:41 GMT

So are you just uploading the file in the GUI over an over? Not sure what your env looks like, how you plan to ingest this data, etc. It doesn't matter necessarily what you call the sourcetype, but that you are using the same sourcetype when you search for the data as when you ingest it.

So is the timestamp correct in Splunk yet? Maybe not to the ms, but is it no longer 6/10? I'm having trouble following along with all of these posts.

Re: Unable to extract new timestamp field

wuming79 — Mon, 12 Jun 2017 15:42:15 GMT

The latest I got the the post below with EpochCOnverter screenshot. Actually, the above screenshots are all different. I think we can skip the above screenshots now since I managed to get the timestamp now but with the milliseconds still having some issues.

Re: Unable to extract new timestamp field

maciep — Mon, 12 Jun 2017 16:21:18 GMT

So the timestamp of the splunk event is correct (minus milliseconds)? You're not just extracting the timestamp to a new field, right? There is a big difference, so want to be sure splunk timestamping is working as expected.

Re: Unable to extract new timestamp field

wuming79 — Mon, 12 Jun 2017 16:31:19 GMT

yup, timestamp working minus milliseconds and I'm no longer extracting it as a new field.

Re: Unable to extract new timestamp field

maciep — Mon, 12 Jun 2017 18:52:31 GMT

Ok. I'm not sure about the millseconds part, maybe %s%3N?

Re: Unable to extract new timestamp field

wuming79 — Tue, 13 Jun 2017 08:06:53 GMT

oh....you were right...example has a dit and I just follow but my data did not have. Now it shows the milliseconds...thanks.