topic Re: Why is Splunk not indexing in milli seconds? in Getting Data In

Why is Splunk not indexing in milli seconds?

siva_cg — Thu, 30 Aug 2018 09:32:48 GMT

Hi All,

I configured an input in which the timestamp field is in format 20180830112930314 (%Y%m%d%H%M%S%3N). The same has been configured in props.conf on Splunk Indexers, but still, I am seeing event time as 2018/08/30 11:29:30.000. I mean Splunk is showing 000 as milli seconds even if the timestamp field has milli seconds other than 000. Could you please help me to find out the issue? Thanks in advance.

Re: Why is Splunk not indexing in milli seconds?

harsmarvania57 — Thu, 30 Aug 2018 09:59:40 GMT

Hi @siva_cg,

Can you please let us know, universal forwarder is sending data directly to Indexers or it goes to Heavy Forwarder and then Indexer ? Also it will be good to if you will provide your props.conf stanza.

Re: Why is Splunk not indexing in milli seconds?

siva_cg — Tue, 29 Sep 2020 21:05:20 GMT

Hi @harsmarvania57,

Data Flow is: UF -> Intermediate Forwarder -> Indexer
props.conf
[st]
TZ = GMT
FIELD_DELIMITER = ,
TIMESTAMP_FIELDS = Timestamp
TIME_FORMAT = %Y%m%d%H%M%S%3N
SHOULD_LINEMERGE = false

TIME_PREFIX = ^

LINE_BREAKER = (^\d{17})

MAX_TIMESTAMP_LOOKAHEAD = 17

Re: Why is Splunk not indexing in milli seconds?

harsmarvania57 — Thu, 30 Aug 2018 10:09:46 GMT

Your intermediate forwarder is Universal Forwarder or Heavy Forwarder ? If it is Heavy Forwarder, in that case you need to apply props.conf on Heavy Forwarder not on Indexer. If it is Universal forwarder then please provide sample data

Re: Why is Splunk not indexing in milli seconds?

siva_cg — Thu, 30 Aug 2018 11:34:10 GMT

It is Universal forwarder. Due to sensitivity of data, everything is masked and only event is provided. Apologies.
20180830113004270,1,45,09e3ab1a,IRxxxxxx,USAxxxxxxxx,,,IRxxxx,,989368837464,,xxxxxxxx,00000000,0,0.0

Re: Why is Splunk not indexing in milli seconds?

harsmarvania57 — Thu, 30 Aug 2018 12:29:37 GMT

I have tested props.conf which you have provided with sample data and it is not extracting any time stamp so I guess splunk is assigning default timestamp which is on server at the time of indexing data.

When you use FIELD_DELIMITER, at that time I guess you need to provide all field name as well. So i have tested below props.conf with sample data and it is working fine.

[st]
TZ = GMT
FIELD_DELIMITER = ,
FIELD_NAMES = FIELD1,FIELD2,FIELD3,FIELD4,FIELD5,FIELD6,FIELD7,FIELD8,FIELD9,FIELD10,FIELD11,FIELD12,FIELD13,FIELD14,FIELD15,FIELD16
TIME_FORMAT = %Y%m%d%H%M%S%3N
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 17

Sample Data which I used

20180830113004270,1,45,09e3ab1a,IR1234,USA1234,,,IR3456,,989368837464,,hasdghghj,00000000,0,0.0

Re: Why is Splunk not indexing in milli seconds?

mstjohn_splunk — Thu, 30 Aug 2018 20:53:51 GMT

Hey @siva_cg,

Did @harsmarvania57 's solution in the comment chain work? Let me know so that I can convert it to an answer. Then, you can approve it, and also, give him some sweet sweet upvotes

Re: Why is Splunk not indexing in milli seconds?

siva_cg — Fri, 31 Aug 2018 07:50:38 GMT

it worked well. Thank you @harsmarvania57

Re: Why is Splunk not indexing in milli seconds?

harsmarvania57 — Fri, 31 Aug 2018 08:06:08 GMT

Great, I have converted my comment to answer so you can accept & upvote it.