topic Re: Timestamp issue in Getting Data In

Timestamp issue

ips_mandar — Wed, 30 Sep 2020 01:25:34 GMT

I have below sample events in log file-

2019-07-19|23:02:24.213|TEST|XYZ|Test1
2019-07-19|23:02:24.213|TEST|XYZ|Test2
2008-12-31|19:01:22.265|TEST|XYZ|Test3
2008-12-31|19:01:22.359|TEST|XYZ|Test4

Here it can be seen that few of last events are coming for Year 2008 due to which my all events are becoming of 2008 year in splunk although i mentioned time format/time prefix and by default MAX_DAYS_AGO is 5.48 years so it should not index these events ..below is my props.conf-

[sample_st]
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
disabled = false
TIME_PREFIX=^
TIME_FORMAT=%Y-%m-%d|%H:%M:%S.%3N
TRANSFORMS-set= setnull,setparsing,replacedefaultsource

Re: Timestamp issue

skalliger — Wed, 30 Sep 2020 01:25:37 GMT

Hey.

I don't see a MAX_DAYS_AGO in your props.conf stanza provided. Also, according to the props.conf that setting does not prevent old data from being indexed.

To exclude such events, you'd need a transform to send those events to the nullQueue (based on a regex that sends all events to nothing that will match, basically).

There are quite a few posts about how to do this out there. E.g. https://answers.splunk.com/answers/96/how-do-i-exclude-some-events-from-being-indexed-by-splunk.html

Skalli

Re: Timestamp issue

DavidHourani — Mon, 22 Jul 2019 07:42:43 GMT

Hi @ips_mandar,

Try setting it to a smaller value. It should fix it. Set it to 10 for example for that sourcetype :

MAX_DAYS_AGO = 10

Edit:
Here's the definition for this setting, so your older events will only be indexed with a newer date instead, not ignored though :

* Splunk software still indexes events with dates older than 'MAX_DAYS_AGO'
  with the timestamp of the last acceptable event.
* If no such acceptable event exists, new events with timestamps older
  than 'MAX_DAYS_AGO' uses the current timestamp.
* For example, if MAX_DAYS_AGO = 10, Splunk software applies the timestamp
  of the last acceptable event to events with extracted timestamps older
  than 10 days in the past. If no acceptable event exists, Splunk software
  applies the current timestamp.

If you wish to ignore those events completely you can use sed-cmd to filter out any event starting with that 2008 year.

Best regards,
David

Re: Timestamp issue

ips_mandar — Wed, 30 Sep 2020 01:25:40 GMT

Thanks @DavidHourani
I tried setting MAX_DAYS_AGO = 10 in props.conf
but now 2008 year showing as it is and 2019 years showing it as 2009 year ...kindly help here

Re: Timestamp issue

ips_mandar — Wed, 30 Sep 2020 01:25:43 GMT

Thanks @skalliger
I am already using nullqueue and indexqueue ...and also now I tried MAX_DAYS_AGO = 10 in props.conf
but now 2008 year showing as it is and 2019 years showing it as 2009 year ...kindly help here

Re: Timestamp issue

DavidHourani — Mon, 22 Jul 2019 09:07:50 GMT

First event into Splunk should be the 2019 year. If you want to get rid of the 2008 altogether then just add this to your sourcetype :
SEDCMD-test = s/2008.+//g

Re: Timestamp issue

ips_mandar — Mon, 22 Jul 2019 09:49:51 GMT

Thanks
Yes first event in file which I am monitoring is 2019 year still it is changing to 2009 year and how splunk will monitor file top to bottom?
for sedcmd there can be possibility of 2007/2006 or any year as well so I can not hard code it in sedcmd.

Why it is showing me 2009 year ?if require I can share you exact file on email.
Thanks

Re: Timestamp issue

skalliger — Thu, 25 Jul 2019 12:18:51 GMT

Show us your transforms.conf doing the nullQueue sampling. Basically, you need a REGEX matching on the years you want to discard.
With an example event, we might be able to help here.

Skalli