Getting Data In

Timestamp issue

ips_mandar
Builder

I have below sample events in log file-

2019-07-19|23:02:24.213|TEST|XYZ|Test1
2019-07-19|23:02:24.213|TEST|XYZ|Test2
2008-12-31|19:01:22.265|TEST|XYZ|Test3
2008-12-31|19:01:22.359|TEST|XYZ|Test4

Here it can be seen that few of last events are coming for Year 2008 due to which my all events are becoming of 2008 year in splunk although i mentioned time format/time prefix and by default MAX_DAYS_AGO is 5.48 years so it should not index these events ..below is my props.conf-

[sample_st]
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
disabled = false
TIME_PREFIX=^
TIME_FORMAT=%Y-%m-%d|%H:%M:%S.%3N
TRANSFORMS-set= setnull,setparsing,replacedefaultsource
0 Karma

DavidHourani
Super Champion

Hi @ips_mandar,

Try setting it to a smaller value. It should fix it. Set it to 10 for example for that sourcetype :

MAX_DAYS_AGO = 10

Edit:
Here's the definition for this setting, so your older events will only be indexed with a newer date instead, not ignored though :

* Splunk software still indexes events with dates older than 'MAX_DAYS_AGO'
  with the timestamp of the last acceptable event.
* If no such acceptable event exists, new events with timestamps older
  than 'MAX_DAYS_AGO' uses the current timestamp.
* For example, if MAX_DAYS_AGO = 10, Splunk software applies the timestamp
  of the last acceptable event to events with extracted timestamps older
  than 10 days in the past. If no acceptable event exists, Splunk software
  applies the current timestamp.

If you wish to ignore those events completely you can use sed-cmd to filter out any event starting with that 2008 year.

Best regards,
David

0 Karma

ips_mandar
Builder

Thanks @DavidHourani
I tried setting MAX_DAYS_AGO = 10 in props.conf
but now 2008 year showing as it is and 2019 years showing it as 2009 year ...kindly help here

0 Karma

DavidHourani
Super Champion

First event into Splunk should be the 2019 year. If you want to get rid of the 2008 altogether then just add this to your sourcetype :
SEDCMD-test = s/2008.+//g

0 Karma

ips_mandar
Builder

Thanks
Yes first event in file which I am monitoring is 2019 year still it is changing to 2009 year and how splunk will monitor file top to bottom?
for sedcmd there can be possibility of 2007/2006 or any year as well so I can not hard code it in sedcmd.

Why it is showing me 2009 year ?if require I can share you exact file on email.
Thanks

0 Karma

skalliger
SplunkTrust
SplunkTrust

Hey.

I don't see a MAX_DAYS_AGO in your props.conf stanza provided. Also, according to the props.conf that setting does not prevent old data from being indexed.

To exclude such events, you'd need a transform to send those events to the nullQueue (based on a regex that sends all events to nothing that will match, basically).

There are quite a few posts about how to do this out there. E.g. https://answers.splunk.com/answers/96/how-do-i-exclude-some-events-from-being-indexed-by-splunk.html

Skalli

0 Karma

ips_mandar
Builder

Thanks @skalliger
I am already using nullqueue and indexqueue ...and also now I tried MAX_DAYS_AGO = 10 in props.conf
but now 2008 year showing as it is and 2019 years showing it as 2009 year ...kindly help here

0 Karma

skalliger
SplunkTrust
SplunkTrust

Show us your transforms.conf doing the nullQueue sampling. Basically, you need a REGEX matching on the years you want to discard.
With an example event, we might be able to help here.

Skalli

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...