topic Re: How to make Splunk ignore logs older than specific days in Getting Data In

How to make Splunk ignore logs older than specific days

phudinhha — Wed, 01 Jul 2020 07:34:06 GMT

Dear Team,

I had an issue with splunk and had to follow this post:

https://community.splunk.com/t5/All-Apps-and-Add-ons/Amazon-Web-Services-Add-on-s3-generic-error-TypeError-int-object/td-p/439118

to make my splunk works again. However, now Splunk will ingest logs from the beginning of everything. How do I make splunk to ingest logs from the last 7 days / 14 days.

I'm pretty new with Splunk so I really appreciate every input from you guys.

Re: How to make Splunk ignore logs older than specific days

isoutamo — Wed, 01 Jul 2020 10:06:23 GMT

Check from props.conf how to use MAX_DAYS_AGO. It defines how old data splunk will accepts.

Re: How to make Splunk ignore logs older than specific days

gcusello — Wed, 01 Jul 2020 10:16:04 GMT

Hi @phudinhha ,

did you tried with ignoreOlderThan option in inputs.conf?

For more infos see at https://docs.splunk.com/Documentation/Splunk/8.0.4/Admin/Inputsconf

Ciao.

Giuseppe

Re: How to make Splunk ignore logs older than specific days

phudinhha — Wed, 01 Jul 2020 10:50:20 GMT

Hi gcusello,

I tried ignoreOlderThan option in inputs.conf like below:

host = TESTSERVER
index = serverlogs
sourcetype = serverlogs:json
ignoreOlderThan = 14d
recursive = true

Is that okay? Should I use any other solution?

Re: How to make Splunk ignore logs older than specific days

gcusello — Wed, 01 Jul 2020 10:55:49 GMT

Hi @phudinhha ,

this is the solution.

Remember to restart Splunk on Forwarder after update.

Ciao.

Giuseppe

Re: How to make Splunk ignore logs older than specific days

phudinhha — Thu, 02 Jul 2020 02:48:37 GMT

hi @gcusello

I tried but the log stopped coming in for that index. I did restart my whole splunk and still nothing happen. I checked the colddb and db and nothing new added.

in the index, I set up the storage optimization long time ago, and it Reduce tsidx files older than values is 21 days. Is that the root cause ?

Re: How to make Splunk ignore logs older than specific days

gcusello — Thu, 02 Jul 2020 07:18:00 GMT

Hi @phudinhha ,

I don't think, but try to enlarge it.

Check also what's the date format of your logs: if it's dd/mm/yyyy, in the first days of each month there could be a problem related to the Splunk default date format (mm/dd/yyyy).

You can check this viewing logs on operative system and/or searching logs of 1st July on the 7th of January.

Ciao.

Giuseppe

Re: How to make Splunk ignore logs older than specific days

phudinhha — Thu, 02 Jul 2020 08:09:00 GMT

I use "_index_earliest = -15m" and found that they're ingesting log with this format key_name="20200106/20200106T111000Z_20200106T111500Z_8ce5e9c5.log.gz"

However, when i do index=serverlogs, I saw log of July-2nd as well.

Where can we go to check the default splunk format?

Re: How to make Splunk ignore logs older than specific days

gcusello — Thu, 02 Jul 2020 08:45:36 GMT

Hi @phudinhha ,

you should correctly define the TIME_FORMAT and the TIME_PREFIX for your sourcetype.

If you share an example of your your logs and the indication of which is the correct timestamp to take I could help you.

Ciao.

Giuseppe