Getting Data In

How to make Splunk ignore logs older than specific days

phudinhha
Explorer

Dear Team,

 

I had an issue with splunk and had to follow this post:

https://community.splunk.com/t5/All-Apps-and-Add-ons/Amazon-Web-Services-Add-on-s3-generic-error-Typ...

to make my splunk works again. However, now Splunk will ingest logs from the beginning of everything. How do I make splunk to ingest logs from the last 7 days / 14 days.

I'm pretty new with Splunk so I really appreciate every input from you guys.

Labels (2)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @phudinhha ,

did you tried with ignoreOlderThan option in inputs.conf?

For more infos see at https://docs.splunk.com/Documentation/Splunk/8.0.4/Admin/Inputsconf

Ciao.

Giuseppe

0 Karma

phudinhha
Explorer

Hi gcusello,

I tried ignoreOlderThan option in inputs.conf like below:

host = TESTSERVER
index = serverlogs
sourcetype = serverlogs:json
ignoreOlderThan = 14d
recursive = true

Is that okay? Should I use any other solution?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @phudinhha ,

this is the solution.

Remember to restart Splunk on Forwarder after update.

Ciao.

Giuseppe

0 Karma

phudinhha
Explorer

hi @gcusello 

I tried but the log stopped coming in for that index. I did restart my whole splunk and still nothing happen. I checked the colddb and db and nothing new added.

in the index, I set up the storage optimization long time ago, and it Reduce tsidx files older than values is 21 days. Is that the root cause ?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @phudinhha ,

I don't think, but try to enlarge it.

Check also what's the date format of your logs: if it's dd/mm/yyyy, in the first days of each month there could be a problem related to the Splunk default date format (mm/dd/yyyy).

You can check this viewing logs on operative system and/or searching logs of 1st July on the 7th of January.

Ciao.

Giuseppe

0 Karma

phudinhha
Explorer

I use "_index_earliest = -15m" and found that they're ingesting log with this format key_name="20200106/20200106T111000Z_20200106T111500Z_8ce5e9c5.log.gz"

However, when i do index=serverlogs, I saw log of July-2nd as well.

Where can we go to check the default splunk format?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @phudinhha ,

you should correctly define the TIME_FORMAT and the TIME_PREFIX for your sourcetype.

If you share an example of your your logs and the indication of which is the correct timestamp to take I could help you.

Ciao.

Giuseppe

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Check from props.conf how to use MAX_DAYS_AGO. It defines how old data splunk will accepts.

0 Karma
Get Updates on the Splunk Community!

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...

Splunk and Fraud

Watch Now!Watch an insightful webinar where we delve into the innovative approaches to solving fraud using the ...