Getting Data In

How to make Splunk ignore logs older than specific days

phudinhha
Explorer

Dear Team,

 

I had an issue with splunk and had to follow this post:

https://community.splunk.com/t5/All-Apps-and-Add-ons/Amazon-Web-Services-Add-on-s3-generic-error-Typ...

to make my splunk works again. However, now Splunk will ingest logs from the beginning of everything. How do I make splunk to ingest logs from the last 7 days / 14 days.

I'm pretty new with Splunk so I really appreciate every input from you guys.

Labels (2)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @phudinhha ,

did you tried with ignoreOlderThan option in inputs.conf?

For more infos see at https://docs.splunk.com/Documentation/Splunk/8.0.4/Admin/Inputsconf

Ciao.

Giuseppe

0 Karma

phudinhha
Explorer

Hi gcusello,

I tried ignoreOlderThan option in inputs.conf like below:

host = TESTSERVER
index = serverlogs
sourcetype = serverlogs:json
ignoreOlderThan = 14d
recursive = true

Is that okay? Should I use any other solution?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @phudinhha ,

this is the solution.

Remember to restart Splunk on Forwarder after update.

Ciao.

Giuseppe

0 Karma

phudinhha
Explorer

hi @gcusello 

I tried but the log stopped coming in for that index. I did restart my whole splunk and still nothing happen. I checked the colddb and db and nothing new added.

in the index, I set up the storage optimization long time ago, and it Reduce tsidx files older than values is 21 days. Is that the root cause ?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @phudinhha ,

I don't think, but try to enlarge it.

Check also what's the date format of your logs: if it's dd/mm/yyyy, in the first days of each month there could be a problem related to the Splunk default date format (mm/dd/yyyy).

You can check this viewing logs on operative system and/or searching logs of 1st July on the 7th of January.

Ciao.

Giuseppe

0 Karma

phudinhha
Explorer

I use "_index_earliest = -15m" and found that they're ingesting log with this format key_name="20200106/20200106T111000Z_20200106T111500Z_8ce5e9c5.log.gz"

However, when i do index=serverlogs, I saw log of July-2nd as well.

Where can we go to check the default splunk format?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @phudinhha ,

you should correctly define the TIME_FORMAT and the TIME_PREFIX for your sourcetype.

If you share an example of your your logs and the indication of which is the correct timestamp to take I could help you.

Ciao.

Giuseppe

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Check from props.conf how to use MAX_DAYS_AGO. It defines how old data splunk will accepts.

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...