topic Re: Indexes not saving past 2 weeks. in Knowledge Management

Indexes not saving past 2 weeks.

JJE — Thu, 08 Aug 2024 00:03:01 GMT

I want to start of by saying I am extremely new to splunk, so please bear with me, I'm not sure at all if I'm on the right track so please feel let me know if I need to try something else.

I have two Cisco ASA5506s are used as firewalls. Searching for either of their hostnames only yields results for about 17 days or so. So if today is the 1st day, it will overwrite the 17th day to record tomorrows logs. Since all I was doing was just trying to get a total view of how many total entries it's pulling from all indexes I wasn't sure which index could be the reason why it's not logging past 17 days. Poking around I found that the _syslog and _metrics indexes both only had logs 14-15 days old. So that lead me to modify the indexes.conf file, however this did not help log the firewalls past 17 days. What else should I be looking for? These devices see millions of hits daily, so that could possibly be contribiting to this as well.

Previous: Indexes.conf

[default] serviceSubtaskTimingPeriod = 30 serviceInactiveIndexesPeriod = 60 enableRealtimeSearch = true timePeriodInSecBeforeTsidxReduction = 604800 serviceMetaPeriod = 25 defaultDatabase = main rotatePeriodInSecs = 60 rtRouterThreads = 0 enableTsidxReduction = false maxHotIdleSecs = 0 bucketRebuildMemoryHint = auto suspendHotRollByDeleteQuery = false maxHotSpanSecs = 7776000 suppressBannerList = maxBucketSizeCacheEntries = 0 hotBucketTimeRefreshInterval = 10 maxHotBuckets = 3 processTrackerServiceInterval = 1 maxDataSize = auto maxRunningProcessGroups = 8 minRawFileSyncSecs = disable enableDataIntegrityControl = false minStreamGroupQueueSize = 2000 maxMetaEntries = 1000000 throttleCheckPeriod = 15 tstatsHomePath = volume:_splunk_summaries\$_index_name\datamodel_summary tsidxReductionCheckPeriodInSec = 600 maxBloomBackfillBucketAge = 30d datatype = event syncMeta = true partialServiceMetaPeriod = 0 frozenTimePeriodInSecs = 188697600 maxGlobalDataSizeMB = 0 quarantinePastSecs = 77760000 compressRawdata = true coldToFrozenScript = coldPath.maxDataSizeMB = 0 enableOnlineBucketRepair = true repFactor = 0 rtRouterQueueSize = 10000 maxTimeUnreplicatedWithAcks = 60 assureUTF8 = false maxTimeUnreplicatedNoAcks = 300 rawChunkSizeBytes = 131072 memPoolMB = auto homePath.maxDataSizeMB = 0 warmToColdScript = maxWarmDBCount = 300 minHotIdleSecsBeforeForceRoll = auto coldToFrozenDir = maxTotalDataSizeMB = 500000 maxConcurrentOptimizes = 6 maxRunningProcessGroupsLowPriority = 1 streamingTargetTsidxSyncPeriodMsec = 5000 journalCompression = gzip quarantineFutureSecs = 2592000 splitByIndexKeys = sync = 0 serviceOnlyAsNeeded = true [_audit] bucketRebuildMemoryHint = 0 compressRawdata = 1 coldPath = $SPLUNK_DB\audit\colddb minHotIdleSecsBeforeForceRoll = 0 enableTsidxReduction = 0 enableOnlineBucketRepair = 1 suspendHotRollByDeleteQuery = 0 enableDataIntegrityControl = 0 thawedPath = $SPLUNK_DB\audit\thaweddb tstatsHomePath = volume:_splunk_summaries\audit\datamodel_summary homePath = $SPLUNK_DB\audit\db rtRouterThreads = syncMeta = 1 maxTotalDataSizeMB = 5120 rtRouterQueueSize = [_internal] bucketRebuildMemoryHint = 0 syncMeta = 1 maxHotSpanSecs = 432000 compressRawdata = 1 coldPath = $SPLUNK_DB\_internaldb\colddb minHotIdleSecsBeforeForceRoll = 0 maxDataSize = 1000 enableOnlineBucketRepair = 1 suspendHotRollByDeleteQuery = 0 enableDataIntegrityControl = 0 thawedPath = $SPLUNK_DB\_internaldb\thaweddb tstatsHomePath = volume:_splunk_summaries\_internaldb\datamodel_summary homePath = $SPLUNK_DB\_internaldb\db rtRouterThreads = enableTsidxReduction = 0 maxTotalDataSizeMB = 25600 frozenTimePeriodInSecs = 188697600 rtRouterQueueSize = [_introspection] bucketRebuildMemoryHint = 0 syncMeta = 1 compressRawdata = 1 coldPath = $SPLUNK_DB\_introspection\colddb minHotIdleSecsBeforeForceRoll = 0 enableTsidxReduction = 0 enableOnlineBucketRepair = 1 suspendHotRollByDeleteQuery = 0 enableDataIntegrityControl = 0 thawedPath = $SPLUNK_DB\_introspection\thaweddb homePath = $SPLUNK_DB\_introspection\db rtRouterThreads = maxDataSize = 1024 maxTotalDataSizeMB = 5120 frozenTimePeriodInSecs = 1209600 rtRouterQueueSize = [_telemetry] bucketRebuildMemoryHint = 0 syncMeta = 1 compressRawdata = 1 coldPath = $SPLUNK_DB\_telemetry\colddb minHotIdleSecsBeforeForceRoll = 0 enableTsidxReduction = 0 enableOnlineBucketRepair = 1 suspendHotRollByDeleteQuery = 0 enableDataIntegrityControl = 0 thawedPath = $SPLUNK_DB\_telemetry\thaweddb homePath = $SPLUNK_DB\_telemetry\db rtRouterThreads = maxDataSize = 256 maxTotalDataSizeMB = 500 frozenTimePeriodInSecs = 63072000 rtRouterQueueSize = [_thefishbucket] bucketRebuildMemoryHint = 0 syncMeta = 1 compressRawdata = 1 coldPath = $SPLUNK_DB\fishbucket\colddb minHotIdleSecsBeforeForceRoll = 0 enableTsidxReduction = 0 enableOnlineBucketRepair = 1 suspendHotRollByDeleteQuery = 0 enableDataIntegrityControl = 0 thawedPath = $SPLUNK_DB\fishbucket\thaweddb tstatsHomePath = volume:_splunk_summaries\fishbucket\datamodel_summary homePath = $SPLUNK_DB\fishbucket\db rtRouterThreads = maxDataSize = 500 maxTotalDataSizeMB = 500 frozenTimePeriodInSecs = 188697600 rtRouterQueueSize = [history] bucketRebuildMemoryHint = 0 syncMeta = 1 compressRawdata = 1 coldPath = $SPLUNK_DB\historydb\colddb minHotIdleSecsBeforeForceRoll = 0 enableTsidxReduction = 0 enableOnlineBucketRepair = 1 suspendHotRollByDeleteQuery = 0 enableDataIntegrityControl = 0 thawedPath = $SPLUNK_DB\historydb\thaweddb tstatsHomePath = volume:_splunk_summaries\historydb\datamodel_summary homePath = $SPLUNK_DB\historydb\db rtRouterThreads = maxDataSize = 10 maxTotalDataSizeMB = 500 frozenTimePeriodInSecs = 604800 rtRouterQueueSize = [main] enableOnlineBucketRepair = 1 bucketRebuildMemoryHint = 0 syncMeta = 1 minHotIdleSecsBeforeForceRoll = 0 compressRawdata = 1 coldPath = $SPLUNK_DB\defaultdb\colddb maxHotBuckets = 10 maxDataSize = auto_high_volume maxConcurrentOptimizes = 6 suspendHotRollByDeleteQuery = 0 enableDataIntegrityControl = 0 thawedPath = $SPLUNK_DB\defaultdb\thaweddb tstatsHomePath = volume:_splunk_summaries\defaultdb\datamodel_summary homePath = $SPLUNK_DB\defaultdb\db rtRouterThreads = enableTsidxReduction = 0 maxHotIdleSecs = 86400 maxTotalDataSizeMB = 10240 rtRouterQueueSize = [splunklogger] disabled = true bucketRebuildMemoryHint = 0 compressRawdata = 1 coldPath = $SPLUNK_DB\splunklogger\colddb minHotIdleSecsBeforeForceRoll = 0 enableTsidxReduction = 0 enableOnlineBucketRepair = 1 suspendHotRollByDeleteQuery = 0 enableDataIntegrityControl = 0 thawedPath = $SPLUNK_DB\splunklogger\thaweddb homePath = $SPLUNK_DB\splunklogger\db rtRouterThreads = syncMeta = 1 maxTotalDataSizeMB = 500 rtRouterQueueSize = [summary] bucketRebuildMemoryHint = 0 compressRawdata = 1 coldPath = $SPLUNK_DB\summarydb\colddb minHotIdleSecsBeforeForceRoll = 0 enableTsidxReduction = 0 enableOnlineBucketRepair = 1 suspendHotRollByDeleteQuery = 0 enableDataIntegrityControl = 0 thawedPath = $SPLUNK_DB\summarydb\thaweddb tstatsHomePath = volume:_splunk_summaries\summarydb\datamodel_summary homePath = $SPLUNK_DB\summarydb\db rtRouterThreads = syncMeta = 1 maxTotalDataSizeMB = 500 rtRouterQueueSize = [volume:_splunk_summaries] path = $SPLUNK_DB

Modified indexes.conf:

[default] serviceSubtaskTimingPeriod = 30 serviceInactiveIndexesPeriod = 60 enableRealtimeSearch = true timePeriodInSecBeforeTsidxReduction = 604800 serviceMetaPeriod = 25 defaultDatabase = main rotatePeriodInSecs = 60 rtRouterThreads = 0 enableTsidxReduction = false maxHotIdleSecs = 0 bucketRebuildMemoryHint = auto suspendHotRollByDeleteQuery = false maxHotSpanSecs = 7776000 suppressBannerList = maxBucketSizeCacheEntries = 0 hotBucketTimeRefreshInterval = 10 maxHotBuckets = 3 processTrackerServiceInterval = 1 maxDataSize = auto maxRunningProcessGroups = 8 minRawFileSyncSecs = disable enableDataIntegrityControl = false minStreamGroupQueueSize = 2000 maxMetaEntries = 1000000 throttleCheckPeriod = 15 tstatsHomePath = volume:_splunk_summaries\$_index_name\datamodel_summary tsidxReductionCheckPeriodInSec = 600 maxBloomBackfillBucketAge = 30d datatype = event syncMeta = true partialServiceMetaPeriod = 0 frozenTimePeriodInSecs = 188697600 maxGlobalDataSizeMB = 0 quarantinePastSecs = 77760000 compressRawdata = true coldToFrozenScript = coldPath.maxDataSizeMB = 0 enableOnlineBucketRepair = true repFactor = 0 rtRouterQueueSize = 10000 maxTimeUnreplicatedWithAcks = 60 assureUTF8 = false maxTimeUnreplicatedNoAcks = 300 rawChunkSizeBytes = 131072 memPoolMB = auto homePath.maxDataSizeMB = 0 warmToColdScript = maxWarmDBCount = 300 minHotIdleSecsBeforeForceRoll = auto coldToFrozenDir = maxTotalDataSizeMB = 500000 maxConcurrentOptimizes = 6 maxRunningProcessGroupsLowPriority = 1 streamingTargetTsidxSyncPeriodMsec = 5000 journalCompression = gzip quarantineFutureSecs = 2592000 splitByIndexKeys = sync = 0 serviceOnlyAsNeeded = true [_audit] bucketRebuildMemoryHint = 0 compressRawdata = 1 coldPath = $SPLUNK_DB\audit\colddb minHotIdleSecsBeforeForceRoll = 0 enableTsidxReduction = 0 enableOnlineBucketRepair = 1 suspendHotRollByDeleteQuery = 0 enableDataIntegrityControl = 0 thawedPath = $SPLUNK_DB\audit\thaweddb tstatsHomePath = volume:_splunk_summaries\audit\datamodel_summary homePath = $SPLUNK_DB\audit\db rtRouterThreads = syncMeta = 1 maxTotalDataSizeMB = 5120 rtRouterQueueSize = [_internal] bucketRebuildMemoryHint = 0 syncMeta = 1 maxHotSpanSecs = 432000 compressRawdata = 1 coldPath = $SPLUNK_DB\_internaldb\colddb minHotIdleSecsBeforeForceRoll = 0 maxDataSize = 1000 enableOnlineBucketRepair = 1 suspendHotRollByDeleteQuery = 0 enableDataIntegrityControl = 0 thawedPath = $SPLUNK_DB\_internaldb\thaweddb tstatsHomePath = volume:_splunk_summaries\_internaldb\datamodel_summary homePath = $SPLUNK_DB\_internaldb\db rtRouterThreads = enableTsidxReduction = 0 maxTotalDataSizeMB = 51200 frozenTimePeriodInSecs = 188697600 rtRouterQueueSize = archiver.enableDataArchive = 0 metric.enableFloatingPointCompression = 1 selfStorageThreads = tsidxWritingLevel = [_introspection] bucketRebuildMemoryHint = 0 syncMeta = 1 compressRawdata = 1 coldPath = $SPLUNK_DB\_introspection\colddb minHotIdleSecsBeforeForceRoll = 0 enableTsidxReduction = 0 enableOnlineBucketRepair = 1 suspendHotRollByDeleteQuery = 0 enableDataIntegrityControl = 0 thawedPath = $SPLUNK_DB\_introspection\thaweddb homePath = $SPLUNK_DB\_introspection\db rtRouterThreads = maxDataSize = 1024 maxTotalDataSizeMB = 5120 frozenTimePeriodInSecs = 1209600 rtRouterQueueSize = [_telemetry] bucketRebuildMemoryHint = 0 syncMeta = 1 compressRawdata = 1 coldPath = $SPLUNK_DB\_telemetry\colddb minHotIdleSecsBeforeForceRoll = 0 enableTsidxReduction = 0 enableOnlineBucketRepair = 1 suspendHotRollByDeleteQuery = 0 enableDataIntegrityControl = 0 thawedPath = $SPLUNK_DB\_telemetry\thaweddb homePath = $SPLUNK_DB\_telemetry\db rtRouterThreads = maxDataSize = 256 maxTotalDataSizeMB = 500 frozenTimePeriodInSecs = 63072000 rtRouterQueueSize = [_thefishbucket] bucketRebuildMemoryHint = 0 syncMeta = 1 compressRawdata = 1 coldPath = $SPLUNK_DB\fishbucket\colddb minHotIdleSecsBeforeForceRoll = 0 enableTsidxReduction = 0 enableOnlineBucketRepair = 1 suspendHotRollByDeleteQuery = 0 enableDataIntegrityControl = 0 thawedPath = $SPLUNK_DB\fishbucket\thaweddb tstatsHomePath = volume:_splunk_summaries\fishbucket\datamodel_summary homePath = $SPLUNK_DB\fishbucket\db rtRouterThreads = maxDataSize = 500 maxTotalDataSizeMB = 500 frozenTimePeriodInSecs = 188697600 rtRouterQueueSize = [history] bucketRebuildMemoryHint = 0 syncMeta = 1 compressRawdata = 1 coldPath = $SPLUNK_DB\historydb\colddb minHotIdleSecsBeforeForceRoll = 0 enableTsidxReduction = 0 enableOnlineBucketRepair = 1 suspendHotRollByDeleteQuery = 0 enableDataIntegrityControl = 0 thawedPath = $SPLUNK_DB\historydb\thaweddb tstatsHomePath = volume:_splunk_summaries\historydb\datamodel_summary homePath = $SPLUNK_DB\historydb\db rtRouterThreads = maxDataSize = 10 maxTotalDataSizeMB = 500 frozenTimePeriodInSecs = 604800 rtRouterQueueSize = [main] enableOnlineBucketRepair = 1 bucketRebuildMemoryHint = 0 syncMeta = 1 minHotIdleSecsBeforeForceRoll = 0 compressRawdata = 1 coldPath = $SPLUNK_DB\defaultdb\colddb maxHotBuckets = 10 maxDataSize = auto_high_volume maxConcurrentOptimizes = 6 suspendHotRollByDeleteQuery = 0 enableDataIntegrityControl = 0 thawedPath = $SPLUNK_DB\defaultdb\thaweddb tstatsHomePath = volume:_splunk_summaries\defaultdb\datamodel_summary homePath = $SPLUNK_DB\defaultdb\db rtRouterThreads = enableTsidxReduction = 0 maxHotIdleSecs = 86400 maxTotalDataSizeMB = 10240 rtRouterQueueSize = [splunklogger] disabled = true bucketRebuildMemoryHint = 0 compressRawdata = 1 coldPath = $SPLUNK_DB\splunklogger\colddb minHotIdleSecsBeforeForceRoll = 0 enableTsidxReduction = 0 enableOnlineBucketRepair = 1 suspendHotRollByDeleteQuery = 0 enableDataIntegrityControl = 0 thawedPath = $SPLUNK_DB\splunklogger\thaweddb homePath = $SPLUNK_DB\splunklogger\db rtRouterThreads = syncMeta = 1 maxTotalDataSizeMB = 500 rtRouterQueueSize = [_syslog] bucketRebuildMemoryHint = 0 compressRawdata = 1 coldPath = $SPLUNK_DB\_syslog\colddb minHotIdleSecsBeforeForceRoll = 0 enableTsidxReduction = 0 enableOnlineBucketRepair = 1 suspendHotRollByDeleteQuery = 0 enableDataIntegrityControl = 0 thawedPath = $SPLUNK_DB\_syslog\thaweddb tstatsHomePath = volume:_splunk_summaries\_syslog\datamodel_summary homePath = $SPLUNK_DB\_syslog\db rtRouterThreads = syncMeta = 1 maxTotalDataSizeMB = 10240 frozenTimePeriodInSecs = 7776000 rtRouterQueueSize = [_metrics] bucketRebuildMemoryHint = 0 compressRawdata = 1 coldPath = $SPLUNK_DB\_metrics\colddb minHotIdleSecsBeforeForceRoll = 0 enableTsidxReduction = 0 enableOnlineBucketRepair = 1 suspendHotRollByDeleteQuery = 0 enableDataIntegrityControl = 0 thawedPath = $SPLUNK_DB\_metrics\thaweddb tstatsHomePath = volume:_splunk_summaries\_metrics\datamodel_summary homePath = $SPLUNK_DB\_metrics\db rtRouterThreads = syncMeta = 1 maxTotalDataSizeMB = 10240 frozenTimePeriodInSecs = 7776000 rtRouterQueueSize = [summary] bucketRebuildMemoryHint = 0 compressRawdata = 1 coldPath = $SPLUNK_DB\summarydb\colddb minHotIdleSecsBeforeForceRoll = 0 enableTsidxReduction = 0 enableOnlineBucketRepair = 1 suspendHotRollByDeleteQuery = 0 enableDataIntegrityControl = 0 thawedPath = $SPLUNK_DB\summarydb\thaweddb tstatsHomePath = volume:_splunk_summaries\summarydb\datamodel_summary homePath = $SPLUNK_DB\summarydb\db rtRouterThreads = syncMeta = 1 maxTotalDataSizeMB = 500 rtRouterQueueSize = [volume:_splunk_summaries] path = $SPLUNK_DB

Re: Indexes not saving past 2 weeks.

gcusello — Thu, 08 Aug 2024 06:22:15 GMT

Hi @JJE ,

one additional information: did you received logs until the 31st of July and logs stopped at the 1st of August?

if this is true, the issue is that you're receiving logs from your firewalls with an European date format (dd/mm/yyyy) and you didn't declared the date format, in this case Splunk tries to recognize timestamp and did it until the 31st of July using the standard america format (mm/dd/yyyy), so101/08/2024 is read as the 8th of January.

Force the time format in props.conf for that sourcetype:

TIME_FORMAT = %d/%m/%Y %H:%M:%S

If you didn't solved, could you share a sample of your logs and props.conf?

The indexes.conf isn't relevant for the time format.

Only for your information: indexes in Splunk are only a recipient for the logs, but there isn't any information about logs, infact you can store different logs in the same index: an index isn't a database table where you have to define every data information .

Ciao.

Giuseppe

Re: Indexes not saving past 2 weeks.

PickleRick — Thu, 08 Aug 2024 06:23:34 GMT

1. You should not use underscores in names for your indexes. Underscores denote Splunk's internal indexes. As _metrics is - that's Splunk's internal metrics index.

2. Retention period is one thing but if you exceed index size limits oldest bucket will get rolled to frozen (by default it will be deleted). As typically firewall logs (assuming you're logging network sessions) are very "noisy", that's what I'd suspect

If you have an all-in-one setup the easiest way to check the index size would be to go to Settings->Indexes

Re: Indexes not saving past 2 weeks.

JJE — Thu, 08 Aug 2024 15:07:30 GMT

Okay I'll see if removing the _ helps. Thank you.

Re: Indexes not saving past 2 weeks.

isoutamo — Thu, 08 Aug 2024 16:05:24 GMT

Usually those underscore indexes are restricted only for admin user access. As @PickleRick said those are reserved for Splunk’s own usage, not for regular data. If you need to use those as a regular user, you must separately grant access to those.

Re: Indexes not saving past 2 weeks.

JJE — Thu, 08 Aug 2024 16:52:03 GMT

Here's what as in my Props.conf. I cannot share logs.

[SUMS]
EVENT_BREAKER_ENABLE=true
EVENT_BREAKER=(At\s[0-2][0-9]:[0-6][0-9]:[0-6][0-9]\s-\d{4}\s-)

Re: Indexes not saving past 2 weeks.

gcusello — Fri, 09 Aug 2024 05:52:27 GMT

Hi @JJE ,

I'm not interested on your logs, only to the timestamp format!

Anyway, check if the timestamp format has the format I described and in this case use the TIME_FORMAT option in props.conf.

Ciao.

Giuseppe

Re: Indexes not saving past 2 weeks.

JJE — Tue, 13 Aug 2024 14:30:13 GMT

I apologize but could you break this process down barney style for me?

Re: Indexes not saving past 2 weeks.

JJE — Thu, 15 Aug 2024 12:47:40 GMT

Just to clarify. Every device on this network is being logged by splunk, but these two firewalls are the only two that have this problem. All the other devices can pull logs normally, so I don't believe the time format is the issue.

Re: Indexes not saving past 2 weeks.

PickleRick — Thu, 15 Aug 2024 14:37:13 GMT

"Every device on the network" doesn't have to necessarily be identically configured. That's from experience.

Also - we don't know your data, we don't know how your data is onboarded.

Check your events as they come with something like

index=whatever_index_you're_using host=your_router | head 10

And run this over "all time (real-time)" - that's practically the only use case I've ever seen where real-time search is actually useful.

See the timestamp in the event itself, see the timestamp Splunk uses (either parsed out of the event or not recognized and assumed to be something).

That's to check if your data is OK.

BTW, if all your routers' logs are getting indexed in the same index there is no way (unless you have a very botched distributed indexing setup which I assume you haven't) that data from the same index for those hosts is rolled and for other hosts is retained.