Getting Data In

Data going directly to frozen

norbertt911
Path Finder

Hello,

I have a new index - it's a monster - eating up my disk space. Until I move it to the physical server I need to fix it.

Well, I limited maxTotalDataSizeMB, seem working but the cold storage skipped landed in frozen directly, so I cannot search it.

The hot/warm storage is "local" on VM, the cold, frozen, thawed is an S3.

The optimal idea is 7 days in hot/warm (if over maxTotalDataSizeMB then faster) then go cold for 90 days (no size limit) then thawed for 1 year (no size limit).

here is my current setting

archiver.enableDataArchive = 0
/opt/splunk/etc/system/default/indexes.conf archiver.maxDataArchiveRetentionPeriod = 0
/opt/splunk/etc/system/default/indexes.conf assureUTF8 = false
bucketRebuildMemoryHint = 0
coldPath = /mnt/archive_s3/SPLUNK_DB/indexname/colddb
/opt/splunk/etc/system/default/indexes.conf coldPath.maxDataSizeMB = 0
coldToFrozenDir = /mnt/archive_s3/SPLUNK_DB/indexname/Frozenarchive
/opt/splunk/etc/system/default/indexes.conf coldToFrozenScript =
compressRawdata = 1
/opt/splunk/etc/system/default/indexes.conf datatype = event
/opt/splunk/etc/system/default/indexes.conf defaultDatabase = main
enableDataIntegrityControl = 0
enableOnlineBucketRepair = 1
/opt/splunk/etc/system/default/indexes.conf enableRealtimeSearch = true
enableTsidxReduction = 0
frozenTimePeriodInSecs = 3024000
homePath = $SPLUNK_DB/indexname/db
/opt/splunk/etc/system/default/indexes.conf homePath.maxDataSizeMB = 0
/opt/splunk/etc/system/default/indexes.conf hotBucketTimeRefreshInterval = 10
/opt/splunk/etc/system/default/indexes.conf indexThreads = auto
/opt/splunk/etc/system/default/indexes.conf journalCompression = gzip
/opt/splunk/etc/system/default/indexes.conf maxBloomBackfillBucketAge = 30d
/opt/splunk/etc/system/default/indexes.conf maxBucketSizeCacheEntries = 0
maxConcurrentOptimizes = 6
maxDataSize = auto_high_volume
maxGlobalDataSizeMB = 0
maxHotBuckets = 10
maxHotIdleSecs = 86400
/opt/splunk/etc/system/default/indexes.conf maxHotSpanSecs = 7776000
maxMemMB = 20
/opt/splunk/etc/system/default/indexes.conf maxMetaEntries = 1000000
/opt/splunk/etc/system/default/indexes.conf maxRunningProcessGroups = 8
/opt/splunk/etc/system/default/indexes.conf maxRunningProcessGroupsLowPriority = 1
/opt/splunk/etc/system/default/indexes.conf maxTimeUnreplicatedNoAcks = 300
/opt/splunk/etc/system/default/indexes.conf maxTimeUnreplicatedWithAcks = 60
maxTotalDataSizeMB = 76800
maxWarmDBCount = 200
/opt/splunk/etc/system/default/indexes.conf memPoolMB = auto
minHotIdleSecsBeforeForceRoll = 0
/opt/splunk/etc/system/default/indexes.conf minRawFileSyncSecs = disable
/opt/splunk/etc/system/default/indexes.conf minStreamGroupQueueSize = 2000
/opt/splunk/etc/system/default/indexes.conf partialServiceMetaPeriod = 0
/opt/splunk/etc/system/default/indexes.conf processTrackerServiceInterval = 1
/opt/splunk/etc/system/default/indexes.conf quarantineFutureSecs = 2592000
/opt/splunk/etc/system/default/indexes.conf quarantinePastSecs = 77760000
/opt/splunk/etc/system/default/indexes.conf rawChunkSizeBytes = 131072
/opt/splunk/etc/system/default/indexes.conf repFactor = 0
rotatePeriodInSecs = 60
rtRouterQueueSize =
rtRouterThreads =
selfStorageThreads =
/opt/splunk/etc/system/default/indexes.conf serviceInactiveIndexesPeriod = 60
/opt/splunk/etc/system/default/indexes.conf serviceMetaPeriod = 25
/opt/splunk/etc/system/default/indexes.conf serviceOnlyAsNeeded = true
/opt/splunk/etc/system/default/indexes.conf serviceSubtaskTimingPeriod = 30
/opt/splunk/etc/system/default/indexes.conf splitByIndexKeys =
/opt/splunk/etc/system/default/indexes.conf streamingTargetTsidxSyncPeriodMsec = 5000
/opt/splunk/etc/system/default/indexes.conf suppressBannerList =
suspendHotRollByDeleteQuery = 0
/opt/splunk/etc/system/default/indexes.conf sync = 0
syncMeta = 1
thawedPath = /mnt/archive_s3/SPLUNK_DB/indexname/thaweddb
/opt/splunk/etc/system/default/indexes.conf throttleCheckPeriod = 15
/opt/splunk/etc/system/default/indexes.conf timePeriodInSecBeforeTsidxReduction = 604800
/opt/splunk/etc/system/default/indexes.conf tsidxReductionCheckPeriodInSec = 600
tsidxWritingLevel =
tstatsHomePath = volume:_splunk_summaries/$_index_name/datamodel_summary
/opt/splunk/etc/system/default/indexes.conf warmToColdScript =

I assume this is the issue coldPath.maxDataSizeMB = 0 why skip cold, but not sure.

I appreciated if somebody could fix my settings.

 

Labels (1)
0 Karma
Get Updates on the Splunk Community!

Thanks for the Memories! Splunk University, .conf24, and Community Connections

Thank you to everyone in the Splunk Community who joined us for .conf24 – starting with Splunk University and ...

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...