Thanks for responding! Yep, I thought that to be the case. I appreciate the confirmation -we'll go about this in another way on our end. ... View more

Hey All, Thanks for taking time out to reply back with feedback on this. I appreciate it!. 🙂 Knowing suggesting a possible answer often depends on more specific details, here's the scoop on our setup: Approximate daily data ingest rate: 7 GB Data retention requirements: "online", searchable data - 90 days, "offline", non-searchable frozen data - 1 year CPUs: 8 cores, utilization typically stays below 50% (when watching top for a bit, it hovered around 20-25%) RAM: 16 GB -Disks: • Storage is a RAID5 grouping of seven physical 2 TB, 7200 RPM, SAS near-line disks • Available storage after RAID is 10.913 TB • This storage is partitioned into 4 logical volumes, each of size: 2,793.774 GB • We see these four volumes as /data01 through /data04 • There is no difference in size, speed, or capability between these four volumes • All four volumes continuously compete for the attention of the same seven physical disks We only need backups in order to meet our data retention needs or to reconstitute the indexer in the event of an outage. We could move backups elsewhere if necessary. Should we continue to keep the hot & warm buckets on /data01 and place the cold bucket on /data02, frozen on /data03? Knowing that our disks aren't particularly fast, do we need to consider faster disks for our hot and warm buckets? Thanks! ... View more

Thank you guys! This information you have provided has been extremely helpful to me. ... View more

So if the storage needs for my main index are 456 GB, would I also set the maxTotalDataSizeMB to 456000? or so you typically set the maxTotalDataSizeMB to a value that is smaller that your total index space needs? ... View more

Makes sense! I think it must have been even worse before, because we had the index restricted to 140 GB, but the retention period was the default, which I believe is ~6 years. So we had storage that was sized to retain data for 112 days, but wouldn't delete until the data was 6 years old. I'm surprised that we didn't have this space problem earlier. ... View more

That is true. Just checked, and it looks like there are 2 hot buckets, 71 warm buckers in db folder ... View more

Sure! Here it is: [default] minRawFileSyncSecs = disable throttleCheckPeriod = 15 rotatePeriodInSecs = 60 compressRawdata = true quarantinePastSecs = 77760000 quarantineFutureSecs = 2592000 maxTotalDataSizeMB = 140000 maxHotIdleSecs = 0 maxMetaEntries = 1000000 serviceMetaPeriod = 25 syncMeta = true assureUTF8 = false frozenTimePeriodInSecs = 31556952 blockSignatureDatabase = _blocksignature maxWarmDBCount = 300 maxConcurrentOptimizes = 3 coldToFrozenDir = blockSignSize = 0 maxHotBuckets = 3 enableRealtimeSearch = true maxHotSpanSecs = 7776000 coldToFrozenScript = memPoolMB = auto partialServiceMetaPeriod = 0 suppressBannerList = rawChunkSizeBytes = 131072 sync = 0 maxRunningProcessGroups = 20 defaultDatabase = main maxDataSize = auto ... View more

Very strangely, my $splunk_home/etc/system/default/indexes.conf. is missing all the tstatsHomePath entries. But I definitely know not to edit that file (big no-no). I have no idea how those entries are missing. Just this once, would it be permissible to add the tstatsHomePath entries to the default indexes.conf file, or would that make my situation even worse? ... View more

Update: The errors I was seeing above were definitely due to those slaves being blocked over port 8089. I followed your suggestion and ran a wget on my slaves. I then worked with our firewall administrator to allow traffic from the slaves back to the license master over port 8089. It now works. I have a setup now with a central license master running on my search head and the other instances of Splunk Enterprise pointing back to the search head license master as license slaves. I have a feeling that setting up my licensing schema in this recommended way will likely resolve that "duplicate license" issue I was previously seeing. I will keep an eye on it. Many thanks for your help! ... View more

Thanks for the advice! I did attempt to set up one of my search heads as the license master and point the rest of my indexers to it as license slaves. Unfortunately, when I try to point the rest of my indexers that server as slaves, they each display one of the two following error messages: Bad Request — In handler 'localslave': editTracker failed, reason='Unable to connect to license master: https://:8089 Connect to=https://:8089 timed out; exceeded 30sec' -or- Splunkd daemon is not responding: ("Error connecting to /services/licenser/localslave/license: ('The read operation timed out',)",) Does this potentially point to network connectivity issues from slave to master? Permissions issues? ... View more