I found the only one system index ( summary is working, others are disabled )     ... View more

/opt/splunk/etc/system/local/indexes.conf (I tried to rename the indexes.conf to o_indexes.conf and restart the splunk ) In addition, I try to upgrade to latest version of splunk and still no luck. <>_<> [_audit] archiver.enableDataArchive = 0 bucketRebuildMemoryHint = 0 compressRawdata = 1 enableDataIntegrityControl = 0 enableOnlineBucketRepair = 1 enableTsidxReduction = 0 metric.enableFloatingPointCompression = 1 minHotIdleSecsBeforeForceRoll = 0 rtRouterQueueSize = rtRouterThreads = selfStorageThreads = suspendHotRollByDeleteQuery = 0 syncMeta = 1 tsidxWritingLevel =     /opt/splunk/etc/system/default/indexes.conf contents: ( this file last modified havent been modified since 2 years ago)   1 # Version 8.0.0 2 # DO NOT EDIT THIS FILE! 3 # Changes to default files will be lost on update and are difficult to 4 # manage and support. 5 # 6 # Please make any changes to system defaults by overriding them in 7 # apps or $SPLUNK_HOME/etc/system/local 8 # (See "Configuration file precedence" in the web documentation). 9 # 10 # To override a specific setting, copy the name of the stanza and 11 # setting to the file where you wish to override it. 12 # 13 # This file configures Splunk's indexes and their properties. 14 # 15 16 ################################################################################ 17 # "global" params (not specific to individual indexes) 18 ################################################################################ 19 sync = 0 20 indexThreads = auto 21 memPoolMB = auto 22 defaultDatabase = main 23 enableRealtimeSearch = true 24 suppressBannerList = 25 maxRunningProcessGroups = 8 26 maxRunningProcessGroupsLowPriority = 1 27 bucketRebuildMemoryHint = auto 28 serviceOnlyAsNeeded = true 29 serviceSubtaskTimingPeriod = 30 30 serviceInactiveIndexesPeriod = 60 31 maxBucketSizeCacheEntries = 0 32 processTrackerServiceInterval = 1 33 hotBucketTimeRefreshInterval = 10 34 rtRouterThreads = 0 35 rtRouterQueueSize = 10000 36 selfStorageThreads = 2 37 fileSystemExecutorWorkers = 5 38 39 ################################################################################ 40 # index specific defaults 41 ################################################################################ 42 maxDataSize = auto 43 maxWarmDBCount = 300 44 frozenTimePeriodInSecs = 188697600 45 rotatePeriodInSecs = 60 46 coldToFrozenScript = 47 coldToFrozenDir = 48 compressRawdata = true 49 maxTotalDataSizeMB = 500000 50 maxGlobalRawDataSizeMB = 0 51 maxGlobalDataSizeMB = 0 52 maxMemMB = 5 53 maxConcurrentOptimizes = 6 54 maxHotSpanSecs = 7776000 55 maxHotIdleSecs = 0 56 maxHotBuckets = 3 57 minHotIdleSecsBeforeForceRoll = auto 58 quarantinePastSecs = 77760000 59 quarantineFutureSecs = 2592000 60 rawChunkSizeBytes = 131072 61 minRawFileSyncSecs = disable 62 assureUTF8 = false 63 serviceMetaPeriod = 25 64 partialServiceMetaPeriod = 0 65 throttleCheckPeriod = 15 66 syncMeta = true 67 maxMetaEntries = 1000000 68 maxBloomBackfillBucketAge = 30d 69 enableOnlineBucketRepair = true 70 enableDataIntegrityControl = false 71 maxTimeUnreplicatedWithAcks = 60 72 maxTimeUnreplicatedNoAcks = 300 73 minStreamGroupQueueSize = 2000 74 warmToColdScript= 75 tstatsHomePath = volume:_splunk_summaries/$_index_name/datamodel_summary 76 homePath.maxDataSizeMB = 0 77 coldPath.maxDataSizeMB = 0 78 streamingTargetTsidxSyncPeriodMsec = 5000 79 journalCompression = gzip 80 enableTsidxReduction = false 81 suspendHotRollByDeleteQuery = false 82 tsidxReductionCheckPeriodInSec = 600 83 timePeriodInSecBeforeTsidxReduction = 604800 84 datatype = event 85 splitByIndexKeys = 86 tsidxWritingLevel = 1 87 archiver.enableDataArchive = false 88 archiver.maxDataArchiveRetentionPeriod = 0 89 tsidxTargetSizeMB = 1500 90 metric.tsidxTargetSizeMB = 1500 91 metric.enableFloatingPointCompression = true 92 metric.compressionBlockSize = 1024 93 94 # 95 # By default none of the indexes are replicated. 96 # 97 repFactor = 0 98 99 [volume:_splunk_summaries] 100 path = $SPLUNK_DB 101 102 [provider-family:hadoop] 103 vix.mode = report 104 vix.command = $SPLUNK_HOME/bin/jars/sudobash 105 vix.command.arg.1 = $HADOOP_HOME/bin/hadoop 106 vix.command.arg.2 = jar 107 vix.command.arg.3 = $SPLUNK_HOME/bin/jars/SplunkMR-h1.jar 108 vix.command.arg.4 = com.splunk.mr.SplunkMR 109 vix.env.MAPREDUCE_USER = 110 vix.env.HADOOP_HEAPSIZE = 512 111 vix.env.HADOOP_CLIENT_OPTS = -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr 112 vix.env.HUNK_THIRDPARTY_JARS = $SPLUNK_HOME/bin/jars/thirdparty/common/avro-1.7.7.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/a 113 vix.mapred.job.reuse.jvm.num.tasks = 100 114 vix.mapred.child.java.opts = -server -Xmx512m -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr 115 vix.mapred.reduce.tasks = 0 116 vix.mapred.job.map.memory.mb = 2048 117 vix.mapred.job.reduce.memory.mb = 512 118 vix.mapred.job.queue.name = default 119 vix.mapreduce.job.jvm.numtasks = 100 120 vix.mapreduce.map.java.opts = -server -Xmx512m -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr 121 vix.mapreduce.reduce.java.opts = -server -Xmx512m -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr 122 vix.mapreduce.job.reduces = 0 123 vix.mapreduce.map.memory.mb = 2048 124 vix.mapreduce.reduce.memory.mb = 512 125 vix.mapreduce.job.queuename = default 126 vix.splunk.search.column.filter = 1 127 vix.splunk.search.mixedmode = 1 128 vix.splunk.search.debug = 0 129 vix.splunk.search.mr.maxsplits = 10000 130 vix.splunk.search.mr.minsplits = 100 131 vix.splunk.search.mr.splits.multiplier = 10 132 vix.splunk.search.mr.poll = 2000 133 vix.splunk.search.recordreader = SplunkJournalRecordReader,ValueAvroRecordReader,SimpleCSVRecordReader,SequenceFileRecordReader 134 vix.splunk.search.recordreader.avro.regex = \.avro$ 135 vix.splunk.search.recordreader.csv.regex = \.([tc]sv)(?:\.(?:gz|bz2|snappy))?$ 136 vix.splunk.search.recordreader.sequence.regex = \.seq$ 137 vix.splunk.home.datanode = /tmp/splunk/$SPLUNK_SERVER_NAME/ 138 vix.splunk.heartbeat = 1 139 vix.splunk.heartbeat.threshold = 60 140 vix.splunk.heartbeat.interval = 1000 141 vix.splunk.setup.onsearch = 1 142 vix.splunk.setup.package = current 143 144 ################################################################################ 145 # index definitions 146 ################################################################################ 147 148 [main] 149 homePath = $SPLUNK_DB/defaultdb/db 150 coldPath = $SPLUNK_DB/defaultdb/colddb 151 thawedPath = $SPLUNK_DB/defaultdb/thaweddb 152 tstatsHomePath = volume:_splunk_summaries/defaultdb/datamodel_summary 153 maxMemMB = 20 154 maxConcurrentOptimizes = 6 155 maxHotIdleSecs = 86400 156 maxHotBuckets = 10 157 maxDataSize = auto_high_volume 158 159 [history] 160 homePath = $SPLUNK_DB/historydb/db 161 coldPath = $SPLUNK_DB/historydb/colddb 162 thawedPath = $SPLUNK_DB/historydb/thaweddb 163 tstatsHomePath = volume:_splunk_summaries/historydb/datamodel_summary 164 maxDataSize = 10 165 frozenTimePeriodInSecs = 604800 166 167 [summary] 168 homePath = $SPLUNK_DB/summarydb/db 169 coldPath = $SPLUNK_DB/summarydb/colddb 170 thawedPath = $SPLUNK_DB/summarydb/thaweddb 171 tstatsHomePath = volume:_splunk_summaries/summarydb/datamodel_summary 172 173 [_internal] 174 homePath = $SPLUNK_DB/_internaldb/db 175 coldPath = $SPLUNK_DB/_internaldb/colddb 176 thawedPath = $SPLUNK_DB/_internaldb/thaweddb 177 tstatsHomePath = volume:_splunk_summaries/_internaldb/datamodel_summary 178 maxDataSize = 1000 179 maxHotSpanSecs = 432000 180 frozenTimePeriodInSecs = 2592000 181 182 [_audit] 183 homePath = $SPLUNK_DB/audit/db 184 coldPath = $SPLUNK_DB/audit/colddb 185 thawedPath = $SPLUNK_DB/audit/thaweddb 186 tstatsHomePath = volume:_splunk_summaries/audit/datamodel_summary 187 188 [_thefishbucket] 189 homePath = $SPLUNK_DB/fishbucket/db 190 coldPath = $SPLUNK_DB/fishbucket/colddb 191 thawedPath = $SPLUNK_DB/fishbucket/thaweddb 192 tstatsHomePath = volume:_splunk_summaries/fishbucket/datamodel_summary 193 maxDataSize = 500 194 frozenTimePeriodInSecs = 2419200 195 196 # this index has been removed in the 4.1 series, but this stanza must be 197 # preserved to avoid displaying errors for users that have tweaked the index's 198 # size/etc parameters in local/indexes.conf. 199 # 200 [splunklogger] 201 homePath = $SPLUNK_DB/splunklogger/db 202 coldPath = $SPLUNK_DB/splunklogger/colddb 203 thawedPath = $SPLUNK_DB/splunklogger/thaweddb 204 disabled = true 205 206 [_introspection] 207 homePath = $SPLUNK_DB/_introspection/db 208 coldPath = $SPLUNK_DB/_introspection/colddb 209 thawedPath = $SPLUNK_DB/_introspection/thaweddb 210 maxDataSize = 1024 211 frozenTimePeriodInSecs = 1209600 212 213 [_telemetry] 214 homePath = $SPLUNK_DB/_telemetry/db 215 coldPath = $SPLUNK_DB/_telemetry/colddb 216 thawedPath = $SPLUNK_DB/_telemetry/thaweddb 217 maxDataSize = 256 218 frozenTimePeriodInSecs = 63072000 219 220 [_metrics] 221 homePath = $SPLUNK_DB/_metrics/db 222 coldPath = $SPLUNK_DB/_metrics/colddb 223 thawedPath = $SPLUNK_DB/_metrics/thaweddb 224 datatype = metric 225 #14 day retention 226 frozenTimePeriodInSecs = 1209600 227 splitByIndexKeys = metric_name ... View more

Thanks for the reply.   For disk usage: /dev/sda1 used around 11% only   and the screen capture for the index like this:   I tried to read the error log message but there are tone of lines.... and failed to find useful error log.... ... View more