Hi, I noticed that, right after a log rotation, the data is not being indexed anymore. Data is still going through /var/log/myapp.log and /var/log/messages (rsyslog UDP), so it all arrives on the machine (at 100%), but it is not being indexed. On Splunk, I am monitoring logs arriving with real-time searches. Before log-rotate, everything is fine and logs are arriving on a regular basis. After logrotate, I do not get anything anymore. Here is my inputs.conf: [monitor:///var/log/myapp.log] sourcetype = myappsourcetype crcSalt = <SOURCE> crcSalt = 2048 disabled = 0 My log rotate conf: "/var/log/myapp.log" { monthly size 100M rotate 30 compress postrotate /bin/kill -HUP `cat /var/run/syslogd.pid 2>/dev/null` &> /dev/null || true endscript } My sourcetype shouldn't be the problem as it only contains some side fields extractions. Thank you very much in advance ... View more

Hello, When restoring buckets to thawedb, Splunk Web does not want to restart. Here is the procedure: A Warm Bucket containing the main index is stored on a backup server I download the bucket into /tmp/ I chown all the files and directories of the bucket with splunk:splunk Splunk fsck on the bucket directory Move the bucket to thawedb create meta.dirty restart splunk Here are the main lines of the crash log file: Received fatal signal 6 (Aborted). Cause: Signal sent by PID 31817 running under UID 497. Crashing thread: SplunkdSpecificInitThread Backtrace: [0x00007F2E41B385F7] gsignal + 55 (/lib64/libc.so.6) [0x00007F2E41B39CE8] abort + 328 (/lib64/libc.so.6) [0x00007F2E41B31566] ? (/lib64/libc.so.6) [0x00007F2E41B31612] ? (/lib64/libc.so.6) [0x0000000000BBE5E7] _ZN14IndexerService35disableIndexesAndReinitGlobalConfigERKN9__gnu_cxx17__normal_iteratorIPK3StrSt6vectorIS2_SaIS2_EEEESA_ + 503 (splunkd) [0x0000000000BBF005] _ZN14IndexerService18initPerIndexConfigEP9StrVectorb + 309 (splunkd) [0x0000000000BC042C] _ZN14IndexerService12reloadConfigERK14IndexConfigRef + 428 (splunkd) [0x0000000001011BF3] _ZN9EventLoop20internal_runInThreadEP13InThreadActorb + 291 (splunkd) [0x0000000000BBD9BB] _ZN14IndexerService16loadLatestConfigEP14IndexConfigRef + 411 (splunkd) [0x0000000000BBDB65] _ZN14IndexerService16loadLatestConfigEv + 21 (splunkd) [0x0000000000BBDEA2] _ZN14IndexerServiceC2Ev + 786 (splunkd) [0x0000000000BBE231] _ZN14IndexerService14_new_singletonEv + 65 (splunkd) [0x00000000009AF647] _ZN25SplunkdSpecificInitThread4mainEv + 135 (splunkd) [0x00000000010A423E] _ZN6Thread8callMainEPv + 62 (splunkd) [0x00007F2E41ECCDC5] ? (/lib64/libpthread.so.0) [0x00007F2E41BF9C9D] clone + 109 (/lib64/libc.so.6) Linux / splunk.myhost.local / 4.4.10-22.54.amzn1.x86_64 / #1 SMP Tue May 17 22:45:04 UTC 2016 / x86_64 Last few lines of stderr (may contain info on assertion failure, but also could be old): 2016-05-24 14:47:05.530 -0700 splunkd started (build cae2458f4aef) splunkd: /home/build/build-src/ember/src/pipeline/indexer/IndexerService.cpp:928: void IndexerService::disableIndexesAndReinitGlobalConfig(const const_iterator&, const const_iterator&): Assertion `0 && "Cannot disable the default index."' failed. 2016-05-24 14:52:51.066 -0700 splunkd started (build cae2458f4aef) 2016-05-24 14:54:51.823 -0700 Interrupt signal received 2016-05-24 14:55:12.773 -0700 splunkd started (build cae2458f4aef) splunkd: /home/build/build-src/ember/src/pipeline/indexer/IndexerService.cpp:928: void IndexerService::disableIndexesAndReinitGlobalConfig(const const_iterator&, const const_iterator&): Assertion `0 && "Cannot disable the default index."' failed. Thread: "SplunkdSpecificInitThread", did_join=0, ready_to_run=Y, main_thread=N What am I missing here? Thank you very much in advance ... View more

Hello, I am currently designing a backup/restore procedure and I am wondering: Does Splunk automatically rotate all the buckets being restored? I am wondering if I should manually rotate the buckets when I do my incremental backup, or if I should save them all and let Splunk rotate them when I restore them? If Splunk is able to load, in thawdb, let's say, 600 warm buckets for a limit of 300 and rotate to cold the corresponding ones and then move the cold ones to frozen, then all I will have to do, for backup, is: - Upload recently warmed up ones - Delete backup of buckets older than frozen time from long-term storage If not, I will keep a directory for each warm and cold buckets and do the upload/rotation/deletion at every incremental backup round. On the restore, I will restore them by age from oldest to most recent, which is much more painful. Thanks a lot ... View more