I encountered problem with ITSI each time I tries to upgrade or install a new deployment. - upgrading ITSI on version 2.6 on a search-head cluster, to 3.1 - installing a new 3.0.0 or 3.1.2 on a search-head cluster. Each time I push the ITSI bits from the deployer and wait for the sh rolling restart. Usually when a problem occurs, the symptoms are : ITSI panels not loading, permissions issues, and nothing in my configure > services and teams even for my admin user. Looking in the logs, I see in index=_internal source=*itsi_migration.log* that one of the shpeer tried to start the install/migration but failed because of permissions of "teams" missing. I checked, there are no teams in my ITSI (in the manager or in the kvstore collection) I also see errors on some peers about sh captain not ready. example : 2018-06-04 15:29:22,979 INFO [itsi.migration] [itsi_migration] [run_migration] [23748] Enable UI Exception: Failed to import Team settings. ITSI will not work properly until the Team settings are imported. See [http://docs.splunk.com/Documentation/ITSI/3.0.1/Configure/Installationandconfigurationconsiderationsandissues#Run_script_to_set_the_default_team_to_Global this documentation page] for instructions on how to resolve this issue. raise Exception(error_msg) File "S:\splunk\etc\apps\SA-ITOA\lib\itsi\upgrade\itsi_migration.py", line 3269, in run_migration Traceback (most recent call last): 2018-06-04 15:29:22,976 ERROR [itsi.migration] [itsi_migration] [run_migration] [23748] Migration failed from version:None, to version:3.1.2 ... View more

I have ITSI 3.0, and on a regular basis it is reporting a kvstore connection test. The problem is that the check triggers a false positive error message in splunkd.log internal logs In ITSI 2.* the message was like 05-09-2017 06:04:18.605 -0400 ERROR HttpListener - Exception while processing request from 127.0.0.1 for /servicesNS/nobody/SA-ITOA/storage/collections/data/dummy_collection_nvfjdnvjkfdnvjkfnvjkfnvernvjfnvjkfsdnvuenvkjfnvjka?output_mode=json: Could not find object id=dummy_collection_nvfjdnvjkfdnvjkfnvjkfnvernvjfnvjkfsdnvuenvkjfnvjka In ITSI 3.0, it now looks like 01-23-2018 13:00:01.622 -0800 ERROR HttpListener - Exception while processing request from 127.0.0.1 for /servicesNS/nobody/SA-ITOA/storage/collections/data/this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore?output_mode=json: Could not find object id=this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore Can I find a way to drop those event? I do not want to index them. ... View more

I have several similar alerts and I would like to regroup them. But each alerts has to send the email to particular person, how can I do that, and add some logic. ... View more

I have Linux servers with Splunk, and the process monit to check my processed. But sometimes I see an issue where monit restarts Splunk unexpectedly. ... View more

Too many search jobs found in the dispatch directory (found=4596, warning level=4000). This could negatively impact Splunk's performance, consider removing some of the old search jobs. We see this error often on my search head. I tried to clean my jobs and empty the dispatch, but it came back a few hours after. When looking at the artifacts, they are mostly quite recent (last 24h) and are real-time scheduled searches linked to alerts. ... View more

I want to create the sourcetype AAA, that is not listed on the sourcetype manager. But when I go to settings > sourcetypes and add it, when I save, Splunk says "sourcetype already exists" Why ? ... View more

I am on an instance where I have no access to the license manager page, or where I never log in, or when I am not an admin (Splunk cloud or sandbox, in my case). How can I set up a scheduled email alert to tell me when I exceed the license usage capacity? ... View more

I had a SplunkStorm project, and I was sending data directly with 5 different inputs. Upload small file on the web UI Splunk Universal forwarder over SSL TCP port UDP port API inputs with special api key What are the options for SplunkCloud ? ... View more

I installed a splunk forwarder on windows, and setup monitoring of remote wmi. Configured splunk to run on a domain user, with permissions. wmi.conf [WMI:getmylogs] disabled = 0 event_log_file = Security index = default interval = 5 server = secure.hiddencastle.kp But cannot retrieve anything. I do not see any wplunk-wmi.exe process and I double check that I can access the remote logs with the user using wbemtest.exe wbemtest ... View more

I Installed an Amazon Ubuntu using the preconfigured splunk AMI. Splunk is installed on /opt/splunk , with the indexes and the dispatch folder The problem is that my root "/" partition is very small (2GB), and the indexes are filling it. in particular /opt/splunk/var/lib/splunk with the indexes. How to relocate the indexes to a larger partition that I mounted (EBS in my case) ? ... View more

Trying to start a splunkforwarder on SunOs. splunkforwarder-6.2.0-237341-SunOS10-sparc.tar.Z It fails to start, and we get those errors : ld.so.1: splunkd: fatal: libc.so.1: version `SUNW_1.22.6' not found (required by file /opt/splunkforwarder/lib/libarchive.so.13) ld.so.1: splunkd: fatal: libc.so.1: open failed: No such file or directory Killed ld.so.1: splunkd: fatal: libc.so.1: version `SUNW_1.22.6' not found (required by file /opt/splunkforwarder/lib/libarchive.so.13) ld.so.1: splunkd: fatal: libc.so.1: open failed: No such file or directory Killed ... View more

I have a centralized server with all my logs per instance /var/log/database/hostA/report.log /var/log/database/hostA/report.log.1 /var/log/database/hostA/report.log.2 /var/log/database/hostB/report.log /var/log/database/hostB/report.log.1 /var/log/database/hostB/report.log.2 /var/log/database/hostC/report.log /var/log/database/hostC/report.log.1 /var/log/database/hostC/report.log.2 /var/log/database/hostD/report.log /var/log/database/hostD/report.log.1 /var/log/database/hostD/report.log.2 etc ... and a monitoring on each [monitor://var/log/database/hostA/] sourcetype=report host=hostA Except that the splunk monitor consider that all my reports.log are duplicates.B 10-06-2014 18:06:41.767 -0700 ERROR TailingProcessor - File will not be read, seekptr checksum did not match (file=/var/log/database/hostA/report.log). Last time we saw this initcrc, filename was different. You may wish to use a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info. If I want to use crcSalt= , it will cause all my rotated versions to be indexed. Any better options ? ... View more

They are many features using objects named "summary", this is confusing, please clarify. what are the differences between all those paths ? $SPLUNK_HOME/var/lib/splunk/summary/db $SPLUNK_HOME/var/lib/splunk/defaultdb/summary $SPLUNK_HOME/var/lib/splunk/defaultdb/datamodel_summary In savedsearches, what means auto_summarize and alert.action=summary ... View more

I found this in my splunkd.log and It seems linked to the setting rdnsMaxDutyCycle in limits.conf I assume that it triggers when my reverse dns resolution takes too long. what is too long, and what is the base value used to compare ? 8-27-2014 12:51:28.048 +0800 WARN TcpInputConfig - reverse dns lookups appear to be excessively slow, this may impact receiving from network inputs. 66.688940 % time is greater than configured rdnsMaxDutyCycle=10 %. Current lookup: host::XX.XX.XX.XX ... View more

I am trying to localize where my events are located. - in which indexer - in which index - in which bucket Can I use a search to do that ? ... View more