It appears they are left-overs from your previous versions. Check it out if the version you are using has the parameters in the docs - they are there in the docs for 2.x versions but no longer appear in 3.x versions. If that's the case you can comment them out in the db_connections.conf and restart splunk. ... View more

There is a known issue in this regards - if there are any duplicate tokens for HEC in search/local/inputs.conf the splunkd/splunkweb will fail to start and puts log message on the screen. WARNING: web interface does not seem to be available! Run to find the duplicate token in inputs.conf $ splunk btool --debug inputs list | grep "token =" | cut -d "=" -f 2 |sort |uniq -c | awk '{ if ($1 > 1) print $2}' The fix has been included in 7.0.6+ and 7.1.3+. ... View more

As of this writing where the latest version is 7.1.2.1, this is not a supported feature. The steps you have done works for the case of the migration from standalone to single site cluster. But it is not working for the case where you migrate to multisite cluster as the buckets created in single site cluster/standalone don't have site information. This will be implemented and available in 7.2+, where a parameter, "constrain_singlesite_buckets" is available in server.conf. ... View more

Please try the below in the "server.conf" [httpServer] replyHeader.X-XSS-Protection= 1; mode=block replyHeader.Content-Security-Policy = script-src 'self'; object-src 'self' [sslConfig] sendStrictTransportSecurityHeader=true OR the easier way, you can consider to block scanner from connecting to the port.. acceptFrom = "list of server name or ip addresses to include all SHs/Deployer,Indexers/CM, HF, LM, Deployment Srver,127.0.0.1,.. " Implementing this parameter needs thorough testing to ensure it doesn't break Splunk Services and make sure to include 127.0.0.1 this is mandatary. https://docs.splunk.com/Documentation/Splunk/7.1.2/Admin/Serverconf for example, in server.conf, *[httpServer] acceptFrom = shd*.abc.com, idx*.abc.com, cm.abc.com,deployer.abc.com,LM.abc.com,127.0.0.1 * ... View more

This can happen when token for Http Event collector is used multiple times in different "http" stanzas. You can confirm if it is the case or not by running the below; $SPLUNK_HOME/bin/splunk btool ---debug inputs list | grep "token =" |cut -d "=" -f 2 | sort |uniq -c | awk '{if ($1>1) print $1,$2 }' If you find tokens used more than 1 it can cause crash. Make sure the tokens are only used once. We are implementing fixes so that it puts ERROR messages when it detects duplicate tokens used for HEC and continue, no more crashes. ... View more

It turned out to be hitting an unexpected corner case. SHCluster used to have ES installed and now moved out of it - but it had side-effect of causing this error. As the error message suggests it appears something to do with capability. Firstly checked the role_admin which is supposed to inherit itoa_admin via GUI but it didn't have or use "splunk btool --debug authorize list role_admin ". By default it is included in "apps/itsi/default/authorize.conf" but it's been overwritten by a left-over from ES install setup for the role. [role_admin] in etc/system/local/authorize.conf importRoles = can_delete;ess_admin;ess_analyst;ess_user;power;user After adding itoa_admin to role_admin it immediately allows me to get in. There is a documentation addressing the similar situation but unfortuately it is under "Upgrade Splunk ITSI" section. http://docs.splunk.com/Documentation/ITSI/3.1.2/Configure/UpgradeSplunkITServiceIntelligence It would have been better if it is under steps for installation page. Just FYI, Before upgrading, make sure the Splunk admin role inherits from the itoa_admin role. The default settings for admin role inheritance for ITSI are contained in $SPLUNK_HOME/etc/apps/itsi/default/authorize.conf. Problems can occur when these settings have been modified in $SPLUNK_HOME/etc/system/local/authorize.conf which takes precedence over the ITSI .conf file settings. Do the following: Use the CLI btool command and look at the line importRoles to make sure itoa_admin, itoa_analyst, and itoa_user are listed. For example: ./splunk btool authorize list role_admin To add the itoa roles, do one of the following: From the UI, navigate to Settings > Access Controls> Roles > admin > Inheritance. Add itoa_admin, itoa_analyst and itoa_user to Selected roles if necessary. Alternatively, open $SPLUNK_HOME/etc/system/local/authorize.conf. Make sure itoa_admin, itoa_analyst and itoa_user are listed in the [role_admin] stanza for the importRoles setting as shown below. [role_admin] importRoles = itoa_admin;itoa_analyst;itoa_user;power;user If they are not, add them manually. Even if it still doesn't allow you to get in, check this out and try "splunk cmd python itsi_reset_default_team.py" http://docs.splunk.com/Documentation/ITSI/3.0.1/Configure/Installationandconfigurationconsiderationsandissues#Run_script_to_set_the_default_team_to_Global ... View more

This is a UI bug and will be corrected. Correct way for match_type for multiple fields is like, WILDCARD(A*), WILDCARD(B*) as described in spec. match_type = A comma and space-delimited list of < match_type >(< field_name >) specification to allow for non-exact matching The available match_type values are WILDCARD, CIDR, and EXACT. EXACT is the default and does not need to be specified. Only fields that should use WILDCARD or CIDR matching should be specified in this list ... View more

It didn't work even after stopping the anti-virus scanner on workstations but we were able to work out a simpler command that worked. msiexec /i "splunkforwarder-7.0.0-c8a78efdd40f-x64-release.msi" AGREETOLICENSE=1 DEPLOYMENT_SERVER="192.168.0.1:8089" SPLUNKPASSWORD=splunkpassword /qn /l*v %windir%\ccm\logs\INSTALL_Splunk.log And from there we were able to push configurations using deployment server. If you are not able to work it out even after that, you can contact splunk support with msi log and procmon data generated by following steps below; === Set Procmon to collect events for all processes during the repro : ==== 1 Launch Procmon, this should immediately bring up the Process Monitor Filter dialogue 2 If the Process Monitor Filter dialogue is not showing, launch it by going to Filter | Filter... 3 Reset the list of filters 4 OK the dialogue 5 Ensure that File | Capture Events is ticked 6 Reproduce whatever issue it is that we are interested in; Use /l*vx for msiexec instead of /l*v) so that it puts debugging logs. 7 Go to File | Save... 8 Under "Events to save:" ensure that "All events" is selected 9 Under "Format:" ensure that "Native Process Monitor Format (PML)" is selected 10 Choose appropriate Path: 11 OK ... View more

Here's a possible situation. Your indexers appear to be under stress and not performing fast enough for the inputs. Then due to the busy indexers, your forwarders also unable to send data and put logs like " TailReader - Could not send data to output queue " and at the same time your application rolls log data over to another directory where UF is not monitoring files. Here's the suggestion - please share it with your application team. i) Your application team has to change the rotation mechanism to like below; > How to rotate: Rotate files as before but keep the rotated file in the same directory as "/logs" so that splunk forwarder keeps reading the rest regardless of the name change. > When to move rotated files to the other directory. Use below command and retrieve "sptr=", this value has to be the same as the file size of "ls -l" or stat . Then you can move it to the other safely. "command sample" > ./splunk cmd btprobe -d ../var/lib/splunk/fishbucket/splunk_private_db --file /logs/abcACSLog_20180517.txt "command resutl" > key=0x340aa4a716415f07 scrc=0xac3413d846a972c9 sptr=14822 fcrc=0xc7cee9b9b8bf07ca flen=0 mdtm=1429060083 wrtm=1429060148 Or there are some other ways like, rotate file first and then move the previously rotated file later. But this is not safe way to move. ii) Your forwarder input has to be changed to keep reading the rotated file. ---- input config [monitor://logs/abcACSLog.txt] ===> [monitor://logs/abcACSLog*.txt] to cover the rotated files. Or just add one more monitor to cover the backup directory. ... View more

Splunk command, "btprobe" can tell you how much it read which you can compare with actual file size. Moreover it doesn't require password either. $ ./splunk cmd btprobe -d /home/fwd652/splunkforwarder/var/lib/splunk/fishbucket/splunk_private_db --file /home/fwd652/splunkforwarder/var/log/splunk/splunkd.log Using logging configuration at /home/fwd652/splunkforwarder/etc/log-cmdline.cfg. key=0xf002616871e8cce3 scrc=0x6d35eb146477b364 sptr=9881445 fcrc=0x4d8af36f7d891662 flen=0 mdtm=1510000676 wrtm=1510000679 In the result of "btprobe", find value for "sptr" = 9881445. This indicate how much splunk read the file. Below is against a static file, splunkd.log.1, below shows "sptr == file size". $ ./splunk cmd btprobe -d /home/fwd652/splunkforwarder/var/lib/splunk/fishbucket/splunk_private_db --file /home/fwd652/splunkforwarder/var/log/splunk/splunkd.log.1 Using logging configuration at /home/fwd652/splunkforwarder/etc/log-cmdline.cfg. key=0xc1965e5752d40688 scrc=0x121343f558ae9a0f *sptr=25000066 fcrc=0x4d8af36f7d891662 flen=0 mdtm=1508580949 wrtm=1508580949* $ ls -l /home/fwd652/splunkforwarder/var/log/splunk/splunkd.log.1 -rw------- 1 splunk splunk 25000066 Oct 21 06:15 ../var/log/splunk/splunkd.log.1 ... View more

This turned out that old dbx version sitting in the apps directory broke the UI. Even if it's disabled it still affects the Data input list in the GUI. Removing old dbx app fixed the issue. ... View more