Hello Splunkers! 🎉 You can now easily install Splunk Enterprise and the Universal Forwarder using this handy script. It supports all available versions and can be installed on any Linux distribution. For detailed installation steps, please visit : https://github.com/PraxisForge/Install_Splunk Upgrade universal forwarder version (nix) #Splunk Enterprise(nix) #Universal Forwarder(nix) #Upgrade Splunk Enterprise version(nix) ... View more

For SQS S3 and Cloudtrail just clone your input and create multiple inputs.  Just watch resources (CPU, Load Average) on the host where the inputs are defined.  If you modify any python code then the TA will no longer be supported and will break with upgrades. ... View more

Running into the same modular loading issue. Installing on HF on-prem and this issue is occurring.    With Py 2 being given an official end-date, I would like to know if there the community has found a way to get this working on Py3. ... View more

Here's the source from docs: https://docs.splunk.com/Documentation/Splunk/8.1.1/Installation/RunSplunkasadifferentornon-rootuser In section "Run Splunk Enterprise as a different or non-root user": It says: "On *nix based systems, you can run Splunk Enterprise as a user other than root. This is a Splunk best practice and you should configure your systems to run the software as a non-root user where possible."   ... View more

@networkthinking -- re your comment "not able to award points" -- are you able to "Accept" the answer? If not we can file a bug report. Let me know! ... View more

"First install TA_BOX_APP from splunk and configure with box details and restart the server" -- what details, I tried using the oauth info from the Box app install, looks like authentication works, but I am still getting the 400 errors. 04-01-2016 12:24:59.306 -0500 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\BoxAppForSplunk\bin\box.py"" HTTP Request error: 400 Client Error: Bad Request This used to work prior to upgrading to 6.3 (and the Box app is not rated for 6.3, so I'm not too surprised.) ... View more

2 forwarders is the easy method. very reliable. The complex one is : in outputs.conf : to define 2 tcpout groups, in inputs.conf : create symlinks to the file, add an extra monitor on the symlink, add crcSALT to avoid conflict. then and use _TCP_ROUTING rules on inputs.conf the risk is that if one group is blocked, both will be blocked. ... View more

Hi rmorlen, You can try the solution provided here http://answers.splunk.com/answers/107574/track-users-logging-in-via-sso.html or use the LDAP add on http://apps.splunk.com/app/1852 which enables Splunk to perform nativ LDAP queries and browser for the user on your LDAP server. Hope that helps ... cheers, MuS ... View more

Usually it is some problem with sendmail. Did you check the python.log source too, usually that does yield other clues, like this one showing diskspace being exhausted: /opt/splunk/var/log/splunk/python.log.1:2014-06-19 17:41:10,702 -0400 ERROR sendemail:112 - Sending email. subject /opt/splunk/var/log/splunk/python.log.1:2014-06-19 17:41:10,702 -0400 ERROR sendemail:348 - (452, '4.4.5 Insufficient disk space; try again later', u'splunk@splunk-search-head-2.x.x.x') while sending mail to: x@x.com If that doesn't show anything, the next troubleshooting step is to run sendmail from the command line as the splunk user: /usr/sbin/sendmail -f testfrom@yourdomain.com testfto@yourdomain.com This will show any possible errors. If you are in linux, you can also check /var/log/maillog ... View more

I've the same problem, is there a clear way to set search app permissions on all search head nodes? I had to connect on a search head to update search app permissions in manage apps I've also seen that local.meta has then been updated Also what are the differences between default.meta and local.meta? Many thanks! ... View more

We do something similar to what you are referring to. Yes, outputs.conf on the searchhead to forward data to the indexer. We have an indexapp that contains a local directory with indexes.conf in it. Whenever we add an index we update that file and deploy to all of our indexers and any searchhead who need the index list. We use a script to "enable" the indexes without restarting the indexers. echo Enter in Admin password read pw for indexer in indexer1 indexer2 indexer3 do echo $indexer echo ------------- curl -k -u admin:$pw https://$indexer:8089/servicesNS/admin/indexerbase/configs/conf-indexes/_reload curl -k -u admin:$pw https://$indexer:8089/services/data/indexes/_reload echo sleep 1 done echo ------------- echo Indexes reloaded ... View more

The reason this is happening is because the pass4SymmKey/shared-secret on your new member was not matching the secret of the rest of the cluster. When you ran the CLI command splunk add shcluster-member -current_member_uri https://server2:8089, an authentication token was generated on the search head server1 and put in its $HOME/.splunk directory. add shcluster-member proxies the request to one of the current members (server 2) and tries to get the current node added via the captain. This proxy process fails due to authentication, when the invalid token is used. What you will need to do is. 1. bring down the node that is being added and fails. 2. go to server.conf of that node and change its pass4SymmKey to correct shared secret. or re init the node. 3. delete the $HOME/.splunk folder so that the CLI gets a new key 4. restart splunk 5. try the add shcluster-member command again. ... View more

I encountered the same issue. Try going to IE settings, select Compatibility View Mode. Remove Splunk from the list of website and unselected the checkbox option for "Display intranet sites in Compatibility View". This solved my problem. 🙂 ... View more

You can use Splunkit, comes with both an indexing benchmark as well as a search benchmark. https://apps.splunk.com/app/749/ ... View more

You can run the splunk btool --debug tool. I had several "No spec file for: ... .conf" entries after moving from 5.0.4 to 6.2.3. Then I remembered I had some sandboxing app with copies of ALL the default configuration files in it. I removed the app, restarted Splunk, and everything appeared to work after that. My guess is that there are some incompatibilities between the different configuration stanzas and syntax of 5/6 and that while the migration will overwrite the conflicts in the /default paths, it won't touch conflicting settings if they are somewhere else (i.e., /apps). I thought Splunk 5 actually included some deprecated setting stanzas from 4 to avoid this but it doesn't seem to be 100% true going from 5 to 6. ... View more

They are not in the internal indexes ( or so i believe 😉 ) , but rather they are in the $SPLUNK_HOME/var/run/splunk/dispatch/your_search_with_"cryptic"_name_dir/ in one of the logs, or csv there ( search.log ? ) You should be able to download this from the job-inspector . ... View more

Could provide the DEBUG message from the Search Inspector ? Also, since you're changing the conf file, you're doing refresh/restart of splunk after change right? ... View more

Great! Thanks. Now when will the new TimePicker be out? 🙂 ... View more

If you only want to consider searches that explicitly list index=specific then you should easily be able to deduce that from the above two ways of loading saved searches. ... View more

Cool. I forgot to post the doc link: http://docs.splunk.com/Documentation/Splunk/6.0.1/SearchReference/map ... View more

If you are collecting process-level information for Splunk processes using the S.o.S app's ps_sos.sh scripted input, you can break down your daily search workload between scheduled and ad-hoc searches like so: `set_sos_index` sourcetype=ps host=<indexer or search-head host> | multikv | `get_splunk_process_type` | search type="searches" | rex field=ARGS "_--user=(?<search_user>.*?)_--" | rex field=ARGS "--id=(?<sid>.*?)_--" | rex field=sid "remote_(?<search_head>[^_]*?)_" | eval is_remote=if(like(sid,"%remote%"),"remote","local") | eval is_scheduled=if(like(sid,"%scheduler_%"),"scheduled","ad-hoc") | eval is_realtime=if(like(sid,"%rt_%"),"real-time","historical") | eval is_subsearch=if(like(sid,"%subsearch_%"),"subsearch","generic") | eval search_type=is_remote.", ".is_scheduled.", ".is_realtime | timechart span=1d dc(sid) AS "Search count" by is_scheduled Note that you'l need to run this search from within the context of the S.o.S app for the macros it uses to be available. You will also need for the ps_sos.sh scripted input to have been running for several days on the instance that you are targeting the search against. ... View more

Thank you. That worked. ... View more

Thanks. I will do that. ... View more