sslConfig is case sensitive. ... View more

Is the file structured data? If not, you probably have INDEXED_EXTRACTIONS defined for this source, sourcetype, or globally; this will cause issues with the crc. While the issue manifests itself as TailingProcessor errors, btprobe behaves in the same way and is an easy test. You can replicate this with the license_usage.log, here with the setting sitting outside of any stanza (not recommended!): > head -1 ./splunk/etc/system/local/props.conf INDEXED_EXTRACTIONS = CSV > ./splunk/bin/splunk cmd btprobe -d ./splunk/var/lib/splunk/fishbucket/splunk_private_db/ --file ./splunk/var/log/splunk/license_usage.log Using logging configuration at ./splunk/etc/log-cmdline.cfg. terminate called after throwing an instance of 'FileContentException' what(): Ran out of data while looking for end of header Aborted (core dumped) btprobe crashes! We don't index license_usage.log! > ./splunk/bin/splunk search 'index=_internal | stats count by source' source count ----------------------------------------------------------------------- ----- ./splunk/etc/splunk.version 1 ./splunk/var/log/splunk/conf.log 5 ./splunk/var/log/splunk/first_install.log 4 ./splunk/var/log/splunk/metrics.log 68079 ./splunk/var/log/splunk/migration.log.2016-02-18.19-59-41 12 ./splunk/var/log/splunk/mongod.log 412 ./splunk/var/log/splunk/scheduler.log 48 ./splunk/var/log/splunk/splunkd-utility.log 65 ./splunk/var/log/splunk/splunkd.log 1837 ./splunk/var/log/splunk/splunkd_access.log 166 ./splunk/var/log/splunk/splunkd_stderr.log 10 ./splunk/var/log/splunk/splunkd_ui_access.log 2363 ./splunk/var/log/splunk/web_access.log 131 ./splunk/var/log/splunk/web_service.log 1101 If we rectify this, btprobe works, and we index the log: > sed -i 's/^INDEXED_EXTRACTIONS/#INDEXED_EXTRACTIONS/g' ./splunk/etc/system/local/props.conf > ./splunk/bin/splunk restart &>/dev/null > ./splunk/bin/splunk search 'index=_internal | stats count by source' source count ----------------------------------------------------------------------- ----- ./splunk/etc/splunk.version 2 ./splunk/var/log/splunk/conf.log 6 ./splunk/var/log/splunk/first_install.log 4 ./splunk/var/log/splunk/license_usage.log 11877 ./splunk/var/log/splunk/metrics.log 69386 ./splunk/var/log/splunk/migration.log.2016-02-18.19-59-41 13 ./splunk/var/log/splunk/mongod.log 631 ./splunk/var/log/splunk/scheduler.log 66 ./splunk/var/log/splunk/splunkd-utility.log 107 ./splunk/var/log/splunk/splunkd.log 3100 ./splunk/var/log/splunk/splunkd_access.log 214 ./splunk/var/log/splunk/splunkd_stderr.log 17 ./splunk/var/log/splunk/splunkd_ui_access.log 2366 ./splunk/var/log/splunk/web_access.log 134 ./splunk/var/log/splunk/web_service.log 1835 > ./splunk/bin/splunk cmd btprobe -d ./splunk/var/lib/splunk/fishbucket/splunk_private_db/ --file ./splunk/var/log/splunk/license_usage.log Using logging configuration at ./splunk/etc/log-cmdline.cfg. key=0x96c781d205619afe scrc=0xf665afd1903bda74 sptr=4720849 fcrc=0xa3bd152f008960fc flen=0 mdtm=1455235563 wrtm=1455236118 ... View more

Please avoid setting maxHotSpanSecs = 86400. This can cause undesired behavior if the setting aligns with the hour or the day, and can cause a bucket explosion - in some cases the creation of millions of buckets at midnight has been observed, each containing a single event. 86399 or 86401 are both fine, just avoid the exact alignment of 1 hour or 1 day. ... View more

This differs between linux and windows installs. splunk-6.3.0-aa7d4b1ccb80-Linux-x86_64> find . -name Cookie.py ./lib/python2.7/Cookie.py PS C:\Users\winders\Desktop\splunk-6.3.0-aa7d4b1ccb80-windows-64> Get-ChildItem -Path . -Name Cookie.py -Recurse splunk\Python-2.7\Lib\Cookie.py ... View more

To create a new dashboard named "newdash": curl -ku admin:changeme https://localhost:8089/servicesNS/admin/search/data/ui/views/ -d 'name=newdash' -d 'eai:data=<put your xml here>' To edit an existing dashboard with the name "newdash": curl -ku admin:changeme https://localhost:8089/servicesNS/nobody/search/data/ui/views/newdash -d 'eai:data=<put your xml here>' Take a look at the keys that are returned from the endpoint, that should give you a pretty good idea of what you can change. http://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTknowledge#data.2Fui.2Fviews.2F.7Bname.7D ... View more

They are pretty sneaky, and have bitten me more than once. I didn't even notice them on the first pass! ... View more

Are the double quotes cut and pasted? They look like fancy smart quotes to me, but I am not sure if that is reformatting. You might try re-entering them manually. The '>' is an indication that the shell is awaiting further input, and one smart quote would have the same effect: > splunk search “index=_internal earliest=-1m@m" -auth 'admin:[email protected]' > > ^C ... View more

We’ve just released version 1.4 of the Splunk SDK for Java. This version brings support for Java version 8, including the ability to manually configure the SSL security protocol of your choosing! Get the bits here: http://dev.splunk.com/goto/sdk-java-jar Read the docs here: http://dev.splunk.com/view/java-sdk/SP-CAAAEK3 ... View more

istat is another alternative to stat, available on aix. ... View more

You can recompile the jar after editing the HttpService.java to assign the appropriate SSLContext if SSLv3 is not appropriate: --- /splunk/src/splunk-sdk-java/splunk-sdk-java-1.3.1/splunk/com/splunk/HttpService.java 2014-08-21 14:07:12.000000000 -0400 +++ /splunk/src/splunk-sdk-java/patch/splunk/com/splunk/HttpService.java 2014-12-16 00:12:59.655857080 -0500 @@ -397,7 +397,7 @@ } }; try { - SSLContext context = SSLContext.getInstance("SSL"); + SSLContext context = SSLContext.getInstance("TLSv1.2"); context.init(null, trustAll, new java.security.SecureRandom()); return new SSLv3SocketFactory(context.getSocketFactory()); } catch (Exception e) { @@ -408,7 +408,7 @@ private static final class SSLv3SocketFactory extends SSLSocketFactory { private final SSLSocketFactory delegate; - public static final String[] PROTOCOLS = {"SSLv3"}; + public static final String[] PROTOCOLS = {"TLSv1.2"}; private SSLv3SocketFactory(SSLSocketFactory delegate) { this.delegate = delegate; Reference: http://docs.oracle.com/javase/7/docs/technotes/guides/security/StandardNames.html#SSLContext ... View more

This is most likely splunk phoning home to apps.splunk.com to check for updates. You can verifiy if this is the case by looking up the destination IP: dig +short apps.splunk.com 54.187.11.200 dig +short -x 54.187.11.200 ec2-54-187-11-200.us-west-2.compute.amazonaws.com. You should see a correlating event in the splunkd.log, but I believe only if there is an error: WARN HTTPClient - Connect to=apps.splunk.com:443 timed out; exceeded 10sec ERROR ApplicationUpdater - Error checking for update, URL=/api/apps:resolve/checkforupgrade: Connect to=https://apps.splunk.com timed out; exceeded 10sec ERROR HTTPClient - Cannot resolve IP of host=apps.splunk.com: No such host is known. If you would like to disable this behavior, check this answer: http://answers.splunk.com/answers/28616/how-can-automatic-update-checking-be-disabled-for-splunkbase-apps.html ... View more

Are you connecting a <= 5.x license slave to a >= 6.x master? It would appear that this message is to be expected in this scenario due to a differing argument count/differing capabilities. This can be safely ignored, or the logging level of the LMTracker category can be set above WARN to avoid logging the condition. ... View more

Or you might try: 'http://download.splunk.com/releases/&lt;version&gt;/&lt;product&gt;/&lt;platform&gt;/&lt;product&gt;-&lt;version&gt;-&lt;build&gt;-&lt;os&gt;.&lt;format&gt;' > curl 'http://download.splunk.com/releases/6.0.1/universalforwarder/linux/splunkforwarder-6.0.1-189883-linux-2.6-x86_64.rpm' -s| md5 92b2dbca6b9abb200c32c6d29b5c3a31 > curl 'http://download.splunk.com/releases/6.0.1/universalforwarder/linux/splunkforwarder-6.0.1-189883-linux-2.6-x86_64.rpm.md5' MD5(splunkforwarder-6.0.1-189883-linux-2.6-x86_64.rpm)= 92b2dbca6b9abb200c32c6d29b5c3a31 ... View more

In splunkd.log you should receive a message similar to: WARN HttpListener - Can't handle request for threads already in use ... View more