Presumably the sleep delay allows PAM to be fully spun up so that ulimit settings are applied properly when Splunk starts. Another alternative and more details are discussed in George Starcher's blog post Splunk Ulimits and You. ... View more

DB Connect is not intended for searching databases, it's intended to pull data into Splunk for indexing. It doesn't allow you to just query a database and display it in the Splunk WebUI. ... View more

No Splunk tarball install needs make or make install . The tarball contains binaries and python scripts and libraries. ... View more

mfrost8, please change your comment to an Answer so it can be accepted and upvoted! ... View more

Look for index=_internal sourcetype=splunkd or source=*python_modular* ERROR over the last hour or so (it runs every 5 minutes). If they aren't merging it generally means the process is erroring out on an entry in one of the CSVs being merged--header issue, problem with field contents of a line, etc. It will be an iterative process, as the merge process bails after the first error. ... View more

Usually the script will throw an error about this. Field names with spaces is another one (has to be underscores). ... View more

You will need to review your asset files for problems. Based on cases I have dealt with (from my Splunker hat), I would check for IPv6 addresses which can be represented multiple ways, not all of which we parse well. (Samples in a support case to file bugs would be helpful and appreciated.) This looks like the key error: 2016-07-22 09:08:38,411 ERROR pid=80905 tid=asset file=lookup_modinput.py:streaming_merge_task:298 | status="Error processing record" count="1518" record="fields(ip=... Find the line mentioned following the above part of the error, and look at that specific record, as well as the records before and after it. Something around there is causing the python to fail. Check for hidden characters in a text editor, mis-matched quotes, etc. If you are feeling industrious (make a backup copy), you can modify the output.py file to change it's logging level: # normal logging level level=logging.INFO # additional logging level=logging.DEBUG Change should be made on line 19 of output.py . Switch the logging level back after you've tested, or you will cause your logs to fill up with unnecessary stuff. Modular inputs run every 5 minutes. ... View more

"Accepted" == solved. ... View more

A couple of FYIs: If you run the modular input command manually (as suggested in "a" above) for these merges to try to debug what's going on with them, the results will not wind up in the correct SA-IdentityManagement context. In my tests they wound up in Searching & Reporting OR EnterpriseSecurity contexts. We ended up manually moving the results to SA-IdentityManager so as not to have to run the search again. Modular input throws an error if there are spaces in any extra field names you pull in. Errors like #2 may be masked by other problems, like inputs.conf stanza in one context and transforms.conf stanza in another (which may happen if you use the WebUI). ... View more

I just had a customer with this problem on Splunk 6.4.1, ESS 4.1.0, and Threatstream 4.2.6-1. The Threatstream app had to be completely removed; disabling it was not enough. ... View more

Currently, if the IDX takes longer than the restart_timeout to come back on-line, the CM marks the IDX as "down". Counter-intuitively, this frees up a slot in the CM's IDX restart queue, and it moves to the next IDX. Of course, this impacts the total number of IDXes available. In addition, the CM does not take into account SF or RF when doing a rolling restart. E.g., it doesn't check to see if the current IDX it's restarting has the last searchable copy if other IDXes are marked as down. The only way to avoid this issue and keep the data 100% searchable throughout a maintenance like this is to make a multi-site cluster (which could be in the same DC) and use the -site-by-site flag as mentioned in the link that mbrown mentioned. If you have large numbers of buckets per IDX, this can increase the amount of time it takes an IDX to restart. Generally, you will see fewer issues if you keep the IDXes at under 100K buckets each. One of our primary index cluster developers gave a talk at our user's conference last year, and provided some recommendations for cluster tuning based on bucket counts. Slide 15 has a table with recommendations on tuning 'service_interval' (on the CM), 'heartbeat_period' (on IDXes), 'heartbeat_timeout' (on the CM), as well as a few other settings. You can read the slides here: https://conf.splunk.com/session/2015/conf2015_Dxu_Splunk__Deploying_IndexerClusteringTips.pdf A recording of the talk is available here: https://conf.splunk.com/session/2015/recordings/2015-splunk-68.mp4 As I stated in my comment above, if you have large numbers of buckets with timestamp issues, this can cause problems with cluster rolling restarts. If a timestamp is far in the past compared to the current time, and you're using time-based retention, this will cause buckets to roll prematurely, creating many small buckets. At a minimum, consider running splunk remove excess-buckets [index-name] periodically, particularly if you have had any IDX outages, as the CM does not remove excess buckets automatically. ... View more

Another point, if you have large numbers of buckets with timestamp issues, this can cause problems with cluster rolling restarts. Reducing the overall number of buckets in the cluster can help reduce restart times. At a minimum, consider running: splunk remove excess-buckets [index-name] ... View more

Have a look at these two Answer posts: Mongod fails to start due to SSL cert expiration Mongod fails to start It may also be a permissions problem in the mongod tree, so verify that as well. ... View more

Another work around is simply moving, deleting, or renaming the duplicate app. ... View more

There's also that problem in SHC for version 6.2.6 that can be resolved with an upgrade. link text SHC: $SPLUNK_HOME/var/run/splunk/dispatch may fill with old artifacts after 6.2.6 upgrade. ... View more

splunktcp-ssl and tcp-ssl are two separate input stanza types. splunktcp-ssl is intended for receiving data from Splunk forwarders and allows the key connection_host . tcp-ssl is intended for encrypted communication coming in unparsed (e.g. from 3rd party systems) and does not allow the connection_host key. Reference: Inputs.conf spec ... View more

If you it turns out that is the fix, please leave docs feedback on the spec page so the documentation team can update, and edit your answer with the details. 🙂 ... View more

Take a look at Duane Waddle's SSL Best Practices talk: slide, video As a first step I would suggest testing via Splunk's OpenSSL on the command line from the UF endpoint to verify connectivity: splunk cmd openssl s_client -connect <ip>:<port> -showcerts This will dump the certs which you can then copy to a file and check manually: splunk cmd openssl x509 -text -noout -in <file>.crt Other gotchas: - Splunk expects PEM certs and keys. - If older than version 6.1, key file must be unencrypted, and key and cert must be in separate files. - Splunkd expects key, cert, and root-cert all in one file You can cut the noise in Splunk by searching for the specific IP or hostname that you are testing with, limited to the _internal index. Since nothing is being sent from the endpoint, log into it and tail the logs (*NIX) or use event viewer (Windows). ... View more

Glad I was able to help. Changed to an answer. ... View more

There are some examples in the developer's docs: Charts Using Simple XML. You may also wish to download the Dashboard Examples for Splunk 6. So you can do it, but it's not as simple as plugging in an option tag, unfortunately, as you can do for height. Dashboards and Visualizations Reference. ... View more