Your right, you have to stop all of the SHC members, run those deletes then start the cluster. ... View more

First, I assume you meant to say default here: How do we eliminate our version of the config and revert back to the one in the local directory? If you want to remove all the local conf files here is a simple command line that will do it very quickly on a search head. WARNING: backup of etc/apps recommended first! splunk btool check --debug | egrep ^Checking.+/apps.+/local | sed -e 's/Checking: //' | xargs rm You could add another grep in the list if you want to limit it to only certain apps. Be careful, good luck! ... View more

Your hot/warm storage should always be on direct-attach storage (disk, ssd) or SAN. NFS if fine for cold storage and it gives you the option to expand the available space if you need longer retention times or you add more indexing volume. I'm not sure why 6.4.x is giving you more problems but not using NFS for hot/warm has been a recommendation for a looooong time. The usual recommendation is you should get enough hot/warm storage to hold data for 2-4 weeks as that time window will probably be wide enough for most user searches (on the faster disks). After that it can roll to cold since relatively fewer searches will be run for 30 days or greater (your mileage may vary!). ... View more

Please tell me that your hot/warm location is NOT on an NFS mount. From the installation guide: If you use NFS, note the following: Do not use NFS to host hot or warm index buckets as a failure in NFS can cause data loss. NFS works best with cold or frozen buckets. Do not use NFS to share cold or frozen index buckets amongst an indexer cluster, as this potentially creates a single point of failure. Splunk Enterprise does not support "soft" NFS mounts. These are mounts that cause a program attempting a file operation on the mount to report an error and continue in case of a failure. Only "hard" NFS mounts (mounts where the client continues to attempt to contact the server in case of a failure) are reliable with Splunk Enterprise. Do not disable attribute caching. If you have other applications that require disabling or reducing attribute caching, then you must provide Splunk Enterprise with a separate mount with attribute caching enabled. Do not use NFS mounts over a wide area network (WAN). Doing so causes performance issues and can lead to data loss. ... View more

Do you see any messages in the dbx2.log file? ... View more

What jkat54 said. If your syslog server doesn't have a good log rotation configuration, then the forwarder will be monitoring 100's or 1000's of files and that consumes a lot of memory and cpu. I suspect that the oomkiller is killing the splunk process. You should consider adding the following inputs.conf setting: ignoreOlderThan = nonnegative integer [s|m|h|d] * Causes the monitored input to stop checking files for updates if their modtime has passed this threshold. This improves the speed of file tracking operations when monitoring directory hierarchies with large numbers of historical files (for example, when active log files are colocated with old files that are no longer being written to). * As a result, do not select a cutoff that could ever occur for a file you wish to index. Take downtime into account! Suggested value: 14d , which means 2 weeks * A file whose modtime falls outside this time window when seen for the first time will not be indexed at all. * Default: 0, meaning no threshold. ... View more

One thing you need to be aware of about join is that it has some significant limits OOB. You can alter them in limits.conf if needed: [join] subsearch_maxout = integer * Maximum result rows in output from subsearch to join against. * Defaults to 50000 subsearch_maxtime = integer * Maximum search time (in seconds) before auto-finalization of subsearch. * Defaults to 60 subsearch_timeout = integer * Maximum time to wait for subsearch to fully finish (in seconds). * Defaults to 120 ... View more

Under the category: RTFM http://docs.splunk.com/Documentation/DBX/2.3.0/DeployDBX/Architectureandperformanceconsiderations ... View more

A Splunk universal forwarder does NOT include python so it is using python from the OS. You are missing a python library required by the myScript.py. You will need to find the missing library and add it to your python install. ... View more

You have a couple choices: Use JMS to collect data with the app: JMS Messaging Modular Input https://splunkbase.splunk.com/app/1317/ Roll your own into your Java code using the SDK: http://dev.splunk.com/java ... View more

Yes, the app will run on Centos. You need to have the Splunk universal forwarder running on your domain controllers. You then configure and deploy the Splunk Add-on for Microsoft Windows to those forwarders. If you DC's are very busy, you will probably need to also set the following on the forwarders as well: limits.conf [thruput] maxKBps = The default value is 256 which is often not enough to keep up with the log traffic from busy DC's. Setting this to 0 is unlimited. If you have bandwidth limitations you will need to find a happy median. You may have to live with some latency in indexing the events. If you have a dev/test DC, I ALWAYS recommend you test on that one before deploying to production. ... View more

Set the values back to INFO after you figure out your problem. ... View more

On the Splunk instance that is monitoring the files, navigate to the $SPLUNK_HOME/etc directory and edit the file: log.cfg modify the following settings and change INFO to DEBUG category.TailingProcessor=INFO category.WatchedFile=INFO category.ArchiveProcessor=INFO category.TailReader=INFO save the file. Restart the Splunk instance. Take a look at the log: $SPLUNK_HOME/var/log/splunk/splunkd.log Look for the names of the files you were monitoring, the debug information should tell you why they were skipped. ... View more

Are you using Splunk Enterprise Security? If so, what version? ... View more

Sorry - I forgot you have it under a powershell stanza. If you can't make that work, maybe you can run it under a script stanza where you could use the interval=0 setting powershell -executionPolicy RemoteSigned -command .$SplunkHome\bin\scripts\metric.ps1 ... View more

Depending on what you are trying to do there are 3 techniques that are used to make historical searches run faster: Summary indexing Report acceleration Data model acceleration You should read this part of the knowledge management doc to help you pick the right technology: http://docs.splunk.com/Documentation/Splunk/6.4.2/Knowledge/Aboutsummaryindexing ... View more

There is a setting in limits.conf: indexed_realtime_use_by_default = * Should we use the indexedRealtime mode by default * Precedence: SearchHead * Defaults to false This might have the effect of delaying your "real time" search results if it is enabled on your search head. ... View more

You could be more explicit in describing what you are trying to do. You could create a DNS alias for all your indexers and forward to the DNS name. You could also create a VIP using a load-balancer and forward to the VIP. Finally if you just want the http events balanced among your indexers, you could set up a heavy forwarder, send all the events to it and then have it load balance it's ouput (via outputs.conf) to all indexers. Pick one. ... View more

it's stdev(count) not stddev. ... View more

If https://pbdc-splk-01:8089 is a linux box, look at iptables. If windows, windows firewall or other endpoint protection product. Also make sure that splunk is running and that the management port hasn't been moved to some other port. ... View more