For the comparison of sourcetype=foo vs log_type=foo, it all depends on how log_type is defined. If log_type=foo can be inferred to be only a keyword in the text of the event, then for many cases it will be equivalently efficient. We can rule out buckets for both cases in the bloom filter. However, for cases where 'foo' is present in your events, but is not actually the value of log_type, then sourcetype=foo will be more efficient, because we can retreive only the events where sourcetype=foo from the index, while events that have 'foo' but where log_type is not 'foo' will have to be post-filtered after the rules to identify the value are identified. If you have a wide variety of possible sources of log_type=foo, then the field based search will tend to diverge more in efficiency as compared to sourcetype=foo. In short, use sourcetypes, host, and source when they make sense, if it's a search that will run over significant data or will be saved and reused. However the difference between searching on source/sourcetype/host and a field is often not large enough to make mangling those fields for your use-case worth it. Typically, if that kind of mangling turned out to be worth it, it would be sufficient to use an indexed field in any event. ... View more

In your first example, you're getting command not found because you are running the python script using the shell, not python. You haven't shown how you are running it so I'm not sure what's going wrong, and I'm not familiar enough with the app to guess how it is intended to be invoked. In your first comment, you appear to be running a python script which expects to run in the Splunk environment, using splunk modules. However, it is being invoked with (presumably) the system python and certainly without the PYTHONPATH. Please try /opt/splunk/bin/splunk cmd python /opt/splunk/etc/apps/Splunk_TA_nessus/bin/nessus2splunk.py ... although I'm still not familiar enough with the app to know if that will produce the desired results. In the second comment, it seems you have the script set up as a scripted input. It is complaining that /opt/nessus/input does not exist. Does it? The error message's guess that $SPLUNK_HOME may not be set is implausible here, as splunkd will set $SPLUNK_HOME before launching the script. ... View more

Generally speaking I would suggest, in a clustered environment, separation of data collection from cluster management. The idea of clustering is redundancy and replace-ability. Collecting data on a single clustering node conflicts with that idea and goal. A Universal Forwarder causing outages is pretty surprising, though its certainly possible. Usually local log collection is more reliable than collection over network filesystems or similar, but remote log collection is a valid strategy. ... View more