I downvoted this post because things ... View more

I downvoted this post because things ... View more

You can see the running Configs via REST, but not the on disk configs, for the On disk configs you would need to create a custom search command that would call Btool. There have been a few written do to that, REST: | rest splunk_server=local /services/configs/conf-props | transpose Splunk_server can be any sever conf-props gets your props, conf-transforms gets you transforms conf-inputs gets you inputs conf-server gets you server..... etc Here is the Splunk Base app for Btool as well https://splunkbase.splunk.com/app/4182/#/overview ... View more

So this gets to the structure of how splunk works, the searches are dispatched to the indexer for them to do the work, so the number of threads open is not where you are running into the constraint, I would still assume it is related to the indexer returning the results quick enough for the search head. I would increase the number of concurrent searches you allow and see if that changes what you see on the SHC. Also look at the I/O on the indexer. That is so often the bottle neck that is why I keep pointing back that way. ... View more

It is the number of threads that are in the threadpool, controlling the number of threads that are able to be addressed by the scheduler ... View more

aaraneta, So the first thing you want to do is click save as: ![alt text][1] Then after you click this, select Alert: [1]: /storage/temp/207661-cap1.jpgThen You will want to select the time window that you want the search to run in and the frequency, then select the add action from the triggered actions section, selecting send email: If you have not setup your email server here is a guide to doing that. Splunk Email setup and Configuration ... View more

So you need to look into the /etc/init.d/splunk service, and determine who was the original bootstart user, that is the user that the files need to be owned by. Then the user that you need to use to change the permissions needs to have sudo privileges, so they need to have the ability to make changes to the user permissions. This might be the root user in your case. ... View more

aab5272, the first question is often why? Is it to avoid additional firewall rules or for data protection etc? If you want to send to, two different data centers from the same forwarder, you do not have to have a heavy forwarder as an intermediate. should you want / have a business case to have a heavy forwarder in between the forwarding nodes, and the indexing layer you can, but often there are other solutions that might have better performance. What is the end result you are trying to accomplish? and what are the constraints that are making you think you need a heavy forwarder in place at each data center? ... View more

Assuming that you are not I/O bound, which might be something to look into and double check, as well as your Indexers are able to keep up which is often the source of latency. You could up the number of executor_workers in your server.conf [shclustering] executor_workers = * Number of threads that can be used by the search head clustering threadpool. * Defaults to 10. A value of 0 will be interpreted as 1. But again I would evaluate if your indexers are not able to keep up with the additional calls during your more heavily loaded windows. That is often where I see imparted latency is from the indexing layer, not the SHC. ... View more

Painmonsoror, I don't know that in a clustered environment you have a better option, so I would do that. because Slave-apps takes the highest precedent. So I would do it how you suggest. Good luck! ... View more

https://splunkbase.splunk.com/app/1467/#/details Installation Step 1: App installation Install the Cisco Networks (cisco_ios) App on your search head Install the Cisco Networks Add-on (TA-cisco_ios) on your search head AND indexers/heavy forwarders Syslog input: Enable a UDP input with a custom port number on your Splunk forwarder or Splunk indexer. Set the sourcetype to cisco:ios or syslog Smart Call Home input: Enable a TCP input with a custom port number on your Splunk forwarder or Splunk indexer. Set the sourcetype to Cisco:SmartCallHome Step 2: Configure your Cisco devices Cisco IOS This includes all IOS variants. Not all commands are supported on all models Basic logging and timestamping service timestamps debug datetime msec localtime show-timezone year service timestamps log datetime msec localtime show-timezone year service sequence-numbers logging trap informational logging host [YOUR SYSLOG/SPLUNK SERVER IP] transport udp port [YOUR UDP PORT]        Enable change auditing archive log config   logging enable   logging size 200   notify syslog contenttype plaintext   hidekeys ! login on-failure log login on-success log logging userinfo !        Monitor interface changes General logging event trunk-status global logging event link-status global    Interface level logging event trunk-status logging event spanning-tree logging event status          MAC move notifications, STP logging, IP SLA logging etc. mac address-table notification mac-move spanning-tree logging ip sla logging traps ip dhcp limit lease log ip dhcp conflict logging ip nat log translations syslog xconnect logging pseudowire status ntp logging epm logging        For DHCP utilization logging on your devices, do this for each pool utilization mark high 80 log        For ARP threshold logging, do this on your SVIs and IP interfaces arp log threshold entries 2048        TrustSec If you are using Cisco TrustSec, add the following cts sxp log binding-changes cts logging verbose        ACL logging General Remember to add the log or log-input keyword to your access list entries if you want to enable access list logging Access list correlation tags ip access-list logging hash-generation        CPU and Memory Utilization logging This generates CPU and memory notifications. CPU notifications if the CPU has been over 80% for more than 5 seconds. Memory if there is less than 20000KB. process cpu threshold type total rising 80 interval 5 memory free low-watermark processor 20000 memory free low-watermark io 20000        Smart Call Home logging Enabling Smart Call home allows you to get extended device information from your devices at specific intervals ip http client source-interface [SOURCE INTERFACE] ! service call-home   call-home   contact-email-addr [YOUR.EMAIL@ADDR.ESS] site-id ["YOUR_SITE_NAME"] profile "Splunk"     destination transport-method http     destination address http http://[YOUR SYSLOG/SPLUNK SERVER IP]:[YOUR SMART CALL HOME TCP PORT]   subscribe-to-alert-group diagnostic severity debug     subscribe-to-alert-group environment severity debug     subscribe-to-alert-group inventory     subscribe-to-alert-group inventory periodic daily 22:30 !        NX-OS This includes all NX-OS variants. Not all commands are supported on all models Basic logging and timestamping logging logfile messages 6 logging server [YOUR SYSLOG/SPLUNK SERVER IP] 6 use-vrf [YOUR MGMT VRF] logging timestamp milliseconds logging monitor 6        Enable change auditing This feature is not supported on the NX-OS platform Monitor interface changes General    logging message interface type ethernet description logging event link-status default logging event trunk-status default          Interface level logging event port link-status logging event port trunk-status          MAC move notifications, STP logging, IP SLA logging etc. mac address-table notification mac-move ntp logging        ACL logging General Remember to add the log or log-input keyword to your access list entries if you want to enable access list logging NX-OS ACL logging logging level acllog 6 acllog match-log-level 6 logging logfile messages 6        Smart Call Home logging Enabling Smart Call home allows you to get extended device information from your devices at specific intervals callhome   site-id ["YOUR_SITE_NAME"]   email-contact [YOUR.EMAIL@ADDR.ESS]   phone-contact [YOUR PHONE NUMBER]   streetaddress S01   destination-profile Splunk   destination-profile Splunk format XML   destination-profile Splunk transport-method http   destination-profile Splunk http http://[YOUR SYSLOG/SPLUNK SERVER IP]:[YOUR SMART CALL HOME TCP PORT]   destination-profile Splunk alert-group Diagnostic   destination-profile Splunk alert-group EEM   destination-profile Splunk alert-group environmental   destination-profile Splunk alert-group inventory   destination-profile Splunk alert-group license   destination-profile Splunk alert-group linecard-hardware   destination-profile Splunk alert-group supervisor-hardware   destination-profile Splunk alert-group system   destination-profile Splunk alert-group test   transport http use-vrf [YOUR MGMT VRF]   enable   periodic-inventory notification interval  1   periodic-inventory notification timeofday 22:30 ... View more