Hi all, so I've been trying to ingest cisco netflow logs into my splunk environment, and finally got the logs in with Splunk Stream. However, there's a field "src_content" which seems to be unable to parse or read by splunk, and its appearing as symbols. I'm suspecting itt is due to cisco netflow sending them via High-Speed Logging. Is there a template for splunk to decode these? It looks like this for eg. src_content:
"��`��f
^P
,
d�
N�q������l��
z�so#(��� ... View more

Hi splunk community, In my environment I have 1 indexer each in 2 different datacenters, 1 searchhead, 1 clustermaster , X forwarders. Is it possible for the following configuration in server.conf, with 1 searchpeer each, in 2 sites, if replication is not required? Again, I do not need replication. site_replication_factor = origin:1, total:1 site_search_factor = origin:1, total:1   Or can the searchpeer in the different datacenters, form a single site, 2 node cluster, with replication_factor=1 search_factor=1   I know that the recommendation is for latency to be <100ms. Thanks for your inputs! ... View more

Hi splunk community, So on one of the search heads my SHC environment's /opt/splunk/var/lib/splunk/kvstorebackup/ directory, besides the kvdump.tar.gz, there seems to be multiple folders of tmpKVRestore_* Those folders seem to contain the apps and the .json data. I would like to ask if those folders are safe to be removed, and if those folders are caused by restoring splunk kvstore on the search head, which splunk did not remove it as a means of housekeeping after the successful restore? Thanks. ... View more

Hi all, I am having this issue whereby the logs are not coming in from the UF > Local HF > Regional HF > Indexer There was also a local setting on limits.conf which could it possibly cause the blockup? [thruput] maxKBps = 30   On the Local HF splunkd.log, these are the errors - 02-17-2021 00:25:12.661 -0500 WARN TailReader - Could not send data to output queue (parsingQueue), retrying... 02-17-2021 00:25:17.673 -0500 INFO TailReader - ...continuing. 02-17-2021 00:25:23.864 -0500 ERROR PipelineComponent - Monotonic time source didn't increase; is it stuck? 02-17-2021 00:25:24.866 -0500 ERROR PipelineComponent - Monotonic time source didn't increase; is it stuck? 02-17-2021 00:25:27.681 -0500 WARN TailReader - Could not send data to output queue (parsingQueue), retrying... 02-17-2021 00:25:29.392 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.10.10.10_8089_hostname.com_hostname_C83 02-17-2021 00:25:30.533 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.10.10.10_8089_hostname.com_hostname_C83 02-17-2021 00:25:42.501 -0500 WARN HttpListener - Connection from 127.0.0.1:49251 didn't send us any data, disconnecting 02-17-2021 00:25:54.856 -0500 ERROR PipelineComponent - Monotonic time source didn't increase; is it stuck? 02-17-2021 00:25:57.724 -0500 INFO TailReader - ...continuing. 02-17-2021 00:26:04.858 -0500 ERROR PipelineComponent - Monotonic time source didn't increase; is it stuck? 02-17-2021 00:26:12.748 -0500 WARN TailReader - Could not send data to output queue (parsingQueue), retrying... 02-17-2021 00:26:16.869 -0500 ERROR PipelineComponent - Monotonic time source didn't increase; is it stuck? 02-17-2021 00:26:19.874 -0500 ERROR PipelineComponent - Monotonic time source didn't increase; is it stuck? 02-17-2021 00:26:30.777 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.10.10.10_8089_hostname.com_hostname_C83 02-17-2021 00:26:42.625 -0500 WARN HttpListener - Connection from 127.0.0.1:49305 didn't send us any data, disconnecting 02-17-2021 00:26:44.558 -0500 INFO TailReader - Continuing... 02-17-2021 00:26:49.560 -0500 WARN TailReader - Could not send data to output queue (parsingQueue), retrying... 02-17-2021 00:27:31.935 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.10.10.10_8089_hostname.com_hostname_C83 02-17-2021 00:27:32.885 -0500 INFO TailReader - ...continuing. 02-17-2021 00:27:33.088 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.10.10.10_8089_hostname.com_hostname_C83 02-17-2021 00:27:37.886 -0500 WARN TailReader - Could not send data to output queue (parsingQueue), retrying... 02-17-2021 00:27:41.867 -0500 ERROR PipelineComponent - Monotonic time source didn't increase; is it stuck? 02-17-2021 00:27:42.635 -0500 WARN HttpListener - Connection from 127.0.0.1:49403 didn't send us any data, disconnecting 02-17-2021 00:27:47.902 -0500 INFO TailReader - ...continuing. 02-17-2021 00:27:56.078 -0500 WARN TcpInputProc - Stopping all listening ports. Queues blocked for more than 300 seconds 02-17-2021 00:27:56.078 -0500 INFO TcpInputProc - Stopping IPv4 port 514 02-17-2021 00:27:56.078 -0500 INFO TcpInputProc - Stopping IPv4 port 5514 02-17-2021 00:27:56.078 -0500 INFO TcpInputProc - Stopping IPv4 port 9997 02-17-2021 00:27:57.915 -0500 WARN TailReader - Could not send data to output queue (parsingQueue), retrying... 02-17-2021 00:28:33.100 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.10.10.10_8089_hostname.com_hostname_C83 02-17-2021 00:28:34.741 -0500 INFO TailReader - Continuing... 02-17-2021 00:28:37.968 -0500 INFO TailReader - ...continuing. 02-17-2021 00:28:39.753 -0500 WARN TailReader - Could not send data to output queue (parsingQueue), retrying... 02-17-2021 00:28:41.866 -0500 ERROR PipelineComponent - Monotonic time source didn't increase; is it stuck? 02-17-2021 00:28:42.906 -0500 WARN HttpListener - Connection from 127.0.0.1:49528 didn't send us any data, disconnecting 02-17-2021 00:28:42.976 -0500 WARN TailReader - Could not send data to output queue (parsingQueue), retrying... 02-17-2021 00:28:44.768 -0500 INFO TailReader - Continuing... 02-17-2021 00:28:49.769 -0500 WARN TailReader - Could not send data to output queue (parsingQueue), retrying...02-17-2021 00:25:12.661 -0500 WARN TailReader - Could not send data to output queue (parsingQueue), retrying... 02-17-2021 00:25:17.673 -0500 INFO TailReader - ...continuing. 02-17-2021 00:25:23.864 -0500 ERROR PipelineComponent - Monotonic time source didn't increase; is it stuck? 02-17-2021 00:25:24.866 -0500 ERROR PipelineComponent - Monotonic time source didn't increase; is it stuck? 02-17-2021 00:25:27.681 -0500 WARN TailReader - Could not send data to output queue (parsingQueue), retrying... 02-17-2021 00:25:29.392 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.10.10.10_8089_hostname..com_hostname_C83 02-17-2021 00:25:30.533 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.10.10.10_8089_hostname.com_hostname_C83 02-17-2021 00:25:42.501 -0500 WARN HttpListener - Connection from 127.0.0.1:49251 didn't send us any data, disconnecting 02-17-2021 00:25:54.856 -0500 ERROR PipelineComponent - Monotonic time source didn't increase; is it stuck? 02-17-2021 00:25:57.724 -0500 INFO TailReader - ...continuing. 02-17-2021 00:26:04.858 -0500 ERROR PipelineComponent - Monotonic time source didn't increase; is it stuck? 02-17-2021 00:26:12.748 -0500 WARN TailReader - Could not send data to output queue (parsingQueue), retrying... 02-17-2021 00:26:16.869 -0500 ERROR PipelineComponent - Monotonic time source didn't increase; is it stuck? 02-17-2021 00:26:19.874 -0500 ERROR PipelineComponent - Monotonic time source didn't increase; is it stuck? 02-17-2021 00:26:30.777 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.10.10.10_8089_hostname.com_hostname_C83 02-17-2021 00:26:42.625 -0500 WARN HttpListener - Connection from 127.0.0.1:49305 didn't send us any data, disconnecting 02-17-2021 00:26:44.558 -0500 INFO TailReader - Continuing... 02-17-2021 00:26:49.560 -0500 WARN TailReader - Could not send data to output queue (parsingQueue), retrying... 02-17-2021 00:27:31.935 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.10.10.10_8089_hostname.com_hostname_C83 02-17-2021 00:27:32.885 -0500 INFO TailReader - ...continuing. 02-17-2021 00:27:33.088 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.10.10.10_8089_hostname.com_hostname_C83 02-17-2021 00:27:37.886 -0500 WARN TailReader - Could not send data to output queue (parsingQueue), retrying... 02-17-2021 00:27:41.867 -0500 ERROR PipelineComponent - Monotonic time source didn't increase; is it stuck? 02-17-2021 00:27:42.635 -0500 WARN HttpListener - Connection from 127.0.0.1:49403 didn't send us any data, disconnecting 02-17-2021 00:27:47.902 -0500 INFO TailReader - ...continuing. 02-17-2021 00:27:56.078 -0500 WARN TcpInputProc - Stopping all listening ports. Queues blocked for more than 300 seconds 02-17-2021 00:27:56.078 -0500 INFO TcpInputProc - Stopping IPv4 port 514 02-17-2021 00:27:56.078 -0500 INFO TcpInputProc - Stopping IPv4 port 5514 02-17-2021 00:27:56.078 -0500 INFO TcpInputProc - Stopping IPv4 port 9997 02-17-2021 00:27:57.915 -0500 WARN TailReader - Could not send data to output queue (parsingQueue), retrying... 02-17-2021 00:28:33.100 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.10.10.10_8089_hostname.com_hostname_C83 02-17-2021 00:28:34.741 -0500 INFO TailReader - Continuing... 02-17-2021 00:28:37.968 -0500 INFO TailReader - ...continuing. 02-17-2021 00:28:39.753 -0500 WARN TailReader - Could not send data to output queue (parsingQueue), retrying... 02-17-2021 00:28:41.866 -0500 ERROR PipelineComponent - Monotonic time source didn't increase; is it stuck? 02-17-2021 00:28:42.906 -0500 WARN HttpListener - Connection from 127.0.0.1:49528 didn't send us any data, disconnecting 02-17-2021 00:28:42.976 -0500 WARN TailReader - Could not send data to output queue (parsingQueue), retrying... 02-17-2021 00:28:44.768 -0500 INFO TailReader - Continuing... 02-17-2021 00:28:49.769 -0500 WARN TailReader - Could not send data to output queue (parsingQueue), retrying... ... View more

Hi, we are using splunk 8.0.6 with LDAP authentication in a SHC, and with a few local splunk users. We would be adding MFA (RSA) soon. My question is, how will the local users be tied to the RSA MFA, and will it be possible to only exclude local splunk users from MFA? Does RSA support SAML authentication, should I change from using LDAP to SAML if it does, and ultimately my intention is to exclude local users from the MFA using the logontype=splunk in url. ... View more

Hi everyone, so one of our SHC member will be moved to another datacenter, as such the IP assigned will be different. It is a 3 SHC with 1 Deployer, without IDX Clustering/MN. Currently the setup of the SHC is with IP. I would like to know of the implications/complexity of changing the IP of one of the SHC members. Is it possible to do this just by reinitializing the SHC member? Are there any official/unofficial Splunk guides in doing this? Thanks. ... View more

Hello, I am facing an issue on my 3 member SHC whereby I have used the deployer to push a local folder with the deployer_push_mode=full. This ends up at <app>/local/inputs.conf on all 3 SHs. This inputs.conf has settings which are independent to the SH it is on, eg. SSL cert name. So for now, the inputs.conf is the same throughout the cluster, which is wrong as the certs are named differently on each SH.   The thing is, I only need this inputs.conf on only 1 SH which is receiving logs from Splunk UBA.  Splunk UBA is now forwarding logs to the SH1. My question is,  should the SH1 go down, how do i configure SH2 inputs.conf to point to its cert path instead of the configuration from deployer which is the path for SH1. I'm not sure if I can setup each of the 3 SHs /opt/splunk/etc/<app>/local/inputs.conf differently as it will affect SHC raft issues? ... View more

Regarding the migration of knowledge objects from a standalone searchhead 'search' app to searchhead clustering environment, the docs recommends - Migrate_settings_to_a_search_head_cluster  Copy the .../search/local directory in the temporary directory to a new app directory, such as search_migration_app, in the temporary directory. Do not name this new app "search." Following which after pushing from the deployer, should the knowledge objects be moved back to the search app via the search head?? The concern is that users will be using the 'search' and 'search_migration_app' and have to look through the 'search' and 'search_migration_app' for the objects. Should the knowledge objects remain in the 'search_migration_app' and new objects to be saved into 'search' app? Is there a way to bulk move the objects easily from the 'search_migration_app' to 'search' app? ... View more