You need to have admin role together with list_settings capability in order to send alert email when SMTP auth is used. ... View more

Can you try concat the certs into a single pem file, and have TLS_CACERT pointing at it an also commented out TLS_CACERTDIR attribute, like below: TLS_REQCERT never TLS_CACERT /opt/splunk/etc/openldap/certs/Your_Cert_Chain.pem #TLS_CACERTDIR /opt/splunk/etc/openldap/certs ... View more

For Kafka connect to ingest metrics data - we need to use collectd as documented here: https://www.splunk.com/blog/2018/04/25/unleashing-data-ingestion-from-apache-kafka.html Once collectd is installed, below is an example of a connector to send collectd metrics to a Splunk metrics index The Splunk metrics index is optimized for ingesting and retrieving metrics. For more information, see the Metrics manual. curl http://localhost:8083/connectors -X POST -H "Content-Type: application/json" -d '{ "name": "metrics-via-collectd", "config": { "connector.class": "com.splunk.kafka.connect.SplunkSinkConnector", "tasks.max": "1", "topics":"collectd", "splunk.sourcetypes": "collectd_http", "splunk.indexes": "kafka_metrics", "splunk.hec.uri": "https://localhost:8088", "splunk.hec.token": "727ac06e-5150-4f0f-a85d-c7b070176a2c", "splunk.hec.ack.enabled" : "false", "splunk.hec.ack.poll.interval" : "20", "splunk.hec.ack.poll.threads" : "2", "splunk.hec.event.timeout" : "120", "splunk.hec.ssl.validate.certs": "false", "splunk.hec.raw" : "true", "splunk.hec.raw.line.breaker" : "####" } }' Then, you can use below sample data to send to kafka. bin/kafka-console-producer.sh --broker-list localhost:9092 --topic collectd [{"values":[164.9196798931339196],"dstypes":["derive"],"dsnames":["value"],"time":1505356687.894,"interval":10.000,"host":"collectd","plugin":"protocols","plugin_instance":"IpExt","type":"protocol_counter","type_instance":"InOctets"}] Then, check from Splunk metrics index by using below command to confirm metrics data is indexed correctly. | mcatalog values(metric_name) WHERE index=kafka_metrics AND metric_name=protocols.protocol_counter.InOctets.value ... View more

I did that kafka setup before and below is based on centos 7. yum upgrade yum install java-1.8.0-openjdk download kafka Here is the splunkbase link but it will point you to Github https://splunkbase.splunk.com/app/3862/ Once you downloaded kafka, untar it and rename the directory tar -xzf kafka_2.11-2.1.0.tgz mv kafka_2.11-2.0.0 kafka Start kafka zookeeper and server cd kafka bin/zookeeper-server-start.sh config/zookeeper.properties > zookeeper.log & bin/kafka-server-start.sh config/server.properties > kafka_server.log & Create kafka topic bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test Below command can be used to list/delete topic as well. bin/kafka-topics.sh --list --zookeeper localhost:2181 bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic test Send some event to topic "test" so that you can get it from kafka connect later bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test this is a test message ^C Check the worker properties file. Make sure the following settings are set correctly. key.converter=org.apache.kafka.connect.storage.StringConverter value.converter=org.apache.kafka.connect.storage.StringConverter key.converter.schemas.enable=false value.converter.schemas.enable=false internal.key.converter.schemas.enable=false internal.value.converter.schemas.enable=false Start Splunk-kafka-connect bin/connect-distributed.sh config/connect-distributed.properties > Kafka_connect.log & Create HEC connector in Splunk Create splunk-kafka-connect task curl http://localhost:8083/connectors -X POST -H "Content-Type: application/json" -d '{ "name": "test-single-event", "config": { "connector.class": "com.splunk.kafka.connect.SplunkSinkConnector", "tasks.max": "1", "topics":"test", "splunk.sources": "test_kafka_event", "splunk.indexes": "kafka_event", "splunk.hec.uri": "https://localhost:8088", "splunk.hec.token": "26faccb6-a0af-45d6-996e-7df97afb81fd", "splunk.hec.raw": "false", "splunk.hec.ack.enabled": "false", "splunk.hec.ack.poll.interval": "10", "splunk.hec.ack.poll.threads": "1", "splunk.hec.ssl.validate.certs": "false", "splunk.hec.http.keepalive": "true", "splunk.hec.max.http.connection.per.channel": "4", "splunk.hec.total.channels": "8", "splunk.hec.max.batch.size": "500", "splunk.hec.threads": "1", "splunk.hec.event.timeout": "300", "splunk.hec.socket.timeout": "120", "splunk.hec.track.data": "true" } }’ ... View more

Did you try enable the maintenance mode and then restart the CM? ... View more

You can run below search on your cluster master to get a list of bucket that have status="bucket hasn't rolled yet" | rest splunk_server=local /services/cluster/master/fixup level=replication_factor | table title, latest.reason | rename latest.reason AS LatestReason | rename totle AS bucketID | regex LatestReason="bucket hasn't rolled yet" | table buckekID Once you got the bucketId, simple run below command on your Cluster Master will roll the bucket. curl -k -u admin:changme https://localhost:8089/services/cluster/master/control/control/roll-hot-buckets -d "bucket_id=<BUCIET_ID>” For example, curl -k -u admin:changeme https://localhost:8089/services/cluster/master/control/control/roll-hot-buckets -d "bucket_id=_internal~4520~11111111-1111-1111-1111-111111111111” IF you prefer, you can do this in batch mode. e.g. cat /var/tmp/bucketId.txt os~100~2FC3562D-9D2D-49CB-A598-89E47397E5D7 _internal~4523~11111111-1111-1111-1111-111111111111 for i in `cat /var/tmp/bucketId.txt`; do curl -k -u admin:changeme https://localhost:8089/services/cluster/master/control/control/roll-hot-buckets -d "bucket_id=$i"; done ... View more

Here is some script that you can use to test the Splunk port by using openssl. #!/usr/bin/env bash # OpenSSL requires the server IP and port number SERVER=127.0.0.1:8089 DELAY=1 ciphers=$(openssl ciphers 'ALL:eNULL' | sed -e 's/:/ /g') echo Obtaining cipher list from $(openssl version). for cipher in ${ciphers[@]} do echo -n Testing $cipher... result=$(echo -n | openssl s_client -cipher "$cipher" -connect $SERVER 2>&1) if [[ "$result" =~ ":error:" ]] ; then error=$(echo -n $result | cut -d':' -f6) echo NO \($error\) else if [[ "$result" =~ "Cipher is ${cipher}" || "$result" =~ "Cipher :" ]] ; then echo YES else echo UNKNOWN RESPONSE echo $result fi fi sleep $DELAY done A sample run again Splunk 7.1.2 with port 8089 will below result. # ./openssl.sh | grep YES Testing ECDHE-RSA-AES256-GCM-SHA384...YES Testing ECDHE-RSA-AES256-SHA384...YES Testing AES256-GCM-SHA384...YES Testing ECDHE-RSA-AES128-GCM-SHA256...YES Testing ECDHE-RSA-AES128-SHA256...YES Testing AES128-GCM-SHA256...YES Testing AES128-SHA256...YES ... View more

The above procedure work most of the case. However, I did hit "Unauthorized" when running above curl command even the username/password is correct. I use below alternative and it works. First, login to splunk first. # ./splunk login Splunk username: admin Password: Then, check the session key stored under your login profile. # cd ~/.splunk # ls -l authToken_xxxxxxxxxxx_8089 # cat authToken_xxxxxxxxxxx_8089 <auth><username>admin</username><sessionkey>861QHAwxbgNbssLPHed92VEkK4P^lAVlalKCmY9v1xf5Q16zlQJMri39U21oc7jTyJg9xtk4vi3dFpOqBZT3mqYSWu_y^E2vppzCVovekuFX48a</sessionkey><cookie>splunkd_8089</cookie></auth> Run the above curl again with the above session token. # curl -k -H "Authorization: Splunk 861QHAwxbgNbssLPHed92VEkK4P^lAVlalKCmY9v1xf5Q16zlQJMri39U21oc7jTyJg9xtk4vi3dFpOqBZT3mqYSWu_y^E2vppzCVovekuFX48a" https://localhost:8089/servicesNS/nobody/system/storage/collections/data/assetcollection/5b6953c89787925c6501af61 ... View more

You need to know the _key associated with the record that you want to delete first. Suppose you've a lookup called asset_kvlookup and is under collection=assetcollection. Run below search will show you all the _key associated with the record. | inputlookup asset_kvlookup|eval assetkey = _key Assume the key assetkey return "5b6953c89787925c6501af61", you can use this key to remove the record by running below curl. To display the record that is going to delete. # curl -k -u admin:changeme https://localhost:8089/servicesNS/nobody/system/storage/collections/data/assetcollection/5b6953c89787925c6501af61 To delete the record. # curl -k -u admin:changeme -X DELETE https://localhost:8089/servicesNS/nobody/system/storage/collections/data/assetcollection/5b6953c89787925c6501af61 ... View more

Configuration from Splunk App for Stream is stored inside kvstore. If you want to make a backup, you can run below command. cd $SPLUNK_HOME/bin ./splunk cmd python ../etc/apps/splunk_app_stream/bin/kvstore_debug.py ... View more

You can run below command to check the streamfwd version. pwd /opt/splunk/etc/apps/Splunk_TA_stream/linux_x86_64/bin ./streamfwd --version terminate called after throwing an instance of 'std::runtime_error' what(): locale::facet::_S_create_c_locale name not valid Aborted (core dumped) If you got the same error about locale, try to export the LC_ALL="en_US.UTF-8" before you start splunk. For example, pwd /opt/splunk/bin export LC_ALL="en_US.UTF-8" ./splunk start ... View more

I enabled the following DEBUG for WMI. $SPLUNK_HOME/etc/log.cfg [splunkd] category.ExecProcessor=DEBUG $SPLUNK_HOME/etc/log-cmdlog.cfg category.WMI=DEBUG Then, I got the following from splunkd.log 01-03-2018 17:03:31.713 +0800 DEBUG ExecProcessor - message from ""E:\Program Files\Splunk\bin\splunk-wmi.exe"" WMI - getEventLogWql: DESC: chk=4294967295, low=110155, hi=4294967295 (10.13.18.59: Security) 01-03-2018 17:03:31.713 +0800 DEBUG ExecProcessor - message from ""E:\Program Files\Splunk\bin\splunk-wmi.exe"" WMI - Event log wql "SELECT Category, CategoryString, ComputerName, EventCode, EventIdentifier, EventType, Logfile, Message, RecordNumber, SourceName, TimeGenerated, TimeWritten, Type, User FROM Win32_NTLogEvent WHERE Logfile = "Security" AND RecordNumber > 110155" (10.13.18.59: Security) 01-03-2018 17:03:31.713 +0800 DEBUG ExecProcessor - message from ""E:\Program Files\Splunk\bin\splunk-wmi.exe"" WMI - Executing query wql="SELECT Category, CategoryString, ComputerName, EventCode, EventIdentifier, EventType, Logfile, Message, RecordNumber, SourceName, TimeGenerated, TimeWritten, Type, User FROM Win32_NTLogEvent WHERE Logfile = "Security" AND RecordNumber > 110155" (10.13.18.59: Security) From the above debug log, the message for Security event shows the rec-id hits the limit of unsigned int type: 01-03-2018 17:03:31.713 +0800 DEBUG ExecProcessor - message from ""E:\Program Files\Splunk\bin\splunk-wmi.exe"" WMI - getEventLogWql: DESC: chk=4294967295, low=110155, hi=4294967295 (10.13.18.59: Security) The limit of unsigned int is 4294967295 (0xffffffff). Here's a link for the limitation of Microsoft WQL api: https://social.technet.microsoft.com/Forums/windowsserver/en-US/78e6d555-0f5d-4def-92d5-14d3ad6ee558/eventlog-problems-with-query As stated in the link, the rec ids are limited to 32 bit unsigned int. WMI does not work if the rec id goes beyond that point. You can try to configure event logs to smaller sizes so that the logs rotate before rec id hitting the limit. ... View more