All Apps and Add-ons

Any quick startup guide to do end-to-end testing from kafka to Splunk.

Splunk Employee
Splunk Employee

If you got any cheat sheet that I can setup kafka to send event to Splunk, that would help.

0 Karma

0 Karma

Splunk Employee
Splunk Employee

I did that kafka setup before and below is based on centos 7.

  1. yum upgrade

  2. yum install java-1.8.0-openjdk

  3. download kafka
    Here is the splunkbase link but it will point you to Github

  4. Once you downloaded kafka, untar it and rename the directory

    tar -xzf kafka_2.11-2.1.0.tgz
    mv kafka_2.11-2.0.0 kafka

  5. Start kafka zookeeper and server

    cd kafka
    bin/ config/ > zookeeper.log &
    bin/ config/ > kafka_server.log &

  6. Create kafka topic

    bin/ --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

    Below command can be used to list/delete topic as well.

    bin/ --list --zookeeper localhost:2181
    bin/ --zookeeper localhost:2181 --delete --topic test

  7. Send some event to topic "test" so that you can get it from kafka connect later

    bin/ --broker-list localhost:9092 --topic test

    this is a test message

  8. Check the worker properties file. Make sure the following settings are set correctly.
  9. Start Splunk-kafka-connect

    bin/ config/ > Kafka_connect.log &

  10. Create HEC connector in Splunk

  11. Create splunk-kafka-connect task

    curl http://localhost:8083/connectors -X POST -H "Content-Type: application/json" -d '{
    "name": "test-single-event",
    "config": {
    "connector.class": "com.splunk.kafka.connect.SplunkSinkConnector",
    "tasks.max": "1",
    "splunk.sources": "test_kafka_event",
    "splunk.indexes": "kafka_event",
    "splunk.hec.uri": "https://localhost:8088",
    "splunk.hec.token": "26faccb6-a0af-45d6-996e-7df97afb81fd",
    "splunk.hec.raw": "false",
    "splunk.hec.ack.enabled": "false",
    "splunk.hec.ack.poll.interval": "10",
    "splunk.hec.ack.poll.threads": "1",
    "splunk.hec.ssl.validate.certs": "false",
    "splunk.hec.http.keepalive": "true",
    "": "4",
    "": "8",
    "splunk.hec.max.batch.size": "500",
    "splunk.hec.threads": "1",
    "splunk.hec.event.timeout": "300",
    "splunk.hec.socket.timeout": "120",
    "": "true"

0 Karma