Here is a capture of traffic between the Splunk Modular Input for Kafka and HEC. The quote before C3M is causing the json not to lint. This was generated from heka-flood with the ascii-only option on. Unfortunately, I did not capture the traffic from heka-flood to kafka or kafka to Splunk Modular Input for Kafka. Splunk Modular Input for Kafka fails consistently with traffic from heka-flood or the kafka provided consumer test. POST /services/collector HTTP/1.1^M Authorization: Splunk C28170BC-215A-44A8-8229-B05D708ECDD3^M Content-Length: 425^M Content-Type: application/json; charset=UTF-8^M Host: localhost:8088^M Connection: Keep-Alive^M User-Agent: Apache-HttpAsyncClient/4.1 (Java/1.7.0_101)^M ^M {"event":"Tue Jul 19 20:34:48 UTC 2016 name=kafka_msg_received event_id= msg_body={\"logger\":\"\",\"type\":\"logfile\",\"tags\":[\"ih\",\"services\"],\"host\":\"pod.test.heka-sq05z\",\"message\":\"hekabench: pod.test.heka-sq05z - `U)G\\7{Cu| KYih@_E@@TU*:|cP\\"C3M <_M.r:B58eRNT![IbBoN(.7[6J,:w\/1\/+c+rZ\\#;Pi5>\\-\\yjo*7us<8.LHP4vb'0`3W\"}","source":"kbrown2-kafka-logs","time":"1468960488565","sourcetype":"infinitehome"}è<8e><8e>W-º^H^@m^A^@^@m^A^@^@^@^@^@^@^@^@^@^@^@^@^@^@^H^@E^@^A_-Á@^@@^F^MÖ^?^@^@^A^?^@^@^A^_<98>©äàVý<89>=ï <80>^X^D^@ÿS^@^@^A^A^H ^EÍ´<84>^EÍ´<80>HTTP/1.1 400 Bad Request^M Date: Tue, 19 Jul 2016 20:34:48 GMT^M Content-Type: application/json; charset=UTF-8^M X-Content-Type-Options: nosniff^M Content-Length: 64^M Connection: Keep-Alive^M X-Frame-Options: SAMEORIGIN^M Server: Splunkd^M ^M {"text":"Invalid data format","code":6,"invalid-event-number":0} ... View more

[root@splunk-xl kafka_mod]# pwd /opt/splunk/var/lib/splunk/modinputs/kafka_mod [root@splunk-xl kafka_mod]# ls -la total 128 drwx------ 2 root root 4096 Aug 1 21:05 . drwx------. 4 root root 4096 Aug 1 19:08 .. -rw------- 1 root root 82 Aug 1 21:05 a2Fma2EwOjkwOTIsa2Fma2ExOjkwOTIsa2Fma2EyOjkwOTI=_logs_0 -rw------- 1 root root 82 Aug 1 21:05 a2Fma2EwOjkwOTIsa2Fma2ExOjkwOTIsa2Fma2EyOjkwOTI=_logs_1 -rw------- 1 root root 83 Aug 1 21:05 a2Fma2EwOjkwOTIsa2Fma2ExOjkwOTIsa2Fma2EyOjkwOTI=_logs_10 [...] [root@splunk-xl kafka_mod]# cat a2Fma2EwOjkwOTIsa2Fma2ExOjkwOTIsa2Fma2EyOjkwOTI=_logs_0 {"kafka_partition": 0, "kafka_topic": "logs", "kafka_partition_offset": 306627169} ... View more

Memory Limits - did that Data Output to Splunk - getting BAD REQUEST from HEC because it doesn't handle some characters properly (fails every time when generating messages with heka-flood in ascii mode or kafka-consumer-perf-test) so cannot test with HEC Multi Processes - did that Getting 400 msg/ sec vs 17000 with splunk add-on I have not seen any case studies which show what kind of performance can be expected with either Kafka consumer. ... View more

Please answer: With the Splunk Add-on for Kafka, where can I see the consumer lag? I should not have mentioned The Splunk Modular Input for Kafka in the same question. ... View more