Kafka Messaging Modular Input: Messages are consumed, but why do they never show up in the index?

[ahmed_khalifa]

1. Modular input is installed
2. It is connected properly and I can see in the logs of splunk and Kafka that the connection happens.
3. No error messages (except for the Failed to load class "org.slf4j.impl.StaticLoggerBinder". message which I read it should be ignored).
4. The Messa ges are consumed ( I do run the kafka consumer offset checker and it shows that the splunk group id is at the latest offset alway).
5. Nothing shows on the index ?
6. Tried changing index to no avail
7. No Errors in splunkd.log
8. Created my own handler (copied the default handler) and added it.
9. Put 'System.out.println' on the "HandleMessage' and "SetParams" methods but I do not see them in the logs
10. Thought maybe this is not the best way to log so I actually put code to create outside files and put the log into them but the files are never created

Any ideas? I am at the end of my attempts here. Is there a specific format for the message? What am I missing here?

[petehmrc]

Tried a custom one, as well as csv, but no joy. Is there a sourcetype that will simply ingest whatever data it gets 'as is'?

[petehmrc]

What would be quite helpful is... could you send us a sample inputs.conf from someone who has this already up and running? It would maybe then help us to compare and contrast and rule out anything obvious

[Damien_Dallimor]

There is nothing in your inputs.conf that looks out of place.

How are your searching ? correct index ? correct time range ?

Any errors ? search "index=_internal ExecProcessor error kafka.py"

[petehmrc]

Have tried index=main, searching on the eventsource and the host... nada. Time range is All time. _internal index ExecProcessor error query just yeilds three messages regarding slf4j, which we have read is expected.

[petehmrc]

The weird thing is that the modular input is definitely reading through the consumer group as we can see the offset increasing. Just not reaching the index.

[Damien_Dallimor]

What is your Splunk architecture and where/how have you installed the Mod Input ?

[petehmrc]

Running splunk enterprise on my local machine with the default license. In order to install we simply copy over the app and the inputs.conf into the required folder before starting splunk.

[petehmrc]

We've been trying to debug this. It looks like the call on line 377 of ModularInput.StateCheckerThread which makes a rest call to Splunk. This call seems to be using HTTPS, and as we are developing against a local instance the connection setup fails as we have the default SSL cert on the server.

This does not appear in the logs anywhere (AFAICT) either as an error or an exception trace and it doesn't seem to be possible to disable this check. As a result the Kafka connector seems to assume it has been disabled and throws all the messages away.

Is there anything we can do to get this working?