Amazon Kinesis Modular Input: How to resolve "com.splunk.modinput.kinesis.KinesisModularInput$MessageReceiver.connect(Unknown Source)" errors?

[jmajumdar]

Hi -
We have download the Amazon Kinesis Modular Input add-on from : https://splunkbase.splunk.com/app/1856/#/details
Unfortunately we are unable to get it to work. We are getting the following issues. Can you guide us in the right direction?

After configuring the Kinesis data input in our on-prem server, we are getting many errors like this :

11-14-2016 11:32:26.551 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py"        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.(Worker.java:89)
11-14-2016 11:32:26.551 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py"        at com.splunk.modinput.kinesis.KinesisModularInput$MessageReceiver.connect(Unknown Source)

We are trying to determine the issue? Can you assist ? Here is how our $SPLUNK_HOME/etc/apps/Splunk_TA_aws/local/aws_kinesis_tasks.conf file looks like :

[test]
account = AWS
encoding =
index = aws
init_stream_position = TRIM_HORIZON
region = us-east-1
sourcetype = aws:kinesis
stream_names = test

can you tell us what we are missing here? or what we are doing wrong? Thank you

[Damien_Dallimor]

The Amazon Kinesis Modular Input has nothing to do with Splunk_TA_aws , so it's an irrelevant comparison.

Regarding errors running Amazon Kinesis Modular Input......

Have you followed the docs correctly ? The troubleshooting steps are useful ie: correct Java version ?
What does your inputs.conf stanza look like ?

[jmajumdar]

Thanks Damien for the reply , below is how the inputs.conf stanza look like . We need to figure out what is this error means and how to correct it . We are getting data from other sources of AWS thru the AWS app, so it is not the firewall issue. this is the error:
11-14-2016 17:19:42.194 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py" at com.splunk.modinput.kinesis.KinesisMo
dularInput.startMessageReceiverThread(Unknown Source)

vi inputs.conf

[kinesis://name]

connection settings

app_name = test
stream_name = test
kinesis_endpoint = https://kinesis.us-east-1.amazonaws.com

LATEST or TRIM_HORIZON

initial_stream_position = TRIM_HORIZON

aws_access_key_id = some key secret
aws_secret_access_key = some key secret

message reader settings

backoff_time_millis =
num_retries =
checkpoint_interval_millis =

message handler

message_handler_impl =
message_handler_params =

additional startup settings

additional_jvm_propertys =

data output

One of [stdout | hec ]. Defaults to stdout.

output_type = stdout

For hec(HTTP Event Collector) output

hec_port =

Defaults to 1

hec_poolsize =
hec_token =

1 | 0

hec_https = 0

1 | 0

hec_batch_mode = 0

numeric value

hec_max_batch_size_bytes =

numeric value

hec_max_batch_size_events =

in milliseconds

hec_max_inactive_time_before_batch_flush =
index = aws
sourcetype = aws:kinesis

[jmajumdar]

Thank you Damien , Yes you are correct, JAVA was not in the path : now I put the java in the path of user that is running the splunk (which is root) , now I am no longer getting the above error , I am getting new error in the message : Unable to initialize modular input "kinesis" defined inside the app "kinesis_ta": Introspecting scheme=kinesis: script running failed (exited with code 1). Got this when I ran with scheme

/opt/splunk/etc/apps/Splunk_TA_aws/bin/aws_kinesis.py --scheme
Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/aws_kinesis.py", line 9, in
from splunktalib.common import log
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunktalib/common/log.py", line 11, in
from splunktalib.splunk_platform import make_splunkhome_path
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunktalib/splunk_platform.py", line 86
res[section] = {item[0]: item[1] for item in parser.items(section)}
^
SyntaxError: invalid syntax

I will go thru the trouble shoot docs again .
echo $JAVA_HOME
/usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java
[root@server~]# echo $PATH
/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/root/bin:JAVA_HOME

[Damien_Dallimor]

You didn't answer this :

Have you followed the docs correctly ? The troubleshooting steps are useful ie: correct Java version ?

If you are the same person that emailed me a log dump , I am going to presume you do not have the correct Java version.