All Apps and Add-ons

Amazon Kinesis Modular Input: How to resolve "com.splunk.modinput.kinesis.KinesisModularInput$MessageReceiver.connect(Unknown Source)" errors?

Explorer

Hi -
We have download the Amazon Kinesis Modular Input add-on from : https://splunkbase.splunk.com/app/1856/#/details
Unfortunately we are unable to get it to work. We are getting the following issues. Can you guide us in the right direction?

After configuring the Kinesis data input in our on-prem server, we are getting many errors like this :

11-14-2016 11:32:26.551 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py"        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.(Worker.java:89)
11-14-2016 11:32:26.551 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py"        at com.splunk.modinput.kinesis.KinesisModularInput$MessageReceiver.connect(Unknown Source)

We are trying to determine the issue? Can you assist ? Here is how our $SPLUNKHOME/etc/apps/SplunkTAaws/local/awskinesis_tasks.conf file looks like :

[test]
account = AWS
encoding =
index = aws
init_stream_position = TRIM_HORIZON
region = us-east-1
sourcetype = aws:kinesis
stream_names = test

can you tell us what we are missing here? or what we are doing wrong? Thank you

0 Karma

Ultra Champion

The Amazon Kinesis Modular Input has nothing to do with SplunkTAaws , so it's an irrelevant comparison.

Regarding errors running Amazon Kinesis Modular Input......

Have you followed the docs correctly ? The troubleshooting steps are useful ie: correct Java version ?
What does your inputs.conf stanza look like ?

0 Karma

Explorer

Thanks Damien for the reply , below is how the inputs.conf stanza look like . We need to figure out what is this error means and how to correct it . We are getting data from other sources of AWS thru the AWS app, so it is not the firewall issue. this is the error:
11-14-2016 17:19:42.194 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py" at com.splunk.modinput.kinesis.KinesisMo
dularInput.startMessageReceiverThread(Unknown Source)

vi inputs.conf

[kinesis://name]

connection settings

appname = test
stream
name = test
kinesis_endpoint = https://kinesis.us-east-1.amazonaws.com

LATEST or TRIM_HORIZON

initialstreamposition = TRIM_HORIZON

awsaccesskeyid = some key secret
aws
secretaccesskey = some key secret

message reader settings

backofftimemillis =
numretries =
checkpoint
interval_millis =

message handler

messagehandlerimpl =
messagehandlerparams =

additional startup settings

additionaljvmpropertys =

data output

One of [stdout | hec ]. Defaults to stdout.

output_type = stdout

For hec(HTTP Event Collector) output

hec_port =

Defaults to 1

hecpoolsize =
hec
token =

1 | 0

hec_https = 0

1 | 0

hecbatchmode = 0

numeric value

hecmaxbatchsizebytes =

numeric value

hecmaxbatchsizeevents =

in milliseconds

hecmaxinactivetimebeforebatchflush =
index = aws
sourcetype = aws:kinesis

0 Karma

Explorer

Thank you Damien , Yes you are correct, JAVA was not in the path : now I put the java in the path of user that is running the splunk (which is root) , now I am no longer getting the above error , I am getting new error in the message : Unable to initialize modular input "kinesis" defined inside the app "kinesis_ta": Introspecting scheme=kinesis: script running failed (exited with code 1). Got this when I ran with scheme

/opt/splunk/etc/apps/SplunkTAaws/bin/awskinesis.py --scheme
Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk
TAaws/bin/awskinesis.py", line 9, in
from splunktalib.common import log
File "/opt/splunk/etc/apps/SplunkTAaws/bin/splunktalib/common/log.py", line 11, in
from splunktalib.splunkplatform import makesplunkhomepath
File "/opt/splunk/etc/apps/Splunk
TAaws/bin/splunktalib/splunkplatform.py", line 86
res[section] = {item[0]: item[1] for item in parser.items(section)}
^
SyntaxError: invalid syntax

I will go thru the trouble shoot docs again .
echo $JAVAHOME
/usr/lib/jvm/jre-1.7.0-openjdk.x86
64/bin/java
[root@server~]# echo $PATH
/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/root/bin:JAVA_HOME

0 Karma

Ultra Champion

You didn't answer this :

Have you followed the docs correctly ? The troubleshooting steps are useful ie: correct Java version ?

If you are the same person that emailed me a log dump , I am going to presume you do not have the correct Java version.

0 Karma