Hi -
We have download the Amazon Kinesis Modular Input add-on from : https://splunkbase.splunk.com/app/1856/#/details
Unfortunately we are unable to get it to work. We are getting the following issues. Can you guide us in the right direction?
After configuring the Kinesis data input in our on-prem server, we are getting many errors like this :
11-14-2016 11:32:26.551 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py" at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.(Worker.java:89)
11-14-2016 11:32:26.551 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py" at com.splunk.modinput.kinesis.KinesisModularInput$MessageReceiver.connect(Unknown Source)
We are trying to determine the issue? Can you assist ? Here is how our $SPLUNK_HOME/etc/apps/Splunk_TA_aws/local/aws_kinesis_tasks.conf file looks like :
[test]
account = AWS
encoding =
index = aws
init_stream_position = TRIM_HORIZON
region = us-east-1
sourcetype = aws:kinesis
stream_names = test
can you tell us what we are missing here? or what we are doing wrong? Thank you
The Amazon Kinesis Modular Input has nothing to do with Splunk_TA_aws , so it's an irrelevant comparison.
Regarding errors running Amazon Kinesis Modular Input......
Have you followed the docs correctly ? The troubleshooting steps are useful ie: correct Java version ?
What does your inputs.conf stanza look like ?
Thanks Damien for the reply , below is how the inputs.conf stanza look like . We need to figure out what is this error means and how to correct it . We are getting data from other sources of AWS thru the AWS app, so it is not the firewall issue. this is the error:
11-14-2016 17:19:42.194 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py" at com.splunk.modinput.kinesis.KinesisMo
dularInput.startMessageReceiverThread(Unknown Source)
vi inputs.conf
[kinesis://name]
app_name = test
stream_name = test
kinesis_endpoint = https://kinesis.us-east-1.amazonaws.com
initial_stream_position = TRIM_HORIZON
aws_access_key_id = some key secret
aws_secret_access_key = some key secret
backoff_time_millis =
num_retries =
checkpoint_interval_millis =
message_handler_impl =
message_handler_params =
additional_jvm_propertys =
output_type = stdout
hec_port =
hec_poolsize =
hec_token =
hec_https = 0
hec_batch_mode = 0
hec_max_batch_size_bytes =
hec_max_batch_size_events =
hec_max_inactive_time_before_batch_flush =
index = aws
sourcetype = aws:kinesis
Thank you Damien , Yes you are correct, JAVA was not in the path : now I put the java in the path of user that is running the splunk (which is root) , now I am no longer getting the above error , I am getting new error in the message : Unable to initialize modular input "kinesis" defined inside the app "kinesis_ta": Introspecting scheme=kinesis: script running failed (exited with code 1). Got this when I ran with scheme
/opt/splunk/etc/apps/Splunk_TA_aws/bin/aws_kinesis.py --scheme
Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/aws_kinesis.py", line 9, in
from splunktalib.common import log
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunktalib/common/log.py", line 11, in
from splunktalib.splunk_platform import make_splunkhome_path
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunktalib/splunk_platform.py", line 86
res[section] = {item[0]: item[1] for item in parser.items(section)}
^
SyntaxError: invalid syntax
I will go thru the trouble shoot docs again .
echo $JAVA_HOME
/usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java
[root@server~]# echo $PATH
/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/root/bin:JAVA_HOME
You didn't answer this :
Have you followed the docs correctly ? The troubleshooting steps are useful ie: correct Java version ?
If you are the same person that emailed me a log dump , I am going to presume you do not have the correct Java version.