Hello Everyone,
I am using the below in indexes.conf file, but the script never got executed instead the frozen files are deleted.
frozenTimePeriodInSecs = 1382400
coldToFrozenScript = "/opt/splunk/bin/python" "/opt/splunk/etc/apps/atl-cold-to-frozen-s3/bin/coldToFrozenS3.py"
Do I need to create sub folders on S3 bucket? because manually executing the script, without the sub folders works.
However if I manually execute the script something like "python coldtofrozens3.py arguments", its copying the data to s3.
Also I have tried coldToFrozenDir=, which is working.
But the coldtofrozen script never works. I am unable to test the script, as i am losing the frozen data.
Can some one please help/suggest what is going wrong here.
We have upgraded splunk indexer to 7.x and enabled boot start as Splunk user after changing the owner to splunk for $SPLUNKHOME
And finally able to automate cold to frozen s3.
The following worked as splunk user:
./splunk cmd /opt/splunk/bin/python /opt/splunk/etc/apps/atl-cold-to-frozen-s3/bin/coldToFrozenS3.py /opt/splunkindexes//cold/testbk_rb_6D93D52F011A
and we added the following under /opt/splunk/etc/slave-apps/_cluster/local
indexes.conf:
frozenTimePeriodInSecs = 1382400
coldToFrozenScript = "/opt/splunk/bin/python" "/opt/splunk/etc/apps/atl-cold-to-frozen-s3/bin/coldToFrozenS3.py"
following by splunk indexer restart
We have upgraded splunk indexer to 7.x and enabled boot start as Splunk user after changing the owner to splunk for $SPLUNKHOME
And finally able to automate cold to frozen s3.
The following worked as splunk user:
./splunk cmd /opt/splunk/bin/python /opt/splunk/etc/apps/atl-cold-to-frozen-s3/bin/coldToFrozenS3.py /opt/splunkindexes//cold/testbk_rb_6D93D52F011A
and we added the following under /opt/splunk/etc/slave-apps/_cluster/local
indexes.conf:
frozenTimePeriodInSecs = 1382400
coldToFrozenScript = "/opt/splunk/bin/python" "/opt/splunk/etc/apps/atl-cold-to-frozen-s3/bin/coldToFrozenS3.py"
following by splunk indexer restart
We have decided to upgrade splunk to 7.0, as splunk is unable to execute aws. We couldn’t resolve permission issues.
It doesn't seem likely that the Splunk version would cause (or remedy) the issue you were having with permissions.
Splunk 7 has support for S3, so that’s the reason for upgrade. We are not going to use the coldtofrozen script anymore
Please let us know how you did this. Thanks!!
I have used cold to frozen s3 script to achieve this. The steps are mentioned in the 'answer' section above. However we have upgraded to splunk 7.x as coldtofrozens3 script did not work in splunk 6.x
I'd also be interested to know how you got on with this. It's currently an unsupported feature and I'm unclear how to set it up.
@ basu42002 ,
Can you explain how you set this up?
Thanks
Have you searched your internal splunk logs for any reference to the script? This may show you that the script is or is not running, and if it's producing any errors.
Did you restart Splunk after making the change to indexes.conf?
12-14-2017 23:44:43.621 +0000 INFO IndexWriter - idx=, Initializing, params='[300,period=60,frozenTimePeriodInSecs=1382400,coldToFrozenScript="/opt/splunk/bin/python" "/opt/splunk/etc/apps/atl-cold-to-frozen-s3/bin/coldToFrozenS3.py"
this is from the splunkd.log, I have restarted splunkindexer.
The changes are reflected, I see this script is being referenced in the log after restart.
However the script never executed.
From the logs, I see
ERROR BucketMover - coldToFrozenScript /usr/bin/python: /opt/splunk/lib/libssl.so.1.0.0: version `OPENSSL_1.0.0' not found (required by /usr/bin/python)
Is it possible to help in resolving this problem.
When you test the app manually you should use Splunk's python interpreter:
./bin/splunk cmd python <script name>
Thank you, if I run it manually it is working, but otherwise it doesn't work when splunk runs automatically:
./splunk cmd /opt/splunk/bin/python /opt/splunk/etc/apps/atl-cold-to-frozen-s3/bin/coldToFrozenS3.py /opt/splunkindexes/xxx/cold/testbk_bkt
Looks like this is permission issue , as splunk user it is not able to execute was
can't open file '/usr/local/bin/aws': [Errno 13] Permission denied
Were you able to solve the permission issue?
When you run it manually, are you using the same account that splunk runs as? If not, check the permissions.
I don't see any issues with permissions, also I am using the same account that splunk is running as.
Any suggestions please.
I have tried with and without double quotes:
"/opt/splunk/bin/python" "/opt/splunk/etc/apps/atl-cold-to-frozen-s3/bin/coldToFrozenS3.py"
But it did not work.