topic Re: s3 add-on adding data in Getting Data In

s3 add-on adding data

jonsaenz — Wed, 20 Feb 2013 23:13:50 GMT

Splunk is installed
s3 add-on is installed.

I have gone to data inputs and added the amazon s3 bucket we wanted.
However, for the life of me I can't figure out how to actually pull in the data from s3 to do anything with.

Can anyone tell me what I need to do?

Cheers.

Re: s3 add-on adding data

khourihan_splun — Mon, 27 Jan 2014 18:27:12 GMT

Here is what you need to do:

$SPLUNK_HOME/etc/apps/s3/README/inputs.conf.spec
inputs.conf.spec

[s3://]

key_id =
* This is Amazon key ID.

secret_key =
* This is the secret key.

http://docs.splunk.com/Documentation/Splunk/6.0.1/AdvancedDev/ModInputsExample

Re: s3 add-on adding data

sbutto — Fri, 31 Aug 2018 18:55:11 GMT

I have, from a Splunk python script, developed coldToFrozenPlusS3Uplaod.py that encrypts and uploads frozen buckets to S3.

It can be found here: https://github.com/marboxvel/Encrypt-upload-archived-Splunk-buckets

Re: s3 add-on adding data

Splunk_rocks — Tue, 29 Sep 2020 23:12:04 GMT

Hi Sbuttio,
Im using your script here is sample one getting issues

import sys, os, gzip, shutil, subprocess, random, gnupg
import boto
import datetime
import time
import tarfile

applyLogging is a python script named applyLogging.py that exists at the same level of this script.

If the file applyLogging.py doesn't exist where this file is located, the import statement will fail.

sys.path.append(script_path)
import applyLogging

CHANGE THIS TO YOUR ACTUAL ARCHIVE DIRECTORY!!!

ARCHIVE_DIR = "/splunk/index/splunk/archiveindex"

ARCHIVE_DIR = os.path.join(os.getenv('SPLUNK_HOME'), 'frozenarchive')

script_path = '/opt/splunk/etc/apps/Encrypt-upload-archived-Splunk-buckets-master/coldToFrozenPlusS3Uplaod.py'
log_file_path = '/opt/splunk/var/log/splunk/'

gnu_home_dir = '' #where the gpg directory is. For example /home/s3/.gnupg/

gnu_home_dir = /home/splunkq/.gnupg

reciepient_email = '' #the email the gpg uses to encrypt the files

reciepient_email = xxyxy@gmail.com

Enabling the logging system

logger = applyLogging.get_module_logger(app_name='SplunkArchive',file_path=log_file_path)

all are same

Re: s3 add-on adding data

Splunk_rocks — Sun, 10 Feb 2019 02:07:27 GMT

Can you please help me