Getting Data In
Highlighted

s3 add-on adding data

Engager

Splunk is installed
s3 add-on is installed.

I have gone to data inputs and added the amazon s3 bucket we wanted.
However, for the life of me I can't figure out how to actually pull in the data from s3 to do anything with.

Can anyone tell me what I need to do?

Cheers.

Tags (3)
Highlighted

Re: s3 add-on adding data

Splunk Employee
Splunk Employee

Here is what you need to do:

$SPLUNK_HOME/etc/apps/s3/README/inputs.conf.spec
inputs.conf.spec

[s3://]

key_id =
* This is Amazon key ID.

secret_key =
* This is the secret key.

http://docs.splunk.com/Documentation/Splunk/6.0.1/AdvancedDev/ModInputsExample

0 Karma
Highlighted

Re: s3 add-on adding data

Explorer

I have, from a Splunk python script, developed coldToFrozenPlusS3Uplaod.py that encrypts and uploads frozen buckets to S3.

It can be found here: https://github.com/marboxvel/Encrypt-upload-archived-Splunk-buckets

0 Karma
Highlighted

Re: s3 add-on adding data

Path Finder

Hi Sbuttio,
Im using your script here is sample one getting issues

import sys, os, gzip, shutil, subprocess, random, gnupg
import boto
import datetime
import time
import tarfile

applyLogging is a python script named applyLogging.py that exists at the same level of this script.

If the file applyLogging.py doesn't exist where this file is located, the import statement will fail.

sys.path.append(script_path)
import applyLogging

CHANGE THIS TO YOUR ACTUAL ARCHIVE DIRECTORY!!!

ARCHIVE_DIR = "/splunk/index/splunk/archiveindex"

ARCHIVEDIR = os.path.join(os.getenv('SPLUNKHOME'), 'frozenarchive')

scriptpath = '/opt/splunk/etc/apps/Encrypt-upload-archived-Splunk-buckets-master/coldToFrozenPlusS3Uplaod.py'
log
file_path = '/opt/splunk/var/log/splunk/'

gnuhomedir = '' #where the gpg directory is. For example /home/s3/.gnupg/

gnuhomedir = /home/splunkq/.gnupg

reciepient_email = '' #the email the gpg uses to encrypt the files

reciepient_email = xxyxy@gmail.com

Enabling the logging system

logger = applyLogging.getmodulelogger(appname='SplunkArchive',filepath=logfilepath)

all are same

0 Karma
Highlighted

Re: s3 add-on adding data

Path Finder

Can you please help me

0 Karma