All Apps and Add-ons

Specifying sourcetype for a Cloud Storage Bucket Input Using the Splunk Add-on for Google Cloud Platform

gn694
Communicator

I have set up a Cloud Storage Bucket input using the Splunk Add-on for Google Cloud Platform.

I do not see a way to easily configure the sourcetype.  I added a sourcetype line in google_cloud_storage_buckets.conf, but when restarting Splunk it showed "Invalid key in stanza [______] in /opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/local/google_cloud_storage_buckets.conf, line 8: sourcetype (value: ______)"

 

The only way I was able to get my desired sourcetype set was to edit the python script which is setting it (which is not a long-term solution as I may need to add more inputs of different sourcetypes later)

Even then, after updating the script to set the desired sourcetype, the config I set up for this sourcetype (regarding event breaking) is not working - even though it does work properly when adding the data from a local copy of the file in the GCP bucket.  Is there anything the script is doing to the files from the bucket that is mucking with their format.... or a reason why the settings in props.conf for the sourcetype are not being applied to the files being pulled down from the GCP bucket?

Labels (1)
0 Karma

abdelmajidint
New Member

It's hard to set up a Cloud Storage Bucket input using the Splunk Add-on for the Google Cloud Platform. You need to create a new bucket, which doesn't happen automatically. My friend from https://fastupload.io/en gave me a "Pro tip:" you can use "gs://<BUCKET_NAME>/" as the bucket name without having to worry about it being too long as Splunk will ignore that part of the URL. Another way is to use REST API calls but first create a bucket with gsutil mb gs://<BUCKET_NAME>/.

0 Karma

venkatasri
SplunkTrust
SplunkTrust

@gn694 there is no sourcetype setting in add-on it supports only following storage bucket conf, Have you tried configuring them and test it. I assume modinputs python script will allocate one of the sourcetype supported by add-on. 

google:gcp:buckets:csvdata ->	CSV contents of objects present in the Cloud Storage Bucket	
google:gcp:buckets:jsondata ->	JSON contents of objects present in the Cloud Storage Bucket	
google:gcp:buckets:xmldata ->	XML contents of objects present in the Cloud Storage Bucket	
google:gcp:buckets:data	-> Generic source type for the contents of other file extensions. For example, txt, avro, and parquet	None
[<name>]
google_credentials_name = stanza name in google_credentials.conf
google_project = the project name
bucket_name = bucket name
index =
polling_interval = 

 ----

An upvote would be appreciated and Accept solution if it helps!

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...