All Apps and Add-ons

Specifying sourcetype for a Cloud Storage Bucket Input Using the Splunk Add-on for Google Cloud Platform


I have set up a Cloud Storage Bucket input using the Splunk Add-on for Google Cloud Platform.

I do not see a way to easily configure the sourcetype.  I added a sourcetype line in google_cloud_storage_buckets.conf, but when restarting Splunk it showed "Invalid key in stanza [______] in /opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/local/google_cloud_storage_buckets.conf, line 8: sourcetype (value: ______)"


The only way I was able to get my desired sourcetype set was to edit the python script which is setting it (which is not a long-term solution as I may need to add more inputs of different sourcetypes later)

Even then, after updating the script to set the desired sourcetype, the config I set up for this sourcetype (regarding event breaking) is not working - even though it does work properly when adding the data from a local copy of the file in the GCP bucket.  Is there anything the script is doing to the files from the bucket that is mucking with their format.... or a reason why the settings in props.conf for the sourcetype are not being applied to the files being pulled down from the GCP bucket?

Labels (1)
0 Karma

New Member

It's hard to set up a Cloud Storage Bucket input using the Splunk Add-on for the Google Cloud Platform. You need to create a new bucket, which doesn't happen automatically. My friend from gave me a "Pro tip:" you can use "gs://<BUCKET_NAME>/" as the bucket name without having to worry about it being too long as Splunk will ignore that part of the URL. Another way is to use REST API calls but first create a bucket with gsutil mb gs://<BUCKET_NAME>/.

0 Karma


@gn694 there is no sourcetype setting in add-on it supports only following storage bucket conf, Have you tried configuring them and test it. I assume modinputs python script will allocate one of the sourcetype supported by add-on. 

google:gcp:buckets:csvdata ->	CSV contents of objects present in the Cloud Storage Bucket	
google:gcp:buckets:jsondata ->	JSON contents of objects present in the Cloud Storage Bucket	
google:gcp:buckets:xmldata ->	XML contents of objects present in the Cloud Storage Bucket	
google:gcp:buckets:data	-> Generic source type for the contents of other file extensions. For example, txt, avro, and parquet	None
google_credentials_name = stanza name in google_credentials.conf
google_project = the project name
bucket_name = bucket name
index =
polling_interval = 


An upvote would be appreciated and Accept solution if it helps!

0 Karma
Get Updates on the Splunk Community!

Build Scalable Security While Moving to Cloud - Guide From Clayton Homes

 Clayton Homes faced the increased challenge of strengthening their security posture as they went through ...

Mission Control | Explore the latest release of Splunk Mission Control (2.3)

We’re happy to announce the release of Mission Control 2.3 which includes several new and exciting features ...

Cloud Platform | Migrating your Splunk Cloud deployment to Python 3.7

Python 2.7, the last release of Python 2, reached End of Life back on January 1, 2020. As part of our larger ...