Specifying sourcetype for a Cloud Storage Bucket Input Using the Splunk Add-on for Google Cloud Platform

[gn694]

I have set up a Cloud Storage Bucket input using the Splunk Add-on for Google Cloud Platform.

I do not see a way to easily configure the sourcetype.  I added a sourcetype line in google_cloud_storage_buckets.conf, but when restarting Splunk it showed "Invalid key in stanza [______] in /opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/local/google_cloud_storage_buckets.conf, line 8: sourcetype (value: ______)"

The only way I was able to get my desired sourcetype set was to edit the python script which is setting it (which is not a long-term solution as I may need to add more inputs of different sourcetypes later)

Even then, after updating the script to set the desired sourcetype, the config I set up for this sourcetype (regarding event breaking) is not working - even though it does work properly when adding the data from a local copy of the file in the GCP bucket.  Is there anything the script is doing to the files from the bucket that is mucking with their format.... or a reason why the settings in props.conf for the sourcetype are not being applied to the files being pulled down from the GCP bucket?

[abdelmajidint]

It's hard to set up a Cloud Storage Bucket input using the Splunk Add-on for the Google Cloud Platform. You need to create a new bucket, which doesn't happen automatically. My friend from https://fastupload.io/en gave me a "Pro tip:" you can use "gs://<BUCKET_NAME>/" as the bucket name without having to worry about it being too long as Splunk will ignore that part of the URL. Another way is to use REST API calls but first create a bucket with gsutil mb gs://<BUCKET_NAME>/.

[venkatasri]

@gn694 there is no sourcetype setting in add-on it supports only following storage bucket conf, Have you tried configuring them and test it. I assume modinputs python script will allocate one of the sourcetype supported by add-on. 

google:gcp:buckets:csvdata ->	CSV contents of objects present in the Cloud Storage Bucket	
google:gcp:buckets:jsondata ->	JSON contents of objects present in the Cloud Storage Bucket	
google:gcp:buckets:xmldata ->	XML contents of objects present in the Cloud Storage Bucket	
google:gcp:buckets:data	-> Generic source type for the contents of other file extensions. For example, txt, avro, and parquet	None

[<name>]
google_credentials_name = stanza name in google_credentials.conf
google_project = the project name
bucket_name = bucket name
index =
polling_interval = 

 ----

An upvote would be appreciated and Accept solution if it helps!