All Apps and Add-ons

Specifying sourcetype for a Cloud Storage Bucket Input Using the Splunk Add-on for Google Cloud Platform

gn694
Communicator

I have set up a Cloud Storage Bucket input using the Splunk Add-on for Google Cloud Platform.

I do not see a way to easily configure the sourcetype.  I added a sourcetype line in google_cloud_storage_buckets.conf, but when restarting Splunk it showed "Invalid key in stanza [______] in /opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/local/google_cloud_storage_buckets.conf, line 8: sourcetype (value: ______)"

 

The only way I was able to get my desired sourcetype set was to edit the python script which is setting it (which is not a long-term solution as I may need to add more inputs of different sourcetypes later)

Even then, after updating the script to set the desired sourcetype, the config I set up for this sourcetype (regarding event breaking) is not working - even though it does work properly when adding the data from a local copy of the file in the GCP bucket.  Is there anything the script is doing to the files from the bucket that is mucking with their format.... or a reason why the settings in props.conf for the sourcetype are not being applied to the files being pulled down from the GCP bucket?

Labels (1)
0 Karma

venkatasri
SplunkTrust
SplunkTrust

@gn694 there is no sourcetype setting in add-on it supports only following storage bucket conf, Have you tried configuring them and test it. I assume modinputs python script will allocate one of the sourcetype supported by add-on. 

google:gcp:buckets:csvdata ->	CSV contents of objects present in the Cloud Storage Bucket	
google:gcp:buckets:jsondata ->	JSON contents of objects present in the Cloud Storage Bucket	
google:gcp:buckets:xmldata ->	XML contents of objects present in the Cloud Storage Bucket	
google:gcp:buckets:data	-> Generic source type for the contents of other file extensions. For example, txt, avro, and parquet	None
[<name>]
google_credentials_name = stanza name in google_credentials.conf
google_project = the project name
bucket_name = bucket name
index =
polling_interval = 

 ----

An upvote would be appreciated and Accept solution if it helps!

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!