Export logs from Splunk Cloud to GCP Pub/Sub

[bhagyashriyan]

Hi Splunkers,

I would like to export logs (raw/csv) out of Splunk cloud periodically to send it to gcp pub/sub. How can I achieve this. Appreciate ideas here.

[tscroggins]

Hi @bhagyashriyan,

This is a challenge in Splunk Cloud at scale. If you manage data flow at the source or integration layers, you may prefer to tee your data to both Splunk Cloud and Google Cloud Pub/Sub at one of those layers.

Otherwise, you can execute relatively simple saved searches in Splunk Cloud using an external client and stream the output to Google Cloud Pub/Sub.

For example:

index=foo | fields - _raw | table *

will return _time and all fields available at search time from the search's execution context (user and app). Note that results are returned in _time reversed order, newest to oldest.

In Google Cloud, you can use a combination of low cost services to periodically execute the search via the Splunk Cloud REST API in batches over fixed _time intervals and stream the results to Google Cloud Pub/Sub.

[bhagyashriyan]

Hi @tscroggins ,

Could you please elaborate on running searches in GCP using rest api?

Thanks

[tscroggins]

Hi @bhagyashriyan,

Any Google Cloud solution that allows you to submit HTTP requests, parse HTTP responses, and publish messages to a Google Cloud Pub/Sub topic can use the Splunk Cloud REST API, assuming REST API access is enabled and granted to the source Google Cloud egress address(es).

You can execute Splunk searches using the Splunk Cloud REST API search/jobs endpoint. Here's a simple Bash command-line example using curl, jq, and gcloud:

curl -s -u username:password https://<deployment-name>.splunkcloud.com:8089/services/search/jobs -d search="| makeresults count=10" -d exec_mode=oneshot -d output_mode=json | jq -r '.results[] | tojson | @sh' | while IFS= read message; do gcloud pubsub topics publish mytopic --message=${message}; done

Replace <deployment-name> with your Splunk Cloud stack name and mytopic with your Google Cloud Pub/Sub topic name. This example assumes gcloud is already correctly configured. You can also use Splunk Cloud access tokens instead of username/password authentication.

See https://docs.splunk.com/Documentation/SplunkCloud/latest/RESTTUT/RESTandCloud  and https://docs.splunk.com/Documentation/SplunkCloud/latest/RESTREF/RESTsearch#search.2Fjobs for more information.

I don't work in Google Cloud day to day, so I recommend browsing the Google Cloud documentation for inspiration.