Splunk Search

Best way to mass and bulk export searched fields and events for data warehousing.

suhprano
Path Finder

It appears that there are several ways to bulk export data from Splunk.
-rest API
-search query option: outputcsv
-cli search option: -output csv -maxout
-sendemail

I'm loading data into Splunk and there's a lot (in the millions) of records/events I would like to export as a flat file, ideally to a CSV. I plan on using these flat files as a source in our ETL jobs to populate our data warehouse.
The Splunk searches grab only certain fields I extracted from the inputs.

I know there are some hard limits to certain procedures regarding how many records we can extract per search.

What would be the best way to bulk export this data from Splunk?
I expect the number of records/events from search results should be in the millions.

Thanks!

Tags (2)

lguinn2
Legend

There is another choice: you can set your indexer(s) to forward data to "3rd party systems." This will resend the raw data. See http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Forwarddatatothird-partysystemsd for details.

You could get around the "millions of events" problems with the other methods, such as outputcsv. Just export more frequently (like every hour or even more often).

lguinn2
Legend

I am not sure what the upper bound is -- I don't know that there is a limit for outoutcsv, since it simply writes to a file on the splunk server.

0 Karma

suhprano
Path Finder

Thanks for the info. I was hoping I could extract fields out of the raw and only bulk export certain columns out of the indexed data. I don't think forwarding raw data to a 3rd party system gives me much room for customizing how the flat file should look.
Maybe exporting more frequently is the way to go with this large amount of data. Do you know what the limit is for outputcsv? Maybe the cli search is better suited since it has the -maxoutput 0 option.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...