Getting Data In

Streaming CSV output

iroddis
New Member

I'm querying very large data sets from Splunk several times a day. During days with a lot of data, I'll get an OOM on the search head. Some notes:

  • Pulled via the export REST endpoint
  • Only contain distributed streaming commands (until the end)
  • Last distributed streaming command is "fields" to limit data coming back from the search head
  • Last item is a "table" command to get CSV output
  • Already querying in the smallest reasonable timeframe for business requirements

The problem is that "table" is not a streaming command. Is there a command that will take the streaming results, format them as a CSV (header row, data rows), and pass them on without buffering the entire result set on the search head? The expected columns and column order are already defined, so it should just be a filter on the search head to stream the header, then format each record as it's returned from the indexers.

0 Karma

anmolpatel
Builder

https://docs.splunk.com/Documentation/SplunkCloud/8.0.2001/SearchReference/Table
look at Command Type sub heading. The alternative is the fields command.

Note: the table is just displaying the results in a tabular format. If you want to output the result into a csv file look at @rich7177 comment.

0 Karma

iroddis
New Member

Thanks, I read that reference. Despite what the document says, using just the fields command and output_mode=csv returns data with three columns: _time, _system, and _raw. The _raw is a brace-enclosed key-value pair list of the data I'm after. That's not a useful CSV format: I need the key-value pairs as a CSV.

0 Karma

Richfez
SplunkTrust
SplunkTrust

To where do the results have to be passed on to?

The command table doesn't really generate CSV output, so if you are getting CSV output it's via some other mechanism.

And have you looked at outputcsv or outputlookup depending on where it is you are sending this data to?

0 Karma

iroddis
New Member

Sorry, I should have mentioned that the REST query specifies output_mode=csv . That, in conjunction with the table command, returns CSV-formatted data.

outputcsv isn't very useful for me: I'm a user of the system, not an admin, and don't have access to the search heads to retrieve locally generated files.

0 Karma
Get Updates on the Splunk Community!

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...