Streaming CSV output

[iroddis]

I'm querying very large data sets from Splunk several times a day. During days with a lot of data, I'll get an OOM on the search head. Some notes:

• Pulled via the export REST endpoint
• Only contain distributed streaming commands (until the end)
• Last distributed streaming command is "fields" to limit data coming back from the search head
• Last item is a "table" command to get CSV output
• Already querying in the smallest reasonable timeframe for business requirements

The problem is that "table" is not a streaming command. Is there a command that will take the streaming results, format them as a CSV (header row, data rows), and pass them on without buffering the entire result set on the search head? The expected columns and column order are already defined, so it should just be a filter on the search head to stream the header, then format each record as it's returned from the indexers.

[anmolpatel]

https://docs.splunk.com/Documentation/SplunkCloud/8.0.2001/SearchReference/Table
look at Command Type sub heading. The alternative is the fields command.

Note: the table is just displaying the results in a tabular format. If you want to output the result into a csv file look at @rich7177 comment.

[iroddis]

Thanks, I read that reference. Despite what the document says, using just the fields command and output_mode=csv returns data with three columns: _time, _system, and _raw. The _raw is a brace-enclosed key-value pair list of the data I'm after. That's not a useful CSV format: I need the key-value pairs as a CSV.

[Richfez]

To where do the results have to be passed on to?

The command table doesn't really generate CSV output, so if you are getting CSV output it's via some other mechanism.

And have you looked at outputcsv or outputlookup depending on where it is you are sending this data to?

[iroddis]

Sorry, I should have mentioned that the REST query specifies output_mode=csv . That, in conjunction with the table command, returns CSV-formatted data.

outputcsv isn't very useful for me: I'm a user of the system, not an admin, and don't have access to the search heads to retrieve locally generated files.