Getting Data In

Streaming CSV output

iroddis
New Member

I'm querying very large data sets from Splunk several times a day. During days with a lot of data, I'll get an OOM on the search head. Some notes:

  • Pulled via the export REST endpoint
  • Only contain distributed streaming commands (until the end)
  • Last distributed streaming command is "fields" to limit data coming back from the search head
  • Last item is a "table" command to get CSV output
  • Already querying in the smallest reasonable timeframe for business requirements

The problem is that "table" is not a streaming command. Is there a command that will take the streaming results, format them as a CSV (header row, data rows), and pass them on without buffering the entire result set on the search head? The expected columns and column order are already defined, so it should just be a filter on the search head to stream the header, then format each record as it's returned from the indexers.

0 Karma

anmolpatel
Builder

https://docs.splunk.com/Documentation/SplunkCloud/8.0.2001/SearchReference/Table
look at Command Type sub heading. The alternative is the fields command.

Note: the table is just displaying the results in a tabular format. If you want to output the result into a csv file look at @rich7177 comment.

0 Karma

iroddis
New Member

Thanks, I read that reference. Despite what the document says, using just the fields command and output_mode=csv returns data with three columns: _time, _system, and _raw. The _raw is a brace-enclosed key-value pair list of the data I'm after. That's not a useful CSV format: I need the key-value pairs as a CSV.

0 Karma

Richfez
SplunkTrust
SplunkTrust

To where do the results have to be passed on to?

The command table doesn't really generate CSV output, so if you are getting CSV output it's via some other mechanism.

And have you looked at outputcsv or outputlookup depending on where it is you are sending this data to?

0 Karma

iroddis
New Member

Sorry, I should have mentioned that the REST query specifies output_mode=csv . That, in conjunction with the table command, returns CSV-formatted data.

outputcsv isn't very useful for me: I'm a user of the system, not an admin, and don't have access to the search heads to retrieve locally generated files.

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...