Getting Data In

Infos over REST API from CSV Files

bardill
Explorer

Is there a default possibility for a user
get a list of all CSV files for an app
to determine the size of the individual CSV files

0 Karma

woodcock
Esteemed Legend

No, but you can calculate it with a search in a dashboard panel like this one:

<panel>
  <title>Lookup table details (beware "Bundle too large" replication errors) - WARNING: may take a long time to complete; sizes and percentages are APPROXIMATE</title>
  <table>
    <title>The warning will be this: Bundle Replication: Problem replicating config (bundle) to search peer ' <hostname>:8089 ', HTTP response code 413 (HTTP/1.1 413 Content-Length of <size here> too large (maximum is 838860800)). Content-Length of <size here> too large (maximum is 838860800) (Unknown write error)</title>
    <search>
      <query>|rest/services/data/lookup-table-files splunk_server=local
| search eai:acl.app="$env:app$"
| rename dispatch.* AS *
| rename eai:acl.* AS *
| map maxsearches=99 search="
| inputlookup $$title$$
| rename COMMENT1of3 AS \"Some field names have single-quotes which will cause this error:\"
| rename COMMENT3of3 AS \"{map}: Failed to parse templatized search for field 'Bad Field's Name Here'\"
| rename COMMENT3of3 AS \"So rename those fields before we process them to replace ' with _\"
| rename *'*'*'*'* AS *_*_*_*_*, *'*'*'* AS *_*_*_*, *'*'* AS *_*_*, *'* AS *_*
| eval T3MpJuNk_bytes=0, T3MpJuNk_cols=0, T3MpJuNk_field_names=\",\"
| foreach _*
    [ eval T3MpJuNk_bytes = T3MpJuNk_bytes + coalesce(len('<<FIELD>>'), 0)
    | eval T3MpJuNk_cols = T3MpJuNk_cols + 1
    | eval T3MpJuNk_field_names = T3MpJuNk_field_names . \"<<FIELD>>\"]
| rename _* AS *, T3MpJuNk_* AS _T3MpJuNk_*
| foreach *
    [ eval _T3MpJuNk_bytes = _T3MpJuNk_bytes + coalesce(len('<<FIELD>>'), 0)
    | eval _T3MpJuNk_cols = _T3MpJuNk_cols + 1
    | eval _T3MpJuNk_field_names = _T3MpJuNk_field_names . \"<<FIELD>>\"]
| rename COMMENT AS \"Account for the commas, too!\"
| eval bytes = bytes + (cols - 1)
| stats sum(_T3MpJuNk_bytes) AS bytes count AS lines first(_T3MpJuNk_cols) AS cols first(_T3MpJuNk_field_names) AS field_names
| rename COMMENT AS \"Account for the header line, too!\"
| eval bytes = bytes + (len(field_names) - 1)
| eval title=\"$$title$$\"
| eval owner=\"$$owner$$\"" 
| eval bytes = coalesce(bytes, 0)
| addtotals row=false col=true labelfield=title label="$TOTAL_FIELD_VALUE$" 
| eval "bytes/line" = if(title=="$TOTAL_FIELD_VALUE$", "N/A", round(coalesce(bytes/lines, 0), 2))
| eval owner = if(title=="$TOTAL_FIELD_VALUE$", "N/A", owner)
| eval cols  = if(title=="$TOTAL_FIELD_VALUE$", "N/A", coalesce(cols, "N/A"))
| eval MB = round(bytes / 1024 / 1024, 2)
| eval bundlePct = round(100 * bytes / 838860800, 2)
| eval status=case(
   title=="$TOTAL_FIELD_VALUE$", if((bundlePct < 90),                         "OK", "DANGEROUS TERRITORY"),
   true(),                       if((bundlePct < 25 AND lines < 10000000), "OK", "Consider KVStore"))
| sort 0 - bytes
| table title status bundlePct owner bytes MB lines cols bytes*line
| eval _drilldown  = if(title=="$TOTAL_FIELD_VALUE$", "*", title)</query>
      <earliest>0</earliest>
      <latest></latest>
      <sampleRatio>1</sampleRatio>
    </search>
    <option name="count">100</option>
    <option name="dataOverlayMode">none</option>
    <option name="drilldown">cell</option>
    <option name="percentagesRow">false</option>
    <option name="rowNumbers">false</option>
    <option name="totalsRow">false</option>
    <option name="wrap">true</option>
    <drilldown target="_blank">
      <link>/manager/$env_app$/data/lookup-table-files?app=$env:app$&amp;app_only=1&amp;count=100&amp;search=$row._drilldown$</link>
    </drilldown>
  </table>
</panel>
0 Karma

usd0872
Path Finder

Awesome!

I had tried something similar before but couldn't figure out how to get the size of a whole row. Didn't think about going through each field separately.
Quite resource intensive, though, as you mention in the code itself.

Doesn't work well for me when using the REST endpoint

|rest/services/data/lookup-table-files splunk_server=local

This one works better for me:

|rest /servicesNS/$env:user$/$env:app$/data/lookup-table-files splunk_server=local

And all the $TOTAL_FIELD_VALUE$ cause problems in my dashboard as I use a form, replacing it with #TOTAL_FIELD_VALUE# works fine.

Still, awesome!

0 Karma

somesoni2
Revered Legend

AFAIK, the REST endpoint for lookup table files doesn't show size information for the lookup.

0 Karma

bardill
Explorer

in The Moment i do this so
https://server:0000/services/data/lookup-table-files
but i have no size.

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...

Beyond Detection: How Splunk and Cisco Integrated Security Platforms Transform ...

Financial services organizations face an impossible equation: maintain 99.9% uptime for mission-critical ...

Customer success is front and center at .conf25

Hi Splunkers, If you are not able to be at .conf25 in person, you can still learn about all the latest news ...