Infos over REST API from CSV Files

bardill — Tue, 26 Mar 2019 10:58:33 GMT

Is there a default possibility for a user
get a list of all CSV files for an app
to determine the size of the individual CSV files

Re: Infos over REST API from CSV Files

bardill — Tue, 26 Mar 2019 11:09:11 GMT

in The Moment i do this so
https://server:0000/services/data/lookup-table-files
but i have no size.

Re: Infos over REST API from CSV Files

somesoni2 — Tue, 26 Mar 2019 14:12:41 GMT

AFAIK, the REST endpoint for lookup table files doesn't show size information for the lookup.

Re: Infos over REST API from CSV Files

woodcock — Wed, 27 Mar 2019 06:12:11 GMT

No, but you can calculate it with a search in a dashboard panel like this one:

<panel>
  <title>Lookup table details (beware "Bundle too large" replication errors) - WARNING: may take a long time to complete; sizes and percentages are APPROXIMATE</title>
  <table>
    <title>The warning will be this: Bundle Replication: Problem replicating config (bundle) to search peer ' <hostname>:8089 ', HTTP response code 413 (HTTP/1.1 413 Content-Length of <size here> too large (maximum is 838860800)). Content-Length of <size here> too large (maximum is 838860800) (Unknown write error)</title>
    <search>
      <query>|rest/services/data/lookup-table-files splunk_server=local
| search eai:acl.app="$env:app$"
| rename dispatch.* AS *
| rename eai:acl.* AS *
| map maxsearches=99 search="
| inputlookup $$title$$
| rename COMMENT1of3 AS \"Some field names have single-quotes which will cause this error:\"
| rename COMMENT3of3 AS \"{map}: Failed to parse templatized search for field 'Bad Field's Name Here'\"
| rename COMMENT3of3 AS \"So rename those fields before we process them to replace ' with _\"
| rename *'*'*'*'* AS *_*_*_*_*, *'*'*'* AS *_*_*_*, *'*'* AS *_*_*, *'* AS *_*
| eval T3MpJuNk_bytes=0, T3MpJuNk_cols=0, T3MpJuNk_field_names=\",\"
| foreach _*
    [ eval T3MpJuNk_bytes = T3MpJuNk_bytes + coalesce(len('<<FIELD>>'), 0)
    | eval T3MpJuNk_cols = T3MpJuNk_cols + 1
    | eval T3MpJuNk_field_names = T3MpJuNk_field_names . \"<<FIELD>>\"]
| rename _* AS *, T3MpJuNk_* AS _T3MpJuNk_*
| foreach *
    [ eval _T3MpJuNk_bytes = _T3MpJuNk_bytes + coalesce(len('<<FIELD>>'), 0)
    | eval _T3MpJuNk_cols = _T3MpJuNk_cols + 1
    | eval _T3MpJuNk_field_names = _T3MpJuNk_field_names . \"<<FIELD>>\"]
| rename COMMENT AS \"Account for the commas, too!\"
| eval bytes = bytes + (cols - 1)
| stats sum(_T3MpJuNk_bytes) AS bytes count AS lines first(_T3MpJuNk_cols) AS cols first(_T3MpJuNk_field_names) AS field_names
| rename COMMENT AS \"Account for the header line, too!\"
| eval bytes = bytes + (len(field_names) - 1)
| eval title=\"$$title$$\"
| eval owner=\"$$owner$$\"" 
| eval bytes = coalesce(bytes, 0)
| addtotals row=false col=true labelfield=title label="$TOTAL_FIELD_VALUE$" 
| eval "bytes/line" = if(title=="$TOTAL_FIELD_VALUE$", "N/A", round(coalesce(bytes/lines, 0), 2))
| eval owner = if(title=="$TOTAL_FIELD_VALUE$", "N/A", owner)
| eval cols  = if(title=="$TOTAL_FIELD_VALUE$", "N/A", coalesce(cols, "N/A"))
| eval MB = round(bytes / 1024 / 1024, 2)
| eval bundlePct = round(100 * bytes / 838860800, 2)
| eval status=case(
   title=="$TOTAL_FIELD_VALUE$", if((bundlePct < 90),                         "OK", "DANGEROUS TERRITORY"),
   true(),                       if((bundlePct < 25 AND lines < 10000000), "OK", "Consider KVStore"))
| sort 0 - bytes
| table title status bundlePct owner bytes MB lines cols bytes*line
| eval _drilldown  = if(title=="$TOTAL_FIELD_VALUE$", "*", title)</query>
      <earliest>0</earliest>
      <latest></latest>
      <sampleRatio>1</sampleRatio>
    </search>
    <option name="count">100</option>
    <option name="dataOverlayMode">none</option>
    <option name="drilldown">cell</option>
    <option name="percentagesRow">false</option>
    <option name="rowNumbers">false</option>
    <option name="totalsRow">false</option>
    <option name="wrap">true</option>
    <drilldown target="_blank">
      <link>/manager/$env_app$/data/lookup-table-files?app=$env:app$&amp;app_only=1&amp;count=100&amp;search=$row._drilldown$</link>
    </drilldown>
  </table>
</panel>

Re: Infos over REST API from CSV Files

usd0872 — Tue, 29 Sep 2020 23:53:03 GMT

Awesome!

I had tried something similar before but couldn't figure out how to get the size of a whole row. Didn't think about going through each field separately.
Quite resource intensive, though, as you mention in the code itself.

Doesn't work well for me when using the REST endpoint

|rest/services/data/lookup-table-files splunk_server=local

This one works better for me:

|rest /servicesNS/$env:user$/$env:app$/data/lookup-table-files splunk_server=local

And all the $TOTAL_FIELD_VALUE$ cause problems in my dashboard as I use a form, replacing it with #TOTAL_FIELD_VALUE# works fine.

Still, awesome!

topic Re: Infos over REST API from CSV Files in Getting Data In

Infos over REST API from CSV Files

Re: Infos over REST API from CSV Files

Re: Infos over REST API from CSV Files

Re: Infos over REST API from CSV Files

Re: Infos over REST API from CSV Files