Does Splunk have an easier way to get FileSize to human readable format?

[tb5821]

I'm surprised splunk doesn't have an easier way to get a human readable format by passing it the field you want it to work its magic on al-la du -h

So here's my question, I have a field called fs which is file size but its in bytes. I want to convert that to the proper Kb,Mb,Gb format... whats the best way to do this?

[jameswatts]

How about just installing Humanize?

Convert numbers, bytes, and timestamps into fuzzy, human-friendly units! Using the humanize library from https://github.com/jmoiron/humanize

Here are some example searches:
index=_internal | head | eval foo=random() | humanize command=naturalsize foo
index=_internal earliest=-7h | bin _time span=10m | eval foo=_time | stats first(foo) as foo by _time | humanize command=naturaltime foo

https://splunkbase.splunk.com/app/3104/

[jpvlsmv]

Ancient thread necropsy, but here's a better macro (IMO). It's ugly but it works just like the -h option on many GNU tools.
Usage:

| eval readable_size=`readable(size)`

Definition: (as seen in Settings -> Adv Search -> Search macros -> new:

if( $num$ < 1024, tostring($num$), if ( (floor($num$/pow(1024,floor(log($num$,1024))))) < 10
     , ( (tostring((floor($num$/pow(1024,floor(log($num$,1024)))))) + ".") + tostring(round((($num$/pow(1024,floor(log($num$,1024))))-(floor($num$/pow(1024,floor(log($num$,1024))))))*10))) + (substr("KMGTPEZY",floor(log($num$,1024)),1))
     , ( tostring((floor($num$/pow(1024,floor(log($num$,1024)))))) + (substr("KMGTPEZY",floor(log($num$,1024)),1)) )
   ) )

Not an eval-based definition (unchecked)
Arguments: num
Validation Expression: !isnum($num$)
Validation Error Message: Numeric value required

My key observation for the algorithm is that the log base 1024 will give you the "scale"-- KB or PB or whatever, by dropping the fractional part (i.e. log_10(5.6MB) = 2 -> M).

In working on this, I used meaningful names and replace-all'd them to fundamental eval functions. Here's the pseudocode:

if $num$ < 1024:
  printf("%4d", $num$)
else
  if $num$ reduces to a single digit
    # print in the form x.yS
    printf( "%d.%d%c", whole_part(reduction), 1st digit of frac_part(reduction), KMGTPEZY suffix appropriate for this scale
  else # This is actually the most common case.  The result is just the whole part of the reduction and the suffix
    printf("%3d%s", whole_part(reduction), suffix)

Hope this helps somebody
--Joe

[alacercogitatus · Accepted solution]

I have this setup. You can put it into $SPLUNK_HOME/etc/system/local/props.conf and it should work. You may want to have your field called "b" or "bytes" for it to work, or change the config to match your fields. This will auto-calculate kilos, megas and gigas automatically.

[host::*]
priority = 100
EVAL-kilobytes = if(isnotnull(kilobytes),kilobytes,bytes/1024)
EVAL-megabytes = if(isnotnull(megabytes),megabytes,bytes/1024/1024)
EVAL-gigabytes = if(isnotnull(gigabytes),gigabytes,bytes/1024/1024/1024)
EVAL-kb = if(isnotnull(kb),kb,b/1024)
EVAL-mb = if(isnotnull(mb),mb,b/1024/1024)
EVAL-gb = if(isnotnull(gb),gb,b/1024/1024/1024)

You can also use a macro (in the Search UI):
Manager -> Advanced Search -> Search Macros

Name: resize(1)
Arguments: bytes
definition = eval kilobytes = $bytes$/1024| eval megabytes=kilobytes/1024|eval gigabytes/1024

Then you can do your_search | resize(fs)

[alacercogitatus]

weird. anyway, glad that worked for you!

[tb5821]

Found the issue, the definition needs to be:

eval kilobytes=($fs$/1024) | eval megabytes=kilobytes/1024 |eval gigabytes=megabytes/1024

[tb5821]

now getting:
Error in 'eval' command: Failed to parse the provided arguments. Usage: eval dest_key = expression

[alacercogitatus]

you need the "backticks". They are the key under Esc on the keyboard.

[tb5821]

Error in 'SearchParser': Missing a search command before '''.

...| 'resize(fs)'

[alacercogitatus]

See my edit. You will need backticks around "resize(fs)". The answer system stripped them for me.

[tb5821]

Can I do it in search? I don't have access to anything but the search UI

[tfujita_splunk]

This could be solution for you.

https://community.splunk.com/t5/Dashboards-Visualizations/Smart-conversion-of-large-numbers-to-human...

| makeresults count=35
```THIS SECTION IS JUST CREATING SAMPLE VALUES.```
| streamstats count as digit
| eval val=pow(10,digit-1), val=val+random()%val
| foreach bytes [eval <<FIELD>>=val]
| table digit val bytes
| fieldformat val=tostring(val,"commas")

```THE FOLLOWING LINES MAY BE WHAT ACHIEVES THE FORMAT YOU ARE LOOKING FOR.```
| fieldformat bytes=printf("% 10s",printf("%.2f",round(bytes/pow(1024,if(bytes=0,0,floor(min(log(bytes,1024),10)))),2)).case(bytes=0 OR log(bytes,1024)<1,"B ", log(bytes,1024)<2,"KiB", log(bytes,1024)<3,"MiB", log(bytes,1024)<4,"GiB", log(bytes,1024)<5,"TiB", log(bytes,1024)<6,"PiB", log(bytes,1024)<7,"EiB", log(bytes,1024)<8,"ZiB", log(bytes,1024)<9,"YiB", log(bytes,1024)<10,"RiB", log(bytes,1024)<11,"QiB", 1=1, "QiB"))

If you can install app or ask admin on your to install app,

installing add-on Numeral system macros for Splunk enables you to use macros numeral_binary_symbol(1) or numeral_binary_symbol(2).

Example

| makeresults count=35
```THIS SECTION IS JUST CREATING SAMPLE VALUES.```
| streamstats count as digit
| eval val=pow(10,digit-1), val=val+random()%val
| foreach bytes [eval <<FIELD>>=val]
| table digit val bytes
| fieldformat val=tostring(val,"commas")

```THE FOLLOWING LINES MAY BE WHAT ACHIEVES THE FORMAT YOU ARE LOOKING FOR.```
| fieldformat bytes=printf("% 10s",`numeral_binary_symbol(bytes,2)`)

Numeral system macros for Splunk

https://splunkbase.splunk.com/app/6595

Usage:

How to convert a large number to string with expressions of long and short scales, or neither.

https://community.splunk.com/t5/Splunk-Search/How-to-convert-a-large-number-to-string-with-expressio...