Splunk Search

Does Splunk have an easier way to get FileSize to human readable format?

tb5821
Communicator

I'm surprised splunk doesn't have an easier way to get a human readable format by passing it the field you want it to work its magic on al-la du -h

So here's my question, I have a field called fs which is file size but its in bytes. I want to convert that to the proper Kb,Mb,Gb format... whats the best way to do this?

Labels (1)
Tags (4)
0 Karma
1 Solution

alacercogitatus
SplunkTrust
SplunkTrust

I have this setup. You can put it into $SPLUNK_HOME/etc/system/local/props.conf and it should work. You may want to have your field called "b" or "bytes" for it to work, or change the config to match your fields. This will auto-calculate kilos, megas and gigas automatically.

[host::*]
priority = 100
EVAL-kilobytes = if(isnotnull(kilobytes),kilobytes,bytes/1024)
EVAL-megabytes = if(isnotnull(megabytes),megabytes,bytes/1024/1024)
EVAL-gigabytes = if(isnotnull(gigabytes),gigabytes,bytes/1024/1024/1024)
EVAL-kb = if(isnotnull(kb),kb,b/1024)
EVAL-mb = if(isnotnull(mb),mb,b/1024/1024)
EVAL-gb = if(isnotnull(gb),gb,b/1024/1024/1024)

You can also use a macro (in the Search UI):
Manager -> Advanced Search -> Search Macros

Name: resize(1)
Arguments: bytes
definition = eval kilobytes = $bytes$/1024| eval megabytes=kilobytes/1024|eval gigabytes/1024

Then you can do your_search | resize(fs)

View solution in original post

jameswatts
Explorer

How about just installing Humanize?

Convert numbers, bytes, and timestamps into fuzzy, human-friendly units! Using the humanize library from https://github.com/jmoiron/humanize

Here are some example searches:
index=_internal | head | eval foo=random() | humanize command=naturalsize foo
index=_internal earliest=-7h | bin _time span=10m | eval foo=_time | stats first(foo) as foo by _time | humanize command=naturaltime foo

https://splunkbase.splunk.com/app/3104/

0 Karma

jpvlsmv
Path Finder

Ancient thread necropsy, but here's a better macro (IMO). It's ugly but it works just like the -h option on many GNU tools.
Usage:

| eval readable_size=`readable(size)`

Definition: (as seen in Settings -> Adv Search -> Search macros -> new:

if( $num$ < 1024, tostring($num$), if ( (floor($num$/pow(1024,floor(log($num$,1024))))) < 10
     , ( (tostring((floor($num$/pow(1024,floor(log($num$,1024)))))) + ".") + tostring(round((($num$/pow(1024,floor(log($num$,1024))))-(floor($num$/pow(1024,floor(log($num$,1024))))))*10))) + (substr("KMGTPEZY",floor(log($num$,1024)),1))
     , ( tostring((floor($num$/pow(1024,floor(log($num$,1024)))))) + (substr("KMGTPEZY",floor(log($num$,1024)),1)) )
   ) )

Not an eval-based definition (unchecked)
Arguments: num
Validation Expression: !isnum($num$)
Validation Error Message: Numeric value required

My key observation for the algorithm is that the log base 1024 will give you the "scale"-- KB or PB or whatever, by dropping the fractional part (i.e. log_10(5.6MB) = 2 -> M).

In working on this, I used meaningful names and replace-all'd them to fundamental eval functions. Here's the pseudocode:

if $num$ < 1024:
  printf("%4d", $num$)
else
  if $num$ reduces to a single digit
    # print in the form x.yS
    printf( "%d.%d%c", whole_part(reduction), 1st digit of frac_part(reduction), KMGTPEZY suffix appropriate for this scale
  else # This is actually the most common case.  The result is just the whole part of the reduction and the suffix
    printf("%3d%s", whole_part(reduction), suffix)

Hope this helps somebody
--Joe

alacercogitatus
SplunkTrust
SplunkTrust

I have this setup. You can put it into $SPLUNK_HOME/etc/system/local/props.conf and it should work. You may want to have your field called "b" or "bytes" for it to work, or change the config to match your fields. This will auto-calculate kilos, megas and gigas automatically.

[host::*]
priority = 100
EVAL-kilobytes = if(isnotnull(kilobytes),kilobytes,bytes/1024)
EVAL-megabytes = if(isnotnull(megabytes),megabytes,bytes/1024/1024)
EVAL-gigabytes = if(isnotnull(gigabytes),gigabytes,bytes/1024/1024/1024)
EVAL-kb = if(isnotnull(kb),kb,b/1024)
EVAL-mb = if(isnotnull(mb),mb,b/1024/1024)
EVAL-gb = if(isnotnull(gb),gb,b/1024/1024/1024)

You can also use a macro (in the Search UI):
Manager -> Advanced Search -> Search Macros

Name: resize(1)
Arguments: bytes
definition = eval kilobytes = $bytes$/1024| eval megabytes=kilobytes/1024|eval gigabytes/1024

Then you can do your_search | resize(fs)

alacercogitatus
SplunkTrust
SplunkTrust

weird. anyway, glad that worked for you!

0 Karma

tb5821
Communicator

Found the issue, the definition needs to be:

eval kilobytes=($fs$/1024) | eval megabytes=kilobytes/1024 |eval gigabytes=megabytes/1024

0 Karma

tb5821
Communicator

now getting:
Error in 'eval' command: Failed to parse the provided arguments. Usage: eval dest_key = expression

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

you need the "backticks". They are the key under Esc on the keyboard.

0 Karma

tb5821
Communicator

Error in 'SearchParser': Missing a search command before '''.

...| 'resize(fs)'

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

See my edit. You will need backticks around "resize(fs)". The answer system stripped them for me.

0 Karma

tb5821
Communicator

Can I do it in search? I don't have access to anything but the search UI

tfujita_splunk
Splunk Employee
Splunk Employee

This could be solution for you.

https://community.splunk.com/t5/Dashboards-Visualizations/Smart-conversion-of-large-numbers-to-human...

| makeresults count=35
```THIS SECTION IS JUST CREATING SAMPLE VALUES.```
| streamstats count as digit
| eval val=pow(10,digit-1), val=val+random()%val
| foreach bytes [eval <<FIELD>>=val]
| table digit val bytes
| fieldformat val=tostring(val,"commas")

```THE FOLLOWING LINES MAY BE WHAT ACHIEVES THE FORMAT YOU ARE LOOKING FOR.```
| fieldformat bytes=printf("% 10s",printf("%.2f",round(bytes/pow(1024,if(bytes=0,0,floor(min(log(bytes,1024),10)))),2)).case(bytes=0 OR log(bytes,1024)<1,"B ", log(bytes,1024)<2,"KiB", log(bytes,1024)<3,"MiB", log(bytes,1024)<4,"GiB", log(bytes,1024)<5,"TiB", log(bytes,1024)<6,"PiB", log(bytes,1024)<7,"EiB", log(bytes,1024)<8,"ZiB", log(bytes,1024)<9,"YiB", log(bytes,1024)<10,"RiB", log(bytes,1024)<11,"QiB", 1=1, "QiB"))

 

If you can install app or ask admin on your to install app,

installing add-on Numeral system macros for Splunk enables you to use macros numeral_binary_symbol(1) or numeral_binary_symbol(2).

Example

| makeresults count=35
```THIS SECTION IS JUST CREATING SAMPLE VALUES.```
| streamstats count as digit
| eval val=pow(10,digit-1), val=val+random()%val
| foreach bytes [eval <<FIELD>>=val]
| table digit val bytes
| fieldformat val=tostring(val,"commas")

```THE FOLLOWING LINES MAY BE WHAT ACHIEVES THE FORMAT YOU ARE LOOKING FOR.```
| fieldformat bytes=printf("% 10s",`numeral_binary_symbol(bytes,2)`)

 

Numeral system macros for Splunk

https://splunkbase.splunk.com/app/6595


Usage:

How to convert a large number to string with expressions of long and short scales, or neither.

https://community.splunk.com/t5/Splunk-Search/How-to-convert-a-large-number-to-string-with-expressio...

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...