topic Re: calculate storage and remove part of log in Splunk Enterprise

Is it possible to calculate storage and remove part of log?

SplunkySplunk — Tue, 21 Feb 2023 15:10:21 GMT

Hello

Is it possible to calculate the storage that part of log is taking ?
I have a log file that contains a message that i want to calculate the storage it takes

after getting the numbers, is it possible to exclude it from index ?

Thanks

Re: calculate storage and remove part of log

jotne — Sun, 19 Feb 2023 12:28:59 GMT

You can do some like this:

<your search> | eval len=len(_raw) | stats sum(len)

It will give you size in bytes. This may also take some time depending of how much data you have how long back you do search.

To remove the same data, you can delete them. You need to give admin the delete role first:

<your search> | delete

Re: calculate storage and remove part of log

PickleRick — Sun, 19 Feb 2023 13:49:13 GMT

Depends on what you need it for and when.

One thing is length of the raw data itself. And that was already shown. Another thing is how much space in splunk indexes is consuned by storing this particular event. And for this there is no clear single answer as it depends on how good the event (and whole bucket data) compresses, if you extract any additional indexed files and so on.

About excluding from indexes - again depends on what you really want. If you want to really prevent long events from getting indexed (why not just truncate them?), you'd have to do some check using ingest-time eval and then route to nullqueue based on the result of that check. If, however, you just want to not show long events in the results just filter your events with

| where len(_raw)<120

or whatever limit you want.

Don't use delete command. It has its very limited uses but generally shouldn't be used.