topic Re: How is your volume usage calculated in Knowledge Management

How is your volume usage calculated

bwouters — Fri, 19 Jan 2018 12:58:13 GMT

Hi all

I managed to generate a log file which I would need to use to display certain graphs.
This logfile only increases a few times an hour and only with a few KB so I should be able to stay way below the 500MB trial license limit.
However, after a few hours into using Splunk, I'm almost reaching the limit ?!
I would expect that the Splunk Enterprise server processes the file once (or updates his DB when the logfile changes.. but is he processing the WHOLE again?) and puts the data in his DB, ready to be used for a search.

Can someone help me to determine where this 400+ MB is coming from? As far as I can see, I only configured 1 file to be monitored and everything else is disabled.. all other scripts and files.

Re: How is your volume usage calculated

mayurr98 — Fri, 19 Jan 2018 13:24:51 GMT

No, it does not duplicate any data by itself. So I do not think Splunk is processing the whole file again.Even if it processes Splunk checks first and last 256 bytes of the file if they are same then it does not index again.
Basically, when data gets indexed by an indexer, it counts towards your daily total.
See here for more:

http://docs.splunk.com/Documentation/Splunk/latest/Admin/HowSplunklicensingworks

One way to check this is event count and dig into Splunk data.

| tstats count where index=<your_index>

cross-check your file and see the line count.

let me know if this helps you!

Re: How is your volume usage calculated

horsefez — Fri, 19 Jan 2018 13:27:29 GMT

Hi,

could you give the config you have made for the monitor stanza in inputs.conf?

Re: How is your volume usage calculated

bwouters — Fri, 19 Jan 2018 13:38:47 GMT

/opt/splunk/etc/system/local$ cat inputs.conf 
[default]
host = SPLUNK

[monitor://$SPLUNK_HOME/etc/splunk.version]
disabled = 1

[monitor://$SPLUNK_HOME/var/log/splunk]
disabled = 1

[monitor://$SPLUNK_HOME/var/log/splunk/license_usage_summary.log]
disabled = 1

[batch://$SPLUNK_HOME/var/spool/splunk/...stash_new]
disabled = 1

[batch://$SPLUNK_HOME/var/spool/splunk]
disabled = 1

I used the web UI to do this.. I would expect that whatever I do using the web UI, is also put in the input.conf?

Re: How is your volume usage calculated

horsefez — Fri, 19 Jan 2018 13:50:43 GMT

Yes, you are right, but all those stanzas seem to still be disabled

Re: How is your volume usage calculated

bwouters — Fri, 19 Jan 2018 13:51:41 GMT

If I dig a bit deeper, I have like thousands of events per minute..
When I check what these events are, they are exactly the same
Line 1

2018-01-18 15:37:59,722 TRACE [HTTP worker thread 9] HttpInterface - [IP] [RequestId = ID] HTTP response send to IP
StatusCode = OK
Headers = 
  ResourceEvaluationCount: 1
  Cache-Control: max-age=7200
  Duration: 1 ms
  Content-Encoding: gzip
Body = <?xml version="1.0" encoding="utf-8"?>
<Session id="ID" xmlns="urn">
  <State>Created</State>
</Session>

Line 2

2018-01-18 15:37:59,722 TRACE [HTTP worker thread 9] HttpInterface - [IP] [RequestId = ID] HTTP response send to IP
StatusCode = OK
Headers = 
  ResourceEvaluationCount: 1
  Cache-Control: max-age=7200
  Duration: 1 ms
  Content-Encoding: gzip
Body = <?xml version="1.0" encoding="utf-8"?>
<Session id="ID" xmlns="urn">
  <State>Created</State>
</Session>

This data is put on 2 different lines and there are even more than 2 lines with exactly the same data..
Probably I did something wrong here? Or how can I finetune this to have it only on one line?

Re: How is your volume usage calculated

bwouters — Fri, 19 Jan 2018 13:54:13 GMT

Yes, because I wanted to see if these were the cause of it

Re: How is your volume usage calculated

mayurr98 — Fri, 19 Jan 2018 13:54:26 GMT

have you checked the same in file? check the file with the same time do the file also have multiple events?

Re: How is your volume usage calculated

bwouters — Fri, 19 Jan 2018 14:07:12 GMT

In the log file there is indeed twice the same logging but then I counted the amount of repetitions in Splunk -> 14 times

Re: How is your volume usage calculated

bwouters — Tue, 29 Sep 2020 17:45:28 GMT

Data Input is configured as:
Set host -> Constant value
Host Field valie -> DEMO
Set the Source Type -> Manual
Source Type -> SessionsLogs
Index -> main
Whitelist -> Splunk*

When checking the indexes, I see that 'main' only has 139MB logged to it.
If I add all the indexes together, I only have 152 MB

SourceType configuration:
Destination app: Search & Reporting
Category: Metrics
Indexed Extractions: none
Event Breaks: Auto
Timestamp: Auto
CHARSET: UTF-8
NO_BINARY_CHECK: true
SHOULD_LINEMERGE: true
category: Metrics
disabled: false
pulldown_type: true