DNA- C Metrics and Log volume

[Eldemallawy]

hi,

Wondering if there is a document or guidance on how to estimate the  volume of data ingested in Splunk by pulling data from DNA Centre using the Splunk Add-on: Cisco DNA Center Add-on.

Cheers, Ahmed.

[kiran_panchavat]

@Eldemallawy 

In Splunk, EPS (Events Per Second) is a metric used to measure the rate at which events are ingested into the Splunk indexer. The formula to calculate EPS is relatively straightforward:

EPS = (Total Number of Events) / (Time Duration in Seconds)

To calculate EPS, you need to count the total number of events that were indexed within a specific time duration (usually one second) and then divide that count by the duration in seconds.

For example, if you want to calculate the EPS over a 1-minute window (60 seconds) and you have indexed 3,000 events during that time:

EPS = 3000 / 60 = 50 events per second. This means you are indexing, on average, 50 events per second during that 1-minute period.

[kiran_panchavat · Accepted solution]

@Eldemallawy 

1. Identify and prioritize the data types within the environment.

2. Install the free license version of Splunk.

3. Take the highest priority data type and start ingesting its data into Splunk, making sure to start adding servers/devices slowly so the data volume does not exceed the license.  If data volumes are too high, pick a couple of servers/devices from the different types, areas, or locations to get a good representation of the servers/devices.

4. Review the data to ensure that the correct data is coming in. If there is unnecessary data being ingested, that data can be dropped to further optimize the Splunk implementation.

5. Make any adjustments to the Splunk configurations needed, and then watch the data volume over the next week to see the high, low, and average size of the data per server/device.

6. Take these numbers and calculate them against the total number of servers/devices to find the total data volume for this data type

7. Repeat this process for the other data types listed until you are completed.

[Eldemallawy]

Many thanks @kiran_panchavat, much appreciated.

Cheers, Ahmed.

[kiran_panchavat]

@Eldemallawy 

Estimating the Splunk data volume within an environment is not an easy task due to several factors: number of devices, logging level set on devices, data types collected per device, user levels on devices, load volumes on devices, volatility of all data sources, not knowing what the end logging level will be, not knowing which events can be discarded.

Estimate Indexing Volume

1. Verify raw log sizes.

2. Daily, Peak, retained, future volume.

3. Total number of data sources and hosts.

4. Add volume estimates to data source inventory/spreadsheet.

Estimate index volume size:

1. For syslog type data, index occupies ~50% of original size.

2. 15% of raw data ( compression )

3. 35% for associated index files.

[kiran_panchavat]

@Eldemallawy 

1. Try this (gives the amount of license used for indexes)

index=_internal sourcetype=splunkd source=*license_usage.log type=Usage 
| stats sum(b) as bytes by idx 
| eval mb=round(bytes/1024/1024,3)

If you want overall, then you can use this timechart version

index=_internal sourcetype=splunkd source=*license_usage.log type=Usage 
| timechart span=1d sum(b) as usage_mb
| eval usage_mb=round(usage_mb/1024/1024,3)

For per index, you can use this

index=_internal sourcetype=splunkd source=*license_usage.log type=Usage 
| bucket span=1d _time 
| stats sum(b) as bytes by _time idx 
| eval mb=round(bytes/1024/1024,3)

2. Setup a Monitoring Console:-

https://docs.splunk.com/Documentation/Splunk/latest/DMC/DMCoverview 

[Eldemallawy]

Thanks Kiran,

I was looking for a way to estimate the volume of data that will be ingested into Splunk before installing it. This will help me calculate the License cost. 

Therefore, is there a way to estimate the volume of DNA-C metrics based on number of LAN / WLAN devices?

Cheers, Ahmed.