Flood Splunk with test data to performance test real time dashboards and to war game

[neleisla]

Hi,

I want to flood splunk with a high number of test data to be able to identify flaws in the current alerting and monitoring systems I have in place. The test data will:
- Check whether there is any data loss i.e. no alerts sent etc
- Identify any performance issues with real time dashboards
- Help identify flaws in human process

The test data should not interfere with the real data being recorded and should be easily removed from Splunk logs.

Can anyone suggest the best way to do this?

Thanks
N

[timpacl]

Another consideration is the impact of large ingestion of test data on your data retention across all indexes. If you operate near the maxVolumeDataSizeMB, the test data can cause your other data to drop/archive early due to drive space considerations.

[adonio]

Hello neleisla,
you can achieve this with the Event Generator, read here:
https://splunkbase.splunk.com/app/1924/
download here:
https://github.com/splunk/eventgen
another option is to create a script that generates huge dumb files and have splunk constantly monitor that file.
if you want the data to not interfere with existing data, just make sure you are writing it to a different index and that no role can search that index by default.
If it is a clustered environment, it will be very difficult to get rid of this data, if it is not clustered, you can remove the data simply by stopping splunk: ./splunk stop and then cleaning data

  ./splunk clean eventdata -index YourDumDataIndex

BTW, data onboarded by method provided above will count against your license so be prepared

hope t helps