How can I get datasets/logs for monitoring and analysis?

kattey — Thu, 05 Oct 2023 15:31:27 GMT

Hello, good day
I am very new to Splunk, i and my team want to work on a mini project using splunk cloud with the topic "Splunk Enterprise: An organization's go-to in detecting cyberthreats"
how/where can i easily get datasets/logs that i can use in splunk for monitoring and analysis. and what best way should we go about this topic?

Re: datasets/logs

inventsekar — Thu, 05 Oct 2023 12:37:36 GMT

Hi @kattey ... please check these things:

1) As i heard, the Splunk Essentials app got some sample data.

https://splunkbase.splunk.com/app/3435

2) and then you can find some sample data in this repo:

https://github.com/splunk/botsv3

3) and then, there is an app.. EventGen. very difficult to configure and very worst documentation. i would suggest this as last resort. thanks.

4) Splunk Datasets Add-On: This Splunk add-on provides a variety of sample data sets, including security logs, for you to work with. You can download and install the add-on directly from Splunkbase: https://splunkbase.splunk.com/app/3245/

5) Boss of the SOC (BOTS) datasets: You've already mentioned BOTS v1-3, but don't forget about BOTS v4, which was released later. You can find it here: https://github.com/splunk/botsv4

6) Elastic Common Data Model (ECS) sample data: Although intended for the Elastic Stack, you can adapt these sample logs for use in Splunk. The repository contains logs from various sources, such as network traffic, security events, and web server logs: https://github.com/elastic/ecs/tree/master/generated/samples

6) Sample Log Generator: This tool generates synthetic logs that you can customize to fit your needs. While not real-world data, it can be useful for testing specific scenarios or queries: https://github.com/ErikEJ/SqlQueryStress

7) NIST National Vulnerability Database (NVD) data feeds: NVD provides various data feeds containing vulnerability information. While not logs per se, this data can be useful for exploring security-related data in Splunk: https://nvd.nist.gov/vuln/data-feeds

😎 SecRepo: You've already mentioned this repository, but I'd like to emphasize its value as it contains various sample logs from different sources: http://www.secrepo.com/

9) https://github.com/gfek/Real-CyberSecurity-Datasets

10) https://github.com/shramos/Awesome-Cybersecurity-Datasets

11) https://www.secrepo.com/

hope this helps you and other splunkers.. thanks. karma / upvotes appreciated by all, thanks.

Re: datasets/logs

gcusello — Thu, 05 Oct 2023 12:37:51 GMT

Hi @kattey ,

how much do you know Splunk?

if you start from scratch you need to learn hot to ingest data in Splunk and how to search on Splunk.

Data sources com from you infrastructure, if you haven't, you could use an automatic generator, but it isn't another stack to learn!

Best way, is to search in Community answers about basic learning (e.g. Search Tutorial) and getting data in.

Then you should define a perimeter to identify the data sources to ingest.

Ciao.

Giuseppe

topic Re: datasets/logs in Getting Data In

How can I get datasets/logs for monitoring and analysis?

Re: datasets/logs

Re: datasets/logs