Getting Data In

Loading BOTSV1 JSON into developer Splunk environment

FCTaylor
Explorer

I am new to Splunk and need some serious practice to learn all the cool things Splunk can do. I am trying to load the BOTSV1 JSON dataset into my lab environment so I can start learning the basics of SPL. According to the comments in GitHub this dataset is 120GB uncompressed. This brings up the following two issues.

1) The Splunk web file importer will only load files up to 500MB. How am I supposed to load a 120GB file?

2) The Splunk development license that I received is limited to 10GB, so how am I supposed to load this 120GB file once question #1 is resolved?

I am sure I am not the only one encountering this issue, so forgive me for asking a question that has probably already been answered numerous time. 

Labels (2)
0 Karma
1 Solution

FCTaylor
Explorer

Not only am I new to Splunk, but I am a bit of novice at Linux. Turns out I created my Linux environment using LVM, which seem to have only used 100Gb of the 300Gb disk space I allocated. While attempting to install the Botsv1_Data_Set using the web interface I never saw the notices that I was out of disk space so the install would never compete.

When I ran the install manually using the terminal I finally saw an error message indicating the disk was out of space. Once I resolved my LVM disk space issues the app installed correctly and I was able to run the "index=botsv1 earliest=0" search and get events displayed.

Thank you Stephanie for responding to my posts. I hope this helps some other newbee to Splunk out there.

 

View solution in original post

FCTaylor
Explorer

Not only am I new to Splunk, but I am a bit of novice at Linux. Turns out I created my Linux environment using LVM, which seem to have only used 100Gb of the 300Gb disk space I allocated. While attempting to install the Botsv1_Data_Set using the web interface I never saw the notices that I was out of disk space so the install would never compete.

When I ran the install manually using the terminal I finally saw an error message indicating the disk was out of space. Once I resolved my LVM disk space issues the app installed correctly and I was able to run the "index=botsv1 earliest=0" search and get events displayed.

Thank you Stephanie for responding to my posts. I hope this helps some other newbee to Splunk out there.

 

Stefanie
Builder

The BOTs v1 dataset is 6.1GB compressed and the smaller version is only 135MB compressed. 

Where did you get the BOTs v1 data? Have you looked at https://github.com/splunk/botsv1 ?

 

You can upload your data set to your Splunk server through FTP and install it through the command line or you can try to increase the web upload limit using web.conf.

You would add a stanza like so: 

[settings]
max_upload_size = 1024

where 1024 mb = 1Gb

0 Karma

FCTaylor
Explorer

Does it matter what version of Splunk I am running. I currently have version 8.2 and the GitHub specifically calls out version 6.5.2. I am asking because when I try to install the Botsv1_Data_Set app the server seems to hang and the application never finishes installing.

I can unzip the file to %splunk_home%/etc/apps but after doing that I see the application listed in Application Manager but the "index=botsv1 earlienst=0" command returns no results.

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...