All.. I have inherited the task of learning about an older Splunk installation (4.1.5). It is working just fine, however, it has been starting to show some high-cpu usage and other indications that it is struggling. Given this box has not been touched or changed in a while, but the amount of data it has been tasked with indexing and the number of queries has not been static. I expect that the main reason for the issues being seen are related to the hardware and software being updated to match the load being put on it. I am soliciting ideas on helping me find weak spots and areas of bad implementation since I did not design this thing, I want to look for weak spots. So far, it has been all Splunk research: Found numerous lookups taking place, some with Python scripts, others using external lookup files. Found that the "queue" has been hitting the "max_size" of the queue frequently, but the machine memory is not being utilized fully. Questions: If I have found that there are lookups defined, but files missing, will this cause Splunk to slow down looking for these files or timing out? Is there a way to find out if a lookup is being triggered or being used? Is there a faster or more efficient method of "lookups" than external files or python scripts? Is there a way to increase the max_size of the queue to handle more items in the queue? Is this recommended? Thanks! ... View more

So my question is based on something I am trying to do, but my splunk-foo is not powerful enough to figure this out! So, I have the following scenario: We have a door control system that is not very smart, connected to the network. Whenever the networking team makes significant changes (usually on the holidays) they have to bring the network down. The network "reboot" causes what we call "micros" (one of many interfaces for badge-readers, mag-locks controller, etc..) to go completely brain dead. They require a reboot to come back online. While brain-dead, doors will not open, badges will not be read, etc.. A BAD THING! When this happens the master server that I monitor with Splunk shoot out these errors (see below). Depending on the scope of the network outage, the number of lines (micros) that it reports could vary. Each line will have a different time stamp and a different micro number listed. It usually spits them all out in a matter of seconds of each other. Example: 02:15:00.425 dbmgr : E - SUP invalid table -1 from micro 105 02:15:00.429 dbmgr : E - SUP invalid table -1 from micro 33 02:15:00.476 dbmgr : E - SUP invalid table -1 from micro 305 02:15:00.524 dbmgr : E - SUP invalid table -1 from micro 40 ---- snip ---- 02:15:01.434 dbmgr : E - SUP invalid table -1 from micro 38 02:15:01.536 dbmgr : E - SUP invalid table -1 from micro 21 02:15:01.554 dbmgr : E - SUP invalid table -1 from micro 42 I need to alert on these events and let our on-call people know that it has happened and have them reboot those micros. I know how to do this, but the problem I am having is that each time an alert is found (in realtime) a "help desk ticket" (for the on-call people) gets generated, and equally, the on-call people get a SMS based alert based on each ticket. NOT GOOD. This could create between 10-20 tickets and SMS alerts. I really do not want to piss off the on-call folks with ticket bombs! I would like to be able to summarize the events that Splunk monitors for a short period of time. Then send that summary to the email address that generates the ticket and SMS alert with a list of the events that happened. It would then wait a couple of hours (set up via a threshold) before alerting the on-call staff again. It might take them 15-20 minutes to wake up, login, try to remote reboot the micro or they may have to travel into the office and reboot the micros (30-40 minutes) So I have the alert setup and the threshold setup. Can anyone help me figure out how to summarize the events and have them listed in a single email? ... View more

I am using Barracuda Yosmite Backup software to backup all my servers. Splunk on my Linux box is the only one giving me grief out of 40 others. I know a lot of the errors are changes between the scan and the actual backup process (temp files, etc.. that get created and deleted in seconds of each other). Example output: \Network\sigweb.signaturescience.com\File Systems\splunk_data\splunk\firewall\db\hot_v1_4691\rawdata Error 3020: Object not found Bkp: 35766044 \Network\sigweb.signaturescience.com\File Systems\splunk_data\splunk\firewall\db\hot_v1_4691 Error 3020: Object not found Bkp: 1362042196-1362042194-7792846081280182826.tsidx Error 3020: Object not found Bkp: 1362042198-1362042194-6395177518873184810.tsidx Error 3020: Object not found Bkp: 1362045744-1362041885-2820434847812558834.tsidx \Network\sigweb.signaturescience.com\File Systems\splunk_data\splunk\windows_data\db\hot_v1_618\rawdata Error 3020: Object not found Bkp: 482298675 I just want to ensure that all my log data that I have been collecting, settings, filters, and other core and customization of splunk apps are backed up. I do not need indexes, or stuff that can be regenerated if I had to rebuild the server. So three questions: Is there something special I can do in a pre-backup (to halt or pause, closing open files, prevent file creation, etc..) and post-backup (begin or resume operations)? Is this the best method to follow? What directories should I ensure are being backed up so I can rebuild my Splunk server upon HW failure and have all my data intact, maybe not indexed, but can be indexed? Is there a list of directories to just exclude from backup that are considered temporary or volatile scratch/index data? ... View more