I received this error this morning: "The minimum free disk space (5000MB) reached for /data/splunk/var/splunk/dispatch" I logged into the server (Linux) found that my Splunk volume is 100% used. Unfortunately, I don't currently have the option to add additional disk capacity. So my question is, how can I free up disk space in Splunk? For example, is there a way to purge data within the main index that is over a year old or something like that? What is the recommended way to troubleshoot disk space related issues in Splunk? Thanks! ... View more

Hello, How can I determine the index volume by sourcetype? The reason why I ask is because occasionally I'll have a big spike in my index volume that threatens my license cap and I'm trying to find the best way to determine the cause of the spike. If I can create a chart that shows volume by sourcetype (over X hours) then I can identify the culprit and dig in from there. Or even better, is there a search that I can run that actually identifies the cause of the spike (not just the sourectype)? Thanks! ... View more

So I have this basic search for a line graph visualization: (search goes here) | timechart count Let's say I've had 10 events/hour up until 7:00am this morning. Between 7:00-10:00am I've had zero events. When I render the graph, it "stops" at 7:00am. How can I force the graph to include the zero events between 7:00-10:00am? The reason this is important is that I'm trying to get the graph to communicate that the events have stopped... but it looks like they are still on-going since it doesn't include recent hours of zero events. I hope that makes sense. Thanks for your help! ... View more

We have Splunk Enterprise Security and use the correlation searches along with notable events for incident handling. We also have alerts setup. However, the alerts don't have the incident handling features that correlation searches and notable events do. So, I'd like to move our alerts into correlation searches. However, I'm not sure how to include a count threshold in a correlation search. For example: Is it possible to move this alert: sourcetype="firewall" action="denied" | stats count as Total (Alert when over 1,000 events in 60 minutes) ... into a correlation search? Because when it's a correlation search, it would turn into a notable event for incident handling, which is what I'm trying to do with my alerts. But, I'm not sure how to configure the correlation search to only create a notable event when the event count for the search exceeds X over Y time period. Or, the alternative, would be to have alerts generate notable events, but I'm not sure how to do that either. Thanks! ... View more

Outside of creating an accelerated search or upgrading hardware, is there a way to speed up the search below? This search takes approximately an hour to run on my system and I'm wondering if it's because my search logic is inefficient and if my search syntax/logic can be improved somehow? The search detects denied/blocked outbound remote connections (FTP, SSH, RDP). Thanks! (sourcetype="firewall" OR sourcetype="ips") (dest_port="21" OR dest_port="22" OR dest_port="3389") (action="blocked" OR action="denied") (dest_ip!="10.0.0.0/8" AND dest_ip!="172.16.0.0/12" AND dest_ip!="192.168.0.0/16") | iplocation dest_ip | stats dc(_raw) as Count by src_ip dest_ip Country Region City dest_port protocol sourcetype signature | sort -Count ... View more

I'm getting what I believe are strange results when using the round function to control the number of decimal places. This is a search that I have in a dashboard to calculate how much data (in GB) that I have indexed in the last 24 hours: index="_internal" source="*metrics.log" per_index_thruput| eval GB=kb/1024/1024 | stats sum(GB) as total The result of the above search is: 7.8531551056447. I wanted to round to 2 decimal places so I modified the search to use the round function like this: index="_internal" source="*metrics.log" per_index_thruput| eval GB=round((kb/1024/1024),2) | stats sum(GB) as total However, the result of the above search is: 0.15 (instead of 7.85). So, I tried the same search but this time to 3 decimal places like this: index="_internal" source="*metrics.log" per_index_thruput| eval GB=round((kb/1024/1024),3) | stats sum(GB) as total The result of the above search is: 7.21 which is closer to the expected result of 7.85 but still not 100% accurate. So, I tried 4 decimal places like this: index="_internal" source="*metrics.log" per_index_thruput| eval GB=round((kb/1024/1024),4) | stats sum(GB) as total And the result of the above search is 7.8405 which is almost accurate/correct. But, I don't want 4 decimal places... What am I doing wrong, why does the search with 2 decimal places return 0.15? Thanks! ... View more