I have a search that works most of the time, but sometimes just causes Splunk to crash and requires a restart. I have a ticket opened with Splunk, but they are still not able to figure out what's going on so I thought I would post this. This is the search I use in my dashboard.
<![CDATA[| metadata type=hosts | eval age = now()-lastTime | where age > 300 and age < 86400 | convert ctime(lastTime) | eval field_in_ddhhmmss=tostring((age) , "duration") |rename field_in_ddhhmmss as "Time Offline" lastTime as "Last Update Time" | join host [search sourcetype=systemInfo | rename serial as "Serial Number" isp as "ISP" state as "State" city as "City"] |sort "Time Offline" a | table "Serial Number","Time Offline","Last Update Time","ISP","City","State"]]>
I use it to find computers that were checking in at least 24 hours ago, but have not checked in for the last 5 min. I then use "join" to match to a sourcetype to the host to get some specific data about those hosts. This search is fairly fast and runs in a couple seconds. I was using the same search for months, but this started to happen a couple weeks ago. The only thing that's changed is we have added more hosts. CPU/memory on the Splunk server is low when it crashes, and we're not seeing any spikes when this happens.
... View more