I have a service which we need to monitor discrete states. I only get events if the state changes. I can map these states (e.g. error, init, pending, standby, hot) to numerical values and then graph them on a timechart. With the filldown command I can also maintain the current state to produce a graph that shows the current state at any given time. What I would like to do next is map those numerical values and provide some content. Ideally it would be to replace the y-axis so that instead of -1 through 3 it would display the states as text. Other options that would also work would be to color each column a different color based on the state (for example, make the column chart turn red when it is in an error state), or provide a legend that would make it easier to read. Basically I want someone to be able to look at the timechart and say at a given time what the state was without wondering what a 0, 1, 2 or 3 means. This is what I have thus far: Also, if it gets me the desired effect I could also break it down so RCS, Realtime, and isps do not appear on the same panel but ideally would like to have that functionality as well. Any ideas that might help on this. Splunk 6.6.1 ... View more

I have some files that I need to index daily even though they may not change in content for several days (for example over weekends). The files are generated daily so they have a new creation and modification time. How can I force splunk to automatically index the file daily or use something like creation or modification time? ... View more

We recently installed the Add-on for JIRA and followed the instructions. Everything seems normal and when I opened my Splunk to do the search I can type | jirarest and I see Splunk 6.5 wants to autocomplete the rest of the statement with the following: | jirarest jqlsearch "Reporter=Bob Smith' Since it provides an actual field for Reporter that matches one of our employees, it would seem to indicate that it can access my JIRA instance, however when I run the actual command it says- Search Factory: Unknown search command 'jirarest'. I have ensured the permissions are correct for my user account. Any idea of the problem? Splunk Enterprise 6.5.1, Add-on for JIRA 2.2.0 CentOS7 Running on a dedicate search head ... View more

We have recently implemented ITSI into our environment and are building a glass panel which displays the current state of multiple services running on a system. In our case we have about 16-20 services we monitor on each system and about 8 systems. If I used a regular dashboard I could use a base search to power all the panels with 2 or 3 base searches thus reducing load on the system (in terms of # searches). A regular dashboard's layout is too restrictive so we were hoping to do this with glass panels. I was curious if this was possible in ITSI and how you would implement this? ... View more

I have created a dashboard that has a few filters on it which are used to retrieve specific rows from the MSSQL database. I have managed to substitute the filters into the query, except a timepicker filter is proving more difficult. I want to restrict the query results to the range selected in the time picker rather than retrieving all the data and filtering it afterwards. Ideas on how to do this (besides indexing the data)? I could use either epoch time or a datetime. Splunk 6.3 DBConnect v1.2.1 ... View more

I am trying to determine if there is a way for the Splunk Universal Forwarder to monitor environmental variables. We have an in-house application that uses a particular one to determine whether a service should be actively running certain processes (i.e. the service should only run the process if it is flagged as the active server). Is there a way for the Universal Forwarder to monitor these values? Ideally what I am shooting for is to send an alert when it changes from active to standby (or vice versa). ... View more

We have a file being generated by a vendor that they write data to on a regular basis. I do not need to import the data in the file directly into splunk, but need to monitor the timestamp of the last modified date (Windows system). Based on the documentation they provided me, this file serves also as a checkin file and if the file is not being updated within a certain amount of time it means a critical component is down. What ways can you monitor this last modified date in Splunk? As stated my preference is not have to import the file contents. Splunk 6.2.3 ... View more

We have some critical services we are monitoring on a realtime system so responding in a timely manner is essential. If the services stop we need to be notified. Currently we monitor this with WinHostMon and when one of the services stops we sent out an email indicating it has stopped. Sometimes but not always an event will get logged indicating one of these same services has stopped by a particular user from a certain machine (usually from program interface). These events come in via the application event log. We don’t alert on this currently. Want: We want to be able to incorporate this information into a single alert to do the following: 1. If a user stops the service, send an email that includes the user and source machine 2. If the service stops in another manner (for example through the control panel) where there is no specific user - still send the alert, but just put the user and source machine info as unknown. What I have tried: Did a left join (left side is monitoring the WinHostMon) and if user and source machine are included go ahead and send it or default to Unknown (using fillnull) Problem: I get the alert but sometimes it doesn’t include a username and source user and sometimes it does. I think this is a timing issue as I sometimes see two events show up, one with the default Unknown user and a second with the right info (extractions for these fields have already been set up and working). Currently the alert to set up to run under a cron job 1/minute and the WinHostMon watches the services every 30 seconds. The following is a example of what I have tried that sometimes works and sometimes does not: Type=Service service_name="Critical Service to Watch" State="Stopped" | dedup service_name,host | join type=left service_name,host [ search index=wineventlog Message="Shutdown of the * service on computer * requested by user *" service_name="Critical Service to Watch" | fields host,message,service_name,source_machine,user] |fillnull value=Unknown source_machine,message,user | fields host,Name,service_name,source_machine,message,user Splunk 6.2.3. I am not sure what is best way to handle this, perhaps some kind of lookup that when it sees someone stop the service append it to the log and when the service starts it removes this entry from the lookup file? Anyone have any other ideas? ... View more