Hi folks, This might be elementary, but here goes: I have a Database input that grabs all open tickets from our helpdesk system. This runs every 5 minutes. So I end up with a nice index of table snapshots and I can create nice timecharts with span=5m showing how metrics change (every 5 minutes). But what if I wanted to visualise a pie chart showing currently open vs on-hold tickets? All of that data would come from the last data dump that was done. The quick and dirty solution i found was to set the tme reange as 5 minute window. That way, the chart updates each time a data dump is done and ignores the data from teh last dump because it happened more than 5 minutes ago. I guess this works, but if I change the pooling interval to 10 minutes, I'd have to change the time window to match the polling rate in all of my dashboards. I can't help thinking that there is a better way. Regards, Okolo ... View more

Hi folks, I've searched for an answer to this but haven't found anything that matches what I'm experiencing. For clarity, I am in Jamaica. We are in the EST time zone (GMT-5) but we do not observe daylight savings. I started indexing device syslog messages forwarded from my Network Monitoring System via UDP:514. My NMS is on a windows server set to local time (GMT-5). If an event occurs at 12:01pm Jamaica time (GMT-5), when I search for it in Splunk, that event has a _time of 7:01am (GMT-10). If I look at the syslog event viewer in my NMS, it shows the correct local time of 12:01pm (GMT-5). Based on my research here, I learned that Splunk uses the time and zone of the server it is on if a timezone isn't specified in props.conf. I checked the time on the server with the hwclock command at 2:08pm local time (GMT-5). The result: Fri 30 Aug 2013 08:54:43 AM EST This got me confused. The hardware clock is set 5 hours in the past (Which is actually GMT-10) but has the timezone set to EST (GMT-5). How does that affect the way Splunk indexes events? I'm guessing that I should do one of the following: 1. Change the hwclock on the Splunk server to the correct local time (GMT-5) and keep the timezone as EST. 2. Change the hwclock to the correct GMT time (GMT-0) and set the timezone to GMT. What is considered the best practice for Splunk? Is it best to set the hardware clock to GMT or local time? How will this affect previously indexed items? ... View more

This is going to be long, but I hope it presents an interesting problem and hopefully, it has an elegant solution. One of the things that really sold me on Splunk was the ability to throw a huge CSV file at it, Index it as a one-off and quickly create meaningful graphs and reports. The problem is that I have to manually copy and index each CSV snapshot as a one-off, then I have to modify my searches to point to the new source file. I want to automate this in a way that requires no manual intervention. My saved searches and dashboard should always have the latest information. Here's an idea of what the CSV looke like. Created_Time,Request_ID,Request_Type,Request_Status,Completed_Time DD-MM-YYYY HH:MM:SS,#100,incident,open,, DD-MM-YYYY HH:MM:SS,#101,request,closed,DD-MM-YYYY HH:MM:SS, So I'm basically treating tickets as events. _time = Created_Time #100 is open. The next time a take a snapshot of my tickets, #100 might be closed. I would want my dashboards to pick that up and report on that ticket in it's closed state the next time it is run. But Splunk can't change field values after indexing them. If I made Splunk monitor the CSV for changes, and overwrote it with the entire snapshot, it would re-index the file, duplicating all the tickets from the previous snapshot. If I were to overwrite the CSV with only those tickets that have been newly created, over time, my indexed data would contain only open tickets because the closure of the ticket would be missing from the updates. If I updated with newly created tickets plus tickets that have changed state, tickets would be duplicated in the index every time they changed status. Maybe this isn't such a bad thing, but I'm not sure how to search on it without counting the duplicates. Then I thought about using a lookup table. The CSV could be loaded as a lookup file and I update it with all the tickets by ticket number and create_time. That way, with each search, it would do a lookup for the latest version of the field values. I'd then index a copy of ticket numbers and create_Time and do a lookup on the full list of tickets. Does this sound like it can work? I'm just learning how to use Splunk and this has been racking my brain. ... View more

I have a timechart area chart that shows three types of event over time ("Node up", "Node Down' and "Node Rebooted"). I want the "Node UP" series to show in green, "Node Down" in red and "Node Rebooted" in yellow. I was able to get what I wanted using: [0xFFEE33,0x00FF00,0xFF0000] However, I find that from time to time, the colors become rearranged. "Node UP" might become yellow, "Node Down" green and "Node Rebooted" Red. Whenthis happens, I have to rearrange the colors in the charting.seriesColors until I get it right. However, after some time, the colors change again. How can I explicitly set the colors for each series so that each type of event always has the same color? ... View more

Here's my situation. I have automated a SQL lookup on a database and output a .csv file every 10 minutes with field names and events within the last 10 minutes. eg: eventlog-22112012-1410.csv. Each file is copied to a folder on my Splunk server resulting in an ever growing list of files that Splunk indexes. The first line of each file contains the fields and the rest of the lines are values. I have been able to get Splunk to index the files as they grow and it automatically extracts the fields. I have built a nice dashboard based on this data. I have 4 things I really need to ask: Splunk creates a new source for each file, resulting in a large number of sources in the summary page. Is there a way to have Splunk treat all of the files in the folder as one source? Splunk is indexing each file header as values. Therefore my results contain events with values such as Customer_Name=Customer_Name, IP_Address=IP_Address where the field name and value are identical for each file. How can I stop this from happening? Can I delete older files without losing the indexed data? Will Splunk continue to index new files without losing track and re-indexing all the files again? I want to add new fields to the .csv files going forward. Will Splunk be able to automatically detect the new fields even though they don't exist in the older.csv files that have already been indexed? Questions 3 and 4 are most important for me. I don't want to delete files or add fields and break the dashboards that I've already created. Please help if you can. ... View more