Hey everyone. In data we're taking in (SIP call dialogs) we can monitor for negative termination causes. Some are expected, but what we're concerned about is sudden spikes. For instance, if we suddenly see a huge spike in the number of 404 messages, that would indicate a problem and we would want to be notified about it. So, right now the approach I am rolling around in my head is to do a timechart, and then run a delta against each command. Something like this: index=SIP |timechart span=5m count(eval(SIP_CODE="404")) AS 404_COUNT count(eval(SIP_CODE="487")) AS 487_COUNT .... |delta 404_COUNT |delta 487_COUNT ... But at this point I'm kind of stuck on how to calculate the rolling average and standard deviation to figure out whether an event is a "spike" or not. Statistically speaking if we see a delta which is more than 3 standard deviations from the mean it would be significant, and probably indicate something weird (it'd be in the 99.7th percentile). I know other folks have worked on similar things, so I'm curious how you approached it. A simple threshold won't work because our numbers grow and wane throughout the day as they're dependent on business hours. ... View more

Hey everyone. I have a dashboard which was set up with around 8 searches, each using its own search. The dashboard works correctly. I have created a clone to do testing with, and have tested that the clone works. I am now trying to trim this back into using a single search and several post processes. According to the documentation I am doing everything right. So far I switched all of the "searchTemplate" tags to "searchPostProcess" and removed the overlapping search (basically the search index=x sourcetype=y) portion. I saved and the form still loads fine. But then as soon as I add the tags in the beginning of the form search, the dashboard disappears from my app. If I remove those tags, it suddenly re-appears. I have checked with an xml syntax checker and there aren't any errors. Here is the beginning of my dash: <form> <label>BCV IMS Customer Experience Dashboard (Simple)</label> <searchTemplate> index=xs XS_Calling_Number="*$TN$" OR XS_Called_Number="*$TN$" OR XS_Group_Number="*$TN$"|rex field=XS_Local_Call_ID "(?P&lt;XS_Local_Call_ID_Trimmed&gt;\d+)(?=:)" </searchTemplate> <fieldset> <input type="text" token="TN"> <label>Enter Customer 10-Digit TN</label> <default></default> <seed></seed> </input> <input type="time" /> </fieldset> <row> ... Any thoughts? I am going to be opening a service ticket as well but was hoping I may get a faster answer here. ... View more

Hey everyone. I have a ton of customer records with both IP's and ZIP codes. The issue is that the IP's, despite being around the nation, are all registered to our company in NYC so the geolookup doesn't work. I do have the zip codes though. I know there are latitude and longitude DB's out there based on postal codes. Is this functionality built into the google maps app? ... View more

This is hopefully a simple question but I couldn't find the answer in transforms.conf. I am receiving an EXTREMELY high volume data source (network protocols dump). Every event averages 30-60 lines. There are a few conditions where I would like to keep this data (the entire record), everything else should go to the nullQueue. So, is there a way to specify to trash every event that doesn't match a particular regex/set of regex? The regexes could be found in different lines of the file depending on the network protocol. Also, if I used a regex which specifies line start/linebreak, will this only look at the very first line of the event? I have looked at http://docs.splunk.com/Documentation/Splunk/4.3.1/Deploy/Routeandfilterdatad#Keep_specific_events_and_discard_the_rest but it doesn't seem to say how regular expressions work in multiline events. Will a regex save an entire event if the regex matches any part of any line in the event? Or does it only watch the first line? ... View more

Hey everyone. We are working to get a deployment server up and running. However our original manual configurations include in the inputs.conf file a host= field, which specifies the server FQDN (each server one has several, and due to geographical redundancy we have some servers which have the same shortname but different fqdns). We can't use those files with the deployment server since they'd only be valid on a single server. So can that be manually configured in another file at install time of the forwarder in another configuration file? We've looked at modifying server.conf's serverName= field, but that doesn't appear to make any difference. ... View more

Greetings. I am working on a project to take in several fields of data. I would like to analyze each field of a record, and based on its results, place a comment in a new "Notes" field. This way a viewer can easily tell what happened without having to look at a ton of numbers. So, in a simplest case imagine you had three latencies in a csv: 1, 100, 11 If the goal was to have all be below 10. You could run a case statement against the first field, and since it was below 10, it wouldn't do anything. Then checking the second field it would be incredibly out of range, so a note saying "Latency 2 excessively high.\n" (including a newline, if that is possible). Then it would check the third field and place a note in the notes field stating "Latency 3 marginally above threshold." So in the end the contents of the "notes" field would say: Latency 2 excessively high. Latency 3 marginally above threshold. Is this possible? Any ideas on how to accomplish it? In my case I will be analyzing approximately ~20 fields, each will be analyzed with a case statement or an IF statement. I have not yet written the test cases. ... View more

Hey everyone. I have several sources being spread over 4 indexers. I periodically receive error messages stating that space is low on /splunk/hot/{some source name}, specifically "Search peer server-chi-a2.sys.mycompany.net has the following message: You are low in disk space on partition "/splunk/hot/AS-CDR". Indexing has been paused. Will resume when free disk space rises above 2000MB." It shows up for around 10 minutes, and then disappears. Here is my indexes.conf file's contents: [XS] disabled=false homePath=volume:HOT/XS-CDR coldPath=volume:COLD/XS-CDR thawedPath=/splunk/thawed/XS-CDR maxDataSize=auto [AS] disabled=false homePath=volume:HOT/AS-CDR coldPath=volume:COLD/AS-CDR thawedPath=/splunk/thawed/AS-CDR maxDataSize=auto [PBTS] disabled=false homePath=volume:HOT/PBTS coldPath=volume:COLD/PBTS thawedPath=/splunk/thawed/PBTS maxDataSize=auto_high_volume [CMS] disabled=false homePath=volume:HOT/CMS coldPath=volume:COLD/CMS thawedPath=/splunk/thawed/CMS maxDataSize=auto_high_volume [MSP] disabled=false homePath=volume:HOT/MSP coldPath=volume:COLD/MSP thawedPath=/splunk/thawed/MSP maxDataSize=auto [KPI] disabled=false homePath=volume:HOT/KPI coldPath=volume:COLD/KPI thawedPath=/splunk/thawed/KPI maxDataSize=auto [LICENSING] disabled=false homePath=volume:HOT/LICENSING coldPath=volume:COLD/LICENSING thawedPath=/splunk/thawed/LICENSING maxDataSize=auto [CER] disabled=false homePath=volume:HOT/CER coldPath=volume:COLD/CER thawedPath=/splunk/thawed/CER maxDataSize=auto_high_volume [volume:HOT] path=/splunk/hot maxVolumeDataSizeMB=140000 [volume:COLD] path=/splunk/cold maxVolumeDataSizeMB=840000 The /splunk/hot LUN is 150 GB, and the /splunk/cold LUN is ~850GB (both housed on a SAN). I used volume sizing for the configuration as I did because I don't truly know how much space each individual source will use as the devices are incredibly bursty, and this should let splunk control it for me. I also purposely left the volume sizes around 10GB lower than the LUN size. If splunk is listening to its config files, I should never see a <2000MB free error, since at all times I should have at least 10GB free on each LUN. Has anyone else seen this? Is there something wrong with my config that I'm missing? I'd appreciate the help. Thanks! ... View more

Just curious - is it possible to exclude the totals row and column generated by addtotals and addcoltotals from the heatmap visualization? They always show up as the hottest things on the map, distracting from the values which are actually important. ... View more

Hey everyone. I am working with telephone records, and am trying to work around Splunk's inability to search for literal asterisks(*). To work around I am using a regex to select only records starting with * or #, and then I am trying to use a case statement in eval to figure out what type of feature is being used by our customer. Example values of MYSOURCEFIELD (not exhaustive): *67, #31, *82 Here is the search currently, it only searches for the first 2 cases: index=MYSOURCE|regex MYSOURCEFIELD="(\*|#)(31|67|82|65|77|87)"|eval Feature_Code=case(like(MYSOURCEFIELD,"%31"),"Caller ID Blocking Per Line",like(MYSOURCEFIELD,"%67"),"Caller ID Blocking Per Call")|table Feature_Code This returns nothing, even though I know a significant number of both are being utilized and can even be seen without the eval/case statement. Any suggestions on how to make the case statement work would be really appreciated, thank you. ... View more

Hey everyone, Is there a way to show the indexed time of an event (as opposed to the timestamp)? I am trying to see if my forwarders are capable of keeping up with the volume of data that I am sending - I seem to be missing a lot of events and I want to make sure I'm not losing them when my pruning process runs. ... View more

Hey everyone. In my data's filename there is an indicator which indicates the type of data that it is. It can either be "eu" or "cabs." I am currently running into issues with this search: index=cms| rex field=source "/home/cemp/CEMP/(?<EUORCABS>eu|cabs)/.*"|where EUORCABS=eu It won't run. I have verified that the EUORCABS column is successfully created by running this search: index=cms| rex field=source "/home/cemp/CEMP/(?<EUORCABS>eu|cabs)/.*"|table EUORCABS All records have either a value of "eu" or "cabs" (without quotes). I have also tried casting it to a string, but no luck. Am I missing something obvious here, or is splunk not searching properly? ... View more

Hey everyone. I am starting to investigate the google maps app. However before I put too much effort into it, I would like to use it to show our representation across the country (I work for a large carrier with several million customers). From the records we receive we have access to the town where a user is located. I would like to be able to show a fog map that shows our presence, instead of little markers on the map. Has anyone does anything like this? Is it possible? My concern is what happens to the map if we try to portray the presence of several million customers. I have not yet installed the app, and the link on the map splunkbase page points to a localhost instance instead of the documentation. I am looking for something similar to this: http://www.youtube.com/watch?v=5ANkSgwgkqo. I just know that the google maps one has a direct line of support. ... View more

Greetings everyone. We are using a search against CDR data to calculate the 60 minute period in a day which has the highest number of calls. Currently, we are using this search: index=AS AND sourcetype=AS_CDR earliest=-61m@m latest=-1m@m AND (host=wdv-as03-01.mydomain.net OR host=wdv-as03-02.mydomain.net)|stats count AS "Number Of Calls"|eval "Hour Ending"=strftime(tostring(now()-60), "%m/%d/%Y %H:%M")|table "Hour Ending" "Number Of Calls" It is run every minute, and outputs the number of records from -61 minutes to -1 minutes, and the hour/minute that the search was run. We would love to use timechart to do this at the end of the day as opposed to running the search every minute for scalability purposes, but when we tell timechart to run with a maxspan of 60m, it automatically seems to snap to the hour. The issue this causes is - what if our busiest hour of the day is from 1:32 to 2:32 - that busiest timeframe would be split between the 1pm-2pm period, and the 2pm to 3pm period. We need to be able to see that it was the time between 1:32 and 2:32. Any advice would be very much appreciated. ... View more