This link now redirects to the main splunk doc page. ... View more

In case anyone else's search brings them here first - the new token documentation is here: http://docs.splunk.com/Documentation/Splunk/6.6.2/Alert/EmailNotificationTokens ... View more

Check the splunk log for errors: \opt\splunk\var\log\splunk\splunkd.log ... View more

Ah! Looks like it was fixed in 2008 R2. So now there are 2 different timestamp formats in the logs. e.g. Message=The system time has changed to ?2015?-?12?-?13T13:28:07.492000000Z from ?2015?-?12?-?13T13:18:04.893874600Z. and Message=The system time has changed to 2015-12-12T09:09:14.198Z from 2015-12-12T09:09:14.198Z. So here is my fixed rex (only to the second - decided not to bother with milliseconds): .*to\D+(?<StartYear>\d+)\D+(?<StartMonth>\d+)\D+(?<StartDay>\d+)T(?<StartTime>[^.]+).* from\D+(?<EndYear>\d+)\D+(?<EndMonth>\d+)\D+(?<EndDay>\d+)T(?<EndTime>[^.]+) Here is the full search: source="WinEventLog:System" "system time has changed" | rex field=Message ".*to\D+(?<StartYear>\d+)\D+(?<StartMonth>\d+)\D+(?<StartDay>\d+)T(?<StartTime>[^.]+).* from\D+(?<EndYear>\d+)\D+(?<EndMonth>\d+)\D+(?<EndDay>\d+)T(?<EndTime>[^.]+)" | strcat StartYear "-" StartMonth "-" StartDay "T" StartTime StartTime | strcat EndYear "-" EndMonth "-" EndDay "T" EndTime EndTime | eval StartUnix=strptime(StartTime, "%Y-%m-%dT%H:%M:%S") | eval EndUnix=strptime(EndTime, "%Y-%m-%dT%H:%M:%S") | eval TotalTime=EndUnix - StartUnix | table _time host StartTime EndTime TotalTime ... View more

I think I hit this weirdness as well - this is from Windows system event logs isn't it? I wanted to check how well (or not) our NTP system was working. I used this search: source="WinEventLog:System" "system time has changed" | rex field=Message ".*to (?<StartTime>[^.]+).*from (?<EndTime>[^.]+)\." | eval StartUnix=strptime(StartTime, "%Y-%m-%dT%H:%M:%S") | eval EndUnix=strptime(EndTime, "%Y-%m-%dT%H:%M:%S") | table _time host StartTime EndTime StartUnix EndUnix However, like Lukas, the strptime wasn't doing the conversion. Copying and pasting the text from Splunk into Notepad++ then actually the Message line is: Message=The system time has changed to ?2015?-?12?-?13T13:28:07.492000000Z from ?2015?-?12?-?13T13:18:04.893874600Z. Notice the hidden control codes around the date fields? What on earth were Microsoft thinking? Anyway I have a solution using the rex command. rex field=Message ".*to \D(?<StartYear>\d+)\D-\D(?<StartMonth>\d+)\D-\D(?<StartDay>\d+)T(?<StartTime>.*)Z from \D(?<EndYear>\d+)\D-\D(?<EndMonth>\d+)\D-\D(?<EndDay>\d+)T(?<EndTime>.*)Z" The full search string I used is: source="WinEventLog:System" "system time has changed" | rex field=Message ".*to \D(?<StartYear>\d+)\D-\D(?<StartMonth>\d+)\D-\D(?<StartDay>\d+)T(?<StartTime>.*)Z from \D(?<EndYear>\d+)\D-\D(?<EndMonth>\d+)\D-\D(?<EndDay>\d+)T(?<EndTime>.*)Z" | strcat StartYear "-" StartMonth "-" StartDay "T" StartTime StartTime | strcat EndYear "-" EndMonth "-" EndDay "T" EndTime EndTime | eval StartUnix=strptime(StartTime, "%Y-%m-%dT%H:%M:%S.%9N") | eval EndUnix=strptime(EndTime, "%Y-%m-%dT%H:%M:%S.%9N") | eval TotalTime=EndUnix - StartUnix | table _time host StartTime EndTime StartUnix EndUnix TotalTime ... View more

Hmmm - Thanks BMunson. I will look into that on our hot standby system - it's not a massive file to be honest - only 147 lines so a bit surprised by that (I know some customers have thousands). Can't find anything called tsidxstats under /opt/splunk/var/lib/splunk - although obviously a lot of *.tsidx files in the db dirs. No real documentation out there (just a scattering of tags on here - http://splunk-base.splunk.com/tags/tsidxstats/) but is it generated when I use tscollect or search accelleration? Will see if I can goad a server into caching and then try the transforms.conf change. ... View more

On Splunk indexer, 5.0.2. We have just had a case where the lookup file was definitely being cached (it was feeding to an event creator into our Zenoss monitoring system). Ran the above debug command successfully and it now seems to be reporting back the correct values without having to restart our main Splunk instance (lots of change requests needed for that). I can imagine it was done for speed but I wish I had known that this caching was going on beforehand. Not sure it always caches the file either - I can remember some changes worked without having to restart the Splunk. ... View more

I agree if we switch to syslog then that would have kept us independent but the splunk feed includes failover, buffering in case of network disconnect and fast reconfig of the agents with the deployment host. Depends if you can make a case for your IT budget and management priorities really. I could have set syslogs to forward direct to zenoss but we're hoping for a budget for something better later on. ... View more

Forward another year and a half - we have Splunk Universal agents out everywhere and are about to phase out Scom. We now have an eventtype.conf,tags.conf and lookup table solution that allows us to specify which events we are interested in and then launch a script to raise a relevant alert in our Zenoss monitoring system based on severity and event description in the lookup file. Someday I might document this up here if there is any interest - let me know. ... View more

Try one at a time - then try and expand it, also I notice you have spaces in your regex? If you need 2 then do something like this: props.conf: TRANSFORMS-set = setnullevents1,setnullevents2 and then transforms.conf: [setnullevents1] blah [setnullevents2] blah From our working one to filter out specific hosts and events: [setnullevents] REGEX = (?ms)(EventCode=(17503|6105|6107|6106|6201)\D).*(ComputerName=(COMP-001|COMP-002|COMP-003)) DEST_KEY = queue FORMAT = nullQueue ... View more

Forward about a year - we have deployed Lightweight forwarders to all our systems - and are consistently more stable than the Scom agents. Now upgrading them all to Universal Forwarders. It's all working well - they are forwarding from each host to 2 "hot" Splunk heads. We have visability and graphs of windows events from our entire infrastructure and can quickly set up emergency alerts if we need to. It doesn't directly compete with Scom - we have hundreds of rule sets on Scom that would mean hundreds of Searches on Splunk that which would be unweildy. Great for debugging and views though. ... View more

We tried both and we continued to get events collated into bigger splunk events - i.e. 1 big splunk event that contains a random number of windows events together with no spacing between them. So no splitting on the time/date field. However we found a solution with the help of a splunk tech, Guillaume: I used the shotgun method of creating a props.conf in /etc/system/local With: [source::WinEventLog:Application] LINE_BREAKER=([\r\n](?=\d{2}/\d{2}/\d{2,4} \d{2}:\d{2}:\d{2} [aApPmM]{2})) SHOULD_LINEMERGE=false [source::WinEventLog:System] LINE_BREAKER=([\r\n](?=\d{2}/\d{2}/\d{2,4} \d{2}:\d{2}:\d{2} [aApPmM]{2})) SHOULD_LINEMERGE=false [source::WinEventLog:...] LINE_BREAKER=([\r\n](?=\d{2}/\d{2}/\d{2,4} \d{2}:\d{2}:\d{2} [aApPmM]{2})) SHOULD_LINEMERGE=false It works for system and application but not for WinEventLog:Directory Service and WinEventLog:DNS Server – so I presume the globbing ability of the ellipsis isn't grokked by the Splunk engine anymore too? (I apologise - it has been a long day). Anyway - this should hold people until the patch. ... View more

I moved from this to a separate question here: http://answers.splunk.com/questions/4785/windows-event-log-collection-via-microsoft-scom-2007 Basically it looks like if I want all the events in there then I have to use a Splunk install as a forwarder - with the option of making it a Light Forwarder after it's configured. ... View more

Yes - I had looked at that but unfortunately it only forwards alerts (or events) depending on the rules set. So in order to get all the Application events I would have to get SCOM to look at all the events (instead of just the ones I am interested in). It looks like SCOM prefers to use remote access to the events and doesn't pull them all to the central server - which I can understand but it's annoying to have to deploy another forwarder! ... View more

Hmmmm - have tried a test splunk forwarder - will probably convert to a light forwarder when I have finished configuration testing... This is certainly useful - probably more than using Snare. It does appear to be the only way unless anyone else knows better. Looks like I will just have to have the hassle of a remote forwarder. I assume the remote WMI event log access is even more of an event hog and I like the idea of the buffering after failure. Many thanks! ... View more

Ok - here is a different route... We have Microsoft SCOM (well - we will do - currently migrating from MOM) installed. It has agents on all the windows systems that collect data/events/etc and forward to a central SCOm system. I see from here http://www.splunkbase.com/apps/All/4.x/app:System+Center+Operations+Manager+%28SCOM%29+integration that I can get data from SCOM into Splunk. However I was under the impression that only certain events are forwarded to SCOM for an alert depending upon what you set up. Does anyone know if SCOM can collect all the events which can then be sucked in by Splunk? Or is it only some of them? In which case I need to install Snare as well as Scom agents on all the platforms... Matt ... View more