We have multiple Cisco WSA devices set up in each of the US timezones; each is set to log in local time. But it seems as if the WSA logs don't contain any kind of timezone indicator on them. When I run a search in Splunk, using a user account in Central time, against a WSA device in Eastern time, I end up getting "future" events. Example: ran a search at 8AM central against an eastern WSA device, there were events found with time-stamps of 9AM. Likewise when I run a search looking for lag between index time and timestamp (again from a Central Time account): index=wsa_system sourcetype="cisco:wsa:shd" CliConn=* | eval lag=((_indextime-_time)/(60*60)) All our Eastern devices are reporting negative lag (future timestamps), Central devices are relatively real-time, Mountain devices have approx 1hr lag, Western devices have roughly 2hr lag. I'm trying to set up alerts for high numbers of client connections and need to know: 1) Is there any way to adjust for these time off-sets at search time using our current logs? 2) Is there a way for Splunk to add the time off-sets/zone to the events at indexing time? 3) Is there a way to have the WSA devices add the timezone to the logs before sending? (Or will I need to make a business case that all the WSA devices should log in the same timezone regardless of physical location?) ... View more

Working on better alerting on indexing volume/license usage and the like and I've stumbled across something in-explicable. We have 4 Heavy Forwarders that all have default/outputs.conf with [tcpout] indexAndForward = false Now when I look at: index=_internal host=<nameing convention of Splunk infrastructure devices) source="*metrics.log" group="per_index_thruput" series=main| timechart span=30m per_second(kb) BY host One of the heavy forwarders shows up in the list of hosts (along with the indexers I'd expect) Even more confusing is when I look at: index=_internal source=*license_usage.log type="RolloverSummary" This returns the vast majority of events from our license master with "pool" listed as the "auto_generated_pool_enterprise" as I'd expect. But it also shows all 4 of our heavy forwarders with "pool" listed as "auto_generated_pool_download-trial" or "auto_generated_pool_forwarder" Looking at the Distributed Management Console app, under License Usage... if I look at all pools split by pool, I see all 3 of these pools (download-trial, enterprise and forwarder). When I look at it split by indexer, the list of indexers is in GUID so it's hard to correlate back to device, but there's 10 + "Other" listed... and we only have 5 indexers in our environment... so there's at least 5 more than expected. I'd like to get this cleaned up so we can be certain that we're accurately reporting on which devices are consuming license and at what rate. Anyone know why one HF would be found under index=_internal source="*metrics.log" group="per_index_thruput" series=main and all of them would be found under: index=_internal source=*license_usage.log type="RolloverSummary" As stated above... index and forward is false for all of them. Thank you! ... View more

Not sure how this slipped past us when we installed the Splunk App for Windows Infrastructure, but we've got a sourcetype called ActiveDirectory that is going to our main index that I need to move elsewhere. I believe it's in main due to default settings, but I'm not finding the stanza where the sourcetype ActiveDirectory is being set or what's driving the collection of this data so it's unclear where I should add the index stanza. We currently have: Index windows that contains sourcetype DhcpSrvLog that comes from a custom DHCP app that copies stanzas from Splunk_TA_windows Index wineventlog that contains sourcetypes (coming from Splunk_TA_windows/default/inputs.conf): WinEventLog:Application WinEventLog:Security WinEventLog:System (and coming from TA-DomainController-201R2/default/inputs.conf) WinEventLog:DFS-Replication WinEventLog:Directory-Service Then also the index msad that contains the sourcetypes also coming from TA-DomainController-201R2/default/inputs.conf: MSAD:NT6:Health MSAD:NT6:Netlogon MSAD:NT6:SiteInfo Powershell:ScriptExecutionErrorRecord Powershell:ScriptExecutionSummary We also have perfmon index, but we've disabled all perfmon functions at the request of our DC admins due to performance hits. Our PDC does also has the app TA-DomainController-2012R2 that contains the local/inputs.conf stanza: [admon://ADMonitoring] targetDC = <hostname> baseline=false disabled = false I'd tend to think this is where the index for the ActiveDirectory sourcetype would be set, but I'm not finding any documentation to support that idea. I need to know 1) If it would be best practice to include this sourcetype in one of the existing indexes or if keeping it in its own index would be preferred. 2) Is the is the admon://ADMonitoring stanza the correct place to designate the index for this data, or am I confused about what that stanza does? Thank you! ... View more

Trying to get a basic search of Windows event data to cross reference with fields from ldapsearch. I'm wanting it to take the value of a field (like Account_Name) in a Windows event and use it as a token to run an ldapsearch and return the value of a particular field from AD (like an OU). Rinse and repeat for all the values from the Windows results. I tried a join/append with the ldapsearch as the subsearch, and pondered creating a lookup table of the ldapsearch values. But we've got a very large environment (>140,000 users) that makes the subsearch/lookup-table enormous. Since the Windows search normally only returns a couple dozen events, the processing time of the ldapsearch to return all possible values completely kills the value of the search. I really need to be able to limit the ldapsearch to just he couple dozen main-search values. How do I get <main-search with $tokenvalues$> to pass to | ldapsearch domain=mydomain search="$tokenvalues$" ... View more

I have a list of 200+ IPs that I need to search against source addresses in our firewall data. The search needs to span several months of these logs and we consistently ingest <200 GB/day of this log type. I have uploaded the list of 200+ IPs as a lookup table and can get the following search to run for short time-frames: search index=firewall sourcetype="firewall" | search [|inputlookup SampleIPs.csv | fields SampleIPs | rename SampleIPs as source_address ] But expanding it to a full day ends up taking multiple hours to complete making going back for 3+ months not feasible. Trying to span more than 1 day ends up hitting timeout errors. Anyone have suggestions on how to better structure this type of search to run more efficiently? ... View more