I'm trying to use a lookup table to find servers that are not reporting or have NEVER reported to Splunk. Since I don't have admin access to the Splunk instance, I write the info from our DB of servers to the event log of a specific server and then import it nightly via an alert with a an action of "output results to lookup". When I run the search below, it always returns "Error in 'inputlookup' command: Invalid argument: 'NOT'" - regardless of whether I use the word NOT. | inputlookup siteservers NOT [search host=Server Channel=System ProviderName=EventLog EventID=6013] "Servers" contains the value from the lookup that I want to match to the host field. Event 6013 is just the daily windows uptime event - if it's not found then I have a server that's not reporting. I suspect that the fact that the parameter "local=t" is not available with the inputlookup command may be part of the problem. The lookup must be used with the "local=t" parameter. Our admin tells me that making the lookup table available across the cluster is very difficult. Any ideas on how to make this work would be much appreciated. ... View more

Tried many variations (enclosing arg in quotes, $, and backslash) and got many errors - mostly "expected to be an eval expression that returns a string." Definition: [PACTime(1)] eval PacTime=strftime(relative_time($etime$,"-8h"),"%m/%d/%y %H:%M:%S") Called as EventID=6* | eval dtime=`PACTime(_time)`| table Host, dtime Seems like this would be eval based, but tried both ways. Any help appreciated. ... View more

I am getting a high incidence rate of "Splunk could not get the description for this event." All forwarders are SplunkUniversalForwarders, versions 4.2 thru 5.0.1. Yesterday I got these results from my Linux 4.3.2 indexer: Servers with Splunk Universal Servers with "Splunk Could Forwarder Ver. WinEvents OK not get desc" Total 4.2 27 21 48 4.2.2 4 2 6 4.3.1 1 - 1 4.3.2 4 8 12 5.0.1 172 40 212 Total 208 71 279 Since the results were so scattered among the forwarder versions, I upgraded the indexer to 5.0.2. As the indexer is a Linux box, and I know the event descriptions are extracted from the DLL’s on the clients, I really didn’t expect to see a change. However, since the change I now have 151 servers with “could not get” — over twice what I had yesterday before upgrading the indexer. Now over 75% of my windows events contain “Splunk could not get the description for this event.” The majority of the events are from the security logs, but there is also a significant number of events from the system and application event logs. Descriptions are present when viewed via event viewer on servers in most cases. In a few cases applications do not put descriptions into the application log. A spotcheck of some of the affected servers shows that msaudite.dll file and the security subkey under hklm\system\currentcontrolset\services\eventlog\security are present. Operating systems are also a mix — 78 of the effected machines are Server 2008, the rest 2003. Any help would be greatly appreciated. ... View more

Scenario: Project Splunk Deployment: 1 indexer with ~250 Windows forwarders, a few Linux, and various other switches and hardware. Security logs from the project’s 5 domain controllers are considered sensitive, and must go to a separate index on the project indexer so that Splunk apps can be shared with non-IT users. ADDITIONAL INFO: All other security logs should go to the main index. Corporate Splunk Deployment: 4 indexers with ~100 Windows domain controllers (including 5 from this project) forwarding security logs only, and various other routers, switches and hardware. Domain controller security logs go to main index. ADDITIONAL INFO: Corporate wants ONLY the security logs. ADDITIONAL INFO: The project has no access to the corporate Splunk instance, but if we want to get the logs from the DC's, we have to provide the solution to get the logs to the corporate instance. I currently have the project’s 5 DC’s sending their logs to a separate index on the project’s indexer, no problems. I am also working with an intermediate forwarder for the project’s DC’s that forwards to both the project and another indexer (stand-in for corporate Splunk instance). Test clients are working via the intermediate, but not to different indexes. Is it possible to send the DC’s security logs to an alternate index on the project’s indexer and the main index on corporate’s indexers? If so how? ... View more

I'm trying to run a search for a large number (45) of suspect IP addresses. The search runs for 12 hours or more but never returns any results, and on the jobs page always shows "Running (0%)". earliest=06/01/2011:0:0:0 NOT deny ("112.64.161.162" OR "113.142.9.125" OR "118.102.252.227" OR . . . ) |outputcsv 201107111.csv Using outputcsv because I'm expecting more than 10K results based on individual searches on some of the addresses. I know this is an inefficient and expensive search, but it seems that it should eventually complete. ... View more

This is a repeat of HOWTO: query MySQL from Splunk on Linux 64bit, but that solution did not work for me. Running an Intel processor, but tried the solution with both x86 and AMD64 packages. Unlike the previous post, I do have root access. Running Splunk 4.2 on RHEL 5.5. 64bit ... View more