Out of curiosity, any reason why you aren't monitoring these with a single [monitor://] and then setting up sourcetype rules in props.conf . It seems that Splunk 4.1 is more flexibal than previous versions, but I'm still wondering which approach is better. To be clear, here is what I'm suggesting: inputs.conf: [monitor://C:\Program Files (x86)\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1] whitelist = (native.*|(start|stop)Server|System(Out|Err))\.log$ Side Note: BTW, I saw your question where someone recommended using the " crcSalt = <SOURCE> " entry, but you really don't want this setting when you are monitoring regular log files because using this setting will disable splunk's automatic detection and proper handling of rotated log files. Of course, if you change it now, your files may be indexed again, it's tricky. Which is also why it should be generally avoided.) props.conf: [source::...[/\\]WebSphere[/\\]...[/\\]native*.log] sourcetype = WebSphere:NativeStdOutErrLog [source::...[/\\]WebSphere[/\\]...[/\\](start|stop)Server.log] sourcetype = WebSphere:StartStopServerLog [source::...[/\\]WebSphere[/\\]...[/\\]System(Out|Error).log] sourcetype = WebSphere:SystemOutErrLog Note: I use the ackward looking expression [/\\] to match both windows and unix style paths (that way I can use the same rule on either systems). You may want to match more than just WebSpehere and the log name in your path, but this is just to give you a general idea. You probably don't want to match just the file name (without any other portion of the path) then could could have collisions with non-WebSphere logs files. Of course, if you are not running 4.1 yet, then this would probably be your only option. I would be curious if anyone has any pros/cons to the multiple monitor approach vs the traditional props-based (source matching) approach? BTW: I've used this an approach like this to get certain log files to get the correct sourcetypes for quite a number of configurations. When I first started using splunk I really wanted to use the multiple monitor approach (like what is now available in 4.1), however I now see the props-based approach as much more flexible. Think about it this way, when you add a new set of log files, do you want to have to remember the exact name of every sourcetype (especially if your doing this a few months down the road), or would you rather have existing rules setup that do the work for you. Another advantage I see to the props-based approach is that you can interactively probe log files to determine there properties. This is great for new setups and for double-checking yourself after a config change. This feature has saved me TONS of time (and it works against the current config so you can test your props settings without restarting splunk.) To run the tool, open a command window, and cd to splunk's bin directory. Then run the following command: splunk test sourcetype "<path to logfile>" Ok, that was way too long. Sorry. Hopefully this was helpful. ... View more

That depends on the version. If you don't have any internal events being forwarded right now, you can enable forwarding of _internal events via the following: Splunk 4.0 (and earlier): You have to find the entry in inputs.conf that is responsible for monitoring your splunk internal log files and then add the mystical _TCP_ROUTING=* entry. So add something like this to one of your `local/inputs.conf files: [monitor://$SPLUNK_HOME/var/log/splunk] _TCP_ROUTING = * Note: Be aware that $SPLUNK_HOME/var/log/splunk/ log files can change based on platform. Also, in older version of splunk some of the log files were monitored individually, so you would have to add this for each [monitor:] entry. Splunk 4.1 [tcpout] forwardedindex.filter.disable = true See the outputs.conf docs, specifically the section called Configuring which events are forwarded by index. However, if you already have your _internal events forwarded and only want to forward your deployment log events, then you'll probably have to setup some transformers to re-route these events to a different index and then just forward that index. (That's just a guess. I hope, for your sake, it's not that complicated.... If so, we'll need some smarter people to step in.) Hope this helps ... View more

Yeah, I think so. If you are asking if it's possible to take the input, look at a certain field for your calculation, then return a completely different set of results based on your calculations (or whatever) then yes, you do that. That's not a problem. splunk.Intersplunk.outputResults expects a list of dictionaries, so as long as you pass that in then your search command should work. Splunk has no way of know if you are starting with existing event data and augmenting fields, or if you are building a new set of results from scratch--you have the freedom to do whatever you need. That said, you will want to make sure you have your search command setup properly in commands.conf . For example, you should tell splunk that you will not be retaining events, and that you don't want streaming mode (since you will probably want to operate on all results at once, and not a chunk at a time.) I'm guessing you've been on this page already, but here's a link anyways: Custom search commands Read over the "types of commands" section to give you an idea of what attributes you should setup for you command. (You may want to enable supports_getinfo mode and therfore use the isGetInfo() function if you have any kind of arguments that you script will need. I'm not 100% sure what the pros/cons are of using getinfo.) BTW. Be sure to check out $SPLUNK_HOME/etc/apps/search/bin/*.py for more examples. There are lots of examples to help you get started. I'm not sure what you mean by: Can I specify and empty row with just keys ... If you want to pass an empty value to splunk, then you can simply set the value to an empty string. results = [] for my_results in some_list: d = {} d["some_field"] = my_result d["empty_field"] = "" ... results.append(d) splunk.Intersplunk.outputResults(results) In this case "empty_field" would have no value. Because input and output is translated into and out of CSV format there is no way to differentiate between a missing field and an empty field. Also, different rows can have a different set of keys (fields) which is handled automatically by the outputResults() function (as well as multi-valued fields). ... View more

You can't exactly tell splunk to look in a certain column. However, you can tell splunk what comes before your timestamp, and you can write a simple regex to guide splunk to the right column by simply comma counting. This type of logic can be setup using the TIME_PREFIX props.conf entry. Is also a good idea to indicate a time format as well (using TIME_FORMAT ), and with out this sometimes splunk can mis-identify or otherwise miss your event's timestamp. So if your timestamp is always in the second column, try adding a config entry like this to your splunk/etc/local/system/props.conf file: [source::.../path/to/my/csvfile.csv] sourcetype = my_csv_sourcetype [my_csv_sourcetype] TIME_PREFIX = ^(?:"(?:""|[^"]|\\")+"|[^,"]+), TIME_FORMAT = %d-%b-%y # If your events are all on a single line, you should add: SHOULD_LINEMERGE = False BREAK_ONLY_BEFORE_DATE = False Note: This regex should allow for the first column to be quoted or unquoted. And if it's quoted, it a literal quote can be escaped by either doing a double quote (e.g. "" , which is the CSV default way of escaping a quote), or by using backslash-quote (e.g. \" ) which is another popular way of escaping quotes.... Wow, that gets ugly really fast. I haven't tested this will all possible options. Note that if your date and time are in a different column this can get tricky if they are not side by side. But often they are, in which case it's pretty simple do use a config entry like this: TIME_FORMAT = %d-%b-%y,%H:%M:%S Where we simply put a literal comma in between the date and time fields. If you haven't read this already, please take a look at the props.conf reference. Hope this gets you started. ... View more

You want to use the search expression: NOT status="*" Keep in mind that this is not equivalent to: status!="*" I recommend looking at this related question for additional explanation on this: How can I search for a missing field? I should also point out that if the word "status" is part of the event (as shown in your example), then you could simply search like this: NOT status Which could be faster since "status" would be a term in the index, which is a faster operation than comparing an extracted field. (Although, with a small number of events you may not notice the difference.) ... View more

Hmm. This could be due to a limit as to how many fields get extracted by default, but your event doesn't seem quite big enough to be hitting that. The other possible issue could be the dangling equals ( = ), but I don't know why Transaction values are being extracted, but ContentItem values are not. That's strange. I'm thinking you would be better off disabling automatic KV extraction and setting up your own explicit field extraction rules. (Which isn't too difficult, you'll want to rename the stanzas in my example to match your system.) Entry in props.conf : [my_sourcetype] KV_MODE = none REPORT-eq-fields = my_eq_extraction EXTRACT-fields = ^\S+ \S+ \S+ \S+ (?<log_level>\w+)\s* EXTRACT-thread = \[ExecuteThread: '(?<thread>\d+)' EXTRACT-queue = queue: '(?<queue>\S+)' ... Entry in transforms.conf : [my_eq_extraction] REGEX = ^\s+(\S+) += +(.+?)$ FORMAT = $1::$2 You could event prevent the not_set values from being extracted, if you wanted to (just as an example of of the flexibility that you gain by using the regex field extraction approach.) This could be accomplished with: REGEX = ^\s+([A-Za-z_]+) += +((?!not_set).+)$ ... View more

I don't think your issue is that splunk is indexing your event multiple times. Splunk will only index an event once. I think your questions is really about field extraction and multi-value fields. For WinEventLog:Security events, splunk already has setup the necessary field-extraction rules for you, so all the fields you needs should already be available. However, I think what you hitting is the fact that splunk allows multiple value for these fields. (Which is generally preferable over silently dropping one of the values.) I'm assuming that in your example, your are ending up with a multi-value field named Account_Name that contains the values of '-' and APP2$ . So what you probably want to do is simply drop the "-" value, which can be done a couple of different ways. You can build your chart or report and simply drop the "-" value after the fact. Or you can tell splunk to keep only the first or last value of a multi-value field, or filter the multi-value field. Here are a few examples to give you an idea of what's possible: Keep only the first value: sourcetype="WinEventLog:Security" | eval Account_Name=mvindex(Account_Name, 0) | ... Keep only the last value: sourcetype="WinEventLog:Security" | eval Account_Name=mvindex(Account_Name, -1) | ... Remove out the "-" value using mvfilter . (I think this is probably the best option for you) sourcetype="WinEventLog:Security" | eval Account_Name=mvindex(Account_Name!="-") | ... Remove out the "-" value (using a post-search command) sourcetype="WinEventLog:Security" | stats count by Account_Name | search Account_Name!="-" Here is one last search that will NOT work for what you want. (I feel this is an important gotcha to understand when searching with multi-value fields) sourcetype="WinEventLog:Security" Account_Name!="-" | ... Sometimes this may actually appear to work (based on how many events actually contain a dash value, vs how many event contain non-dash values). Here is what's really going on here: Any event that has a field named Account_Name where any of the multi-values is "-" will be dropped from your search results. Therefore, the example event that you provided will be excluded; which means that you will be loosing both the "-" and "APP2$" at the same time. So this approach is not what you want. Side note: If you are fairly new to splunk, I would suggest that you play around with these different options until you get a better feel for how these different approaches work. You may find that such an exercise will end up saving you time in future splunk searches. Just a thought. Additional resources: Functions for eval and where -- For more information about mvindex and mvfilter (and others). Manipulate and evaluate fields with multiple values -- A more general introduction to dealing with multi-value fields. ... View more

Can you explain what you mean by alias by tag? I think you may have a misconception about how tags and field aliases work. Tags: A "tag" is a human-readable (and user assigned) value given to a field/value combination. In other words, a "host" of "192.168.0.1" could be given the "VPN" tag. So you could search for this using tag::host=VPN , as you pointed out..) Splunk has supported the ability to assign tags like this for a long time. Field aliases: A field alias is basically a way to more gracefully accommodate for the fact that sometimes a single field can be extracted with different field names. For example, one sourcetype may extract an IP address as clientip (such as access_common ), whereas others may extract the same value as ip even though they represent the same thing. Of course, it's best to name your fields consistently, but that's not always possible. So in Splunk 4.0 field aliases were introduced. This way, you can indicate that ip is really an alias of clientip , which then allows you to do a search like: clientip=192.168.1.10 . In which case, splunk not only looks for this specific IP address in the clientip field, it now also looks in the ip field too. (BTW, there are other uses for this functionality as well, but this is probably the biggest use-case) So creating an "alias" of a "tag" doesn't really make a lot of sense. If you simply want a field/value pair to have more than one value, then you can simply assign multiple tags for that pair. However, I don't see any of this will help you with your fundamental issue where your host values (e.g. IPs) are changing over time. Keep in mind that tags are not date-effective, they exist across all time as far as splunk is concerned. In other words, there is no way to return different tag value for a single host at different points in time. Date-effective referencing like this can be done with the lookup feature (which was also introduced in Splunk 4.0), so perhaps there is a way for you to leverage that feature to suite your needs. Lookups can also scale better than tags. Take a look at: How many tags is too many tags? Helpful docs: About tags and aliases (be sure to read this entire section, not just the first page.) About lookups and workflow actions ... View more