I'm wondering if your issue is due to your start/end event occurring on the same second, which could mean that the transaction command is seeing these events out of order which could be breaking your startwith/endswith logic, this is just a guess based on some things I've seen before... The following is an ugly and inefficient hack, but it should prove/disprove my theory; and if correct, perhaps this will lead to a better solution. your search | eval _time=if(searchmatch("LS-XUCF"), _time+1, _time) | sort _time | transaction ConsoleId LogonId Item startswith="LS-XUBF" endswith="LS-XUCF" | eval duration=duration-1 The basic idea is to bump your end event to occur after your start event (by shifting the end event by 1 second) and then resorting your events. We then subtract 1 from the duration to restore the proper duration of the transaction (if you care about that field). UPDATE: Whoops. Let's try this again. I copied your sample event logs to a file on my system to reproduce your transaction search. And I'm now convinced that my initial suspicion is correct, but my first example search was incorrect. (Got some of the logic wrong.) | file /tmp/txn_test | rex "[^|]+\|(?<Event>[^|]+)\|(?<Item>[^| ]+)\s*\|(?<ConsoleID>[^| ]+)\s*\|(?<LogonID>[^| ]+)\s*\|" | eval _time=if(searchmatch("LS-XUBF"), _time+1, _time) | sort _time | transaction ConsoleId LogonId Item startswith="LS-XUBF" endswith="LS-XUCF" A perhaps better solution would be to not mess with _time at all, but add a temporary field for sorting purposes, and then dropping it again. Perhaps something like this: ... | eval start_event=if(searchmatch("LS-XUBF"),1,0) | sort _time,start_event | fields - start_event | transaction ConsoleId LogonId Item startswith="LS-XUBF" endswith="LS-XUCF" It's arguable which approach is "better" (it all comes down to whether or not you think screwing with _time is a bad idea or not) but I'll let that up to you. BTW. I could also prove this theory using the on-the-fly file technique by simply moving the start/end events up and down. Which is essentially what we are doing with the sort . Remember that splunk returns all events in reverse time order (so latest events are first.) Therefore in your original sample listing the end events actually occur before the start events. For the sake of clarity, when I used this as the contents of my /tmp/txn_test file: 2010/06/18 22:17:08.041|LS-XUCF|foo |0635A84D13 |2 |1CB0F33E1DEDC8F 2010/06/18 22:17:08.041|LS-XUBF|foo |0635A84D13 |2 |1CB0F33E1376C4C 2010/06/15 19:54:24.178|LS-XUCF|foo |0635A84D13 |2 |1CB0CC475F2180D 2010/06/15 19:54:24.178|LS-XUBF|foo |0635A84D13 |2 |1CB0CC475A3ECA0 And used this search: (and I got the correct results) | file /tmp/txn_test | rex "[^|]+\|(?<Event>[^|]+)\|(?<Item>[^| ]+)\s*\|(?<ConsoleID>[^| ]+)\s*\|(?<LogonID>[^| ]+)\s*\|" | transaction ConsoleId LogonId Item startswith="LS-XUBF" endswith="LS-XUCF" I am curious as to why your events are out of order in the first place. I thought that splunk preserved the proper order of your events. (I'm not certain of this.) I'm assuming that in your actual log file that the LS-XUBF line occurs before the LS-XUCF line. Are you using a load balancer / distributed search? ... View more

In terms of managing external lookup files, you may find this post useful: http://answers.splunk.com/questions/3769/does-outputlookup-append-or-overwrite (but you may have too many events to make this option practical, idduno) ... View more

I added some additional thoughts to my post, but they may not be what you need. Not sure. I do want to point out that you can update your lookup .csv file anytime you want. You don't have to restart splunk to get the updates. Of course if your are looking at adding millions of new records to your lookup table each day,then you probably need to also consider an expiration policy which you would probably want to manage externally as well. ... View more

I'm not sure what you mean by "old logs"? I'm guessing your question has to do with how splunk stores events in general. There is also the discussion about event-retention policies which has to do with how long your events (or logs) are kept around after they have been indexed by splunk, but since your evaluating splunk, I'm guessing you aren't running to that just yet. Splunk stores all log as indexed events in a proprietary database-like "index" under your splunk install location. If your a looking for sizing information, it may be helpful to visit the directory where your data is stored. Out of the box, splunk contains several indexes (sometimes called "databases"). Here is the location of your "main" (default) index: $SPLUNK_HOME/var/lib/splunk/defaultdb The docs should give you a better idea of how this works. I would start here: What's a Splunk index? and follow the various links provided. ... View more

Anyone have an idea on when this kind of feature could become available? This seems like a very natural type of request to me that would be very beneficial to a lot of users. ... View more

What version are you running? The file monitoring is claimed to be much improved in 4.1, have you seen any performance differences in this latest release? ... View more

You can also get some information by searching some internal splunk logs: Based on web access to splunkd web (ip address only) index=_internal source="*web_access.log" earliest=-15m | top clientip User based on interactive searches: index=_internal source=searches | top user User logins based on audit logs: index=_audit action="login attempt" "info=succeeded" There doesn't seem to be any audit of any logout events, unfortunately. ... View more

I guess the other option is to NOT export the tags. I don't know about anyone else, but the default eventtypes bundled in the unix app don't seem to be very thought out to me. ... View more

Yeah, I've looked at the splunk-labs stuff a few times now, but as you pointed out, it seems outdated and not as easy to use as splunk's internal API. (I mean, sure you could reproduce the REST API fairly simply, but it seems a little silly to do so when Splunk already build a full python-based interface that is well maintained and tested.) Thanks for the exported docs. It still seems like splunk should provide an easier way to simply reuse their API from a different machine. (Doing a full splunk install just to get the python libs on a machine seems rather silly too.) ... View more

Certain field extractions are automatic disabled in 4.1 when it appears that the field is not needed. (An interactive search from splunk's web interface all fields are enabled.) I think all you should have to do is add a fields command to your search. Something like: your search | fields src_ip_addr You may also get the same effect by explicitly passing in the required_field_list to your dispatch request. (Was this just called fields in a previous version of the API? Maybe I'm remembering this incorrectly.) It seems like this should do the trick too, but I'm not 100% sure. ... View more

I thought about a couple of those options, but I missed the _serial field, I like that idea the best so far. Thanks for your help! And yeah, if there is any kind of internal feature request for getting the core search engine to only return some sort of sample like this, please add me to the list of requesters. I have a few other searches where I think that kind of thing may be beneficial. ... View more

To answer your question about by usage of sorting and timechart. First, I'm not actually calculating delay in this specific graph, I'm simply trying to make a "timeline" like graph based on the index time rather than the event time. I have a view that shows the normal "timeline" right next to the index-timeline, which lets the admin to quickly determine if indexing delays were caused by a forwarder restart, or find a more complex delay patten. (I had one forwarder that would only forward _internal events ever couple of hours. Upgrading to 4.1 seems to have fixed it..) ... View more

Have you tried contacting splunk support? ... View more

You should have no problem with missing values like this with the default delimiter-based extractions. BTW. I'm not sure if you intended to provide your literal props.conf entry or not. But what you have shown will not work as is. I assuming that you have the following: transforms.conf : [test] DELIMS = "," FIELDS = "name1", "name2", "name3" props.conf [my_source_type] REPORT-test = test ... View more

I'm not 100% sure I understand what you are trying to accomplish exactly, but there are a few ways to attach different searches together. If you would post an example or two of each of your parent and child events then a more specific example search could be provided. (Click "edit" under your question and add your examples to your question that way.) You have a couple of different options: Use a sub-search In this approach you would use a sub-search to gather distinct values of parent in a sub search, which would end up being added as part of your base search. eventtype=child-event [ search eventtype=parent-event | dedup parent_id | fields + parent_id ] Use the map search command. This is similar to the sub-search option except that it allows you to also search on a different time range within the secondary search, so say the child events are always within a 30 minute (before or after) the parent event. You could so something like this: eventtype=parent-event | dedup parent_id | eval end_time=relative_time(_time,"+30m@") | eval start_time=relative_time(_time,"-30m@m") | map search="search eventtype=child-event parent_id=$parent_id$ starttimeu=$start_time$ endtimeu=$end_time$" ... View more

Hmm. I tried your regex in a regex tool and it seems like your regex is actually working. Now it could be that there is some whitespace change when you posted your example, so perhaps you could try the following (slightly modified) regex. You can test your extractions interactively using using the rex search command: source=*sudo.log | rex "(?i)^[^=]*=\w+\s+:\s*(?P<message>.+?)\s*;\s*TTY=" Once your get your regex working with rex you can add it to tie it to a source/sourcetype within one of your local props.conf files: [syslog] EXTRACT-sudo-message = (?i)^[^=]*=\w+\s+:\s*(?P<message>.+?)\s*;\s*TTY= Instead of using [syslog] you could also make it based on the filename ("source") as well. Simply change the stanza name to [source::.../sudo.log] , for example. ... View more

Is the splunk service running? If not try starting both the "Splunk" and "Splunk Web" services. If these are both running, try opening a browser window to http://localhost:8000/ If splunkd.exe is still not running, check in your splunk install folder for the following path: %SPLUNK_HOME%\var\logs\splunk . You should see several logs files. Take a look at the end of splunkd.log and also look at any *crash* log files exist. If you see any errors here, then try updating adding to your question any additional info that you find from these log files. ... View more