I'm not sure how to get you 100% there, without playing with the data, but I'd try to avoid using Transaction, as it slows down the search considerably. I might try one of these options and see how close they get you: [search sourcetype=oracle_xml_* commit | dedup OS_Process | fields OS_Process] | table _time OS_Process Sql_Text | sort OS_Process, _time [search sourcetype=oracle_xml_* commit | dedup OS_Process | fields OS_Process] | stats values(Sql_Text) by OS_Process These will do a subsearch of the logs for anything containing "commit" and then pull out just the OS_Process. That way, you can do a search for just logs that actually have a commit in them, speeding up the search. You can then try to do a stats values(), or just a table with the texts. I think the latter is probably closest to what you want, but I'm not sure whether values() will sort the data oddly. ... View more

If this comes in CSV format, it might be best to use Splunk's wizardry to extract the fields early on: http://www.splunk.com/base/Documentation/latest/Data/Extractfieldsfromfileheadersatindextime If that's not an option (or so you can get up and running first), the following regex should do the trick, for the particular event you've pasted: YourSearch | rex field=_raw "\"(?<Field1>\d*) ms\",\"\d*\",\"(?<Field2>\d*) ms" | stats avg(Field1), avg(Field2) You can take that regular expression and move it into props.conf as well: [my_sourcetype] EXTRACT-extract_two_fields = "(?<Field1>\d*) ms","\d*","(?<Field2>\d*) ms (I don't believe you need to escape the quotes in props.conf -- you may need to put those escapes back in, if not). Should you have Splunk pull out the field names (as per the first link), you'd end up with ResponseTime="72 ms" and Jitter="0 ms" (or whatever those represent). You could then pull out the numbers by doing: YourSearch | rex field=ResponseTime "(?<NumResponseTime>\d*)" | rex field=Jitter "(?<NumJitter>\d*)" Let me know if that answers your question. ... View more

I've been using Ayn's method, myself, for some time, but transaction is painfully slow over a large log. In looking at this particular question, I actually found a vastly better method, courtesy of gkanapathy ( http://answers.splunk.com/questions/1478/sendmail-transactions ). If you're looking for data about a particular from address, it's much faster to filter on those qids first and then look specifically at the To addresses, if the below assumptions are correct: one email has multiple events all events include the qid the recipient address is only extracted once per email you have extracted the qid, sender and recipient fields Using: sourcetype=mail [search sourcetype=mail [email protected] | dedup qid | fields qid ] | stats count by recipient I'm not sure how to do a totals row in the same table.. If you're going to put this in a dashboard, though, you could do the above (sans the stats command) as a hidden search, have one post process for the main table, and then another post process for a single value field with the total number of emails sent. I went ahead and did a test in my environment, comparing tag=mail [search tag=ab_mail [email protected] | dedup ExchangeMSGID | fields ExchangeMSGID ] | stats count by RecipientAddress to tag=mail | transaction ExchangeMSGID maxspan=30s | search [email protected] | stats count by RecipientAddress over the last 30 days. The first one completed in 38 seconds; I killed the second one 4.8% in, after 227 seconds. Now I need to go re-write some of my own reports to use this better method... ... View more

The windows app is going to assume the pulling of logs off the local machine. There are two ways you can get this to work. The first (and easiest) would be just change the source="" in the windows app, and still use WMI to pull the data in. The second (potentially better, depending on your environment) would be to install a splunk forwarder on the domain controller. This will eat up fewer resources (WMI is very resource intensive) and give you more visibility, but does require installing software on your DCs. ... View more

I would take a look at the answer to this question: http://answers.splunk.com/questions/8079/cached-search-results Specifically, the loadjob search command, or the useHistory option. Alternatively, you can always use summary indexing: http://www.splunk.com/wiki/Deploy:Summary_Indexing ... View more

Taking your questions one by one: Does Splunk offer a split view of the events from the three different sources? I would like to see events that occurred at the "same" time, e.g. within a specific second, side by side. You can create your own view, that would offer something similar to that (though less robust than I imagine you're thinking -- e.g., two by two, or stacked), but the general theory with Splunk would be that you look at the event stream, with the events intermingled. If your timestamps are accurate, that will show you everything in one stream. One application's timestamps do not contain the date. I can't modify the application. Can I "suggest" a date to Splunk? I think this should get you up and running, but if you have your logs going in on a regular basis, it should work just fine. I'm not exactly sure how you'd have to set it up if you want to batch import past logs, so if that's your use case, you might need to open a different Splunk answers specificically for that issue. http://answers.splunk.com/questions/9531/extract-timestamp-without-date Can I delete all data from just one of those three sources? The easiest way is to grant yourself the can_delete role and then you can do a search for source="xyz" | delete. Note that this will make it not reported by Splunk, but won't actually remove it from the filesystem. For that, you would need to put each source in a different index and then... How do I delete all the test data I've indexed? From the command line, you can run splunk clean eventdata -index myindex (or without the index, to do the entire system). See this for more: http://www.splunk.com/base/Documentation/latest/Admin/RemovedatafromSplunk#Remove_data_from_indexes_with_the_CLI_clean_command Looking at Manager > Indexes, there are event in _audit and _internal. Where do these come from? Those are internal splunk indexes. One contains audit information (searches run, etc.) and the other contains statistics about throughput, license information, etc. You can look at the contents of them by running a search for index=_audit or index=_internal or even index=_audit OR index=_internal http://www.splunk.com/base/Documentation/latest/Admin/AuditSplunkactivity ... View more

I don't believe you can summary index the sort itself (or at least, that's not what will provide you the best benefit), but if what you're looking for is the worst 10 responses, then you could do a sort limit=10 -timetaken and toss just that in a summary index. That way, instead of going through 8 GB of data it will go through 10 records. If it's a requirement that it be the last 24 hours (e.g., -24h or -24h@h, versus -1d@d), you could get close to that by running the search every hour, or every half hour, and just taking the last reply, though it's going to have some degree of lag. I haven't played with overlapping periods in summary indexes, but I'm sure there is a way of pulling out just the most recent period. Does that make sense? ... View more

I would do a field extraction as follows: (?<Field1>.{3})(?<Field2>.{5})(?<Field3>.{120})$ The dollar sign will ensure that you're grabbing from the end of the line (making sure that you don't start Field1 in a timestamp, or anything like that). You can either toss that in your props.conf, or do a search like: YourSearch | rex field=_raw "(?<Field1>.{3})(?<Field2>.{5})(?<Field3>.{120})$" | table Field1 Field2 Field3 (Note that if your _raw doesn't have at least 128 characters, it won't extract any fields) Added: Your props.conf might look like the below. [yoursourcetype] EXTRACT-FixedFields = (?<Field1>.{3})(?<Field2>.{5})(?<Field3>.{120})$ And you can replace yoursourcetype with (from the documentation): 1. <sourcetype>, the source type of an event. 2. host::<host>, where <host> is the host for an event. 3. source::<source>, where <source> is the source for an event. 4. rule::<rulename>, where <rulename> is a unique name of a source type classification rule. 5. delayedrule::<rulename>, where <rulename> is a unique name of a delayed source type classification rule. These are only considered as a last resort before generating a new source type based on the source seen. See the props.conf doc for more: http://www.splunk.com/base/Documentation/latest/admin/propsconf ... View more

You can add a blacklist, telling splunk to ignore the compressed log files, e.g.: blacklist = \.(gz|bz2|z|zip)$ (From http://www.splunk.com/base/Documentation/4.2/Data/HowLogFileRotationIsHandled ) Edit: Based on Steven Sorkin's reply, this is only true for 4.1.7 and below. 4.2 will not require a blacklist. ... View more

You can have multiple maps on a single dashboard. I have one view with four maps showing, myself. I may have gone a bit too far, after hitting the same error you did, but I'll tell you how I differentiate them. I have different html files in <param name="src"> as you have above. In each html file, I reference a different settings_file and data_file . I also use different variables, since these are being included into the same page. So in my map of countries with the greatest number of phone calls, I have var soBiggestCountries = new SWFObject... and then soBiggestCountries.addVariable... Additionally, where it has the soBiggestCountries.write("...") I use a unique label (in this case, flashcontentBiggestCountries). Perhaps more useful than the explanation, the full HTML file for one of the different maps is: <script type="text/javascript" src="/static/app/amMap/ammap/swfobject.js"></script> <div id="flashcontentBiggestCountries"> <strong>If you see this message you most likley need to adjust your HTML file in this view or you are trying to view the HTML directly instead of in a Splunk view module.</strong> </div> <script type="text/javascript"> // <![CDATA[ var soBiggestCountries = new SWFObject("/static/app/CCM/ammap/ammap.swf", "ammap2", "100%", "400", "8", "#FFFFFF"); soBiggestCountries.addVariable("path", "/static/app/CCM/ammap/"); soBiggestCountries.addVariable("settings_file", escape("/static/app/CCM/ammap/BiggestCountriesCalled_settings.xml")); // you can set two or more different settings files here (separated by commas) soBiggestCountries.addVariable("data_file", escape("/static/app/CCM/xml_out/BiggestCountriesCalled.xml")); soBiggestCountries.write("flashcontentBiggestCountries"); soBiggestCountries.addParam('wmode', 'transparent'); // ]]> </script> My settings_file is the same between them (except where I want functional differences -- different maps, etc.) and the data_file is obviously different. ... View more

I would probably run the search this way: index=omg earliest=-1h latest=now test_name="Base Homepage" OR test_name="Base Storepage - Furniture" OR test_name="Base Product page - Memory Foam" OR test_name="Base Navigation page - Rugs" OR test_name="Base Cart Page" entry_type="Overall" | chart avg(response_time) as Response by test_name | eval Response=Response/1000 | eval Response=round(Response,2) | rename test_name as Page | join Page [ search index=omg earliest=-24h latest=-23h test_name="Base Homepage" OR test_name="Base Storepage - Furniture" OR test_name="Base Product page - Memory Foam" OR test_name="Base Navigation page - Rugs" OR test_name="Base Cart Page" entry_type="Overall" | chart avg(response_time) as Response by test_name | eval Response=Response/1000 | eval Response=round(Response,2) | rename test_name as Page | rename Response as "Previous Response" ] There is quite possibly a more efficient way of running this, but given that they're two non-overlapping time windows, I'd imagine this would probably work. ... View more