I updated the original post with a walkthrough to your problem. ... View more

To start with, if you're going to be dealing with specific data sets, you'll actually find your sub-searches are probably faster then using the transaction command on the entire dataset. Sub-searches can disperse the work to several different indexers while the transaction command resides entirely on the search head and is not map-reducible. That being said, I'll talk about questions on the transaction command. There's a lot to go over with the transaction command. When using multiple fields in the transaction command, Splunk assumes there is a "transitive" relationship and there is no repeat of the first instantiated fields. Meaning, if MID is created in the first line and has ICID on the second, splunk will then assume all events that have the same MID or the same ICID are supposed to be together. To make this explanation simpler, lets take a generic sample: event=1 host=a event=2 host=a cookie=b event=3 cookie=b In the sample above, |transaction host, cookie , would behave as you'd expect. You'll have 1 output that combines all three events. But what does splunk do when host changes but keeps the same cookie? This is the same type of concept that happens in mail, the mail id never changes, but all of the device identifiers change. you might see the following events grouped into a transaction: event=1 host=a event=2 host=a cookie=b event=3 host=b event=4 host=b cookie=b These 4 events have 1 field that links them, the same cookie. However when splunk looks at this "transaction", it sees 2 distinct transactions, here's why (simplified): event=1 host=a Splunk takes the first line and asks, do I have one of my fields in the field list? If the answer is yes, it knows the event is the start of a transaction. Next event, event=2 host=a cookie=b Splunk then asks the same question, does this event have any fields from the field list? Yes, it has 2. Does one field match a already defined field? Yes! Is it the same value? Yes! So this event must be part of the same transaction. Then splunk says, the 2nd field hasn't been set yet, so lets creates the transitive link between host and cookie. event=3 host=b Does this event already have a field that's been instantiated? Yes. Does it match the value of the first transaction? No. Therefore this must be a new transaction. event=4 host=b cookie=b Lastly, does it have an instantiated field? Yes. Does it match? Yes, it matches event 3. Does the 2nd field match anything in transaction2? No, cookie be has not been instantiated yet for the transaction, so lets instantiate it for the 2nd transaction. The outcome: ------------------------ event=1 host=a event=2 host=a cookie=b ------------------------ event=3 host=b event=4 host=b cookie=b Clearly not what we want. I want ALL 4 events in 1 transaction because they have the same cookie. This is where the transaction profiling app comes into play. We make the assumption that instead of a breadcrumb style transaction, you're actually going to want to build a transaction on top of a transaction. So instead of |transaction host cookie you'd really build this as | transaction host| transaction cookie There are some unique challenges with chaining the transaction command together, but the short answer, to make sure you don't lose any lines, prepend an eval expression: eval $uni_field$=if(isnull($uni_field$), (splunk_server+"#"+index+"#"+_cd), $uni_field$) | transaction $uni_field$ this will make sure that any event that does not contain the unifying field is not thrown away. You could then do a starts with ends with transaction for each piece of the transaction, and combine it at the end. Make sure if you chain together transaction commands that have startswith and endswith , you add keepevicted=t so you don't discard lines you may want to use later in the transaction. By using this approach, you can chain as many transactions together as you'd like, and it will capture all the events you're looking to capture. There is a very similar example to this question in the transaction profiling application that deals with multi-tier email systems. Now in your example, the clientip is only printed on 1 line, meaning you would have to use something like this for your search: earliest=-1h@m index="sistemi_legale" host=ironportout* sourcetype=ironport_mail_logs (MID=* OR ICID=* OR DCID=*) |eval eid=splunk_server+"#"+index+"#"+_cd |eval ICID=if(isnull(ICID), eid, ICID) | transaction ICID maxspan=300s startswith="New SMTP ICID" keepevicted="t" |eval DCID=if(isnull(DCID), eid, DCID) | transaction DCID maxspan=300s |eval MID=if(isnull(MID), eid, MID) | transaction MID maxspan=300s |search RemoteIP=1.2.3.4 Splunk will then have to combine every event over the last hour into a transaction and then search for the IP. This is highly inefficient. Using the subsearches will limit the data and let the indexers do all the work so that at the end you know you have only the events you need, and then run the transaction command on the outcome. We are currently working on some new commands to make this process easier, but I hope i've answered your questions on the transaction command. ... View more

Yes it is possible, you need to modify the brush. Let me know if you need more information on it, but you need to set the line stroke. Short answer: <param name="charting.lineBrush">dashedStroke</param> Here's the link: Brush Strokes Long Answer: I like to establish a core brush I use on a chart when I make modifications, in this case, a dashed line brush: <param name="charting.myBrush">dashedStroke</param> <param name="charting.myBrush.dashSize">10</param> <param name="charting.myBrush.dashSpacing">10</param> <param name="charting.myBrush.color">0xFF0000</param> <param name="charting.myBrush.thickness">5</param> Basically, this will create one super ugly thick red dashed line. When dealing with charts, you need to tell splunk how to reference the "series" of data (incase there is more then one line). <param name="charting.myBrushPalette">list</param> <param name="charting.myBrushPalette.brushes">[@myBrush]</param> <param name="charting.chart.lineBrushPalette">@myBrushPalette</param> We are telling splunk to now use a "list" for the series. This list only contains 1 option, therefore it's used for each line on the chart. You can specify additional brushes for line rendering other lines by following the same process. <param name="charting.myBrush">dashedStroke</param> <param name="charting.myBrush.dashSize">10</param> <param name="charting.myBrush.dashSpacing">10</param> <param name="charting.myBrush.color">0xFF0000</param> <param name="charting.myBrush.thickness">5</param> <param name="charting.myBrush2">solidStroke</param> <param name="charting.myBrush2.color">0x00FF00</param> <param name="charting.myBrushPalette">list</param> <param name="charting.myBrushPalette.brushes">[@myBrush,@myBrush2]</param> <param name="charting.chart.lineBrushPalette">@myBrushPalette</param> This will use the dotted line for the "first" series, and a green line for the "second" series, reguardless of name. If you want more control, use the "field" palette to define the name of the series to reference a brush. <param name="charting.myBrush">dashedStroke</param> <param name="charting.myBrush.dashSize">10</param> <param name="charting.myBrush.dashSpacing">10</param> <param name="charting.myBrush.color">0xFF0000</param> <param name="charting.myBrush.thickness">5</param> <param name="charting.myBrush2">solidStroke</param> <param name="charting.myBrush2.color">0x00FF00</param> <param name="charting.myBrushPalette">field</param> <param name="charting.myBrushPalette.fieldBrushes">{Average_Duration:@myBrush,Count:@myBrush2}</param> <param name="charting.myBrushPalette.defaultBrushPalette">@myBrush2</param> <param name="charting.chart.lineBrushPalette">@myBrushPalette</param> This will establish the "count" line as the dotted line, the "Average_Duration" line as the green line, then make any other line the same green line as "Average_Duration". ... View more

Splunk looks at the start of each line for the date information and matches to any known set date format, in your case, it looks to be matching and failing, so it looks to the file name, Director-110509.081528.log. Here's the order splunk looks for a timestamp: Splunk uses the following precedence to assign timestamps to events: 1. Look for a time or date in the event itself using an explicit TIME_FORMAT if provided. Use positional timestamp extraction for events that have more than one timestamp value in the raw data. 2. If no TIME_FORMAT is provided, or no match is found, attempt to automatically identify a time or date in the event itself. 3. If an event doesn't have a time or date, use the timestamp from the most recent previous event of the same source. 4. If no events in a source have a time or date, look in the source (or file) name. 5. For file sources, if no time or date can be identified in the file name, use the modification time on the file. 6. If no other timestamp is found, set the timestamp to the current system time (the time at which the event is indexed by Splunk). You may want to simply remove the date out of the filename and let splunk use the modification time on the file. Hope this helps! ... View more