Did some digging in the dispatch folder, and now I'm really confused. If I look in the events folder, there are a number of *.csv.gz files which, when added up only contain 10,000 events. But if I pull the job from the jobs manager, I can see all 200,000 events. So where are the other 190,000 events stored? I must be missing something. ... View more

I can confirm this behavior in Splunk 4.1. I ran the search * | head 200000 , saved the job, and when I try to load it using loadjob <sid> events=t , I only get 10,000 events. (I looked in limits.conf but didn't see any settings in there) ... View more

I don't think these are standard saved searches. What all splunk apps have you installed? ... View more

Since I posted my last comment. I too am running to an issue where I can't access an object shared at the application level. The only way to access it seems to be to set the access-level to global... very weird. (I'm seeing the same behavior in Splunk 4.0, BTW) ... View more

You are correct, you shouldn't have to export your objects to use them within that app. In my original scenario, my lookups worked within the app they were defined in, but not from another app. You may want to check your settings again, it's really easy to overlook some of the permissions settings. ... View more

Keep in mind that all charting is handled using Flash. So I'm not sure how you would go about changing that, or if it's possible at all. Out of curiosity, was the CSS change necessary for you to see an improved font, or due to a font-size preference? (In other words, did installing DejaVu help, or was it your CSS change that made the difference?) ... View more

I think you are mixing two things here. I think when you said " [lookups/MyFieldLookup] ", you mean " [transforms/MyFieldLookup] ". The [lookups/...] entries seem to be for specific files in the lookups folder, not the name of the lookup (which is in the transforms file). I think my actual problem was I was some how missing the export=system in my [lookups] entry. ... View more

If you have access to your splunk server, you may find that reviewing the search.log for your search job to be helpful. You can find in under $SPLUNK_HOME/var/run/splunk/dispatch/<job_id>/search.log . (You may want to "save" your search, so the job doesn't expire while your looking at the log file.) Look for anything unusual, such as error messages, what all indexes and buckets were accessed, repeating messages, etc. ... View more

Alan, are you having this problem only with built-in lookup tables or with custom lookups as well? I have user defined lookups that are giving me the same error message. I was suspecting that this is a permissions issue because the problem only appears in apps other than the one where the lookup is defined in. All the permission seem to be correct (transforms, props, and lookup/ are all globally shared). The same config worked fine in 4.0. ... View more

I setup the pdf server on Ubuntu 8.04, and installed the DejaVu fonts that were recommend by Ubuntu's firefox package. (Prior to setting up Splunk pdfreport app, we had no X install at all, so I had to install a handful of packages to satisfy firefox's dependencies.) You should be able to install the DejaVu fonts with this: sudo apt-get install ttf-dejavu-extra ttf-dejavu-core ttf-dejavu This may or may not be your issue. For the purpose of tracking down the problem, try opening the PDF with Adobe's reader. Pull up the document properties, check out the font tab and see which fonts are being used in the document. ... View more

NOTE: This approach works with Splunk 3.4.x (possibly earlier) and in Splunk 4.0+. The oneshot mode may be preferable if you are only working with Splunk 4.0+. I've had success by copying log files into the $SPLUNK_HOME/var/spool/splunk folder, which is the default batch mode input. This works best if your historical log files are already on an indexer or forwarding instance. You can add a special ***SPLUNK*** header to the start of your file to give it the original path, if that is important to you. (You can also set index , sourcetype , and host .) You could for example, load old mail logs from your old directory with a set of commands like so: (echo '***SPLUNK**** source=/var/log/mail'; zcat /var/log/old/mail*.gz) > $SPLUNK_HOME/var/spool/splunk/mail.log This uses your shell to match files with simmilar names and copy them to a single location. And you can use other simple shell scripting techniques to make this a simple or complicated as you need. You can also do some data throttling to keep from overwhelming your indexer with some sleep commands. Yeah, not a very high-tech solution, but it can work.) Be careful when loading event that are more than a year old when missing the year portion of the event's timestamp. (For example, syslog files often don't show the year.) One way around this problem is put a timestamp in the filename (or at least a 4 digit year portion of the timestamp). Most often, splunk will recognize this and load the events with the correct date. But it's best to keep an eye out for this problem. You may also want to consider using a separate index for loading historical data. Splunk 4.0+ handles wide date ranges much better than earlier versions, but there could still be advantages to using this approach even with Splunk 4.0. For example, if you aren't sure you have you have your props.conf indexing settings setup correctly yet (timestamp parsing, sourcetype matching, etc), then loading into an independent (and potentially throw-away) index could pay of big time in terms of cleanup time (especially compared to manually finding and deleting incorrectly indexed events.) You can always merge buckets from your temporary index into your desired destination index. ... View more

Thats a good point. I've updated the post to add a disclaimer. I certainly did not intend to present this as a reusable solution; I was simply pointing out a starting point that leverages the indexing metrics. Hopefully the post is clearer about that now. ... View more

More info would be helpful. For example, are you asking for interactive searching purposes, or are you trying to re-use you search results in a form search or dashboard? It is possible to this in either way, but what it's all depends on your scenario. ... View more

I've resorted do doing the permission thing with my summary index searches. I've also tried the submenu thing (you know, name the menu "Summary Index -- DO NOT RUN --", which looks great and leads to no unwanted questions. right.) In combination it's not too bad. The menu disappears if the user doesn't have access. I agree that it really seems like there should really be an easer/better way to do this. I like how views have the isVisible option. I wish something like this were possible with searches. Does anyone know if this has been officially submited as a feature request? ... View more

Have you tried using the charting view instead of the default flashtimeline search view?This might give you what you are looking for. I liked how you could temporarily change your shown fields using the fields command in Splunk 3.x, but it didn't seem possible in Splunk 4, at least until I discovered this trick... You can get to the "Advanced Charting" view from the menu or tack by tacking "charting" to the URL path. Once your in the Advanced Charting view, you can minimize the Chart and formatting areas, and to focus on the results area. Then you can tack on your fields command to your search (something like | fields + field1 field2 ... ). And now you should only see your fields in the "Events Table" results. So you can see only the fields you want, and in the order that you defined. (Unfortunately, it doesn't work for the fields shown in the "Events List" results pane, which is a pain.) ... View more

Just as a word of warning. Multi-line searches can cause some problems in some of splunk's internal logs. This is mostly due to props settings for the log files. Normally this shows up as time-stamping errors in the splunkd.log source. For example, the auditrail, splunkd, and searches (I think) sourcetypes are not setup for multi-line events. And in some places the multi-line searches get logged across lines which causes the timestamping issue. Other times, the searches get wrapped as a single line, and this isn't a problem. (I tracked this down once, I think. But can't remember what the common factor was as to when this seems to happen.) In my own config, I've added SHOULD_LINEMERGE = True to these sourcetypes to make this problem go away. (I realize this probably isn't as efficient, but it seemed like the best workaround) ... View more

Certainly classifying your events at indexed time using a new sourcetype, rather than with an eventtype at search time has the potential to improve your search performance. This, of course, depends on how efficient your existing eventtype declaration is, which is determined by the search terms and how effectively they can be used to select the desired events and filter out the irrelevant ones. The primary advantage of a sourcetype in this scenario, would be that sourcetype is an indexed field and can therefore limit down your results faster than multiple terms in your eventtype. But again, you probably will not notice a difference unless you have a really complex eventtype definition, in which case, it's probably not easy to filter these out into a separate sourcetype in the first place. Of course, there are other potential advantages to using a separate sourcetype, but that's a different question. I am curious about your indexed fields. You state that they are based on your eventtype, but I assume you mean either: You have fields that are indexed based on some source/souretype/host value (setup in props.conf), and that they events are matched at runtime using your eventtype. You are actually using search-time field extractions, when you think they are indexed fields. Using indexed fields should generally not be slower than extracted-fields. The reason why indexed fields are not generally recommended is because they are difficult to setup and maintain and they count against your license quota. But rarely (if ever) should they be slower than search-time fields. (You may want to double check that you have them setup properly in the fields.conf file. One quick and dirty test is to try search for my_indexed_field::value instead of my_indexed_field=value , this forces search to lookup my_indexed_field as an indexed field, rather than by using a search-time extracted fields. I actually end up using this trick frequently since I have some field names that are indexed some places, and extracted other places. For the record, I recommend avoiding this type of confusion as much as possible.) I would recommend that you start by reviewing you existing eventtype definition, and make sure there isn't a more efficient search you could use. The search should be faster if you aren't relying on extracted fields in your eventtype. Try to stick with only indexed terms. Avoid leading * s. If possible, use punct , this can make a huge difference. Play around with different combinations of search terms an see if you can find something that works better. I'm also curious if your search is slow when you search on your fields, or anytime you search against your events in question? Are your other searches slow, or just these events? What kind of timeframe are you searching? ... View more

I've found that a good way to track this kind of information is by leveraging the indexing metrics on the splunk indexer instance. The indexing metrics are captured every 30 seconds for the top 10 source, sourcetype, index, and host. (You can up the number or series in the limits.conf file, under the [metrics] section.) Using metrics, you can look at and monitor trends by total kb, total events, or average events per second, or number of times a specific series shows up in a sample time frame. (There are more ways to split this out, that was just a few stats that I've found helpful.) We have an email alerting saved search setup to run every 5 minutes (from -6m@m to -1m@m) that uses the 'source' metrics to point out any log files that are becoming too chatty (which may lead to exceeding our license usage, which is the primary scenario this search was setup to alert us about.) Here is a slimmed down version of our search: index=_internal sourcetype=splunkd Metrics "group=per_source_thruput" NOT (splunk ("metrics.log" OR splunkd.log OR web_access.log OR splunkd_access.log)) NOT "/var/log/ftpd.debug" NOT wineventlog:security | stats avg(eps) as eps, count, sum(kb) as total_kb, avg(kbps) by series | search eps>=3 count>=3 total_kb>100 The list of NOT s are used to exclude known sources that produce a large amount of logs, and are also unlikely to be run-away-inputs (as apposed to some other poorly written apps that are likely to get stuck an an infinite loop and write out a few gigs of nonsense in a 5 minute window if you don't kill it fast enough.) The limiting search at the end was more trail and error than anything. The idea is that I only want to know about repeating offenders ( count>=3 ), don't bother me with anything too small ( total_kb>100 ), and we have to be seeing a certain sustained level of activity ( eps>3 ). This approach is not at all adaptive based on series, other than to filter them out. But that's a limitation of my search, not the logs. PLEASE NOTE: Please understand that this search provided here is only an example. So don't just copy it, run it, and expect to get sane results on your system. This is simply one possible way of tracking your indexing/usage patterns, but this is merely a starting point, not a solution. I'm been meaning to overhaul this alert. We take aggregate snapshots of the indexing metrics info that end up in the summary index for 15 minute and 24h intervals. (This was setup for long term indexing analysis, since _internal is very short-lived.) I would like to find a way of combining these snapshots with live metrics events to build an alert to show a series that is exceeding the some baseline value which is established based on previous activity, say the last week. For example, Make the alert report any series that is exceeding the previous average by 1.5 standard deviations. (This would take some work, but it seems doable... If I come up with something helpful, perhaps update this post.) ... View more

That's what I thought, but it never seemed to work for me. But I think I found the problem. It appears that splunk-optimize-lex waits for optimize.result to exist first. ... View more