At those rates, the hardware will spend very little I/O and cpu time overall on indexing. When building dashboards it's not shocking if the first takes are somewhat inefficient. With 8 cores, you can expect to see up to 3 cores used by scheduled searches, and at that data load probably less than one core used for indexing. This means that you probably will get around 4-5 concurrent longer running searches before performance will begin to seriously drop off for the box reasons as opposed to disk reasons. I don't know how to estimate real usage where your uptake and time spent using the tool may vary. However if you imagine tens of concurrent search-running users (not idle in the ui), you will almost certainly need to spread the load further or add cores. If your searches will generally be over very recent time, then the operating system should be able to keep most of the working set in RAM, meaning that recent time searches shouldn't care much about SAN performance. Of course historical searches still will. ... View more

The anomalies command, http://www.splunk.com/base/Documentation/latest/SearchReference/Anomalies acts as a filter as well as a labeller. It tries to figure out, for a data stream, which events are unusual. It will pass through more unusual events, and filter out less unusual ones (configurable), and it will apply fields to events indicating how unusual it felt they were. If you used it for a particular thing, such as a web server log, it might help you identify trouble points, changes, and other things of interest. In this case, it seems you are pointing it at all data from a particular host. Since this is probably a heterogeneous set of data, I think anomolies will be very hard pressed to guess which is unexpected. The event you actually are seeing appears to be the listeing of processes running on that system, probably as produced by 'ps', probably as produced by the unix app. Compared to most data you receive, this event is very large, and differently structured, so it would be unsurprising if anomalies found it to be relatively unusual. ... View more

That was an experiment. As a customer I really hate it when companies tell me to file something again, via another means. Internally, I had to do some evil things to make it happen, so not sure you'll get that again 😛 ... View more

One possibility is the search quotas are being blown. If so you should get message banners about this though. ... View more

This is a splunk support issue. Please log a case via the splunk website if you have not done so already. http://www.splunk.com/support For most platforms, the crash log includes a stack trace. Is that missing in your environment? The registers are pretty hard to go on by themselves. Support will want to do the normal dance of "when did it start happening, how often, any correlations etc." The information you provide so far just simply tells us that some part of splunk is following a null pointer. It is quite a lot of work to get from there to what is wrong. It is likely not possible without a fair amount more data gathering. ... View more

It could be a bug. However there is a known behavior difference. In the search view all fields are requested. In a scheduled environment, only the requested fields are generated from the data. it's possbile that there's some further wrinkle about the fields command needing to be at the right point in the pipeline, I don't fully undrestand your search at the moment. You may want to try the search in the 'advanced charting' view which should skip requesting all the fields and possibly allow you to debug. ... View more

Are they still missing if you use |fields and specify that field? (I'd guess yes, but don't know.) ... View more

The quickest and dirtiest thing is probably to read sys.stdin, push it to a tempfile, and then run your script with that tempfile as an argument. Sucks but you can probably get a proof of concept going. ... View more

Yes, but according to my understanding the stdin of a subprocess should match the stdin of the launching process. Just be sure not to read the data in the launching process, and it should be available. We do do some tricks to arrange stdin to be a temporary file, but I assumed those were outside the process, not a python behavior. In the worst case you could use some nasty arrangement with running the other script via os.subprocess and communicating over a pipe. It's also windows differs regarding execv. ... View more

If you haven't already, take a look at http://splunkbase.splunk.com/apps/All/4.x/app:Splunk+UI+examples+app+for+4.1%2B ... View more

They should apply, and if they do not (and have that sourcetype), that is a bug. That configuration is not terribly extensive, so I'm uncertain what differences you are seeing. There could be a more complex case where the behavior relies on certain source:: based rules, which aren't matching for the upload behavior. ... View more

I'm unclear what you'd like the host value to be set to exactly. The default behavior for syslog is to pull the host out of the text of the event, because it should represent the host on which the event happened. This is accomplished by a transforms.conf entry referenced in etc/system/default/props.conf in [syslog]. You could turn it off by overriding the rule in local/props.conf in etc/system or your own app. If you were to turn it off, then host would be set by the udp input to the tcpip address of the host which sent the packet. It may be possible to adjust this to reverse resolve the ip address to a hostname. I know this feature exists for tcp inputs. It suffers from the usual confusion points where the original hostname reversemaps to something unexpected, but should work fine. If you want instead the hostname to be set to the name of the splunk forwarder, you may be able to disable the udp input hostname setting. I'm not sure which of these options gets at your problem. It may be that the NAT is simply hiding the possibility to determine the originating device. Syslog items are timestamped on the receiving side, typically, so the information may already be lost. You may want to look into putting a syslog reciever closer to the data sources, or using syslog over tcp. ... View more

What were you upgrading from? I thought the -2 artifact arrived in 4.0, but it might have been new with 4.1. This is an artifact of CHECK_FOR_HEADER, which is sort of always on for csv files, even if they are given another sourcetype, at least with older versions of 4.1. I'll have to refer to the specific bug, but i thought this would not happen for explicit sourcetype in 4.1.6. Do you expect to be able to get fields that are labelled by the headers of these files? If so you might need this code enabled, and perhaps sourcetype aliasing is the best solution. ... View more

Using the advanced charting view may speed it up a bit. Just getting that many buckets (timeslots) may end up being problematic. If you want to solve this problem for the future, summary indexing might be the right approach. If you want a very very rough idea, you could use the commands which probe index state such as are used to back the searches for the indexing dashboard. These tell you things about where your buckets exist in time, and then some fancy calculations could give you an upper bound on the number of events over that month. If you want just a total, you could try date_month=1 date_year=2011 | stats count by date_day ... View more

I'm feeling stupid tonight, so don't I don't want to get deep into the interaction of MUST_BREAK_AFTER and BREAK_ONLY_BEFORE. If everything is in one of these, you shouldn't need both. We can dig in later if it's needed? The first ruleset is looking for words like 'config' which have somethig besides backslash or s before them. I think you want BREAK_ONLY_BEFORE = ^config Does that work alone? ... View more

Generally speaking, no. This is more of a python / operating sytsem question than Splunk (not that I mind it) so you may get smarter / cleverer answers in python circles, but here are some of the pieces I know. The csv reader is a python library construct, so exists in the program, and has its state stored in the local program memory. The operating system has no idea that it exists. The operating system does know about files, but not about code interpreting them. A separate process will not have access to any information from the first process. There's a fuzzy bit to this, since on unix you can of course use fork() to copy a process and almost all of its state, but exec tosses most of the state out, and windows doesn't support the fork behavior anyway. The combination of these two means the second program will have to set up a csv reader again. However, the concept of the open standard-in/stdin datastream is, as far as I know, passed along to the launched subprocess. So you should be able to just set up the csvreader inside that script. If this is more of a problem of wanting to feed this data from a splunk-specific script to some other script that doesn't know that it will be getting csv, then you've got a relatively tricky little programming problem, where you'll have to have the csv reader happen in one process, which launches and performs some form of IPC to the other process in a manner that process is expecting, whether that's a named pipe, object remoting, or some other scheme. As for the stated problem: we run scripts with splunk python, but your python has pyodbc set up. I think you just skip this question, and set up the csvreader inside your namelookup.py inner script. The csv reader is a geneneric python module, and not specific to splunk, so will work just fine in both installs. The only special caveats for use of csv reader are: be sure to set up the csvreader to handle the header, which will always be present. Since some csv does not have headers, python doesn't do this automatically. You should probably give python a hint that it's OK if the rows / fields are large. Splunk's unstructured data can sometimes be large (big event comes through) and by default python just tosses an exception if this happens. I configured my csv reader to allow 10MB fields, when doing this. ... View more

This is where you probably want to start. Practically speaking, the rest_apps_view capability is almost certainly required, as well as rest_properties_get. Of course for the view to work, you'll need the search capability, and if you want the summary view you'll need get_metadata. For the full interactivity to work you'll need get_typeahead, ... pretty soon you've cloned or inherited the user role. ... View more

Practically speaking, Genti has your answer: make a new field, optionally hide the _time field (with |fields). I think there's an Enhancement Request here for formatting in email alerts. I'd appreciate if you could toss an email to support about it. Product management really grooves on stuff like, if you're willing. Not elaborate, but a thumbnail sketch. Here is our situation/what we are doing Here is what we want splunk to do To meet our needs like so Personally I wonder if this relates to general email formatting/presentation needs. Some questions I'd think to ask as an engineer: Is this needed for CSV, or table, or html output? All? None? Will typing a strftime string into a conf file get the job done? ... View more

restartss definitely works, because the appserver is entirely killed and starts up again. However, the browser cache may have to be defeated with shift-reload. I think your method tells splunkweb to update its cache-defeating url token. ... View more

It is true, typically. It's possible to set up a NAS to perform sufficiently, but the technical bar is higher. Note there's a wide variety of "NAS" hardware out there, from enterprise class down to 2 disks in a tinkertoy box. ... View more