I don't know what's happening at the moment. Someone else at splunk may. However you can probably work around the problem for the moment, by finding the auto-generated field extraction and modifying it (removing the underscores). I seem to recall that these end up in etc/apps/learned/local/transforms.conf ... View more

While it may seem counterintuitive, index = foo in props.conf has no meaning. The code won't do anything on that basis. In order to get the index to be set based on the host key in your data, there must be a TRANSFORMS key pointing to a transform stanza. At the input layer, controlled by inputs.conf, you can pre-set the target index for all data that input stanza receives, which is where you may have picked up this pattern. In this type of arrangement, I think you want to use the same TRANSFORMS-routing key for all four stanzas, set to different values, so that the stanza layering system simply selects a different transform for each data category. Eg. [host::211.167.20.156] TRANSFORMS-routing = to_ironport_index [host::211.167.20.29] TRNAFORMS-routing = to_test_index1 [host::212.171.24.*] TRANSFORMS-routing = to_test_index2 [host::212.171.214.165] TRANSFORMS-routing = to_cisco_asa_index [syslog] TRANSFORMS-routing = to_test_index3 With transforms.conf entries for each that probably look something like: [to_ironport_index] SOURCE_KEY=_raw REGEX=. DEST_KEY=_MetaData:Index FORMAT=ironport ... View more

I'm a bit distrustful of the capital S in Source::udp://514. Also, I thought events should have udp:514 as their source, without slashes. You should check in the search UI. We may default to using _raw as the source key, but I've always set it explicitly. SOURCE_KEY=_raw in the transform. I'm pretty baffled about the \n. Do your single syslog events actually have linebreaks in them? I would bet they don't. ... View more

This makes me wonder about sedcmd with repeat match behavior, and multiline events. Hopefully will come back with something useful later. ... View more

It should try to get all the ones in the main evenlog listing by default. You might want to review the splunkd.log (or in the _internal index) for errors about the wineventlog processor. There is a token in var/lib/splunk/persistentstorage storing a bookmark of where we are in the eventlog stream. Wiping that when splunk is down will cause us to start over. You could use manager or btool to review whether current_only is set to true/1 for your inputs. ... View more

Sounds more like a bug, or an attempt to do this on a light forwarder, than something easy to handle in answers. If it's a light forwarder, this isn't shocking. If not, we should probably work this via support, sorry. ... View more

Given our current strategy of minimal binaries per platforms, running on FreeBSD 7+ without compat6x means abandoning FreeBSD 6, which some of our customers still use, so this isn't an attractive option at the moment. As for ZFS, I, and we, appreciate the ER. It will help product management the most if you can convey why running on ZFS is important to you -- what are the business issues etc. Realistically you can already defeat the check with some minor hacking, but it means you are in a configuration which is not currently undergoing any testing. ZFS specifically, as opposed to other filesystems, does have some issues with high-io applications, especially high I/O and high computation tasks such as Splunk. Since ZFS performs checksum verification for all reads and writes, this means high I/O can reduce CPU avaialbility, for many Splunk loads causing a significant reduction in overall performance. This reduces, to some extent, the attraction of spending a good deal of effort in verifying and qualifying that platform. That said, please do tell us why it is important to do that work, via the ER channel. ... View more

When I'm unsure about this many components I look for verifiable quantities. The debug endpoint is useful. If we distrust that, a sniffer is useful. To that end, I suggested to hexx to possibly try ruling out variables by axing the trusted IP and other settings for now and seeing if user login can be gotten working at all. ... View more

That would work. You can cause the deployment class to incorporate the turn-on-light-forwarder-app if that makes sense to you. Some people seem to choose their mass-msi-rollout tools to manage splunk in an ongoing fashion, but I'm not an expert in that world. ... View more

This is unexpected, and more of a troubleshooting problem than fits neatly in splunk answers. It could be one of several underlying issues, including permissions. If you installed the app yourself, be sure the splunk user has access to the files that the app installed as. The specific error you're seeing is a splunkweb fail of some kind. The web_service.log may have a useful exception which helps clarify. The root problem is often being encountered by splunkd, however, and passed back up to splunkweb. splunkd.log may be full of interesting comments. This app was built for 3.x, and may need some tweaking for 4.x. Consider lifing the search directly out of the savedsearches.conf and see what you get there, for starters. ... View more

Perhaps one might prefer "*Exception:" ... View more

Be sure you're evaluating the speed of the search in context. In a summary report, the fields will not be extracted. In the flashtimeline view in the ui, they always are. Try the 'advanced charting' view for a better sense. The time window is set to the 1 day window, right? ... View more

I'm no apache wizard, but in our example configurations for internal testing, we do things like this: RequestHeader set REMOTE_USER %{REMOTE_USER}s I think you're sending the user as "User" not as "REMOTE_USER". You can do this, but you'd have to tell Splunk to expect this by setting the remoteUser setting in web.conf You can turn on splunkweb debugging if you like to try to see what's going on in more detail here http://www.splunk.com/base/Documentation/latest/admin/ContactSupport#Debug_Splunk_Web Note that this may log all http headers to web_service.log, if you mind that sort of thing. (Of course, you can always delete after). ... View more

It's possible to hide some data so that has been already indexed with the delete command, this makes it no longer searchable. http://www.splunk.com/base/Documentation/4.1.5/SearchReference/Delete http://www.splunk.com/base/Documentation/4.1.5/Admin/RemovedatafromSplunk It isn't reversible (and off by default), so measure twice, cut once. You can force reindexing by a couple of different methods. You could reindex everything using a splunk clean eventdata on your forwarders. You could force reindexing of specific files by copying them to $SPLUNK_HOME/var/log/splunk, though the paths will be a bit different. You can tell splunk to index a particular file regardless of the duplication logic with the oneshot input method: splunk help add oneshot Lastly, a bit dirty, you could get somewhat sneaky and defeat splunk's redundancy checking. If you modify the first 256 bytes of your logfiles, eg by inserting a single character of whitespace at the start of them, it will reindex those files, assuming they are totally new. ... View more

The simple answer is that setting splunk to monitor the specific path should work. If that is not working for you, then I recommend working with splunk support to resolve the problem. ... View more

I sure hope we don't consider # midline in a regex string to be a comment, but at the moment don't know for sure. Perhaps you could try REGEX = \x23Number of transaction records \x23 is just a way to encode the hexidecimal number of that character, eg #. ... View more