Events in the event viewer will generally be blank unless they include a _raw field. Your sv file probably does not have a field (or column) called _raw, which is the text of the event. ... View more

The first release of the universal forwarder for HP-UX 11.22 / 11.31. There are no releases which specifically target the very first release of HP-UX 11. However, the whole topic of HP-UX versioning is fairly complex with patch levels and revisions, and it is difficult for me to comment on this with absolute authority of what might work. It's certainly true that 11i v2 11.22+ and 11iv3 11.31+ are the targets. ... View more

The performance characteristic of indexes for queries is not index size, but rather index density relative to queries. Putting the data in multiple indexes is not likely to help unless you make the data over an order of magnitude more dense relative to the searches are going to run. Meanwhile, if you were to do that, you would want that slice of data to be present on all indexers. Slicing the data up by indexer simply means you won't be able to do any horizontal scaling. ... View more

The most common cause of this symptom is that the data does not stop, but lands instead at an odd place in time. I suggest using an alltime-realtime search at the problem time to review the data, or else simply searching all time for your data to find data in the future or spikes in the past to see where the data might be landing. If that is the cause, frequently adjusting TIME_FORMAT to more accurately reflect the timestamps in your data is the solution. There are other possible problems but they are hard to imagine from the description. This might become a support issue. ... View more

FWIW, MAX_TIMESTAMP_LOOKAHEAD is only important when TIME_FORMAT fails. It is a good safeguard, but it will not increase performance when TIME_FORMAT works correctly. ... View more

This is not a good solution, as you are changing the way splunk guesses timestamps for your entire install. In addition TIME_FORMAT will produce better performance and more stable results. If you wish to create a custom datetime.xml, you should reference it within your sourcetype in props.conf. ... View more

The problems with externally loadbalancing splunk forwarders are tricky and not necessarily obvious. The mechanism by which forwarders splay their load across N indexers relies upon them periodically reconnecting to different indexers. However if you use a loadbalancer, then the forwarders will not know that there are different indexers and will not reconnect. This means they will effectively remain glued to a single indexer permanently via your F5. This has problems for balancing, since forwarders do not typically have equal data loads. It also has trouble for indexer lifecycle goals. If you shut down an indexer, forwarders will shift away from it. If you bring the indexer back up, forwarders will not shift back to it, because they do not know it exists. Additionally, depending upon your configuration, forwarders performing health checks at the time an indexer goes offline or closes its port due to overload may be seen by other forwarders as an indication that their current link is not healthy, potentially leading to an exacerbated problem under high load. If sending your forwarding data reliably through an external load balancer is important to you, please raise it officially with splunk staff via support channels (Enhancement Request). However for now I suggest you use the Splunk built-in load balancing. ... View more

Splunk should establish the date from the filename (though it may constrain it to MAX_DAYS_AGO/HENCE so if modtime and the filename differ really sharply, be careful with that). Then it should proceed to attempt to establiash both time and date from the event text, using your rules. ... View more

Are you sure (?=) acts as a capturing group? I think it's just a zero-width assertion that doesn't capture any text. Why not drop the ?= I'm a little skittish of a sourcetype like 20141013093007_16799.log though. That doesn't sound like a data format, which is what sourcetypes are. It sounds like the time of day at which the data was produced. I would typically want to call this data something like "app_1". Aside: you probably want to disable followTail, it's not really reasonable/safe and splunk only get the new data anyway. FollowTail is just useful when first setting up a forwarder. ... View more

Getting all three regexes is just a matter of making a regex that can repeat-match. You want a regex that will match url:"POST the_url" and not a byte more, and then you want to use the repeating features to pull it out repeatedly, eg |rex … max_match=3 , or in props.conf with a REPORTS this happens by default with eg REGEX=(url\d):"POST ([^"]+)" and FORMAT=$1::$2 The part about calculating on all of them at once is kind of over my head. ... View more

This is great. For what it's worth, the cluster master / indexers communication is bidirectional (maybe double-headed arrow?), and port 9997 is primarily a tradition though I think the GUI provides that default. There's a support generated diagram somewhere that labells SSL-by-default differently from others, but that's synonymous with the default-port 8089 traffic. Technically external apps can use REST API to any nodes in the system, but that's going to be only really useful for custom administration goals or custom troubleshooting goals typically. The search head is definitely the far most common target. ... View more

I moved this to an answer because it was the crucial insight. If you're willing to edit it a bit to be clearer for future readers feel free, or I'm happy to do so. ... View more

Yes, I specifically talked about how this limit is applied. For most Splunk installs, the required free space on the filesystem will increase from 5.x to 6.x. ... View more

It should probably be set to all time if the input is going to ignore it. ... View more

For syslog-udp there is a maximum possible size that the messages can be in the way they are transmitted over the network. Splunk cannot help here. For syslog-tcp, splunk does not even know that it is accepting syslog when accepting the data. There is nothing in splunk that truncates lines other than the TRUNCATE setting described by MuS, which defaults to 10KB. ... View more

This question is unsolvable. To be solvable, the minimum information set would be: The input stanza configuration A description of the paths on disk Other things that are likely needed A review of any messages related to the files in splunkd.log A search over all time to see whether any of the data is collected ... View more

It may well be an issue to use a 2GB mount -- I know we apply this check to index locations and the dispatch location, but I don't know if the check is disabled in the Universal Forwarder. It may not because the constraint is applied by the IndexProcessor (not loaded) and dispatch search machinery (not used, and would only block search anyway). If the limit does apply, for the Universal forwarder, you may want to tune this downward in server.conf [diskUsage] minFreeSpace. Do NOT tune this value down in indexer unless you enjoy risk of data loss (written here for future readers). ... View more

I assumed your search was all-time. Is it? ... View more

Well I misunderstood, because you linked to information about how to anonymize data at input/index time. If you want to anonymize data at search time you already have the answer that you basically can't. ... View more

I think this really is a question for Cisco/Ironport. How can the device be configured to produce its data in a complete way? ... View more

Yeah i think the typical syslog follows the recommended max length specification. Oops, I forgot. ... View more