thomastaylor, Lets break this down a bit... HTTP Event Collector: A Heavy Forwarder is a great option here. You can manage the token and receive HEC inputs on the HWF without the need of the main Splunk install to do anything. As the data is JSON, you'll also get your field extracts "for free" from autokv. Transforming data: Yes you can use a Heavy Forwarder for this. I must caution that there are a number of pitfalls that come with using a HWF to "pre-parse" data before it hits the indexers. Cooked data is larger on the network than uncooked data: https://www.splunk.com/blog/2016/12/12/universal-or-heavy-that-is-the-question.html - Some have theorized that unless you're doing a massive amount of Index time operations, the load on the indexers is actually higher CPU wise too (Still an argument in the community so take this with a grain of salt). Heavy Forwarders tend to cause data in-balance on Indexers (They get sticky to which indexer they send to, due to not having a break in incoming traffic. A common problem for syslog boxes that use a HWF). The Indexers are not given a second chance to parse the data - This means if your main Splunk install needs to do sourcetyping, index renaming or host renaming, it will be unable to (Well there are some special things you can do to cheat here, but it's a bad idea) Creating field extracts: You are unable to create "search time" field extracts with a Heavy Forwarder. The vast majority of TAs you'll find on Splunkbase are search time. Additionally, creating "index time" field extracts comes with a whole list of caveats (NOTE THE CAUTION WARNING: http://docs.splunk.com/Documentation/Splunk/7.1.1/Data/Configureindex-timefieldextraction). While possible, you're opening yourself up to a massive list of potential issues. To name a few: Greater storage requirements (index time fields are stored in the TSIDX files, uncompressed) Lack of flexibility (Once a field is written, it's "burnt" into the index) Potentially extreme CPU overhead at the HWF level Also, no the HWF will not let you use the regex tool - that's for search time field extracts. You'd have to have a dev search head / indexer for it and lift the extracts AND convert them to index time. DO NOT RECOMMEND. TLDR: For HEC, I think it's a great use case for you. For everything else, I'd advise against it. I'd recommend attempting to fix the relationship with whomever owns your Splunk install. You're setting your team and the Splunk owners up for potential issues down the road (and a bunch of up-front work for yourself as nothing on Splunk base will be plug and play). ... View more

Chustar, You're correct that you need to deploy them with the deployer. As you said, a "custom_configs" app is perfect for this. I do see some folks splitting these custom configs up into a few apps, such as: Indexes (for autocomplete). Often there is an indexes app that can be copied from the Indexer Cluster Master Node and re-used. Just remember to adjust the volume configurations or make them present. Outputs app. Often there is an outputs app already configured correctly for Universal Forwarders in the environment on the Deployment Server. A "SHC_settings" app for anything left over. limits.conf, server.conf, alert_actions.conf, etc. Whatever you need to do here that's SHC or SH specific. There's no requirement to split them out into apps, but if you already have apps that do most of the things you'd like to push to the SHC then it makes sense to reuse them. If you prefer a combined app, that's fine too. ... View more

Here's a regex that should work for you: \[(?<Thread>.*?)\]\s(?<TimeStamp>(\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})?)\s:\s(?:T:(?< ResponseTime>\S+\s+\S+))?(.)*(?<Action>(\w{3})):\s\[(?<XML>.*?)\] This should handle the timeouts either existing in the log or not and only create the "timeout" field when they are there. Hope this helps! ... View more

rbardonetorian, "time_before_close" Will cause Splunk to wait a specified amount of time after Splunk has reach an EOF condition. The default of 3 seconds can be too low on systems buffering their writes or very heavily loaded systems. That will cause Splunk to truncate events. Another option that will help is "multiline_event_extra_waittime = true". I'd recommend using this setting in combination with "time_before_close". Between these two settings, Splunk will wait longer for writes to happen when they're "mid event" and that will reduce event truncation significantly. Some draw-backs of "time_before_close" are that Splunk will use extra file descriptors as it is keeping more files open longer. Lastly, don't use "followTail" unless instructed to do so by support. It doesn't sound like it will help in your situation and will likely cause more issues than it solves. ... View more

A couple of issues here: Don't use indexed fields for this. Unless you have a very specific reason for trying to use an indexed field you're just causing more issues than you're solving. Because of your use of indexed fields, the MV_ADD does not work. A fix is going to be using search time fields like so: props.conf: [rf_ip] REPORT_rfip = rf_ip transforms.conf: [rf_ip] REGEX = \b(?<rf_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b MV_ADD = true Lastly, remove your fields.conf. All of this needs to exist on the Search Head, as it's a search time configuration. ... View more

hippe21, You can use a subsearch to accomplish this: |inputlookup test1.csv | search NOT [search index=_internal |dedup host | table host] This search will take your CSV and elemenate hosts found in the subsearch. The results in your case woulkd be a table with: environment,host prod,server102 Obliviously, modify the subsearch and CSV names to suit your environment. If you'd like to look at your data as the only indicator, i'd recommend | tstats: | tstats count, latest(_time) AS last_seen where index=* by sourcetype,host | eval timeDiff=now()-last_seen | search timeDiff>900 Change "900" to how long you'd like to consider something missing in seconds. | tstats is going to be significantly faster than | metadata. ... View more

sassens1, I'm a big fan of using "Input Addons" aka IA-thing. So it sounds like you could the following: Push the default Splunk_TA_Windows to everything that needs it, with no inputs enabled. Create a baseline IA-windows that collects standard logs from all systems and deploy to all. Note - if you need to send some system's logs to specific indexes, then there may have to be mutliple IAs here too. Create N number of specialized IA-* to collect specific logs from specific sets of systems. So I agree with the idea, but use this as an opportunity to make the names make more sense. ... View more