I have a heavy forwarder that I am receiving an array of data on from port 514. In this case, I would like to break out esxi syslog data, and I can this via REGEX quite easily, however when I make the configurations in the HF, it doesn't seperate the data before sending to the indexers. I would like to keep the 'parsing' load off of my indexers if at all possible.
Sorry had to repost using different answers account. So to go back to the discussion.
Splunk cannot natively perform as well as a native syslog service like syslog-ng or rsyslog. If you are having restart issues then I suspect you are not log rotating your files and blacklisting the tgz extension via the inputs definitions. This will cause the UF to check files that haven't changed and won't change again and track them. That has checksum performance issues for the UF on start and ulimits issues for the OS.
My recommendation is log rotate them to archive and blacklist the extension. If storage is an issue mount cheap storage and have log rotate also move the files out of the live monitored path after say a day or two so the UF has a chance to pick them up.
So I stand by my suggestion that using a native syslog service with a solid configuration and log archival is the best solution. If you don't have cheap storage point to mount you can always rely on the Splunk indexed data retention methods if they suit your requirements.
If you are writing to folder structures based on a good naming convention you don't have to edit the inputs every time you add a new device. So edits to index/sourcetype are minimal for new devices of a type you already have defined.