The development environment was easy. My indexer cluster production environment (Splunk 6.4.1) is making this difficult. Question is—what am I missing? I believe I have tried everything except the correct thing to filter out unwanted F5 heartbeat entries from the new Tomcat access logs.
What I have is a new Tomcat deployment and I am using Universal Forwarder (UF) to forward the information. I would prefer not to have to deploy a Heavy Forwarder (HF) for several reasons but may have to. Currently, I have all forwarders pointing to an intermediate forwarder where I would like to filter out these unwanted records prior to indexing. The intermediate forwarder points to the Distributed Management Console (DMC) which sends the data to the clustered indexers.
I have the Splunk Add-on for Tomcat on everything now and I already know that UF cannot filter data prior to forwarding. My final attempt before posting this query is on the intermediate HF I have the following configured under
/opt/splunk/etc/system/local/props.conf;
[default]
TRANSFORMS-set = dropChatter
/opt/splunk/etc/system/local/transforms.conf;
[dropChatter]
REGEX = (?m)(192.168.18.23[12])
DEST_KEY = queue
FORMAT = nullQueue
Several other configurations work as planned but those are all local files—this is forwarded data that travels “round the horn” to the indexers. Any help would be appreciated. Believe I have read almost every PDF and answer on this site to no avail. The Splunk Add-on for Tomcat is also installed on this server and configured the same way under /opt/splunk/etc/apps/Splunk_TA_tomcat/local
As a follow-up, I should have mentioned that the DMC is also the Cluster Master, License Master and a never used search head. The REGEX works on my DevTest 6.5.1 sandbox but I have tried others getting all old school with some (eg. \d+.\d+.\d+.\d+) and regex101.com accepts them all as PRCE compliant. The reason I took the caret out from the REGEX is the F5 will throw a dash in front of the IP on occasion.
I have the TA now installed all over the place with this trial and mostly error attempt of mine. Master/Slave, Deployment/Apps and as a standard install. What is boggling my little mind is there is no standard way of doing any inputs.conf or sourcing the stream - so to speak. Everything else has the standard [monitor:///] or [script:///] but I can not find one that fits this occasion. There is the standard [tomcat] header under the actual application.
As a follow-up, I should have mentioned that the DMC is also the Cluster Master, License Master and a never used search head. The REGEX works on my DevTest 6.5.1 sandbox but I have tried others getting all old school with some (eg. \d+.\d+.\d+.\d+) and regex101.com accepts them all as PRCE compliant. The reason I took the caret out from the REGEX is the F5 will throw a dash in front of the IP on occasion.
I have the TA now installed all over the place with this trial and mostly error attempt of mine. Master/Slave, Deployment/Apps and as a standard install. What is boggling my little mind is there is no standard way of doing any inputs.conf or sourcing the stream - so to speak. Everything else has the standard [monitor:///] or [script:///] but I can not find one that fits this occasion. There is the standard [tomcat] header under the actual application.
Where I had screwed the pooch is in the heading. I changed it to [tomcat:access:log] and restarted splunkd. I reckon once it had the sourcetype to filter against--it cranked right up. Went back today and added all of the F5 address in a regex that has now filtered out all of the chatter. Now to go back and fix access_combined and the IIS servers as well.
Also forgot that the server that handles the "around the horn" issues is also my Central Logging Server. Just do not want to write these log files out and then re-read them in. It also runs Splunk as well.
Hi dschmidt,
You mentioned:
The intermediate forwarder points to the Distributed Management Console (DMC) which sends the data to the clustered indexers.
Did you deploy the DMC on your cluster master? DMC itself only collects and reports on Splunk topology and deployment metrics, and does not send any data.
I think your configurations are correct, except for a potential problem in the REGEX. Maybe you should try tweaking your REGEX a little bit:
REGEX = (?m)^192.168.18.23[12]
Hope this helps. Thanks!
Hunter
REGEX = (?m)(192.168.18.23[12])