Hi! I'm trying to filter out data, and nothing I have tried seems to work.
What we're doing is taking our data inbound from a Heavy Forwarder, and then parsing it on another Heavy Forwarder, then sending it to the Indexer.
My use case is I want to filter out any event with "Closing" anywhere in the event for this particular file mask.
My setup is:
Target event has Source=/var/log/containers/iceservices-sales-32-n65ld_cct_iceservices-sales-22a0f7bd882bd61c179be102ade62c328ff15e5bdd963774f4313e12d877d263.log
Props.conf:
[source::/var/log/containers/iceservices*.log]
TRANSFORMS-ice=ice_drop
Transforms.conf
[ice_drop]
REGEX = Closing
DEST_KEY = queue
FORMAT = nullQueue
I've tried many permutations of this source, with restarts, and nothing seems to work.
Can someone help?
Thanks!
Stephen
If you are sure that your settings are correct (and it looks like they are), then it must be something else. If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier) UNLESS you are using HEC's JSON endpoint (it gets pre-cooked) or INDEXED_EXTRACTIONS (configs go on the UF in that case), then restart all Splunk instances there. When (re)evaluating, you must send in new events (old events will stay broken), then test using _index_earliest=-5m
to be absolutely certain that you are only examining the newly indexed events.
If you are sure that your settings are correct (and it looks like they are), then it must be something else. If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier) UNLESS you are using HEC's JSON endpoint (it gets pre-cooked) or INDEXED_EXTRACTIONS (configs go on the UF in that case), then restart all Splunk instances there. When (re)evaluating, you must send in new events (old events will stay broken), then test using _index_earliest=-5m
to be absolutely certain that you are only examining the newly indexed events.
Thanks all. Yes, I think now that the issue is that the props/transforms was not done on the first HF in the chain. I'm going to work next week with the owner of the other HF to get them to update settings.
I will update/mark as answered when I have confirmation it works.
Thanks!
Stephen
Do you have any indexed extractions defined on the source input ? If yes, this is probably getting pre-formatted and bypassing all parsing in the subsequent layers.
Why do you have a HF sending to another HF?
Where have you placed these configurations (which instance(s))?
@hrottenberg_splunk et al, I asked internally (Hal, I think this gets back to some of the HEC convos we have had) and find out.
-Stephen
Great q Rich. I'm thinking only changes made at the first HF hop will be effective, but am not certain.
Network segmentation, mostly, I believe. I wasn't the one to set that up, unfortunately. (Queue the document on Inherited Deployments).
I will check with the Server Owner, but I think the server where those logs reside just have a HF installed on it, which forwards to our HF and then to Indexers. So then, would it be better to put that on the first HF in the chain?
Thanks!
Stephen
props and transforms need to be on the first instance that parses the data, in this case the HF on the server.
That said, I strongly recommend replacing all of your HFs with universal forwarders (UFs). UFs required fewer resources and can take the place of HFs in all but a few cases. "Network segmentation" is not one of those cases. Uses for HFs include running python scripts (including apps like DB Connect), filtering events, and masking data.
Ah! OK. I think I understand. So only the first HF in the chain can impact/change the data, and if you don't have a use case to do so at that point, use a UF, then you can use a HF later down the line to do that? But once you introduce a 2nd HF in the chain, it's automatically demoted to more of a UF in that sense?
-Stephen