Getting Data In

Help with filtering out data

skirven
Path Finder

Hi! I'm trying to filter out data, and nothing I have tried seems to work.
What we're doing is taking our data inbound from a Heavy Forwarder, and then parsing it on another Heavy Forwarder, then sending it to the Indexer.

My use case is I want to filter out any event with "Closing" anywhere in the event for this particular file mask.

My setup is:

Target event has Source=/var/log/containers/iceservices-sales-32-n65ld_cct_iceservices-sales-22a0f7bd882bd61c179be102ade62c328ff15e5bdd963774f4313e12d877d263.log

Props.conf:
[source::/var/log/containers/iceservices*.log]
TRANSFORMS-ice=ice_drop

Transforms.conf
[ice_drop]
REGEX = Closing
DEST_KEY = queue
FORMAT = nullQueue

I've tried many permutations of this source, with restarts, and nothing seems to work.

Can someone help?
Thanks!
Stephen

1 Solution

woodcock
Esteemed Legend

If you are sure that your settings are correct (and it looks like they are), then it must be something else. If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier) UNLESS you are using HEC's JSON endpoint (it gets pre-cooked) or INDEXED_EXTRACTIONS (configs go on the UF in that case), then restart all Splunk instances there. When (re)evaluating, you must send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

View solution in original post

0 Karma

woodcock
Esteemed Legend

If you are sure that your settings are correct (and it looks like they are), then it must be something else. If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier) UNLESS you are using HEC's JSON endpoint (it gets pre-cooked) or INDEXED_EXTRACTIONS (configs go on the UF in that case), then restart all Splunk instances there. When (re)evaluating, you must send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

View solution in original post

0 Karma

skirven
Path Finder

Thanks all. Yes, I think now that the issue is that the props/transforms was not done on the first HF in the chain. I'm going to work next week with the owner of the other HF to get them to update settings.

I will update/mark as answered when I have confirmation it works.
Thanks!
Stephen

arjunpkishore5
Motivator

Do you have any indexed extractions defined on the source input ? If yes, this is probably getting pre-formatted and bypassing all parsing in the subsequent layers.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Why do you have a HF sending to another HF?
Where have you placed these configurations (which instance(s))?

---
If this reply helps you, an upvote would be appreciated.

skirven
Path Finder

@hrottenberg_splunk et al, I asked internally (Hal, I think this gets back to some of the HEC convos we have had) and find out.
-Stephen

0 Karma

hrottenberg_spl
Splunk Employee
Splunk Employee

Great q Rich. I'm thinking only changes made at the first HF hop will be effective, but am not certain.

0 Karma

skirven
Path Finder

Network segmentation, mostly, I believe. I wasn't the one to set that up, unfortunately. (Queue the document on Inherited Deployments).

I will check with the Server Owner, but I think the server where those logs reside just have a HF installed on it, which forwards to our HF and then to Indexers. So then, would it be better to put that on the first HF in the chain?
Thanks!
Stephen

0 Karma

richgalloway
SplunkTrust
SplunkTrust

props and transforms need to be on the first instance that parses the data, in this case the HF on the server.

That said, I strongly recommend replacing all of your HFs with universal forwarders (UFs). UFs required fewer resources and can take the place of HFs in all but a few cases. "Network segmentation" is not one of those cases. Uses for HFs include running python scripts (including apps like DB Connect), filtering events, and masking data.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

skirven
Path Finder

Ah! OK. I think I understand. So only the first HF in the chain can impact/change the data, and if you don't have a use case to do so at that point, use a UF, then you can use a HF later down the line to do that? But once you introduce a 2nd HF in the chain, it's automatically demoted to more of a UF in that sense?
-Stephen

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.