Getting Data In

Help with filtering out data

skirven
Communicator

Hi! I'm trying to filter out data, and nothing I have tried seems to work.
What we're doing is taking our data inbound from a Heavy Forwarder, and then parsing it on another Heavy Forwarder, then sending it to the Indexer.

My use case is I want to filter out any event with "Closing" anywhere in the event for this particular file mask.

My setup is:

Target event has Source=/var/log/containers/iceservices-sales-32-n65ld_cct_iceservices-sales-22a0f7bd882bd61c179be102ade62c328ff15e5bdd963774f4313e12d877d263.log

Props.conf:
[source::/var/log/containers/iceservices*.log]
TRANSFORMS-ice=ice_drop

Transforms.conf
[ice_drop]
REGEX = Closing
DEST_KEY = queue
FORMAT = nullQueue

I've tried many permutations of this source, with restarts, and nothing seems to work.

Can someone help?
Thanks!
Stephen

1 Solution

woodcock
Esteemed Legend

If you are sure that your settings are correct (and it looks like they are), then it must be something else. If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier) UNLESS you are using HEC's JSON endpoint (it gets pre-cooked) or INDEXED_EXTRACTIONS (configs go on the UF in that case), then restart all Splunk instances there. When (re)evaluating, you must send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

View solution in original post

0 Karma

woodcock
Esteemed Legend

If you are sure that your settings are correct (and it looks like they are), then it must be something else. If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier) UNLESS you are using HEC's JSON endpoint (it gets pre-cooked) or INDEXED_EXTRACTIONS (configs go on the UF in that case), then restart all Splunk instances there. When (re)evaluating, you must send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

0 Karma

skirven
Communicator

Thanks all. Yes, I think now that the issue is that the props/transforms was not done on the first HF in the chain. I'm going to work next week with the owner of the other HF to get them to update settings.

I will update/mark as answered when I have confirmation it works.
Thanks!
Stephen

arjunpkishore5
Motivator

Do you have any indexed extractions defined on the source input ? If yes, this is probably getting pre-formatted and bypassing all parsing in the subsequent layers.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Why do you have a HF sending to another HF?
Where have you placed these configurations (which instance(s))?

---
If this reply helps you, Karma would be appreciated.

skirven
Communicator

@hrottenberg_splunk et al, I asked internally (Hal, I think this gets back to some of the HEC convos we have had) and find out.
-Stephen

0 Karma

hrottenberg_spl
Splunk Employee
Splunk Employee

Great q Rich. I'm thinking only changes made at the first HF hop will be effective, but am not certain.

0 Karma

skirven
Communicator

Network segmentation, mostly, I believe. I wasn't the one to set that up, unfortunately. (Queue the document on Inherited Deployments).

I will check with the Server Owner, but I think the server where those logs reside just have a HF installed on it, which forwards to our HF and then to Indexers. So then, would it be better to put that on the first HF in the chain?
Thanks!
Stephen

0 Karma

richgalloway
SplunkTrust
SplunkTrust

props and transforms need to be on the first instance that parses the data, in this case the HF on the server.

That said, I strongly recommend replacing all of your HFs with universal forwarders (UFs). UFs required fewer resources and can take the place of HFs in all but a few cases. "Network segmentation" is not one of those cases. Uses for HFs include running python scripts (including apps like DB Connect), filtering events, and masking data.

---
If this reply helps you, Karma would be appreciated.
0 Karma

skirven
Communicator

Ah! OK. I think I understand. So only the first HF in the chain can impact/change the data, and if you don't have a use case to do so at that point, use a UF, then you can use a HF later down the line to do that? But once you introduce a 2nd HF in the chain, it's automatically demoted to more of a UF in that sense?
-Stephen

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...