Getting Data In

Add zone tags to data passing HFs

Jasem
Explorer

Hello 

If I has several network zones and a HF in each zone forwarding logs to indexer cluster, what would be the best practices to label my data for identifying zones, for example adding a filed like zone-tag and when somebody searches for zone-tag= DMZ all data sent to  that HF from UF or syslog assets should be retirieved. I have three questions:

1. Do you think it's a good idea? If not, what are the alternatives? 

2. How much overhead in CPU resources and disk space it'll bring to Splunk? How to calculate or estimate?

3. How should it be done ? I mean configurations In each tier, i.e indexing, search, forward. 

Thank you.

Labels (2)
Tags (1)
0 Karma
1 Solution

PickleRick
SplunkTrust
SplunkTrust

Let me add my three cents to this.

As a general rule - adding an indexed field for storing metadata information which is not (easily) available within the event itself is precisely one of the cases when indexed fields is worth using.

Having said that, it can be done in several different ways.

1. It can be added at the source or close to the source. With an UF (or HF, if you have one for pulling modular inputs and such) you'd specify something like:

_meta = source_site::site1

for inputs. The downside is that:

a) You have to be able to push this setting to all forwarders. If you have some manually managed forwarders at the site and someone forgets to put the setting... it's not marked.

b) While you can specify this in [default] stanza for inputs - it will _not_ be applied for windows inputs. That's a relatively well-known idiosyncrasy of windows inputs - they don't inherit from default.

c) If you wanted more than one _meta entry - there is no way of "stacking" those. If you have multple entries, one will overwrite another. So if you wanted two, three or more indexed fields, you'd have to handle each combination of those manually.

2. If the data passes a HF, the field can be added on that HF. One important thing which I'd do differently than in @livehybrid 's answer - I'd not go for TRANSFORMS, I'd use RULESET. The difference is that RULESET will "fire" for any data going through HF. TRANSFORMS will not be applied if the HF receives already parsed data which might happen if you have HFs on your source site or if you're using indexed extractions (and that's the part people often forget!).

3. You could try spinning up the new Edge Processor node and route your data through EP instance and add your field there.

View solution in original post

livehybrid
SplunkTrust
SplunkTrust

Hi @Jasem 

Yes, it's a reasonable and something ive done with customers in the past using an indexed field.

Set a static indexed field at each HF, assume that the HF is common to all UF/syslog data from a given zone.

We found overhead is be pretty negligible for this use case, Indexed-field cost in the TSIDX scales with cardinality × event count, not just event count, but you only have a handful of zone values, so the extra lexicon entries are small. CPU cost comes from evaluating the transform per event so with no regex this is pretty cheap. It gets noticeably more expensive (resource-wise) only if you try to derive the zone from a regex over host/source on every event instead of hardcoding it per HF.

Configure only at the HF tier, nothing on UF/syslog or indexers:

# props.con
[default] TRANSFORMS-zone = set_zone_dmz

# transforms.conf
[set_zone_dmz]
INGEST_EVAL = zone="DMZ"

Repeat with the appropriate static value on each zone's HF. Putting it on [default] applies it to everything that HF parses, which is correct here since the field describes the forwarder's zone, not the data's sourcetype.

You should really then create a fields.conf with your Indexed-Field definition which goes on the SH too.

🌟 Did this answer help you? If so, please consider:

    • Adding karma to show it was useful
    • Marking it as the solution if it resolved your issue
    • Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.

PickleRick
SplunkTrust
SplunkTrust

Let me add my three cents to this.

As a general rule - adding an indexed field for storing metadata information which is not (easily) available within the event itself is precisely one of the cases when indexed fields is worth using.

Having said that, it can be done in several different ways.

1. It can be added at the source or close to the source. With an UF (or HF, if you have one for pulling modular inputs and such) you'd specify something like:

_meta = source_site::site1

for inputs. The downside is that:

a) You have to be able to push this setting to all forwarders. If you have some manually managed forwarders at the site and someone forgets to put the setting... it's not marked.

b) While you can specify this in [default] stanza for inputs - it will _not_ be applied for windows inputs. That's a relatively well-known idiosyncrasy of windows inputs - they don't inherit from default.

c) If you wanted more than one _meta entry - there is no way of "stacking" those. If you have multple entries, one will overwrite another. So if you wanted two, three or more indexed fields, you'd have to handle each combination of those manually.

2. If the data passes a HF, the field can be added on that HF. One important thing which I'd do differently than in @livehybrid 's answer - I'd not go for TRANSFORMS, I'd use RULESET. The difference is that RULESET will "fire" for any data going through HF. TRANSFORMS will not be applied if the HF receives already parsed data which might happen if you have HFs on your source site or if you're using indexed extractions (and that's the part people often forget!).

3. You could try spinning up the new Edge Processor node and route your data through EP instance and add your field there.

livehybrid
SplunkTrust
SplunkTrust

Good shout @PickleRick yes RULESET would work better and is just a quick change, covers already parsed data. 

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Kubernetes Alerts That Work Out of the Box with AutoDetect detectors in Splunk ...

A deployment rolls out slowly, one pod sits not ready for a few minutes, and whoever is on call gets an alert ...

Observability at Your Fingertips Directly From Splunk Platform—for Free

At .conf, we announced our roadmap to bring observability foundations to Splunk Platform. Bringing metrics and ...

Accelerate Splunk App Development with AI

  Learn how to automate the repetitive parts of your build cycle.   Watch On-Demand   Replay the Accelerate ...