We have a requirement to collect data from testing enclaves (that have copies of production devices) to our primary Splunk environment. I have event data going to a separate index through a heavy forwarder. What I am concerned about is the Internal index. I will need to track if I have Splunk UFs on clients in the enclave. Is there a way to mark or tag the data coming through the Heavy Forwarder to indicate that it is coming from that testing environment?
One of the options you have is to add a metadata field to the events, which will require you to update forwarder configuration whenever a host moves between environments. Here is an answer that describes the process.
An alternative approach is to create (and - the tricky part - maintain) a lookup file that maps host names to enclave at search time.
Do the UFs on Testing enclaves follow any particular naming conventions (basically how can you differentiate a UF from Primary vs UF from testing enclave)? How are those UF's receiving configurations, deployment servers or direct?
The flow of data from the Enclave is UF --> Heavy Forwarder --> Production Splunk --> Enclave Specific Index
They are copying devices into the enclave from production so we will have duplicate names (host names and domain names) between the enclaves and in production.
I guess the easiest place would be to hit the events at the HF layer but I'm unsure how I can do that.
Can you try the dbinspect command? Following is a sample which may suit your need.
| dbinspect index=_internal
| stats sum(eventCount) as eventCount by splunk_server