How can I tag or mark _internal events from different environments?

[sniderwj]

We have a requirement to collect data from testing enclaves (that have copies of production devices) to our primary Splunk environment. I have event data going to a separate index through a heavy forwarder. What I am concerned about is the Internal index. I will need to track if I have Splunk UFs on clients in the enclave. Is there a way to mark or tag the data coming through the Heavy Forwarder to indicate that it is coming from that testing environment?

[s2_splunk]

One of the options you have is to add a metadata field to the events, which will require you to update forwarder configuration whenever a host moves between environments. Here is an answer that describes the process.

An alternative approach is to create (and - the tricky part - maintain) a lookup file that maps host names to enclave at search time.

[somesoni2]

Do the UFs on Testing enclaves follow any particular naming conventions (basically how can you differentiate a UF from Primary vs UF from testing enclave)? How are those UF's receiving configurations, deployment servers or direct?

[sniderwj]

The flow of data from the Enclave is UF --> Heavy Forwarder --> Production Splunk --> Enclave Specific Index

They are copying devices into the enclave from production so we will have duplicate names (host names and domain names) between the enclaves and in production.

I guess the easiest place would be to hit the events at the HF layer but I'm unsure how I can do that.

[niketn]

Can you try the dbinspect command? Following is a sample which may suit your need.

| dbinspect index=_internal 
| stats sum(eventCount) as eventCount by splunk_server