topic Re: How can I tag or mark _internal events from different environments? in Splunk Search

How can I tag or mark _internal events from different environments?

sniderwj — Wed, 05 Apr 2017 14:42:34 GMT

We have a requirement to collect data from testing enclaves (that have copies of production devices) to our primary Splunk environment. I have event data going to a separate index through a heavy forwarder. What I am concerned about is the Internal index. I will need to track if I have Splunk UFs on clients in the enclave. Is there a way to mark or tag the data coming through the Heavy Forwarder to indicate that it is coming from that testing environment?

Re: How can I tag or mark _internal events from different environments?

niketn — Wed, 05 Apr 2017 15:46:50 GMT

Can you try the dbinspect command? Following is a sample which may suit your need.

| dbinspect index=_internal 
| stats sum(eventCount) as eventCount by splunk_server

Re: How can I tag or mark _internal events from different environments?

somesoni2 — Wed, 05 Apr 2017 15:59:56 GMT

Do the UFs on Testing enclaves follow any particular naming conventions (basically how can you differentiate a UF from Primary vs UF from testing enclave)? How are those UF's receiving configurations, deployment servers or direct?

Re: How can I tag or mark _internal events from different environments?

sniderwj — Wed, 05 Apr 2017 16:33:09 GMT

The flow of data from the Enclave is UF --> Heavy Forwarder --> Production Splunk --> Enclave Specific Index

They are copying devices into the enclave from production so we will have duplicate names (host names and domain names) between the enclaves and in production.

I guess the easiest place would be to hit the events at the HF layer but I'm unsure how I can do that.

Re: How can I tag or mark _internal events from different environments?

s2_splunk — Wed, 05 Apr 2017 18:04:38 GMT

One of the options you have is to add a metadata field to the events, which will require you to update forwarder configuration whenever a host moves between environments. Here is an answer that describes the process.

An alternative approach is to create (and - the tricky part - maintain) a lookup file that maps host names to enclave at search time.