Assign tag or static field with static value on the UF

[NoSpaces]

Hello to everyone!
I'm not sure how to correctly name this thing, but I will carefully try to explain what I want to achieve.

In our infrastructure we have plenty of Windows Server instances with Universal Forwarder installed.
All servers are divided into groups according to the particular application that the servers host.
For example, Splunk servers have group 'spl,' remote desktop session servers have group 'rdsh,' etc.
Each server has an environment variable with this group value.

By design, the access policy to logs was built on these groups. One group - one index.
Because of it, each UF input stanza has the option "index = group.".
According to this idea, introspection logs of UF agents are related to the SPL (or Splunk) group\index.

And here the nuisance started.
Sometimes UF agents report about errors that demand some things on the running hosts, for example, restarting the agent manually.
I see these errors because I have access to the 'spl' index, but I don't have access to all Windows machines and
I have to notify the machine owner about it manually.

So, the question is how to create a sort of tag or field on the UF that can help me separate all Splunk UF logs by these groups? Maybe I can use our environment variable to achieve it?
I only need to access this field during search time to create various alerts that will notify machine owners instead of me.

[PickleRick]

There are generally two methods - either add metadata during ingestion or dynamically clasify the sources during searching.

The latter approach is usually easiest done with a lookup as @PaulPanther wrote. This way you don't have to touch your sources at all, you just have to keep the lookup contents up to date. The downside is that... you have to keep the lookup contents up to date and have it's desirable that you have a 1-1 mapping between an existing field (like host) and the information you're looking up. Otherwise you need to do some calculated fields evaluated conditionally... and it's getting messy and hard to maintain.

Another approach is to add an indexed field directly at the source or at an intermediate forwarder. The IF approach of course requires you to have that intermediate HF to add a field with a different value for each group. (you can also do it directly at the receiving indexers but then you'd have a hard time differentiating between source - ingest-time lookup? Ugh). You might as well add the _meta setting to a particular input on the source forwarder. For example

_meta = source_env::my_lab

This will give each event ingested via input with this setting an additional indexed field called source_env with a value of "my_lab". It's convenient and it's very useful if you want to do some summaries with tstats but it also has downsides - you need to define it separately at each source (or at least in the [default] stanza for input as well as at [wineventlog] general stanza - default settings are not inherited to wineventlog!). And it must have a static value(s) defined. There is no way to define this setting dynamically like "source_hostname::$hostname" or something like that - you must define it explicitly.

Another thing is that you can only have one _meta entry. Since there are no multiple _meta-something settings, multiple _meta definitions will overwrite each other according to normal precedence rules.

So while you can define multiple indexed fields with

_meta = app_name::testapp1 env::test_env

But you can't define

_meta = app_name::testapp1

in one config file and

_meta = env::test_env

in another. Only one of those fields would be defined (depending on the files precedence).

[NoSpaces]

Thank you for such a detailed explanation.
But let me ask a little bit more...

Did I correctly understand that under "source forwarder" you meant UF?
I am asking because I tried to set meta = testfield::test_value in two different places, in the inputs.conf of the SplunkUniversalForwarder app and in the system inputs.conf, and I couldn't find this new field during the search.

[PickleRick]

Yes. If you're using UF, it's gonna be UF.

You need to set this for either [default] stanza or a specific input (and remember, default settings are not inherited by windows event log inputs).

But that will take care of creating the metadata field. Your search head tier must be aware of this field - you jeed to make an entry in fields.conf

[Your_field]
INDEXED=true
INDEXED_VALUE=false

[NoSpaces]

Still feel myself as a dummy

I set this in the UF inputs.conf:

[splunkd]
_meta = my_field::abc

Then I set this in the feilds.conf of SH:

[my_field]
INDEXED = true
INDEXED_VALUE = false

 
Of course, I applied config on both instances and still can't find new field created in "_meta"

[PickleRick]

Wait. What is that [splunkd] stanza? You have an input called splunkd? What is it ingesting?

[NoSpaces]

Oh my, you are right, I made a stupid mistake!
I so used to use props.conf in the most (personal) cases that I automatically tried to use sourcetype in stanza name.
But I need the input of splunk UF introspection, that generates splunkd 'sourcetype'.
And this monitor named 'monitor://$SPLUNK_HOME\var\log\splunk\splunkd.log'

[isoutamo]

Hi

If you want collect introspection data from UF there are some instructions for it.

• https://docs.splunk.com/Documentation/Splunk/9.4.0/Troubleshooting/ConfigurePIF
• https://community.splunk.com/t5/Getting-Data-In/How-to-collect-introspection-logs-from-forwarders-in...

Depending on what you are really needing it maybe better to use e.g Windows or Unix/Linux TA to collect that information. And remember that you cannot collect all same data from UFs than splunk collect from full enterprise instance.

There is at least one app for this task https://splunkbase.splunk.com/app/3805

r. Ismo

[PickleRick]

Gotta find which stanza defines this input (might be a single file, might be a whole directory) and add the _meta setting there.

In case of my home lab UF, it's

s:\Program Files\SplunkUniversalForwarder\bin>splunk btool inputs list --debug
[...]
s:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\inputs.conf [monitor://s:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log]
s:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\inputs.conf _TCP_ROUTING = *
s:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf _rcvbuf = 1572864
s:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf evt_dc_name =
s:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf evt_dns_name =
s:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf evt_resolve_ad_obj = 0
s:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf host = $decideOnStartup

So in my case I'd need to either add a custom app or at least system\local\inputs.conf (but it's best to avoid putting events into system/local)

with

[monitor://s:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log]
_meta = my_field::something

[PaulPanther · Accepted solution]

You could create a lookup where you group each instance and add information about the responsible colleagues and their mail addresses.

Then you configure it as an automatic lookup and you would have the benefit that you only create one alert because you could iterate over the mail addresses (e.g. $result.recipient$) in your alert.

Define an automatic lookup in Splunk Web - Splunk Documentation

[NoSpaces]

Thank you for the idea; this is a pretty easy solution.
Obviously less complicated than I came up with =D