Getting Data In

Adding additional fields at the Universal Forwarder: Why are apps are fighting over the _meta field?

oliverja
Path Finder

I have App_1 that is adding metadata in the inputs.conf file:

 

 

###### Forwarded WinEventLogs (WEF) ######
[WinEventLog://ForwardedEvents]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 1
## The addon supports only XML format for the collection of WinEventLogs using WEF, hence do not change the below renderXml parameter to false.
renderXml = true
host=WinEventLogForwardHost
index = system-win
_meta = machine_class::workstation

 

 

I now need to uniquely identify the host that the UF runs on. I expect that this would just be "_meta = uf_name::HostnameOfUF", BUT....

This App_1 is distributed to several hosts, and I cannot modify it to uniquely identify anything. Instead, I created a new app (App_2) consisting of just an inputs.conf with

 

 

###### Forwarded WinEventLogs (WEF) ######
[WinEventLog://ForwardedEvents]
_meta = uf_name::HostnameX

 

 

Unfortunately, this never shows up, but I believe this is because the App_1 cannot "merge" the _meta with the _meta contained in App_2.

How can I uniquely identify my host?

Labels (2)
0 Karma
1 Solution

oliverja
Path Finder

So, there are two things preventing a clean solution here. 

1) I cannot put environmental variables in my inputs.conf. This prevents me from making a generic inputs.conf and having it populate correctly as needed to all instances.

  1. Workaround: I have to hard code my UF hostname on each UF instance.  

 

[WinEventLog://ForwardedEvents]
_meta = uf_name::HostnameX​

 

2) I cannot merge the "_meta" data from multiple sources. _meta must be defined in a single file. So....

  1. Workaround: For hosts that need this additional field, I had to create a new app to supersede the original app's _meta value with my new multi-value _meta. (Note: Make sure app name precedence is being thought of. Capital letters hit before alphabetic)

 

[WinEventLog://ForwardedEvents]
_meta = machine_class::workstation wec_hostname::HardcodedHostname

 

 

So now I have 10 unique apps. And they are applied to 10 unique server classes. All with hardcoded values, just because splunk wont accept an environmental variable. Aaaaargh, splunk.

 

If you find yourself in this situation, stop. Its obnoxious and accident prone.  I am dealing with legacy systems, and no search-time extraction can be applied to gather useful data. Apparently this is an unusual use case! But it is the second time I have managed an environment where something similar was needed. 2 for 2...

View solution in original post

0 Karma

oliverja
Path Finder

So, there are two things preventing a clean solution here. 

1) I cannot put environmental variables in my inputs.conf. This prevents me from making a generic inputs.conf and having it populate correctly as needed to all instances.

  1. Workaround: I have to hard code my UF hostname on each UF instance.  

 

[WinEventLog://ForwardedEvents]
_meta = uf_name::HostnameX​

 

2) I cannot merge the "_meta" data from multiple sources. _meta must be defined in a single file. So....

  1. Workaround: For hosts that need this additional field, I had to create a new app to supersede the original app's _meta value with my new multi-value _meta. (Note: Make sure app name precedence is being thought of. Capital letters hit before alphabetic)

 

[WinEventLog://ForwardedEvents]
_meta = machine_class::workstation wec_hostname::HardcodedHostname

 

 

So now I have 10 unique apps. And they are applied to 10 unique server classes. All with hardcoded values, just because splunk wont accept an environmental variable. Aaaaargh, splunk.

 

If you find yourself in this situation, stop. Its obnoxious and accident prone.  I am dealing with legacy systems, and no search-time extraction can be applied to gather useful data. Apparently this is an unusual use case! But it is the second time I have managed an environment where something similar was needed. 2 for 2...

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Yep, unfortunately, you hit a quite unusual use case. In general _meta is very rarely used itself, let alone having to merge separate  settings there. You can create an idea on ideas.splunk.com but I doubt it will get a significant support - it's so rare for anyone to need that.

That's something that could be resolved by an external configuration management tool and configuration templating. But that of course means another tool to manage the configs, more hassle with maintenance and so on.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

https://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/Wheretofindtheconfigurationfiles

"When different copies have conflicting attribute values (that is, when they set the same attribute to different values), it uses the value from the file with the highest priority."

You can verify it by changing names of your apps so the precedence works the other way around.

I don't see any way to "merge" those settings. But you can keep only the setting that uniquely identifies the server (uf_name in your case) and either do an ingest-time lookup if you really want another values as indexed fields or do lookups in search-time based on that field.

VatsalJagani
SplunkTrust
SplunkTrust

@oliverja - On a side note _meta is not suggested to use, try to see if you can achieve what you want to do with one of the following or the combination of them.

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

@VatsalJaganiYou missed the part where it says that we're talking about Universal Forwarder. 😉

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@PickleRick that is true. I was just saying if what we want can be achievable on Indexers through INGEST_EVAL then we don't have to use _meta which as it says on props.conf.spec is not recommended. 

 

@oliverja - Splunk does not look at the right-hand side of = (equals to sign) when deciding whether the attribute is the same or not. So because you have a common stanza name only one will be applicable.

Here is how the file precedence is followed for inputs.conf (OR index time attributes)

  • System local directory -- highest priority
  • App local directories (`apps/A/local/* ... apps/z/local/*`)
  • App default directories (`apps/A/default/* ... apps/z/default/*`)
  • System default directory -- lowest priority

 

So in case you name the Apps as App_1 and App_2, both's default directory has _meta attribute then App_1's attribute will get priority.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

I know but the downside of index-time evals is that it happens on the HF/indexer, not on the UF and sometimes you want to have metadata to uniquely identify the forwarder (not the source! because single forwarder can be processing many sources).

Example scenario - you have several sites on which you have single UFs installed on machines receiving logs from the whole site over WEF. You want to be able to easily distinguish between the origin of the event (which site it came from). You can't trust the values from the event itself (the sources in the rear world are often misconfigured and trusting that half of your environment is called localhost is a bit foolish ;)). And with possible other values you'd have to maintain huge lookups mapping names or IP addreses to sites.

It's much easier and more convenient to just add a field on each UF. If you wanted to do it on the next layer, you'd have to map sources with lookups because - unfortunately - you have no connection-level metadata so you don't know which UF the event came from.

0 Karma

oliverja
Path Finder

This is exactly what is happening here.

Lookups and search-time cant happen here, because the only thing I know (or know for sure) is which forwarder my remote sites are passing through. And I have to keep the existing _meta field because a lot of teams rely on that.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Unfortunately, unless you're using HF, you can't modify the metadata on the fly. That's simply because how the forwarders work. If you're passing data through UF, you're getting a cooked stream where the events are not parsed yet and you're getting only "per channel" metadata for the whole event stream. If you're inputting data on HF, it parses them and adds metadata to every single event (that's also one of the reasons why HFs are not recommended unless definitely needed) so you can dynamically "adjust" that metadata - add files, overwrite them, remove - as well as the raw data itself.

And for now you simply can't "merge" different settings within a single stanza.

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

Make perfect sense. I think the use-case is very good and common it seems, somehow I've never come across it. 

Thanks for the explanation!!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...