Is it possible for a universal forwarder to inject additional data into existing log stream?

[tanium_spence]

I have several universal forwarders (UF) monitoring files on both Windows and Linux endpoints. I would like to "inject data" into the stream of forwarded events that would be made available either by a search-time extraction or injected directly into the log stream as an indexed field.

Here's a specific example: I am monitoring an application that allows for a wide range of log verbosity levels. Unfortunately, the application does NOT write the verbosity level within the log stream that it generates. (The verbosity level IS ONLY available in a registry key or in a text file, depending on the OS. In other words, it can be acquired programmatically.) I'd like to include the value of this log verbosity level variable within the stream of forwarded data, so that I can search against it like I would search against punct or host or sourcetype or what-have-you. In fact, this variable is the most important bit of metadata that I'd like to capture in my example. It arguably deserves promotion to an indexed field for this specific use case.

Is it possible to have a UF include/join/inject additional data that isn't part of an existing log stream? If so, is it possible to have the UF pull said data in a programmatic way, like having the UF read from the registry or read a value from a text file using python or shell or vbscript, etc.?

Answers and comments that need not be offered:
-Please don't key off of my mention of an "indexed field" and hijack the answer. We all know that indexed fields are bad, except when they're not.
-I know I can use a lookup table on my indexer and manually achieve what I'd like to accomplish. I'm only interested in a solution that can be fully automated across a large enterprise of UFs. A lookup table for this purpose will require lots of care and feeding. Let's not go there in this forum since it's already my fall-back option. If no solution is offered here, I'll answer my own question to close the loop to help any n00bs that stumble upon this answer.
-The developers of this application will not change their log format for me. Again, we all know that modifying the source of a log stream is the easiest way to solve problems. Making comments to this effect provide little benefit to the Answers community.

Thanks!

[woodcock]

All of this is possible but I am unaware of any facility in Splunk to do it directly. This is generally called preprocessing the events and there is all kind of "glue" code around the internet that you can borrow for this.

[tanium_spence]

Thanks, Gregg. I'm not surprised by your answer, but it never hurts to ask. I'm going to use a third-party tool to query my endpoints, extract my logging verbosity variable, and create a lookup table on my indexer with the results of the query.

[woodcock]

Give the answer by @MuS a looksee (I unaccepted mine; his is better).

[gjanders]

Another potential alternative would be to have Splunk run a script which adds the required data and spit's it out to standard output where the standard output could be sent to the Splunk indexers...(I don't know if this would work in your situation)

[MuS]

Hi tandem_spence,

Maybe I'm too pedantic here, but to answer your question

Is it possible for a universal forwarder to inject additional data into existing log stream?
Yes, this is possible. There is the _meta option in inputs.conf http://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf which enables you to add any additional key / value pairs to the event you want.

For example: _meta = field1::foo field2::bar will add this field1 = foo and field2 = bar to all events.

I know you added a lot of information in your post and most likely your verbosity level will be dynamic and therefore this approach will not solve your use case.

But as I said, it's more my pedantry / OCD here answering your original question 😉

anyway, Hope this helps ...

cheers, MuS

[woodcock]

Interesting; I will read up on this! Thank you @MuS!