Conditional tagging on an intermediate forwarder

[MichaelM1]

I have a configuration where I have an intermediate forward that is forwarding logs to central indexer that I do not control.  In order to send logs to the indexer they MUST be tagged with a Project_ID and GUIDe in the _META tag otherwise the logs will get rejected.  I would like to set up a multi-tenant like configuration where the intermediate forwarder is used by several projects that have different Project_ID and GUIDe.  The universal forward configuration that I am sending to the clients have the tagging and will be sending their unique project and guide numbers.  The problem that I am having is that the intermediate forwarder (windows) itself is not tagging its own logs when it sends to the indexers.  The clients tag and send their logs just fine.   If I apply the same tagging as the clients on the on the intermediate forwarder it will tag the logs twice or overwrite the tags.

What I want to do is tag only the intermediate forward logs or any logs that are not already tagged.

This is what I tried, and it is not working.  I was attempting to use regex to only add the tags to logs that are not already tagged using this filter ^(?.*Project_ID::) but this is not working.  Any help is appreciated.

Intermediate forwarder:

etc/apps/projtransforms/local/props.conf

[default]
TRANSFORMS-projectid = addprojectid
TRANSFORMS-IntermediateForwarder = addIntermediateForwarder
TRANSFORMS-GUIDe = addGUIDe

etc/apps/projtransforms/transforms.conf

[addprojectid]
REGEX = ^(?.*Project_ID::)
FROMAT = Project_ID::123456
MV_ADD = true

[addGUIDe]
REGEX = ^(?.*GUIDe::)
FROMAT = GUIDe::654321
MV_ADD = true

[addIntermediateForwarder]
REGEX = .*
FORMAT = IntermediateForwarder::XXXXXX
MV_ADD = false

UF Client tagging

/etc/system/local/inputs.conf

[default]
_meta = GUIDe::654321 Project_ID::123456
disabled = 0

[WinEventLog]
_meta = GUIDe::654321 Project_ID::123456
disabled = 0

[perfmon]
_meta = GUIDe::654321 Project_ID::123456
disabled = 0
index = spl_win

[MichaelM1]

@livehybrid This is definitely an intermediate forwarder that I want to setup. And it is forwarding logs from many UFs that I have point to it.  The UF are tagging their logs just fine from the UF configs that they have.  It is just that the IF itself is not tagging its own logs.

@PickleRick This intermediate forward is only receiving data UFs and I don’t have any HF in my architecture

@livehybrid I don’t think that the transforms are working at all.  This is my current config for props and transforms and I have simplified as you and @isoutamo suggested

== props.conf ==

[default]

TRANSFORMS-setCustomMetadata=setCustomMetadata

[host::macdev]

TRANSFORMS-setCustomMetadata=setThisHostMetadata

== transforms.conf ==

[setCustomMetadata]

INGEST_EVAL = IntermediateForwarder:=COALESCE(IntermediateForwarder," <IFhostname>"), debug1:=COALESCE(debug1,"testdebug")

[setThisHostMetadata]

INGEST_EVAL = = IntermediateForwarder:=COALESCE(IntermediateForwarder," <IFhostname>"), GUIDe:=COALESCE(GUIDe,"123456"), ProjectID:=COALESCE(ProjectID,"54321"), debug1:=COALESCE(debug1,"debugonIF")

I get this warning in the splunkd.log that could be related

WARN TcpOutputProc [19444 parsing]  - Pipeline data does not have indexKey. [_path]  =  e:\program files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe\n[_stmid]blah\n[Metadata:Source] = source::WinEventLog\n[MetaData:Host]  =  host::<IFhostname>\n[MetaData:Sourcetype] = sourcetype::WinEvtLog\n[_done] = _done\n[_conf] = source::WinEventLog:host::< IFhostname>:WinEventLog|\n

This this config I would expect that the logs forwarding though from the UFs should get the debug1 but it does not.  I don’t know why it does not seem to be doing anything at all

Thank you for all your help btw

[livehybrid]

Hi @MichaelM1 

Just to check - is your IF a UF or HF? 

If its a UF the transforms might not apply without force_local_processing=true

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[MichaelM1]

This IF is NOT a heavy forwarder.  It is only a light forwarder and does not do any indexing.

I added "force_local_processing=true" to my props.conf and I am finally making progress!!

Now the logs that originate from the IF are tagging with the IntermediateForwarder,GUIDe, Project_ID and the debug1

This is exactly as expected now that the force_local_processing is forcing it to actually use the props and transforms.  (it also has debug1=debugonIF)

Oddly the logs that the IF is forwarding are still NOT tagged with IntermediateForwarder and debug1 tags that I expect them to be.

The logs do have the GUIDe, Project_ID tags since that it configured on the UF clients.

These are my latest configs.

== props.conf ==

[default]

TRANSFORMS-setCustomMetadata=setCustomMetadata

force_local_processing=true

[host:: <IFhostname>]

TRANSFORMS-setCustomMetadata=setThisHostMetadata

== transforms.conf ==

[setCustomMetadata]

INGEST_EVAL = IntermediateForwarder:=COALESCE(IntermediateForwarder," <IFhostname>"), debug1:=COALESCE(debug1,"testdebug")

[setThisHostMetadata]

INGEST_EVAL = = IntermediateForwarder:=COALESCE(IntermediateForwarder," <IFhostname>"), GUIDe:=COALESCE(GUIDe,"123456"), Project_ID:=COALESCE(ProjectID,"54321"), debug1:=COALESCE(debug1,"debugonIF")

[PickleRick]

Ok. I think everyone involved assumed silently that your IF is a HF because... well, because that's how you do it when you want to use transforms. Yes, as you noticed, when you enable local processing, UF can do some props/transforms but I don't remember having it properly described in docs so we don't know if everything is working exactly as it would on a HF or not. I'm not sure if UF is even capable of doing what you want it to do. Normally a full Splunk instance has a setting for splunktcp input which directs data into proper queue depending on whether the data is cooked or cooked and parsed. I suppose either UF is missing this or might not even be able to send the data from external input into typingQueue.

[livehybrid]

Hi @MichaelM1 

Just to check the typo of FROMAT in your transforms.conf setting in the post - is that just in the post and not a typo in your config file?

Also, in your transforms.conf you will need to use SOURCE_KEY=_meta as by default it is _raw, and if a previous GUIDe/ProjectID is set then it should be in _meta.

Im not able to test this at the moment, but let me know how you get on changing the SOURCE_KEY and I will try and test it out properly later!

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[MichaelM1]

You are correct it was just a post typo

I added the SOURCE_KEY=_meta to my transforms and it still is not working as expected.  The logs that from the UFs still get forwarded by the IF but he IF its self is no longer tagging its own logs and therefor getting rejected by the indexers.

This is my transforms.conf now

[addprojectid]
REGEX = ^(?.*Project_ID::)
FORMAT = Project_ID::123456
MV_ADD = true
SOURCE_KEY=_meta

[addGUIDe]
REGEX = ^(?.*GUIDe::)
FORMAT = GUIDe::654321
MV_ADD = true
SOURCE_KEY=_meta

[addIntermediateForwarder]
REGEX = .*
FORMAT = IntermediateForwarder::XXXXXX
MV_ADD = false
SOURCE_KEY=_meta

[livehybrid]

Hi @MichaelM1 

Ive been doing some experimenting here as genuinely interested in this kind of thing. 

Can I just check - your IF is a heavy forwarder, not UF?

I have a UF with the following inputs:

[monitor:///var/log/will.txt]
index=main
sourcetype=debug
_meta=GUIDe::"abc-123-def-456"

[monitor:///var/log/will2.txt]
index=main
sourcetype=debug

[monitor:///var/log/will3.txt]
index=main
sourcetype=debug
_meta=GUIDe::"abc-123-def-456" ProjectID::"TestProject"

I have a receiver (IDX in this case) with the following:

== props.conf ==
#sourcetype=debug
[debug]
TRANSFORMS-debug=extract_GUIDe,setCustomMetadata

== transforms.conf ==
[extract_GUIDe]
SOURCE_KEY = _meta
WRITE_META = true
REGEX = GUIDe::([^\s]+)
FORMAT = ExGUIDe::$1

[setCustomMetadata]
INGEST_EVAL = GUIDe:=COALESCE(GUIDe,"NotSpecified"), ProjectID:=COALESCE(ProjectID,"NotSpecified")

Walking through this a little, we have 3 inputs sending data from the UF with various combos of:

will1 - GUIDe only
will2 - no meta
will3 - GUIDe and ProjectID 
(I realise you're using Project_ID not ProjectID but I missed it before I started writing this up)

When this is sent to Splunk I get the following:

Note that if GUIDe/ProjectID is missing it is replaced with "NotSpecified" as per the INGEST_EVAL

The transform "extract_GUIDe" isnt actually required, It was just part of my Proof-of-concept and workings.

The main thing here is the INGEST_EVAL - What its doing is eval-ing those fields to be a COALESCE of either the field which is sent (if it is) or the "NotSpecified" field (if it isnt). 

So, this deals with incoming traffic from the UF. I then added another set of props/transforms to deal with all data:

== props.conf ==
[default]
TRANSFORMS-setCustomMetadata=setCustomMetadata

[host::macdev]
TRANSFORMS-setCustomMetadata=setThisHostMetadata

== transforms.conf ==
# This is from the previous test
[setCustomMetadata]
INGEST_EVAL = GUIDe:=COALESCE(GUIDe,"NotSpecified"), ProjectID:=COALESCE(ProjectID,"NotSpecified")

[setThisHostMetadata]
INGEST_EVAL = GUIDe:=COALESCE(GUIDe,"999-999-999"), ProjectID:=COALESCE(ProjectID,"MyForwarderLayer")

When I look at data originated on my receiver, which is called "macdev" (which would be your IF) we can see that its getting the 999-999-999 GUIDe stamped on it.

If we look at data from any other host which doesnt have the GUIDe set at source then it uses the "NotSpecified" value from the INGEST_EVAL:

So I *think* this should help solve your problem? Its not completely clear how the receiver you are sending to is processing the GUIDe and dropping if not present, but it would be easy to do with INGEST_EVAL and checking for the presence of the GUIDe field, so I'm assuming its something like that?

[checkGUIDePresent]
INGEST_EVAL queue=IF(GUIDe!="",queue, nullQueue)

Anyway, I really hope this helps! Please let me know how you get on and consider adding karma to this or any other answer if it has helped, and also to accept an answer if it resolves your issue 🙂
Regards

Will 

[MichaelM1]

Thank you for the response.  I have been testing this all day so far today and to be honest nothing I do with the props and transforms is having any effect.

I do know that the IF is sending logs because if I add

/etc/system/local/inputs.conf

[WinEventLog]

_meta = GUIDe::123456 Project_ID::654321

the logs get tagged and indexed.  This does not work for me because it will double tag the logs since the client is adding its proper project and guide tags and then the IF add another project and guide tag.

Nothing I do in the props and transforms seems to change the fact that the logs are not being tagged from the IF itself the logs still get properly forwarders and indexed since they were already tagged by the client UF.

I really don't know why the transform is not working.

Even if I do this it still has no effect

== props.conf ==
[default]
TRANSFORMS-setCustomMetadata=setThisHostMetadata

== transforms.conf ==
[setThisHostMetadata]
INGEST_EVAL = GUIDe:=COALESCE(GUIDe,"999-999-999"), ProjectID:=COALESCE(ProjectID,"MyForwarderLayer")

[PickleRick]

One caveat - it will obviously _not_ work if your IF receives parsed data (from HF, not UF). You can bend-over backwards and re-parse already parsed data but it's not a great idea.

[livehybrid]

Hi @MichaelM1 

You're right that appling the _meta on the WinEventLog props stanza on the IF may double stamp it if it has already been applied on the UF, this is why we need to use the INGEST_EVAL with the COALESCE eval so that if it already exists then it will use the existing value, else uses a default value.

I am a bit confused as to why the props/transforms you applied did not work though. Does it apply to any of the data (e.g. just the IF's own data being sent)? 
You may wish to add a 3rd "debug" field to see if that gets applied to any of your data that reaches the indexers. 

One thing I dont think I asked which is probably pretty important... is your IF a HF or UF? I have been testing using HF as the IF.

@PickleRick you mentioned this wouldnt work if the data comes from a HF, I think @MichaelM1 is sending from a UF here, however in my testing I tried HF->IDX with the INGEST_EVAL transform on the IDX and this does work, despite the data already being parsed by the HF. I'm fairly sure I've used INGEST_EVAL plenty of times on parsed data but I will double check, perhaps I was using it on a RULESET instead. 

@MichaelM1 Let me know if your IF is a UF/HF so I can update my testing 🙂 We will beat this...!

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[isoutamo]

It's exactly like @PickleRick said. If you get parsed data, the only pipeline which can modify it is rulesets (ingest actions).

Event this is not issue in this case, I'm not sure than you @MichaelM1  remember how splunk manages the order of these 

[default]
TRANSFORMS-projectid = addprojectid
TRANSFORMS-IntermediateForwarder = addIntermediateForwarder
TRANSFORMS-GUIDe = addGUIDe

When you add several  TRANSFORMS in own lines Splunk always use ASCII order of those names when it selects the execution order for those! For that reason when you want to define the order you have two options: 

1. name those correctly to get wanted order like TRANSFORMS-01-first, TRANSFORMS-02-second etc.
2. Put all those in one line like TRANSFORMS-all = xyzzy, abcd, 1123-last

As @livehybrid said you should use INGEST_EVAL to set values only if it didn't exists like he present you.

[PickleRick]

Yes, rulesets are the only stuff that works on parsed data.

[MichaelM1]

let me try that again my regex was missing a ! I am also tring to remove the :: as I dont think it is necessary

[addprojectid]
REGEX = ^(?!.*Project_ID)
FORMAT = Project_ID::123456
MV_ADD = true
SOURCE_KEY=_meta

[addGUIDe]
REGEX = ^(?!.*GUIDe)
FORMAT = GUIDe::654321
MV_ADD = true
SOURCE_KEY=_meta

[addIntermediateForwarder]
REGEX = .*
FORMAT = IntermediateForwarder::XXXXXX
MV_ADD = false
SOURCE_KEY=_meta