How to create a custom field at Heavy Forwarder for all sourcetypes ?

[akocak]

Hi All,
Thanks upfront for your time.

I have a task that I am trying to create 2 fields for any sourcetype that visits my Heavy Forwarders on the way to my indexer cluster.
I had created following props.conf at HF to assign values as below and failed :

[mycustomlogg]
EVAL-HF_LVL1_NAME = myservername
EVAL-HF_LVL1_TIME = now()

This might be good question for any admin who wants to have more detail on data travel times within Splunk environment even though it might be prone to lesser performance if succeeded.

Are there anybody who had achieved similar in the past ? I believe I should be able to assign HF_LVL1_NAME with transforms.conf. However, still have no clue for now() function within conf files.

[oangarita]

Hi, I have the same need, could you resolve this part: EVAL-HF_LVL1_TIME = now() ?

Thanks,

[woodcock]

There is a RegEx-injection attack that you can use to match all sourcetypes, like this:

In props.conf:
[(?::){0}*]
TRANSFORMS-arbitrary_string_here = myevalforhfname

In transforms.conf:
[myevalforhfname]
INGEST_EVAL = HFLVL1NAME:="Your hardcoded string value here"

In fields.conf:
[HFLVL1NAME]
INDEXED = True

[akocak]

is this syntax error ? ( := )

INGEST_EVAL :=

[woodcock]

No. The := syntax says overwrite; the = syntax says append.

[woodcock]

You are right, I had the := in the wrong place. I fixed it. In any case this is what the docs say:

* When writing to a _meta field, the default behavior is to add a new
  index-time field even if one exists with the same name, the same way
  WRITE_META works for regular-expression-based extractions. For example, "a=5,
  a=a+2" adds two index-time fields to _meta: "a::5 a::7". You can change this
  by using ":=" after the variable name. For example, setting "a=5, a:=a+2"
  causes Splunk software to add a single "a::7" field.
* NOTE: Replacing index-time fields is slower than adding them. It is best to
  only use ":=" when you need this behavior.
* The ":=" operator can also be used to remove existing fields in _meta
  by assigning the expression null() to them.

[akocak]

Hi Woodcock,
Thanks for the great answer again.
Even though I couldn't find anything related , I assume my config is not working thanks to version 7.0.8. We had recently upgraded to 7.3 , However, I didn't have chance to work on this again. I will update here when implement.

[akocak]

Still have no luck, my configuration doesn't work for some reason.

[woodcock]

If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier), restart all Splunk instances there, send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

[woodcock]

I would open a support case. Come back and let us know what ends up fixing it!

[harsmarvania57]

Hi,

Have a look at INGEST_EVAL parameter in transforms.conf. Ref. doc https://docs.splunk.com/Documentation/Splunk/7.3.1/Data/IngestEval

[akocak]

Hi, Thanks for the answer

I had enabled ingest-eval as in the examples of documentation from my Heavy Forwarder. It still didn't take my custom fields, this is my configuration:
props.conf

[mycustomlogg]
DATETIME_CONFIG =
NO_BINARY_CHECK = true
TRANSFORMS-abc = myevalforhfname,myevalforhfname2
category = Custom
disabled = false

transforms: (has both testing)
[myevalforhfname]
INGEST_EVAL = HFLVL1NAME="myservername"

[myevalforhfname2]
INGEST_EVAL = HFLVL1NAME2=lower("myservername")

fields.conf
[HFLVL1NAME]
INDEXED = True

[HFLVL1NAME2]
INDEXED = True