topic Re: How to create a custom field at Heavy Forwarder for all sourcetypes ? in Getting Data In

How to create a custom field at Heavy Forwarder for all sourcetypes ?

akocak — Wed, 30 Sep 2020 02:15:38 GMT

Hi All,
Thanks upfront for your time.

I have a task that I am trying to create 2 fields for any sourcetype that visits my Heavy Forwarders on the way to my indexer cluster.
I had created following props.conf at HF to assign values as below and failed :

[mycustomlogg]
EVAL-HF_LVL1_NAME = myservername
EVAL-HF_LVL1_TIME = now()

This might be good question for any admin who wants to have more detail on data travel times within Splunk environment even though it might be prone to lesser performance if succeeded.

Are there anybody who had achieved similar in the past ? I believe I should be able to assign HF_LVL1_NAME with transforms.conf. However, still have no clue for now() function within conf files.

Re: How to create a custom field at Heavy Forwarder for all sourcetypes ?

harsmarvania57 — Wed, 25 Sep 2019 09:02:16 GMT

Hi,

Have a look at INGEST_EVAL parameter in transforms.conf. Ref. doc https://docs.splunk.com/Documentation/Splunk/7.3.1/Data/IngestEval

Re: How to create a custom field at Heavy Forwarder for all sourcetypes ?

akocak — Fri, 11 Oct 2019 18:13:51 GMT

Hi, Thanks for the answer

I had enabled ingest-eval as in the examples of documentation from my Heavy Forwarder. It still didn't take my custom fields, this is my configuration:
props.conf

[mycustomlogg]
DATETIME_CONFIG =
NO_BINARY_CHECK = true
TRANSFORMS-abc = myevalforhfname,myevalforhfname2
category = Custom
disabled = false

transforms: (has both testing)
[myevalforhfname]
INGEST_EVAL = HFLVL1NAME="myservername"

[myevalforhfname2]
INGEST_EVAL = HFLVL1NAME2=lower("myservername")

fields.conf
[HFLVL1NAME]
INDEXED = True

[HFLVL1NAME2]
INDEXED = True

Re: How to create a custom field at Heavy Forwarder for all sourcetypes ?

woodcock — Sun, 13 Oct 2019 00:12:28 GMT

There is a RegEx-injection attack that you can use to match all sourcetypes, like this:

In props.conf:
[(?::){0}*]
TRANSFORMS-arbitrary_string_here = myevalforhfname

In transforms.conf:
[myevalforhfname]
INGEST_EVAL = HFLVL1NAME:="Your hardcoded string value here"

In fields.conf:
[HFLVL1NAME]
INDEXED = True

Re: How to create a custom field at Heavy Forwarder for all sourcetypes ?

akocak — Mon, 14 Oct 2019 15:14:45 GMT

is this syntax error ? ( := )

INGEST_EVAL :=

Re: How to create a custom field at Heavy Forwarder for all sourcetypes ?

woodcock — Mon, 14 Oct 2019 16:01:00 GMT

No. The := syntax says overwrite; the = syntax says append.

Re: How to create a custom field at Heavy Forwarder for all sourcetypes ?

akocak — Mon, 14 Oct 2019 19:55:01 GMT

Still have no luck, my configuration doesn't work for some reason.

Re: How to create a custom field at Heavy Forwarder for all sourcetypes ?

woodcock — Mon, 14 Oct 2019 21:06:03 GMT

I would open a support case. Come back and let us know what ends up fixing it!

Re: How to create a custom field at Heavy Forwarder for all sourcetypes ?

oangarita — Wed, 30 Sep 2020 02:39:01 GMT

Hi, I have the same need, could you resolve this part: EVAL-HF_LVL1_TIME = now() ?

Thanks,

Re: How to create a custom field at Heavy Forwarder for all sourcetypes ?

woodcock — Thu, 17 Oct 2019 13:55:33 GMT

If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier), restart all Splunk instances there, send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

Re: How to create a custom field at Heavy Forwarder for all sourcetypes ?

woodcock — Fri, 24 Jan 2020 19:22:54 GMT

You are right, I had the := in the wrong place. I fixed it. In any case this is what the docs say:

* When writing to a _meta field, the default behavior is to add a new
  index-time field even if one exists with the same name, the same way
  WRITE_META works for regular-expression-based extractions. For example, "a=5,
  a=a+2" adds two index-time fields to _meta: "a::5 a::7". You can change this
  by using ":=" after the variable name. For example, setting "a=5, a:=a+2"
  causes Splunk software to add a single "a::7" field.
* NOTE: Replacing index-time fields is slower than adding them. It is best to
  only use ":=" when you need this behavior.
* The ":=" operator can also be used to remove existing fields in _meta
  by assigning the expression null() to them.

Re: How to create a custom field at Heavy Forwarder for all sourcetypes ?

akocak — Tue, 28 Jan 2020 19:42:22 GMT

Hi Woodcock,
Thanks for the great answer again.
Even though I couldn't find anything related , I assume my config is not working thanks to version 7.0.8. We had recently upgraded to 7.3 , However, I didn't have chance to work on this again. I will update here when implement.