Getting Data In

How to create a custom field at Heavy Forwarder for all sourcetypes ?

akocak
Contributor

Hi All,
Thanks upfront for your time.

I have a task that I am trying to create 2 fields for any sourcetype that visits my Heavy Forwarders on the way to my indexer cluster.
I had created following props.conf at HF to assign values as below and failed :

[mycustomlogg]
EVAL-HF_LVL1_NAME = myservername
EVAL-HF_LVL1_TIME = now()

This might be good question for any admin who wants to have more detail on data travel times within Splunk environment even though it might be prone to lesser performance if succeeded.

Are there anybody who had achieved similar in the past ? I believe I should be able to assign HF_LVL1_NAME with transforms.conf. However, still have no clue for now() function within conf files.

0 Karma

oangarita
Explorer

Hi, I have the same need, could you resolve this part: EVAL-HF_LVL1_TIME = now() ?

Thanks,

0 Karma

woodcock
Esteemed Legend

There is a RegEx-injection attack that you can use to match all sourcetypes, like this:

In props.conf:
[(?::){0}*]
TRANSFORMS-arbitrary_string_here = myevalforhfname

In transforms.conf:
[myevalforhfname]
INGEST_EVAL = HFLVL1NAME:="Your hardcoded string value here"

In fields.conf:
[HFLVL1NAME]
INDEXED = True
0 Karma

akocak
Contributor

is this syntax error ? ( := )

INGEST_EVAL :=
0 Karma

woodcock
Esteemed Legend

No. The := syntax says overwrite; the = syntax says append.

0 Karma

woodcock
Esteemed Legend

You are right, I had the := in the wrong place. I fixed it. In any case this is what the docs say:

* When writing to a _meta field, the default behavior is to add a new
  index-time field even if one exists with the same name, the same way
  WRITE_META works for regular-expression-based extractions. For example, "a=5,
  a=a+2" adds two index-time fields to _meta: "a::5 a::7". You can change this
  by using ":=" after the variable name. For example, setting "a=5, a:=a+2"
  causes Splunk software to add a single "a::7" field.
* NOTE: Replacing index-time fields is slower than adding them. It is best to
  only use ":=" when you need this behavior.
* The ":=" operator can also be used to remove existing fields in _meta
  by assigning the expression null() to them.
0 Karma

akocak
Contributor

Hi Woodcock,
Thanks for the great answer again.
Even though I couldn't find anything related , I assume my config is not working thanks to version 7.0.8. We had recently upgraded to 7.3 , However, I didn't have chance to work on this again. I will update here when implement.

0 Karma

akocak
Contributor

Still have no luck, my configuration doesn't work for some reason.

0 Karma

woodcock
Esteemed Legend

If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier), restart all Splunk instances there, send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

0 Karma

woodcock
Esteemed Legend

I would open a support case. Come back and let us know what ends up fixing it!

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

Have a look at INGEST_EVAL parameter in transforms.conf. Ref. doc https://docs.splunk.com/Documentation/Splunk/7.3.1/Data/IngestEval

0 Karma

akocak
Contributor

Hi, Thanks for the answer

I had enabled ingest-eval as in the examples of documentation from my Heavy Forwarder. It still didn't take my custom fields, this is my configuration:
props.conf

[mycustomlogg]
DATETIME_CONFIG =
NO_BINARY_CHECK = true
TRANSFORMS-abc = myevalforhfname,myevalforhfname2
category = Custom
disabled = false

transforms: (has both testing)
[myevalforhfname]
INGEST_EVAL = HFLVL1NAME="myservername"

[myevalforhfname2]
INGEST_EVAL = HFLVL1NAME2=lower("myservername")

fields.conf
[HFLVL1NAME]
INDEXED = True

[HFLVL1NAME2]
INDEXED = True
0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!