Why are props.conf is not applying on universal forwarder?

jkhalouian — Tue, 15 Feb 2022 13:49:58 GMT

Hello. Props.conf is not applying on my universal forwarder. I diagnosed with btool and it seems that the setting are being applied yet when the data gets shipped to my server, it is in raw form.

inputs.conf:

[batch://C:\Data\*\Cloud-Data\to_splunk\(...)?(?i)*_CloudTrail_*] sourcetype = aws-cloudtrail move_policy = sinkhole index = testindex

Props.conf:

[aws-cloudtrail] LINE_BREAKER=((?<=}),(?={"eventVersion")) NO_BINARY_CHECK=true CHARSET=UTF-8 KV_MODE=json SEDCMD-remove_prefix=s/{"Records":\[//g SEDCMD-remove_suffix=s/\]}//g TIME_PREFIX=eventTime TRANSFORMS-index = write-index

transforms.conf

[write-index] SOURCE_KEY = MetaData:Source DEST_KEY = _MetaData:Index REGEX = .*\\Data\\+(?<yeet>.*)\\.*\\to_splunk.* FORMAT = $1

Re: Props.conf not applying

jkhalouian — Tue, 15 Feb 2022 03:31:11 GMT

C:\Users\>"c:\Program Files\SplunkUniversalForwarder\bin\btool.exe" props list --debug aws-cloudtrail C:\Program Files\SplunkUniversalForwarder\etc\appsCloud-Inputs\default\props.conf [aws-cloudtrail] C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf ADD_EXTRA_TIME_FIELDS = True C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf ANNOTATE_PUNCT = True C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf AUTO_KV_JSON = true C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf BREAK_ONLY_BEFORE = C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf BREAK_ONLY_BEFORE_DATE = True C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf CHARSET = UTF-8 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf DATETIME_CONFIG = \etc\datetime.xml C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf DEPTH_LIMIT = 1000 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf HEADER_MODE = C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf KV_MODE = json C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf LEARN_MODEL = true C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf LEARN_SOURCETYPE = true C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf LINE_BREAKER = ((?<=}),(?={"eventVersion")) C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf LINE_BREAKER_LOOKBEHIND = 100 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MATCH_LIMIT = 100000 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_DAYS_AGO = 2000 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_DAYS_HENCE = 2 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_DIFF_SECS_AGO = 3600 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_DIFF_SECS_HENCE = 604800 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_EVENTS = 256 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_TIMESTAMP_LOOKAHEAD = 128 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MUST_BREAK_AFTER = C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MUST_NOT_BREAK_AFTER = C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MUST_NOT_BREAK_BEFORE = C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf NO_BINARY_CHECK = true C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf SEDCMD-remove_prefix = s/{"Records":\[//g C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf SEDCMD-remove_suffix = s/\]}//g C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION = indexing C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION-all = full C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION-inner = inner C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION-outer = outer C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION-raw = none C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION-standard = standard C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs\default\props.conf SHOULD_LINEMERGE = false C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf TIME_PREFIX = eventTime C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf TRANSFORMS = C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf TRANSFORMS-index = write-index C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf TRUNCATE = 10000 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf detect_trailing_nulls = auto C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf maxDist = 100 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf priority = C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf sourcetype = C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf termFrequencyWeightedDist = false C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs\default\props.conf [aws-cloudtrailinsights] C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf ADD_EXTRA_TIME_FIELDS = True C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf ANNOTATE_PUNCT = True C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf AUTO_KV_JSON = true C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf BREAK_ONLY_BEFORE = C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf BREAK_ONLY_BEFORE_DATE = True C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs\default\props.conf CHARSET = UTF-8 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf DATETIME_CONFIG = \etc\datetime.xml C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf DEPTH_LIMIT = 1000 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf HEADER_MODE = C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs\default\props.conf KV_MODE = json C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf LEARN_MODEL = true C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf LEARN_SOURCETYPE = true C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs\default\props.conf LINE_BREAKER = ([\r\n]+) C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf LINE_BREAKER_LOOKBEHIND = 100 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MATCH_LIMIT = 100000 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_DAYS_AGO = 2000 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_DAYS_HENCE = 2 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_DIFF_SECS_AGO = 3600 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_DIFF_SECS_HENCE = 604800 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_EVENTS = 256 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_TIMESTAMP_LOOKAHEAD = 128 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MUST_BREAK_AFTER = C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MUST_NOT_BREAK_AFTER = C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MUST_NOT_BREAK_BEFORE = C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs NO_BINARY_CHECK = true C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION = indexing C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION-all = full C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION-inner = inner C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION-outer = outer C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION-raw = none C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION-standard = standard C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs\default\props.conf SHOULD_LINEMERGE = true C:\Program Files\SplunkUniversalForwarder\etc\appsCloud-Inputs\default\props.conf TIME_PREFIX = digestStartTime C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf TRANSFORMS = C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs\default\props.conf TRANSFORMS-index = write-index C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf TRUNCATE = 10000 C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs\default\props.conf description = CloudTrail C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf detect_trailing_nulls = auto C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs\default\props.conf disabled = false C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf maxDist = 100 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf priority = C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs\default\props.conf pulldown_type = true C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf sourcetype = C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf termFrequencyWeightedDist = false

Re: Why are props.conf is not applying on universal forwarder?

PickleRick — Tue, 15 Feb 2022 05:58:20 GMT

Only a limited subset of functionality is available on universal forwarder.

https://wiki.splunk.com/Community:HowIndexingWorks

See the diagram with UF.

Your transforms and sedcmd won't work on UF.

Re: Why are props.conf is not applying on universal forwarder?

isoutamo — Tue, 15 Feb 2022 06:55:32 GMT

it’s just like @PickleRick said, only small set of props.conf values are used in UF (like charset). The rest are applied on first full splunk enterprise instance (HD, IDX depends on your installation).

r. Ismo

Re: Why are props.conf is not applying on universal forwarder?

gjanders — Tue, 15 Feb 2022 08:24:55 GMT

Also refer to https://www.aplura.com/assets/pdf/where_to_put_props.pdf

topic Why are props.conf is not applying on universal forwarder? in All Apps and Add-ons

Why are props.conf is not applying on universal forwarder?

Re: Props.conf not applying

Re: Why are props.conf is not applying on universal forwarder?

Re: Why are props.conf is not applying on universal forwarder?

Re: Why are props.conf is not applying on universal forwarder?