Why are props.conf is not applying on universal forwarder?

[jkhalouian]

Hello. Props.conf is not applying on my universal forwarder. I diagnosed with btool and it seems that the setting are being applied yet when the data gets shipped to my server, it is in raw form. 

inputs.conf:

[batch://C:\Data\*\Cloud-Data\to_splunk\(...)?(?i)*_CloudTrail_*]
sourcetype = aws-cloudtrail
move_policy = sinkhole
index = testindex

Props.conf:

[aws-cloudtrail]
LINE_BREAKER=((?<=}),(?={"eventVersion"))
NO_BINARY_CHECK=true
CHARSET=UTF-8
KV_MODE=json
SEDCMD-remove_prefix=s/{"Records":\[//g
SEDCMD-remove_suffix=s/\]}//g
TIME_PREFIX=eventTime
TRANSFORMS-index = write-index

transforms.conf

[write-index]
SOURCE_KEY = MetaData:Source
DEST_KEY = _MetaData:Index
REGEX = .*\\Data\\+(?<yeet>.*)\\.*\\to_splunk.*
FORMAT = $1

[PickleRick]

Only a limited subset of functionality is available on universal forwarder.

https://wiki.splunk.com/Community:HowIndexingWorks

See the diagram with UF.

Your transforms and sedcmd won't work on UF.

[gjanders]

Also refer to https://www.aplura.com/assets/pdf/where_to_put_props.pdf

[isoutamo]

Hi

it’s just like @PickleRick said, only small set of props.conf values are used in UF (like charset). The rest are applied on first full splunk enterprise instance (HD, IDX depends on your installation).

r. Ismo

[jkhalouian]

C:\Users\>"c:\Program Files\SplunkUniversalForwarder\bin\btool.exe" props list --debug aws-cloudtrail C:\Program Files\SplunkUniversalForwarder\etc\appsCloud-Inputs\default\props.conf [aws-cloudtrail] C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf ADD_EXTRA_TIME_FIELDS = True C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf ANNOTATE_PUNCT = True C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf AUTO_KV_JSON = true C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf BREAK_ONLY_BEFORE = C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf BREAK_ONLY_BEFORE_DATE = True C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf CHARSET = UTF-8 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf DATETIME_CONFIG = \etc\datetime.xml C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf DEPTH_LIMIT = 1000 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf HEADER_MODE = C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf KV_MODE = json C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf LEARN_MODEL = true C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf LEARN_SOURCETYPE = true C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf LINE_BREAKER = ((?<=}),(?={"eventVersion")) C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf LINE_BREAKER_LOOKBEHIND = 100 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MATCH_LIMIT = 100000 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_DAYS_AGO = 2000 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_DAYS_HENCE = 2 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_DIFF_SECS_AGO = 3600 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_DIFF_SECS_HENCE = 604800 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_EVENTS = 256 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_TIMESTAMP_LOOKAHEAD = 128 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MUST_BREAK_AFTER = C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MUST_NOT_BREAK_AFTER = C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MUST_NOT_BREAK_BEFORE = C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf NO_BINARY_CHECK = true C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf SEDCMD-remove_prefix = s/{"Records":\[//g C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf SEDCMD-remove_suffix = s/\]}//g C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION = indexing C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION-all = full C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION-inner = inner C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION-outer = outer C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION-raw = none C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION-standard = standard C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs\default\props.conf SHOULD_LINEMERGE = false C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf TIME_PREFIX = eventTime C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf TRANSFORMS = C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf TRANSFORMS-index = write-index C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf TRUNCATE = 10000 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf detect_trailing_nulls = auto C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf maxDist = 100 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf priority = C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf sourcetype = C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf termFrequencyWeightedDist = false C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs\default\props.conf [aws-cloudtrailinsights] C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf ADD_EXTRA_TIME_FIELDS = True C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf ANNOTATE_PUNCT = True C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf AUTO_KV_JSON = true C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf BREAK_ONLY_BEFORE = C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf BREAK_ONLY_BEFORE_DATE = True C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs\default\props.conf CHARSET = UTF-8 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf DATETIME_CONFIG = \etc\datetime.xml C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf DEPTH_LIMIT = 1000 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf HEADER_MODE = C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs\default\props.conf KV_MODE = json C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf LEARN_MODEL = true C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf LEARN_SOURCETYPE = true C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs\default\props.conf LINE_BREAKER = ([\r\n]+) C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf LINE_BREAKER_LOOKBEHIND = 100 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MATCH_LIMIT = 100000 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_DAYS_AGO = 2000 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_DAYS_HENCE = 2 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_DIFF_SECS_AGO = 3600 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_DIFF_SECS_HENCE = 604800 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_EVENTS = 256 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MAX_TIMESTAMP_LOOKAHEAD = 128 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MUST_BREAK_AFTER = C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MUST_NOT_BREAK_AFTER = C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf MUST_NOT_BREAK_BEFORE = C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs NO_BINARY_CHECK = true C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION = indexing C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION-all = full C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION-inner = inner C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION-outer = outer C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION-raw = none C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf SEGMENTATION-standard = standard C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs\default\props.conf SHOULD_LINEMERGE = true C:\Program Files\SplunkUniversalForwarder\etc\appsCloud-Inputs\default\props.conf TIME_PREFIX = digestStartTime C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf TRANSFORMS = C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs\default\props.conf TRANSFORMS-index = write-index C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf TRUNCATE = 10000 C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs\default\props.conf description = CloudTrail C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf detect_trailing_nulls = auto C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs\default\props.conf disabled = false C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf maxDist = 100 C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf priority = C:\Program Files\SplunkUniversalForwarder\etc\apps\Cloud-Inputs\default\props.conf pulldown_type = true C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf sourcetype = C:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf termFrequencyWeightedDist = false