topic Re: Why is my custom sourcetype configuration not extracting the host field? in Splunk Search

Why is my custom sourcetype configuration not extracting the host field?

lisaac — Tue, 01 Sep 2015 18:43:31 GMT

I have a file with data similar to the following:

2015:09:01:15:00:00.005 sl200services007 3:INFO SERVER NOFMT 4327963992431091696812 Saving to client_request_map table

I have a custom sourcetype called services:client, and I have set up an index-time extracttion for the host field.

inputs.conf

[monitor://C:\temp\sample_logs2.txt]
disabled = false
index = test
sourcetype = services:client

props.conf

[services:client]
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Custom
description = trans services
pulldown_type = true
TRANSFORMS-host = hostoverride2

transforms.conf

[hostoverride2]
DEST_KEY = MetaData:Host
REGEX = ^[^\s]+\s([^\s]+)
FORMAT = host::$1

For testing, I stop splunk, append data to the monitored file, and I start splunk. Splunk is indexing the data fine, but the host field isn't set based on the regex result. I tested the regex on https://regex101.com/, and I followed the override syntax from http://blogs.splunk.com/2008/04/16/overriding-default-syslog-host-extraction/. Any thoughts on why this isn't working to set the hostname?

Re: Why is my custom sourcetype configuration not extracting the host field?

tskinnerivsec — Tue, 01 Sep 2015 19:13:23 GMT

Since you are trying to over write an indexed field, you probably need to catch it on the indexer at that source level. So try something like this in your props.conf:

[source::…sample_logs2.txt]
TRANSFORMS-host = hostoverride2

instead of :

[services:client]
TRANSFORMS-host = hostoverride2

Re: Why is my custom sourcetype configuration not extracting the host field?

lisaac — Tue, 29 Sep 2020 07:12:43 GMT

Still no go, I removed transforms reference from the sourcetype and added the source based on the example. The ...sample_logs2.txt will catch any file ending in sample_logs2.txt.

My props.conf value has the following:

[cybs:trans_svs]
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Custom
description = cybs trans services
pulldown_type = true

[source::...sample_logs2.txt]
TRANSFORMS-host = hostoverride2

My splunk instance is a single windows 7 host running 6.2.5.

Re: Why is my custom sourcetype configuration not extracting the host field?

tskinnerivsec — Tue, 01 Sep 2015 19:42:05 GMT

Is that props.conf and transforms.conf deployed on your indexers? (These should be on your indexers, or heavy forwarders, if data is being sent through them first. )

If so, you may also want to run this on your indexers to verify what props stanzas are being applied to your indexers:

./splunk btool list props

you can pipe this to a file and review it to verify what the indexer thinks the total applied configuration is.

Re: Why is my custom sourcetype configuration not extracting the host field?

jkat54 — Tue, 01 Sep 2015 20:01:50 GMT

Try this in transforms.conf instead:

[hostoverride2]
REGEX = ^[^\s]+\s([^\s]+)
FORMAT = host::$1
DEST_KEY = MetaData:Host

I'm not certain the order maters but it's worth a shot. Everything else looks good to me.

Re: Why is my custom sourcetype configuration not extracting the host field?

lisaac — Tue, 29 Sep 2020 07:10:00 GMT

I think a ticket with support might be in order, or a test on a Linux host vs. Windows. Debug output shows the source correctly settting the transforms-host correctly.

C:\Program Files\Splunk\etc\apps\search\local\props.conf [source::...sample_logs2.txt]
C:\Program Files\Splunk\etc\system\default\props.conf ANNOTATE_PUNCT = True
C:\Program Files\Splunk\etc\system\default\props.conf AUTO_KV_JSON = true
C:\Program Files\Splunk\etc\system\default\props.conf BREAK_ONLY_BEFORE =
C:\Program Files\Splunk\etc\system\default\props.conf BREAK_ONLY_BEFORE_DATE = True
C:\Program Files\Splunk\etc\system\default\props.conf CHARSET = AUTO
C:\Program Files\Splunk\etc\system\default\props.conf DATETIME_CONFIG = \etc\datetime.xml
C:\Program Files\Splunk\etc\system\default\props.conf HEADER_MODE =
C:\Program Files\Splunk\etc\system\default\props.conf LEARN_SOURCETYPE = true
C:\Program Files\Splunk\etc\system\default\props.conf LINE_BREAKER_LOOKBEHIND = 100
C:\Program Files\Splunk\etc\system\default\props.conf MAX_DAYS_AGO = 2000
C:\Program Files\Splunk\etc\system\default\props.conf MAX_DAYS_HENCE = 2
C:\Program Files\Splunk\etc\system\default\props.conf MAX_DIFF_SECS_AGO = 3600
C:\Program Files\Splunk\etc\system\default\props.conf MAX_DIFF_SECS_HENCE = 604800
C:\Program Files\Splunk\etc\system\default\props.conf MAX_EVENTS = 256
C:\Program Files\Splunk\etc\system\default\props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
C:\Program Files\Splunk\etc\system\default\props.conf MUST_BREAK_AFTER =
C:\Program Files\Splunk\etc\system\default\props.conf MUST_NOT_BREAK_AFTER =
C:\Program Files\Splunk\etc\system\default\props.conf MUST_NOT_BREAK_BEFORE =
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION = indexing
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-all = full
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-inner = inner
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-outer = outer
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-raw = none
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-standard = standard
C:\Program Files\Splunk\etc\system\default\props.conf SHOULD_LINEMERGE = True
C:\Program Files\Splunk\etc\system\default\props.conf TRANSFORMS =
C:\Program Files\Splunk\etc\apps\search\local\props.conf TRANSFORMS-host = hostoverride2
C:\Program Files\Splunk\etc\system\default\props.conf TRUNCATE = 10000
C:\Program Files\Splunk\etc\system\default\props.conf detect_trailing_nulls = auto
C:\Program Files\Splunk\etc\system\default\props.conf maxDist = 100
C:\Program Files\Splunk\etc\system\default\props.conf priority =
C:\Program Files\Splunk\etc\system\default\props.conf sourcetype =

Re: Why is my custom sourcetype configuration not extracting the host field?

tskinnerivsec — Tue, 01 Sep 2015 20:07:11 GMT

Also, are both the props.conf and transforms.conf contained in the same configuration app with no weird default.meta stanzas applied?

Re: Why is my custom sourcetype configuration not extracting the host field?

lisaac — Tue, 01 Sep 2015 20:12:00 GMT

I have a file on the indexer, since this is just a test situation. This is from props and transforms on the single test indexer.

Re: Why is my custom sourcetype configuration not extracting the host field?

tskinnerivsec — Tue, 01 Sep 2015 20:13:42 GMT

Didn't realize this was windows indexer, I may definitely be worth checking with support . . ., this sort of stuff just seems to work on linux splunk servers.

Re: Why is my custom sourcetype configuration not extracting the host field?

jkat54 — Tue, 01 Sep 2015 20:25:13 GMT

Usually yes, but they dont have to be in the same app.

I tend to do apps like this

/ta-appnameInputs/
/ta-appnameProps/
/ta-appnameTransforms/

which is a bit overkill and usually i combine props & transforms into just "props". Reason being inputs can go on forwarders, props and transforms can be on indexers, or heavy forwarders, some usage on universal forwarders, etc. I do this in distributed environments that employ a deployment server.

Re: Why is my custom sourcetype configuration not extracting the host field?

jkat54 — Tue, 01 Sep 2015 20:26:35 GMT

You could go crazy and have both search and index time transformations in different apps, because search configs should only be on search heads and index time configs should only be on ingestion.

Re: Why is my custom sourcetype configuration not extracting the host field?

jkat54 — Tue, 01 Sep 2015 20:33:55 GMT

you might also try to change some small details...

like TRANSFORMS-host to TRANSFORMS-customhost

to rule out any "internal" stuffs related to "host".

Re: Why is my custom sourcetype configuration not extracting the host field?

jkat54 — Tue, 01 Sep 2015 20:34:28 GMT

also look for errors under index=_internal source=splunkd

Re: Why is my custom sourcetype configuration not extracting the host field?

lisaac — Tue, 01 Sep 2015 20:37:13 GMT

I tried the suggestion to no avail.

[hostoverride2]
REGEX = ^[^\s]+\s([^\s]+)
FORMAT = host::$1
DEST_KEY = MetaData:Host

Re: Why is my custom sourcetype configuration not extracting the host field?

tskinnerivsec — Tue, 01 Sep 2015 20:39:59 GMT

I like the idea of changing the TRANSFORMS name, I have definitely gotten bitten in the past buy using reserved words in stanza names before and "host" has a high probability of that.