Splunk Search
Highlighted

Why is my custom sourcetype configuration not extracting the host field?

Path Finder

I have a file with data similar to the following:

2015:09:01:15:00:00.005 sl200services007 3:INFO SERVER NOFMT 4327963992431091696812 Saving to client_request_map table

I have a custom sourcetype called services:client, and I have set up an index-time extracttion for the host field.

inputs.conf

[monitor://C:\temp\sample_logs2.txt]
disabled = false
index = test
sourcetype = services:client

props.conf

[services:client]
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Custom
description = trans services
pulldown_type = true
TRANSFORMS-host = hostoverride2

transforms.conf

[hostoverride2]
DEST_KEY = MetaData:Host
REGEX = ^[^\s]+\s([^\s]+)
FORMAT = host::$1

For testing, I stop splunk, append data to the monitored file, and I start splunk. Splunk is indexing the data fine, but the host field isn't set based on the regex result. I tested the regex on https://regex101.com/, and I followed the override syntax from http://blogs.splunk.com/2008/04/16/overriding-default-syslog-host-extraction/. Any thoughts on why this isn't working to set the hostname?

0 Karma
Highlighted

Re: Why is my custom sourcetype configuration not extracting the host field?

Contributor

Since you are trying to over write an indexed field, you probably need to catch it on the indexer at that source level. So try something like this in your props.conf:

[source::…sample_logs2.txt]
TRANSFORMS-host = hostoverride2

instead of :

[services:client]
TRANSFORMS-host = hostoverride2

0 Karma
Highlighted

Re: Why is my custom sourcetype configuration not extracting the host field?

Path Finder

Still no go, I removed transforms reference from the sourcetype and added the source based on the example. The ...samplelogs2.txt will catch any file ending in samplelogs2.txt.

My props.conf value has the following:

[cybs:transsvs]
MAX
TIMESTAMPLOOKAHEAD = 30
NO
BINARYCHECK = true
SHOULD
LINEMERGE = false
category = Custom
description = cybs trans services
pulldown_type = true

[source::...sample_logs2.txt]
TRANSFORMS-host = hostoverride2

My splunk instance is a single windows 7 host running 6.2.5.

0 Karma
Highlighted

Re: Why is my custom sourcetype configuration not extracting the host field?

Contributor

Is that props.conf and transforms.conf deployed on your indexers? (These should be on your indexers, or heavy forwarders, if data is being sent through them first. )

If so, you may also want to run this on your indexers to verify what props stanzas are being applied to your indexers:

./splunk btool list props

you can pipe this to a file and review it to verify what the indexer thinks the total applied configuration is.

0 Karma
Highlighted

Re: Why is my custom sourcetype configuration not extracting the host field?

Path Finder

I have a file on the indexer, since this is just a test situation. This is from props and transforms on the single test indexer.

0 Karma
Highlighted

Re: Why is my custom sourcetype configuration not extracting the host field?

SplunkTrust
SplunkTrust

Try this in transforms.conf instead:

[hostoverride2]
REGEX = ^[^\s]+\s([^\s]+)
FORMAT = host::$1
DEST_KEY = MetaData:Host

I'm not certain the order maters but it's worth a shot. Everything else looks good to me.

0 Karma
Highlighted

Re: Why is my custom sourcetype configuration not extracting the host field?

Path Finder

I think a ticket with support might be in order, or a test on a Linux host vs. Windows. Debug output shows the source correctly settting the transforms-host correctly.

C:\Program Files\Splunk\etc\apps\search\local\props.conf [source::...samplelogs2.txt]
C:\Program Files\Splunk\etc\system\default\props.conf ANNOTATE
PUNCT = True
C:\Program Files\Splunk\etc\system\default\props.conf AUTOKVJSON = true
C:\Program Files\Splunk\etc\system\default\props.conf BREAKONLYBEFORE =
C:\Program Files\Splunk\etc\system\default\props.conf BREAKONLYBEFOREDATE = True
C:\Program Files\Splunk\etc\system\default\props.conf CHARSET = AUTO
C:\Program Files\Splunk\etc\system\default\props.conf DATETIME
CONFIG = \etc\datetime.xml
C:\Program Files\Splunk\etc\system\default\props.conf HEADERMODE =
C:\Program Files\Splunk\etc\system\default\props.conf LEARN
SOURCETYPE = true
C:\Program Files\Splunk\etc\system\default\props.conf LINEBREAKERLOOKBEHIND = 100
C:\Program Files\Splunk\etc\system\default\props.conf MAXDAYSAGO = 2000
C:\Program Files\Splunk\etc\system\default\props.conf MAXDAYSHENCE = 2
C:\Program Files\Splunk\etc\system\default\props.conf MAXDIFFSECSAGO = 3600
C:\Program Files\Splunk\etc\system\default\props.conf MAX
DIFFSECSHENCE = 604800
C:\Program Files\Splunk\etc\system\default\props.conf MAXEVENTS = 256
C:\Program Files\Splunk\etc\system\default\props.conf MAX
TIMESTAMPLOOKAHEAD = 128
C:\Program Files\Splunk\etc\system\default\props.conf MUST
BREAKAFTER =
C:\Program Files\Splunk\etc\system\default\props.conf MUST
NOTBREAKAFTER =
C:\Program Files\Splunk\etc\system\default\props.conf MUSTNOTBREAKBEFORE =
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION = indexing
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-all = full
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-inner = inner
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-outer = outer
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-raw = none
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-standard = standard
C:\Program Files\Splunk\etc\system\default\props.conf SHOULD
LINEMERGE = True
C:\Program Files\Splunk\etc\system\default\props.conf TRANSFORMS =
C:\Program Files\Splunk\etc\apps\search\local\props.conf TRANSFORMS-host = hostoverride2
C:\Program Files\Splunk\etc\system\default\props.conf TRUNCATE = 10000
C:\Program Files\Splunk\etc\system\default\props.conf detecttrailingnulls = auto
C:\Program Files\Splunk\etc\system\default\props.conf maxDist = 100
C:\Program Files\Splunk\etc\system\default\props.conf priority =
C:\Program Files\Splunk\etc\system\default\props.conf sourcetype =

0 Karma
Highlighted

Re: Why is my custom sourcetype configuration not extracting the host field?

Contributor

Didn't realize this was windows indexer, I may definitely be worth checking with support . . ., this sort of stuff just seems to work on linux splunk servers.

0 Karma
Highlighted

Re: Why is my custom sourcetype configuration not extracting the host field?

SplunkTrust
SplunkTrust

you might also try to change some small details...

like TRANSFORMS-host to TRANSFORMS-customhost

to rule out any "internal" stuffs related to "host".

0 Karma
Highlighted

Re: Why is my custom sourcetype configuration not extracting the host field?

SplunkTrust
SplunkTrust

also look for errors under index=_internal source=splunkd

0 Karma