topic Re: How to configure setting the host field during index time? in Splunk Search

How to configure setting the host field during index time?

nmohammed — Fri, 20 Jan 2017 23:02:48 GMT

we're trying to set the host fields by extracting the name from the events, but it doesn't seem to work and would appreciate if someone can guide through -

example events -

2017-01-20 14:18:55,816 [31] ep7mmn001 ERROR SMS.Shared.ApiCommon.Attribute.smapiAuthorizeAttribute    at System.Convert.FromBase64_Decode(Char* startInputPtr, Int32 inputLength, Byte* startDestPtr, Int32 destLength)
   at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)
   at System.Convert.FromBase64CharArray(Char[] inArray, Int32 offset, Int32 length)
   at Newtonsoft.Json.JsonTextReader.ParseString(Char quote)
   at Newtonsoft.Json.JsonTextReader.ParseValue()
2017-01-20 14:18:55,816 [31] ep8mmn002 ERROR SMS.Shared.ApiCommon.Attribute.smApiAuthorizeAttribute Invalid length for a Base-64 char array or string.
2017-01-20 14:18:55,816 [31] ep8mmn006 ERROR SMS.Shared.ApiCommon.Attribute.smApiAuthorizeAttribute Invalid length for a Base-64 char array or string.

props.conf

[sm-plat]
TRANSFORMS-hostextract=hostextract

[sm-expt]
TRANSFORMS-hostextract=hostextract

[sm-impt]
TRANSFORMS-hostextract=hostextract

[sm-vend]
TRANSFORMS-hostextract=hostextract

transforms.conf

[hostextract]
REGEX = ^[^\]\n]*\]\s+(\w+)
FORMAT = host::$1
DEST_KEY = _MetaData:Host

Thanks

Re: How to configure setting the host field during index time?

hunters_splunk — Sat, 21 Jan 2017 01:10:01 GMT

Hi nmohammed,

Since you are just using one transform stanza, I think you can just use one stanza to reference it in props.conf:

 [sm]
 TRANSFORMS-hostextract=hostextract

Also, the REGEX does not seem quite right and I'm not sure which hostnames it tries to capture from the events. You may need to finetune the REGEX to make sure it extracts hostnames correctly.

Hope it helps. Thanks!
Hunter

Re: How to configure setting the host field during index time?

arkadyz1 — Sat, 21 Jan 2017 02:06:34 GMT

First of all, your regex is OK - I saw that on my own, but regex101.com confirmed it as well. So the problem is in either your stanza in transforms.conf - but it seems OK to me again - or in your referencing the transform from props.conf. Do your events end up in one of those sourcetypes (sm-plat, sm-expt, sm-impt, sm-vend) and how do you specify their sourcetype?

Re: How to configure setting the host field during index time?

nmohammed — Mon, 23 Jan 2017 22:07:18 GMT

Thanks. I tried hunters approach and referenced only one stanza ,but still doesn't work.

[sm]
TRANSFORMS-hostextract=hostextract

arkadzy1 -

we are assigning sourcetypes based on the respective sources -

inputs.conf

[monitor:\\share\sm\sm-expt]
disabled = false
whitelist = .txt$|.log$
index = sm
sourcetype = sm-expt

[monitor:\\share\sm\sm-impt]
disabled = false
whitelist = .txt$|.log$
index = sm
sourcetype = sm-impt

[monitor:\\share\sm\sm-plat]
disabled = false
whitelist = .txt$|.log$
index = sm
sourcetype = sm-plat

[monitor:\\share\sm\sm-vend]
disabled = false
whitelist = .txt$|.log$
index = sm
sourcetype = sm-vend

Not really sure , where we're going wrong. The regex looks fine when tested and the logs are using log4net type of logging format. we have just assigned different sourcetypes for ease in identifying different applications.

Re: How to configure setting the host field during index time?

arkadyz1 — Fri, 27 Jan 2017 16:30:16 GMT

One of the things I just realized: in transforms.conf, you have an underscore ( _) before MetaData, which, I believe, should not be there. Try DEST_KEY = MetaData:Host and see.

Re: How to configure setting the host field during index time?

nmohammed — Mon, 30 Jan 2017 18:38:35 GMT

Did remove the underscore ( _ ) long ago, but did not help. I have changed a few things this morning and finally got it working now.

Changed the Regex
Have to specify the transforms for all the different sourcetypes, add only a single stanza doesn't work. Tested it by adding single stanza (did not work) and then changed and added it for every sourcetype .

[hostextract]
REGEX = (ep\w*\d)
FORMAT = host::$1
DEST_KEY = MetaData:Host

[sm-plat]
TRANSFORMS-hostextract=hostextract

[sm-expt]
TRANSFORMS-hostextract=hostextract

[sm-impt]
TRANSFORMS-hostextract=hostextract

[sm-vend]
TRANSFORMS-hostextract=hostextract

Thanks