Splunk Search

How to configure setting the host field during index time?

nmohammed
Contributor

we're trying to set the host fields by extracting the name from the events, but it doesn't seem to work and would appreciate if someone can guide through -

example events -

2017-01-20 14:18:55,816 [31] ep7mmn001 ERROR SMS.Shared.ApiCommon.Attribute.smapiAuthorizeAttribute    at System.Convert.FromBase64_Decode(Char* startInputPtr, Int32 inputLength, Byte* startDestPtr, Int32 destLength)
   at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)
   at System.Convert.FromBase64CharArray(Char[] inArray, Int32 offset, Int32 length)
   at Newtonsoft.Json.JsonTextReader.ParseString(Char quote)
   at Newtonsoft.Json.JsonTextReader.ParseValue()
2017-01-20 14:18:55,816 [31] ep8mmn002 ERROR SMS.Shared.ApiCommon.Attribute.smApiAuthorizeAttribute Invalid length for a Base-64 char array or string.
2017-01-20 14:18:55,816 [31] ep8mmn006 ERROR SMS.Shared.ApiCommon.Attribute.smApiAuthorizeAttribute Invalid length for a Base-64 char array or string.

props.conf

[sm-plat]
TRANSFORMS-hostextract=hostextract

[sm-expt]
TRANSFORMS-hostextract=hostextract

[sm-impt]
TRANSFORMS-hostextract=hostextract

[sm-vend]
TRANSFORMS-hostextract=hostextract

transforms.conf

[hostextract]
REGEX = ^[^\]\n]*\]\s+(\w+)
FORMAT = host::$1
DEST_KEY = _MetaData:Host

Thanks

0 Karma
1 Solution

nmohammed
Contributor

Did remove the underscore ( _ ) long ago, but did not help. I have changed a few things this morning and finally got it working now.

  1. Changed the Regex
  2. Have to specify the transforms for all the different sourcetypes, add only a single stanza doesn't work. Tested it by adding single stanza (did not work) and then changed and added it for every sourcetype .

[hostextract]
REGEX = (ep\w*\d)
FORMAT = host::$1
DEST_KEY = MetaData:Host

[sm-plat]
TRANSFORMS-hostextract=hostextract

[sm-expt]
TRANSFORMS-hostextract=hostextract

[sm-impt]
TRANSFORMS-hostextract=hostextract

[sm-vend]
TRANSFORMS-hostextract=hostextract

Thanks

View solution in original post

0 Karma

nmohammed
Contributor

Did remove the underscore ( _ ) long ago, but did not help. I have changed a few things this morning and finally got it working now.

  1. Changed the Regex
  2. Have to specify the transforms for all the different sourcetypes, add only a single stanza doesn't work. Tested it by adding single stanza (did not work) and then changed and added it for every sourcetype .

[hostextract]
REGEX = (ep\w*\d)
FORMAT = host::$1
DEST_KEY = MetaData:Host

[sm-plat]
TRANSFORMS-hostextract=hostextract

[sm-expt]
TRANSFORMS-hostextract=hostextract

[sm-impt]
TRANSFORMS-hostextract=hostextract

[sm-vend]
TRANSFORMS-hostextract=hostextract

Thanks

0 Karma

arkadyz1
Builder

First of all, your regex is OK - I saw that on my own, but regex101.com confirmed it as well. So the problem is in either your stanza in transforms.conf - but it seems OK to me again - or in your referencing the transform from props.conf. Do your events end up in one of those sourcetypes (sm-plat, sm-expt, sm-impt, sm-vend) and how do you specify their sourcetype?

0 Karma

arkadyz1
Builder

One of the things I just realized: in transforms.conf, you have an underscore ( _) before MetaData, which, I believe, should not be there. Try DEST_KEY = MetaData:Host and see.

0 Karma

nmohammed
Contributor

Thanks. I tried hunters approach and referenced only one stanza ,but still doesn't work.

[sm]
TRANSFORMS-hostextract=hostextract

arkadzy1 -

we are assigning sourcetypes based on the respective sources -

inputs.conf

[monitor:\\share\sm\sm-expt]
disabled = false
whitelist = .txt$|.log$
index = sm
sourcetype = sm-expt

[monitor:\\share\sm\sm-impt]
disabled = false
whitelist = .txt$|.log$
index = sm
sourcetype = sm-impt

[monitor:\\share\sm\sm-plat]
disabled = false
whitelist = .txt$|.log$
index = sm
sourcetype = sm-plat

[monitor:\\share\sm\sm-vend]
disabled = false
whitelist = .txt$|.log$
index = sm
sourcetype = sm-vend

Not really sure , where we're going wrong. The regex looks fine when tested and the logs are using log4net type of logging format. we have just assigned different sourcetypes for ease in identifying different applications.

0 Karma

hunters_splunk
Splunk Employee
Splunk Employee

Hi nmohammed,

Since you are just using one transform stanza, I think you can just use one stanza to reference it in props.conf:

 [sm]
 TRANSFORMS-hostextract=hostextract

Also, the REGEX does not seem quite right and I'm not sure which hostnames it tries to capture from the events. You may need to finetune the REGEX to make sure it extracts hostnames correctly.

Hope it helps. Thanks!
Hunter

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...