Hi Folks Is there a way to analyze the bandwith used between the SearchHeads and the indexer cluster peers? I know this has many dependencies on the search and its artifacts but we do need to have a rough calculation to size the environment. Any help is appreciated. Cheers, Claudio ... View more

Hi Ninjas Im struggling with the following scenario: I have a heavy forwarder whos collecting a merged data stream called "generic_sourcetype". For example, this stream consists of the following events (format wise): Event 1 Sep 24 18:22:16 - 209.160.24.63 - - [24/Sep/2017:18:22:16.885] "GET /product.screen?productId=WC-SH-A02&JSESSIONID=SD0SL6FF7ADFF4953 HTTP 1.1" 200 3878 "http://www.google.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.46 Safari/536.5" 349 sourcetype:a Event 2 Sep 24 00:15:03 - 209.160.24.63 - - Thu Sep 24 2017 00:15:02.554 www1 sshd[4747]: Failed password for invalid user jabber from 118.142.68.222 port 3187 ssh2 sourcetype:b This comes in as one merged data stream (no i cant influence that) - so i built a "routing" with transforms.conf on the heavy forwarder like: props.conf [generic_sourcetype] TRANSFORMS-route_st = route_st_a, route_st_b transforms.conf [route_st_a] REGEX = sourcetype:a FORMAT = sourcetype::a DEST_KEY = MetaData:Sourcetype [route_st_b] REGEX = sourcetype:b FORMAT = sourcetype::b DEST_KEY = MetaData:Sourcetype So far so good, this config works fine and i got the two sourcetypes indexed properly. Now the problem I have is the following: Those two events have a detailed timestamp after the header with milliseconds which i want to use as the indexed timestamp. So i configured parsing settings in props.conf for both of the sourcetypes (a+b) on the heavyforwarder like: props.conf [a] TIME_PREFIX = $ProperSetting TIME_FORMAT = $ProperSetting MAX_TIMESTAMP_LOOKAHEAD = $ProperSetting [b] TIME_PREFIX = $ProperSetting TIME_FORMAT = $ProperSetting MAX_TIMESTAMP_LOOKAHEAD = $ProperSetting Testing those settings by adding a oneshot with the dedicated sourcetype set during input shows that my configs are correct and the correct timestamp for both events is extracted. But somehow it does not work with my generic stream, it does split it up but it ignores my timestamp configuration and keeps indexing the first timestamp for both events. So it seems that the heavyforwarder assigns a timestamp automatically for the generic_sourcetype and then processes the transfomrs for the sourcetype filtering but then sends the events directly instead of "re-parse" them with the given settings for the new sourcetype. Is this the way splunk handles this kind of data? Or am I missing something (or somewhere)? Thanks as always ... View more

Hi Ninjas Im aware that you can resolve ips & hosts with the built in dnslookup like <search> | lookup dnslookup clientip as dst OUTPUT clienthost as DST_RESOLVED But is there a way to specify the DNS Server to resolve the ips? I think by default i takes the default dns entry of the splunk machine and i would like to specify another one (without touching the command script). Any ideas? ... View more

Hi Ninjas I have two different json logs which looks like this: {"version":"1.1","host":"t800.skynet.com","short_message":"Msg TzjTJPqvUaGqaOpHXFdKXyXVaHiLpbTKfhePqbEtammLeaZVaaTb \r\n","full_message":"Msg TzjTJPqvUaGqaOpHXFdKXyXVaHiLpbTKfhePqbEtammLeaZVaaTb \r\n","timestamp":1484920098.408,"level":4,"_app":"skynet","_level_name":"WARN","_mdcKeybIwMa":"mdcValueNYUGJgYJaTFaWcdicara","_thread_name":"sample","_logger_name":"common.log.json.LogFileProducer","_env":"ut"} and the other one looks like this: {"timestamp":"2017-01-20 14:48:18.428","level":"DEBUG","thread":"sample","logger":"common.log.json.LogFileProducer","msg":"Msg TzjTJPqvUaGqaOpHXFdKXyXVaHiLpbTKfhePqbEtammLeaZVaaTb","mdc":{"mdcKeybIwMa":"mdcValueNYUGJgYJaTFaWcdicara"}} Those files are located on a windows machine where i have installed a Universal Forwarder(6.5x) with the following settings: props.conf: [my_json_sourcetype] INDEXED_EXTRACTIONS = json TIMESTAMP_FIELDS = timestamp And a poper inputs.conf which simply points to the appropriate directory using monitor and applying the sourcetype "my_json_sourcetype". Now the problem which is driving me nuts: The Events come in in correct json format on the indexer, but searching on them shows that not all the fields are extracted, such as all the "mdc" keyvalue fields are ignored completely. Now the fact which is driving me even more nuts: Uploading the exact same log file by using the add data feature directly on my linux indexer with the same sourcetype (copied the props from the forwarder as descibed above) works perfect and extracts all the fields absolutely perfect - including the mdc key value fields. The forwarder or indexer logs show absolutely nothing in that case, everything seems to work "fine". Any ideas why the forwarder handles the data differently and why not all the fields are extracted? And why it works with the same settings on my indexer? Did i miss a setting concerning indexed_extractions on the uf? Any help is appreciated, thx Cheers Salem ... View more