Updated link: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Configurationparametersandthedatapipeline ... View more

Just a tip.   You can clean up your input to make regex simpler.  This example removes the single " and replace the \" with " | rex mode=sed field=_raw "s/[^\\\]\"//g s/\\\\\"/\"/g" ... View more

Here is  a solution that seems to work for us to find origin of those error messages. This search should show if you have problem with too long fields.   index=_internal source="*splunkd.log" "Received metadata string exceeding maxLength"   This is due to that Splunk tries to create a indexed field that has more than 1000.  Looking trough the logs its not easy to see where this come from.  So with some tips from mmccul I have made this search.   index=<your index> | table [| walklex index=<your index> type=field | search NOT field IN (_indextime date_* punct time*pos) | stats count by field | table field | mvcombine field | return $field ] | foreach * [| eval <<FIELD>>_Len=len(<<FIELD>>)] | table *_Len | stats max(*) as * | transpose header_field=column | rename "row 1" as count | sort -count | rename count as Length, column as FieldName | head 10 | rex mode=sed field=FieldName "s/_Len$//"   What it does: `walklex` will list all indexed fields for an index.  Then with some commands make a list of all interesting fields and return it to the table `| foreach * [| eval <<FIELD>>_Len=len(<<FIELD>>)]` Calculate the length of each field. Then we find the the max length for each field and final get the top 10 field with largest bytes. If there are field with 1000 bytes, there are for sure some fields that are longer, so go trough the raw data and see if there are some regex that do eat to much of the line.This may not be the best/fastest way but seems to work for us. ... View more

If you have may fields and do not know the name of them or variable number of fields, you can run it like this: | stats max(*) as max_* ... View more

Here you go:   | makeresults | eval test="( {\"Item Id\": \"1\", \"locationCategoryCodes\": [{\"categoryCodes\": [{\"categoryCode\": \"CAT_1\", \"ruleID\": [\"138563\"]}], \"locationCode\": \"ABC\"}, {\"categoryCodes\": [{\"categoryCode\": \"CAT_1\", \"ruleID\": [\"138563\"]}], \"locationCode\": \"XYZ\"}, {\"categoryCodes\": [{\"categoryCode\": \"CAT_2\", \"ruleID\": [\"138561\"]}], \"locationCode\": \"DEF\"}, {\"categoryCodes\": [{\"categoryCode\": \"CAT_3\", \"ruleID\": [\"138614\"]}], \"locationCode\": \"IJK\"}], \"timestamp\": \"2023-01-27T00:10:32.367 +0000\"} ¤ {\"Item Id\": \"2\", \"locationCategoryCodes\": [{\"categoryCodes\": [{\"categoryCode\": \"CAT_1\", \"ruleID\": [\"138563\"]}], \"locationCode\": \"ABC\"}, {\"categoryCodes\": [{\"categoryCode\": \"CAT_3\", \"ruleID\": [\"138614\"]}], \"locationCode\": \"IJK\"}], \"timestamp\": \"2023-01-27T00:10:32.367 +0000\"})" | makemv test delim="¤" | mvexpand test | rename test as _raw | rex max_match=0 "locationCode.: .(?<location>[^\"]+)" | mvexpand location | stats count by location Everything before the two empty lines are just to create sample data to make proof of concept.       ... View more

Remember as I told you, its not an optimal regex.  IT will hit multiple times in the line and if first word is not 4 or 5 characters log, it will try next word that is 4 or 5 characters long. https://regex101.com/r/7OSbxb/1 Some better: ^(?<mylevel>\w{4,5}) even better ^(?<mylevel>\S+) ... View more

May fault, just assumed TCPDUMP was for TCP only, but according to manual it can look at UDP and ICMP as well.  Bad name of the application 😉   ... View more

Syslog are normal UDP, so tcpdump will not work. ... View more

| rex field=msg "(?<mylevel>\w{4,5})"   I think this is wrong approach.  It will mach any word with 4 or 5 characters within msg field.   Some better would be: | rex field=msg "^(?<mylevel>\w{4-5})" The ^ makes sure text are on start of the line. Even better: | rex field=msg "(?<mylevel>(?:INFO|DEBUG))" or | rex field=msg "(?<mylevel>(?:INFO|DEBUG|ERROR))" ... View more

Fixed error in regex that did not work correctly An even better regex to get just the file name (especially for long path name) (?:[^\\]+\\)+([^.]+).log$ ... View more

In your input.conf on your UF, set a fixed sourcetype for your data, example "my_logdata" Then on your Index server or HF add the following props and transforms add the following.   props.conf [my_logdata] TRANSFORMS-changesourcetype = changesourcetype-my_logdata     transforms.conf [changesourcetype-my_logdata] SOURCE_KEY = MetaData:Source REGEX = ([^\\]+).log DEST_KEY = MetaData:Sourcetype FORMAT = sourcetype::$1   We do use some like this to set sourcetype based on foldername   ... View more

i did read (slak) that you do not need to have two copies of indexes to use index cluster.  It could give you benefit when adding more nodes to the index since it then distribute data over the new index nodes better. There may also be other benefit as well with management of the index nodes etc. PS I have not tested this ... View more

You can do some like this:   <your search> | eval len=len(_raw) | stats sum(len)   It will give you size in bytes.  This may also take some time depending of how much data you have how long back you do search. To remove the same data, you can delete them.  You need to give admin the delete role first:   <your search> | delete   ... View more

Its fixed in 9.0.4. But form the change log, it was fixed just days after 9.0.3 was released. 2022-12-02 SPL-233492, SPL-229404 Invalid key in stanza instrumentation.usage.tlsBestPractices ... View more

Splunk is the only who can fix this and they have confirmed a fix for this in 9.0.4 ... View more

Escaping with regex may make it hard to read, but luckily there is a smart way around it. You can start and stop in regex with \Q  and \E on what you like to get literal. | makeresults | eval _raw= "C:\Program Files (x86)\Google" | regex "\QC:\Program Files (x86)\Google\E"  This makes regex much more readable. https://www.regular-expressions.info/characters.html ... View more

Not sure if you have an optimal regex.  Why do you make a non capturing group of "-" and why a capture group in the named group?  This is some better: rex field=title "(?<titleNEW>.*?):-"  Even better, do not use * in regex when its not needed.  Do a search until you find some that its not included, like this: rex field=title "(?<titleNEW>[^-]+)"   ... View more

[setnull] Do not use stansa name like this.  What happens if you have two app with samme stansa name, it may give you problem Use f.eks.  [remove_firewall_sensor_abcd]  ... View more

Some comments to the above post. Its better to remove stuff at the Universal Forwarder instead of HF or Index. So to remove 4662, add the following to an input.file # Used to block 4662 message    [WinEventLog://Security] blacklist1 = 4662   Or you can do like this.  Block all 4662 message except 4662 with Message="ms-Mcs-AdmPwd" [WinEventLog://Security] whitelist1 = EventCode="^4662$" Message="ms-Mcs-AdmPwd" whitelist2 = EventCode="^((?!4662$)[0-9]*)$"       Take care with this: REGEX=EventCode=(4776|4662|4634|4688|4648|4907|4768|4624||538|560|552|534) This may block all, due to the double ||,   I gess that is a typo. Also it will block 1552, 5525 etc, so here you should use ^ and $ ... View more

This helped us out as well.  Did have same error. ... View more