topic Re: How to drop some field/values before indexing. in Getting Data In

How to drop some field/values before indexing.

zubairsp — Wed, 15 Nov 2023 13:45:46 GMT

Hello everyone, need your help!
We have a data source which is sending huge logs and thus we want to drop useless field values before indexing, we have installed an add-on which extracts a field from raw logs (example: ProductName) and assigns the value of this field as source to the raw logs and similarly uses the same props and transforms to extract rest of the field/values till this part everything works smooth, now the issue is among those extracted fields there is a field which i want to drop before indexing, i understand that there is way to send particular field/value to a nullqueue using props and transforms, below is a sample configuration i tried but it didn't work.

Your support in this is highly appreciated.

Props: [test:syslog] SHOULD_LINEMERGE = false EVENT_BREAKER_ENABLE = true TRANSFORMS-test_source = test_source, test_format_source TRANSFORMS-nullQ=nullFilter REPORT-regex_field_extraction = test_regex_field_extraction, test_file_name_file_path REPORT-dvc = test_dvc Transforms: [test_source] REGEX = ProductName="([^"]+)" DEST_KEY = MetaData:Source FORMAT = source::$1 [test_format_source] INGEST_EVAL = source=replace(lower(source), "\s", "_") [test_dvc] REGEX = ^<\d+>\d\s[^\s]+\s([^\s]+) FORMAT = dvc::"$1" [nullFilter] REGEX = (?mi)XYZData\>(.*)?=\<*?\/XYZData\> FORMAT = remove_field::$1 DEST_KEY = queue FORMAT = nullQueue [test_regex_field_extraction] REGEX = <([\w-]+)>([^<]+?)<\/\1> FORMAT = $1::$2 CLEAN_KEYS = false [test_file_name_file_path] REGEX = ^(.+)[\\/]([^\\/]+)$ FORMAT = source_process_name::$2 source_process_path::$1 SOURCE_KEY = SourceProcessName [test_severity_lookup] filename = test_severity.csv [test_action_lookup] filename = test_action_v110.csv case_sensitive_match = false [drop_useless_fields] INGEST_EVAL = random:=null()

## Below is the sample raw log, i want to drop the field <XYZData>

Sample raw data:

<29>1 2023-11-09T18:34:02.0Y something testEvents - EventFwd [agentInfo@ioud tenantId="1" bpsId="1" tenantGUID="{00000000-0000-0000-0000-000000000000}" tenantNodePath="x\y"] <?xml version="x.0"?> <test_hjgd><MachineInfo><AgentGUID>{iolkiu-5d9b-89iu-3e19-jjkiuhygtf}</AgentGUID><MachineName>something</MachineName><RawMACAddress>xyxyxyxyxz</RawMACAddress><IPAddress>xx.xx.xx.xx</IPAddress><AgentVersion>x.x.x.xx</AgentVersion><OSName>GGG</OSName><TimeZoneBias>-333</TimeZoneBias><UserName>xxx</UserName></MachineInfo><EventList ProductName="something" ProductVersion="xx.x" ProductFamily="Secure"><Event><Time>2023-11-09T18:34:38</Time><Severity/><EventID>jkiuj</EventID><XYZData>T1BHoQUAAAAAAAADQg0AAO0dXVNbu1E/JZP3xNyElJsZ13cA24kbwB7bhPaJMdiAG8DUNjehnf737of26ONIOrID0zycyWCfo13trlar1WolOU31h/qh7tSteqX+VDO1VCs1Vwt1r/6qXqvf1Fu1A9+vAHKvLqF8CtB7dU3QUzVWXfUGsP5COCu1VhOATuHzlmjMCO8enhH+h2qppjoA6AogbfhG/AHwXMDTDOivNe+hegQKM9Uhme6h/BX8PakHeOsBfZbtI/ztwhNSNZhjjfeKZF3B8y1weVIn8HmnJeqor+qcPjtQPobnsfoHSNKBpwFwtyH78L2vjgA+At4jKOnCew9KOtAG5P4bydJUjYAUrmyfSC8XpJ8e1G6p/6grtad+h3/voHwXIKjPD0DvPTztgvax/A2Uf4RSLHsPpXtQ6wpoTqHeDpTvge7+60lQ5tWEti3gbQ7YT6RH5L/j/XuT+eT/Y/5lDi7XIUj3J/UL93QLavr1yjguDenJVqmmDdkHK2VNfHXsmvvrLfztwOcuyY504vjcg3NoDY+CGfT9Ar4nhb2iDc+gR+YwllpFP2xSA8fSIfzNtfw4pj7qPn8Hnx+gp5Guj9eEN6R3X4wf5ItjZUSja10xEg5VH95PQF6k2oO292gsnEP9MehkTGOiDzbfhTb0CPN10WvVvKUnrqm917oU6z2SrrGlov04TqgH5uRvECbPHfIZdwmP8Tvo0fYYNsXN/AaOKPQQh6Qf4z2wdGz5B8Qewr8+/DF0BHVe6x4OWYrvOWxYC/r/HuS5oJZiCxfw+UD2egXPS5D0FfA6BG7ojy/JH4e4IHWbU8N7C+s0ZtumL3DsPpAkaz1TsH+fQl9eUq/EtYt+d0CaGpOlfdLeuA96OyX79L0z6nGXZKriy3b4QHznpBWU/DNgoOT3hd84AV5H1JfHuhc/qTMYgWyj1fVDXHpAD/t7D+i8hT/2Pm9p9voQocx1mmAtOAqWVMYt+kGedaD7Ig1vQ+lKa8T1PT6dXMwOze+o6TlZhKEQg5gZaUg8cGaPz+fv9OiUCGAFlNnL3VLcgF75iry9kVJwOiDpJdmBD0HOEyu+KHN/T/PBa92DBjPHK6DNYpzAHhT96Tn89T1fGaKK3E4o+im3SrQto31aUPJLUzSee35PSdsmS7TLdrRt+eVN0NMKRhha+yNFKIJbLq/i+jfAXAWhEmE9rwaq+GHbcAyLh1xv1JMj8uk4D3aBwi3ZyI4e6WWI62+RL3uvQ239C4dCHi73QrgFl1D3yRpfXwD+9AI6TvFKSeJGhnl4bdDLsRe3voTdxPjgPDyB+epluNq0zXxwAt9r9Z1s4RtY1g1gLbU+TimqaeloU54PCS72MKexuXoBeWN87DkqJX2oz/N9OEd0Q4rmjC830V0XYAPLp6e42bPYy/StSx37iD3TAlr1mWa8qV7JNAr4HdWZ6Gh9VtAY0ep+aXmLXGzxLPb8yp5spjMKvCao1v5IR3bDYg2CUWAXYK8tL+hS5bXhkviasl2IIz7C50eIsffo+YNeK/qYqBWJuW61H7ZnavSEGFn7ccHzxgPVcrCGV5W8DylfgauSo2IF4kqC8fM+6JHf+uAR7fVcrgytDHxbd8YPjWj0NrRGH8leU9GdX+eYYJcw6pGfWOQZ9egNtOce3p8AC+vO9Zwp7duurhnHomn0OQvwQLzSwHamIpTGT9YPeZmuHm8La3z7s7bd2pXuhQWtzG89GHrU74DxRq/Gq/CatA6dq39Bu2YlPjt67ohjxOpjjLPUWmklqPh4aJnskaY6fjS9bFqVxgrFPWW5D2n+Rs9oVhe2zbfJUicUocm8lYYPaX29IK82tWYzod6IrKCkrrynxpCxOzOOcuvG6thR+jb1c8a+Xbcd9HkSGcRg///obkDtuSmylebNHdfY+xI7NAqfuaSs5Jpm3BuylEnS/2Puaag+URbqnHJNn3Uu4zxg3ZhxM94/xs/NaoRW3mGoyCVYZzRip+T5xuRfzYojDpOoLy1BGsuXRLBTElXjdEg7OJLZs8yB81zrdhWcrXJqGBxembG3wJbuw/Oa/P5NkZEz/nbzejavEYzGC/VPxZncvGyL1D2gPNvThrVCUm3HN9cTlSm4UfJmdUckM9cNZbP21HslGaU2xeTcH2Jdbon9fkr7Y8sApg3Zzh82Ii2Ity2nto1jMDCKCNm5GY0+zbwatqUfbsghr4ZtY5vzyK1jamy6eym56dgeJu4z4C7lKeCMnfVqnKNk5WwfdQy97O7P2tmrXGyf7gg+r9V9gpaL4dcvz5thKiE8n9YA5spBNsU4NsfWBtMetX6sKavxy43wD+B9sxp27OzODo+65xckBVMux9t5dewatkz5fHJrpTRcrdMcLTYCfOIztcstjufzjGO61mbLNoae5VXBDcWY0yLncpOw180ocMuPI9gmSkzhsFaqaKRxRGNVVKqwwhmvatk2qRPm0SPvxRmyvLb8LBW77+z5tLoX09h2f+bQzcV2+ziHcj5+qt/z2rBd7c1sYVtJtqNn535k/2jlnLEwdpHCsunIOoN9GM7Z/4bnLyBVmVoKt5lYoTQSMCOhT6+RgNk75rxHLZllv9ysE208t4xPJi0tDZp3zoEvdb7axMipMwbhGk0rL+TG4+FyzrNg33NWyc1BpKADnVdbkwQSG8r85EL9c1gxGPK7olKzqimXcZaMV3GnAD9ysmd+Oa7VsL9MziSdFT9SfajdLk6UjIsTT/4ezwGU/N3JifucXP68cr7SI820sArDzhWlcEX37q5XTqtPKAt0RidqvmS0vEc7B32wSj59Y3QQ5u1mL2UfO0eyIdl+H9Yu+6DvI9XJkK5cx8gXlyIsI66Pl8o+cfo80rbhaeiV9enM3gF9n+gTozGpy3KZ7BpS7Og9/Ee1tnIT6Xxgr9DegCTZ93alxnrnbezsTaU5Sqx6SlHzI2Gz10N/ckjzz0KPfxOX5mAzXfHep9SmBeVobEphOHrJbxRXnwHsInnC2Hj9/BrxfFk5N+JD21D6UMjt+9Uh1PsehUo5nm/iGVTOptrnmzhGQI2ylwxnIF1YF57yzz126fRzdZbBpiqzxoLmVcyt2buCPsT0cgjCufSljgFSddNYHKctCxzRrzufVeP0aca7Vnxm7jZCJwera/l7d66OQb6qmc5Ky3zjl5TPPzYCZeHstpzolHUujk60bcQ8ohpGSzHYKfCS81vm+bOaUFzKo8k9Wyo5Kv/EaY1X470EXsOxxtA8Z86cNSrg7shyx489Ml1I2c8Z3yjrjNp71t6z9p413q+G9yt4z2ofKf6Cs0joSy5ICvukUhondnYtfg48dsdtkxooOXrPWaLWGKA/it7YDD+O1bdOmdnzwiY1/Hsrtk+OQUxf2tih0vK6Wc7GT+j0SltrUzgs9VzBmalV5O7Ju2K3/vnukmIe5SvdHu05K/9t7ok+73ml+p6oua8Vv/f5q90KlZOV9R1Q+96hyYG85M3DWLbjuW4emrzbtjcMwxTqm4T1TcL6JmF9k7C+SVjfJPwZrvVNwvomYX2TsL5JWN8kRHnrm4T1TcL6JmF9k7C+SbjpTULTS5+g3+Yvkl0M8bA5pywgB2uk4wa02BMawxfKnPxLQQ8ossqPI9uKf22Mf2fsALxY3hkcl4+/j8IWxZjl8qaTQbdxy+VNkujA0prx2n55kyKHvpNV2VQDPSrrW1H1KKGFMj/WhHhezCFOqHckPxCH2ZB4D++UqKSswdanrMVHoIUVjeK7re2kq3r6nOKIfrdyTCM3x25yJZIZ7A7e2MteOCuIRhLao75YWzlKqRWDhCIXzilKTlhGXy4m5t6vyUdNiavcrfRP3bneIVS66TkMW8e8Er0kCzHytkp9EccbkKVdR/YKYlCOmx6SdVMYEgX9zD4c96ucmTzWWuK6vSKSqsLgs4+8nuAdg6mSnZVvRe/K6gjjhhuiG9mnfoY5BmwW9WGYyDhdN5cto9pwkDz3TcSbrG237iHoCVxa41kBKuEY38Z1ND7lfq7kew0u6FXapvZPcSy37sE1r5fCRe52on0fFtxa24VxFx925ex4tclzW1bbo91k5aktHmfnYbX2z1twLqaach824HTqnhfrImW2kDv6mqb/mj9GzJYvpD1fKFxriRhnuiTg/+ojFJOzLjQ3IGjBUblPoV8QkeA/goJhd3Vm3T7EIZjq/UJbmDD7DM2u/WNHltDuGE74ZImfXF7RCvc2IHWL3RPq0M3yUFTekZShLOSYveU9y8T7C5nKO4XufIkDZu91c0rIcHJ8MvP3+aumwzrnOOo6sGyGxyNTnwDHmLUnDdoy/TS35Rhci8dAE5FkVmVV/PygMx9UL7vhhyguikjio7lTKj0xrFHt/wl1m9lDi5X/w5cGCK3OXjt/zKShnj4nH0PHYO5EP90i7sOq8Z0NdZWchZjp6QzA3Pr5N2s26e12Cllg91zUCPKDw8pc5AzMkO8Q9nIseIzwddqpvd4zKiqxizbl7FsvgUks4RAYr/uIr8kbvCFvl9S/fsvjQjvll7Nb/N/JrTU/wA=</XYZData><UserInfo>T1BHoQUAAAAAAAADJwEAAIWSTW+CYBCE56eQ3lv10vRANQ1Yw6EfSfXSm1HbmgAlgE35933ehYooxgvszszusLv4muhXiWJ5+tFGuQpt9a1U97rSSDca8vZgUq3A17CpPo1daK5HXaO6NU2hUkvYNc/YemxMlxI7fqKxfE3xWaB1XhHcBhjhytguUMLKK6dr7DeUTkamZURAotHnSykO8pqIzxrPTMM6FLrTvPPTCLm2aHf3lUdZ6r+5XNBnaWfcGte30v6d4OJmrjGVtx3hk1A0Pm1JfM8N+9m0/ptwSJG7abB/RK6OWuUjWKUyykl8tz+iZ26XI/ST/zZNVLpnNXrLWnWLDfYt5x72KBzdPybfZOnDV4G79w/XozETtyriuyoakOEV+veGztv93aFPHRnS7xfX/qWH8=</UserInfo><DestinationUserInfo>T1BHoQUAAAAAAAADJwEAAIWSTW+CYBCE56eQ3lv10vRANQ1Yw6EfSfXSm1HbmgAlgE35933ehYooxgvszszusLv4muhXiWJ5+tFGuQpt9a1U97rSSDca8vZgUq3A17CpPo1daK5HXaO6NU2hUkvYNc/YemxMlxI7fqKxfE3xWaB1XhHcB6xHhhjhuyfgfr7DeUTkamZURAotHnSykO8pqIzxrPTMM6FLrTvPPTCLm2aHf3lUdZ6r+hjhhyujte30v6d4OJmrjGVtx3hk1A0Pm1JfM8N+9m0/ptwSJG7abB/RK6OWuUjWKUyykl8tz+iZ26XI/ST/zZNVLpnNXrLWnWLDfYt5x72KBzdPybfZOnDV4G79w/XozETtyriuyoakOEV+veGztv93aFPHRnS7xfX/qWH8=</DestinationUserInfo><ThreatName/><PolicyName/><TimeSZone>+00</TimeSZone></Event></EventList></XYYPREV_1100>

Re: How to drop some field/values before indexing.

richgalloway — Wed, 15 Nov 2023 14:51:51 GMT

The null queue is for whole events, not individual fields. One can remove fields using SEDCMD in props.conf.

SEDCMD-rm_XYZData = s/XYZData\>.*\<\/XYZData\>//

Re: How to drop some field/values before indexing.

isoutamo — Thu, 16 Nov 2023 08:34:47 GMT

couple of additional comments:

When you are indexing data REPORT-didn't executed it's (as EXTRACT) works only on search time.
When you have several TRANSFORMS stanzas on own lines then those are applied based on those names ASCII order! If you want apply those in specific order
1. put those in one TRANSFORMS-xyz = a, e, c, b, d
2. or ensure that names are evaluated on correct order (use eg 000x, 001y, 002a etc)

One good instructions for index phase https://www.aplura.com/assets/pdf/props_conf_order.pdf. Aplura have some other Cheat Sheets which helps.

r. Ismo

Re: How to drop some field/values before indexing.

PickleRick — Thu, 16 Nov 2023 10:17:01 GMT

"i understand that there is way to send particular field/value to a nullqueue"

You understand wrong, I'm afraid.

As @richgalloway pointed out - you can send _whole events_ to nullQueue if they match certain regex (or other criteria in case you use INGEST_EVAL).

You can use transforms to cut specific parts of the events with regexes.

But in the ingest pipeline Splunk has no knowledge about the search-time fields (the ones created with REPORT or EXTRACT entries as well as calculated fields or field aliases). It only knows the index-time fields (the default metadata ones and custom index-time extractions if any are defined). So if you want to trim your events you'd have to manipulate them with regexes.

But since your events are structured, it'd be probably better to process your events before ingesting them into Splunk with something that can interpret XML and can selectively filter it based on XML structure, not plain regexes.

Re: How to drop some field/values before indexing.

zubairsp — Fri, 17 Nov 2023 08:18:45 GMT

Thank you for your response, i tried SEDCMD as you suggested in our test environment but with g in the last (SEDCMD-rm_XYZData = s/XYZData\>.*\<\/XYZData\>//g) it only works if i don't use the current Add-on, is there anything i missing?

Re: How to drop some field/values before indexing.

zubairsp — Fri, 17 Nov 2023 08:22:44 GMT

Hi there, thank you for your response! can you help by sharing the configuration using (INGEST_EVAL) to trim out this specific part of the event.

Re: How to drop some field/values before indexing.

zubairsp — Fri, 17 Nov 2023 08:29:28 GMT

Thank you for your response! so in my scenario will below work?

Props:

[test:syslog]
SHOULD_LINEMERGE = false
EVENT_BREAKER_ENABLE = true

TRANSFORMS-test_source = nullFilter, test_source, test_format_source

REPORT-regex_field_extraction = test_regex_field_extraction, test_file_name_file_path
REPORT-dvc = test_dvc

Transforms:

[test_source]
REGEX = ProductName="([^"]+)"
DEST_KEY = MetaData:Source
FORMAT = source::$1

[test_format_source]
INGEST_EVAL = source=replace(lower(source), "\s", "_")

[test_dvc]
REGEX = ^<\d+>\d\s[^\s]+\s([^\s]+)
FORMAT = dvc::"$1"

[nullFilter]
REGEX = (?mi)XYZData\>(.*)?=\<*?\/XYZData\>
DEST_KEY = queue
FORMAT = nullQueue

[test_regex_field_extraction]
REGEX = <([\w-]+)>([^<]+?)<\/\1>
FORMAT = $1::$2
CLEAN_KEYS = false

[test_file_name_file_path]
REGEX = ^(.+)[\\/]([^\\/]+)$
FORMAT = source_process_name::$2 source_process_path::$1
SOURCE_KEY = SourceProcessName

[test_severity_lookup]
filename = test_severity.csv

[test_action_lookup]
filename = test_action_v110.csv
case_sensitive_match = false

Re: How to drop some field/values before indexing.

PickleRick — Fri, 17 Nov 2023 10:06:36 GMT

Fiddling with _raw event using INGEST_EVAL could be tricky. You can use the normal eval text functions but I suppose you'd have to set $pd:_raw$ as the destination key (maybe normal _raw would work as well, don't know never tried that; only used it to create new fields).

Re: How to drop some field/values before indexing.

isoutamo — Fri, 17 Nov 2023 11:13:48 GMT

When I need INGEST_EVAL I (almost) always use Splunk GUI to test it. You should replace normal rex command with replace (which is actually rex :-). Then just add all those eval commands in one line.

Re: How to drop some field/values before indexing.

zubairsp — Mon, 27 Nov 2023 05:50:14 GMT

All the other solutions are tricky which needs more time, however for now i settled with SEDCMD which only works with custom sourcetype, while i am still exploring if i find anything which works i will update this post.