Field Extraction when Index Time

[silverKi]

_raw data

[fw4_deny] [ip-address] start_time="1998-07-07 11:21:09" end_time="1998-07-07 11:21:09" machine_name=test_chall_1 fw_rule_id=11290 fw_rule_name=auto_ruleId_1290 nat_rule_id=0 nat_rule_name= src_ip=1xx.1xx.0.x user_id=- src_port=63185 dst_ip=192.168.0.2 dst_port=16992 protocol=6 app_name=- app_protocol=- app_category=- app_saas=no input_interface=eth212 bytes_forward=70 bytes_backward=0 packets_total=1 bytes_total=70 flag_record=S terminate_reason=Denied by Deny Rule is_ssl=no is_sslvpn=no host=- src_country=X2 dst_country=X2

[resource_cnt] [10.10.10.10] time="1998-07-07 11:24:50" machine_name=test_boby_1 cpu_usage=7.0 mem_usage=19.8 disk_usage=5.6 cpu_count=32, cpu_per_usage=3.0-2.9-2.0-2.0-2.0-2.0-0.0-0.0-23.0-7.9-7.0-6.9-19.4-19.0-8.0-7.0-1.0-1.0-16.0-1.0-2.0-2.0-1.0-2.0-24.8-9.0-16.2-8.0-9.0-9.9-5.0-8.1

my props.conf
[secui:fw]
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SEDCMD-duration = s/duration=\d+\s//
SEDCMD-fragment_info = s/fragment_info=\S*\s//
SEDCMD-ingres_if = s/ingres_if=\S*\s//
SEDCMD-input = s/input\sinterface/interface/
SEDCMD-packets_backward = s/packets_backward=\S*\s//
SEDCMD-packets_forward = s/packets_forward=\S*\s//
SEDCMD-pre = s/^[^\[]+//
SEDCMD-terminate_reason = s/\sterminate_reason=-//
SEDCMD-user_auth = s/user_auth=\S*\s//
SEDCMD-userid = s/user_id=\S*\s//
TRANSFORMS-secui_nullq = secui_nullq
TRANSFORMS-stchg7 = secui_resource
TRANSFORMS-stchg8 = secui_session
category = Custom
description = test
disabled = false
pulldown_type = true

<Fields you want to exclude>
fw_rule_name,  app_saas
nat_rule_name,  is_ssl
user_id,  is_sslvpn
app_name,  host
app_protocol,  src_country
app_category,  dst_country

I want to exclude fields that I want to exclude from being extracted at index time. 
Currently, fields that I want to exclude are automatically extracted when searching for fields of interest. 
Is there a way to do this?

[silverKi]

I am trying to exclude unnecessary fields from the firewall log collection. 

I am trying to delete the fields by excluding them, but they are not reflected well, 
so I am curious about the related collection exclusion process.

[PrewinThomas]

@silverKi 

Try below config to remove highlighted fields from the _raw event. Since they’re not in the raw, Splunk won’t auto-extract them at search time.

props.conf
[secui:fw]
TRANSFORMS-removefields = remove_unwanted_fields

transforms.conf
[remove_unwanted_fields]
REGEX = \s?(fw_rule_name|app_saas|nat_rule_name|is_ssl|user_id|is_sslvpn|app_name|host|app_protocol|src_country|app_category|dst_country)=[^ ]*
FORMAT =
DEST_KEY = _raw

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[silverKi]

I use 'SEDCMD-rm<fieldname>'

WHY my sedcmd is not work?

SEDCMD-rm-appname = s/app_name\=.*/\s//
SEDCMD-rm_appsaas = s/app_saas\=\w+\s//

[bowesmana]

You could also look at ingest actions

https://docs.splunk.com/Documentation/Splunk/9.4.2/Data/DataIngest

which gives you a slightly easier way to achieve the same thing - at least it can be done in the UI, so you can get an interactive way of seeing the results of your configuration.