Hi,
I have below raw event. Data is ingested via reading logfiles from dedicated location on monitored server with UF on it. Splunk's default method is not extracting fields as I need. Some fields have nested fields within. Is it possible to do a regex at search time or preferably at index time to do this?
### Tue Apr 11 00:00:06 CDT 2023: logChangeEventForSplunk() called ###
event.id: 00000000-d825-00000-0cd1-00000000000000
event.time.received: Tue Apr 11 00:00:06 CDT 2023
event.time.first.received: Mon Apr 10 23:56:04 CDT 2023
event.title: TESTING XYZ:CPU Load status changed from OK to Critical
event.description: null
event.state: closed
event.severity: unknown
event.receivedOnCiDowntime: false
event.etiHint: CPULoad:Bottlenecked:82.0
event.isLogOnly: false
forwarding.type: notify_and_update
event.solution: null
event.control.transferred.to.name: <none>
event.control.transferred.to.dns.name:
event.control.transferred.to.state: <none>
event.control.transferred.to.external.id:
event.duplicate_count: 0
event.external.id: urn:uuid:00000000000-d825-00000-0cd1-0000000000000
cause.external.id: null
custom attributes:
SubmitCloseKey=true
bsmc_policy_type=xml-ws
history list:
history line 1:
historyLine.timeCreated.1=2023-04-10 23:56:05.624
historyLine.messageKey.1=null
historyLine.modifiedBy.1=System
historyLine.headline.1=null
history line 2:
historyLine.timeCreated.2=2023-04-10 23:56:05.83
historyLine.messageKey.2=null
historyLine.modifiedBy.2=System
historyLine.headline.2=null
history line 3:
historyLine.timeCreated.3=2023-04-11 00:00:06.336
historyLine.messageKey.3=historylines.component.closing.related.events
historyLine.modifiedBy.3=System
historyLine.headline.3=Closing Related Events
Related CI: 0000000000000000000000000
lic_operational2advanced=false
root_candidatefordeletetime=Sun Apr 30 12:22:50 CDT 2023
data_operationisnew=false
lic_type_basic=false
lic_type_asset=false
lic_type_udf=false
type=nt
root_class=nt
lic_type_udi=false
TenantsUses=System Default Tenant
display_label=XYZMACHINE
data_operationstate=0:Normal
host_key=0.0.0.0 DefaultDomain
lic_type_premium=false
monitored_by=XYZ.ABC.com
data_allow_auto_discovery=true
root_actualdeletetime=Sat May 20 12:22:50 CDT 2023
data_teststate=0:Normal
id=0000000000000000000000
type_label=Windows
project=ABC
default_gateway_ip_address_type=IPv4
data_changecorrstate=0:No Change
last_modified_time=Thu Apr 06 15:05:46 CDT 2023
create_time=Thu Jan 26 12:56:33 CST 2023
TenantOwner=System Default Tenant
data_changestate=0:No Change
primary_dns_name=XYZ.ABC.com
contextmenu=itCIs
global_id=00000000000000000000000000
lic_type_management=false
data_testisnew=false
root_lastaccesstime=Mon Apr 10 12:22:50 CDT 2023
lic_type_operational=false
root_iscandidatefordeletion=false
data_source=XYZ: SAMPLE
data_changeisnew=false
data_testcorrstate=0:Normal
track_changes=false
host_iscomplete=true
name=AAAAAAAAAAAAAAAAAAA
data_operationcorrstate=0:Normal
is_save_persistency=false
data_adminstate=0:Managed
lic_type_full=false
root_enableageing=true
data_updated_by=XYZ : ABC
### Tue Apr 11 00:00:06 CDT 2023: logChangeEventForSplunk() finished ###
Don't use the default settings for ingesting data. Those are just guesses and probably are wrong for your data. Every input should specify a sourcetype and that sourcetype should be present in a props.conf file on the indexers. That props.conf file should contain the "Great Eight" settings for the sourcetype:
[mysourcetype]
TIME_PREFIX = ###
TIME_FORMAT = %a %b %d %H:%M:%S %Z %Y
MAX_TIMESTAMP_LOOKAHEAD = 34
SHOULD_LINEMERGE = false
LINE_BREAKER = ()###.* called ###
TRUNCATE = 10000
EVENT_BREAKER_ENABLE = true
EVENT_BREAKER = ()###.* called ###
Of course, this example is just for the sample event. The settings should be changed for each type of data ingested.
Add this setting to extract fields.
KV_MODE = auto
Don't use the default settings for ingesting data. Those are just guesses and probably are wrong for your data. Every input should specify a sourcetype and that sourcetype should be present in a props.conf file on the indexers. That props.conf file should contain the "Great Eight" settings for the sourcetype:
[mysourcetype]
TIME_PREFIX = ###
TIME_FORMAT = %a %b %d %H:%M:%S %Z %Y
MAX_TIMESTAMP_LOOKAHEAD = 34
SHOULD_LINEMERGE = false
LINE_BREAKER = ()###.* called ###
TRUNCATE = 10000
EVENT_BREAKER_ENABLE = true
EVENT_BREAKER = ()###.* called ###
Of course, this example is just for the sample event. The settings should be changed for each type of data ingested.
Add this setting to extract fields.
KV_MODE = auto
Hi @richgalloway
Can you tell me how did you test below? Any makeresults etc. to test it via search?
Thanks!
I used regex101.com to test the regular expressions. The rest of the answer is based on training and experience.
Slight adjustment and worked out great. Thank you @richgalloway !!!