Is it possible to do a regex at search time or preferably at index time to do this?

[mbasharat]

Hi,

I have below raw event. Data is ingested via reading logfiles from dedicated location on monitored server with UF on it. Splunk's default method is not extracting fields as I need. Some fields have nested fields within. Is it possible to do a regex at search time or preferably at index time to do this?

### Tue Apr 11 00:00:06 CDT 2023: logChangeEventForSplunk() called ###

    event.id: 00000000-d825-00000-0cd1-00000000000000

    event.time.received: Tue Apr 11 00:00:06 CDT 2023

    event.time.first.received: Mon Apr 10 23:56:04 CDT 2023

    event.title: TESTING XYZ:CPU Load status changed from OK to Critical

    event.description: null

    event.state: closed

    event.severity: unknown

    event.receivedOnCiDowntime: false

    event.etiHint: CPULoad:Bottlenecked:82.0

    event.isLogOnly: false

    forwarding.type: notify_and_update

    event.solution: null

    event.control.transferred.to.name: <none>

    event.control.transferred.to.dns.name:

    event.control.transferred.to.state: <none>

    event.control.transferred.to.external.id:

    event.duplicate_count: 0

    event.external.id: urn:uuid:00000000000-d825-00000-0cd1-0000000000000

    cause.external.id: null

    custom attributes:

        SubmitCloseKey=true

        bsmc_policy_type=xml-ws

    history list:

        history line 1:

            historyLine.timeCreated.1=2023-04-10 23:56:05.624

            historyLine.messageKey.1=null

            historyLine.modifiedBy.1=System

            historyLine.headline.1=null

        history line 2:

            historyLine.timeCreated.2=2023-04-10 23:56:05.83

            historyLine.messageKey.2=null

            historyLine.modifiedBy.2=System

            historyLine.headline.2=null

        history line 3:

            historyLine.timeCreated.3=2023-04-11 00:00:06.336

            historyLine.messageKey.3=historylines.component.closing.related.events

            historyLine.modifiedBy.3=System

            historyLine.headline.3=Closing Related Events

    Related CI: 0000000000000000000000000

        lic_operational2advanced=false

        root_candidatefordeletetime=Sun Apr 30 12:22:50 CDT 2023

        data_operationisnew=false

        lic_type_basic=false

        lic_type_asset=false

        lic_type_udf=false

        type=nt

        root_class=nt

        lic_type_udi=false

        TenantsUses=System Default Tenant

        display_label=XYZMACHINE

        data_operationstate=0:Normal

        host_key=0.0.0.0 DefaultDomain

        lic_type_premium=false

        monitored_by=XYZ.ABC.com

        data_allow_auto_discovery=true

        root_actualdeletetime=Sat May 20 12:22:50 CDT 2023

        data_teststate=0:Normal

        id=0000000000000000000000

        type_label=Windows

        project=ABC

        default_gateway_ip_address_type=IPv4

        data_changecorrstate=0:No Change

        last_modified_time=Thu Apr 06 15:05:46 CDT 2023

        create_time=Thu Jan 26 12:56:33 CST 2023

        TenantOwner=System Default Tenant

        data_changestate=0:No Change

        primary_dns_name=XYZ.ABC.com

        contextmenu=itCIs

        global_id=00000000000000000000000000

        lic_type_management=false

        data_testisnew=false

        root_lastaccesstime=Mon Apr 10 12:22:50 CDT 2023

        lic_type_operational=false

        root_iscandidatefordeletion=false

        data_source=XYZ: SAMPLE

        data_changeisnew=false

        data_testcorrstate=0:Normal

        track_changes=false

        host_iscomplete=true

        name=AAAAAAAAAAAAAAAAAAA

        data_operationcorrstate=0:Normal

        is_save_persistency=false

        data_adminstate=0:Managed                                                                                                                                       

        lic_type_full=false

        root_enableageing=true

        data_updated_by=XYZ : ABC

### Tue Apr 11 00:00:06 CDT 2023: logChangeEventForSplunk() finished ###

[richgalloway · Accepted solution]

Don't use the default settings for ingesting data.  Those are just guesses and probably are wrong for your data.  Every input should specify a sourcetype and that sourcetype should be present in a props.conf file on the indexers.  That props.conf file should contain the "Great Eight" settings for the sourcetype:

[mysourcetype]
TIME_PREFIX = ###
TIME_FORMAT = %a %b %d %H:%M:%S %Z %Y
MAX_TIMESTAMP_LOOKAHEAD = 34
SHOULD_LINEMERGE = false
LINE_BREAKER = ()###.* called ###
TRUNCATE = 10000
EVENT_BREAKER_ENABLE = true
EVENT_BREAKER = ()###.* called ###

Of course, this example is just for the sample event.  The settings should be changed for each type of data ingested.

Add this setting to extract fields.

KV_MODE = auto

[mbasharat]

Hi @richgalloway 

Can you tell me how did you test below? Any makeresults etc. to test it via search?

Thanks!

[richgalloway]

I used regex101.com to test the regular expressions.  The rest of the answer is based on training and experience.

[mbasharat]

Slight adjustment and  worked out great. Thank you @richgalloway !!!