Unable to use some props.conf settings in Splunk Cloud

[SuhDude]

Hello team,

For context this is a splunk cloud environment with an es and an ad hoc search head.

Today I tried to change an http event collector input from sourcetype _json to wiz.

The wiz events are json events with a date midway through the event.

Sample event (heavily obfuscated as it is company data):

{
  "event": {
    "trigger": {
      "source": "CLOUD_EVENTS",
      "type": "Created",
      "ruleId": "<rule_id>",
      "ruleName": "WIZ-Splunk Integration"
    },
    "event": {
      "name": "<eventname>",
      "eventURL": "<url>",
      "cloudPlatform": "AWS",
      "timestamp": "2024-06-12T03:01:18Z",
      "source": "<amazon source>",
      "category": "List",
      "path": null,
      "actor": {
        "name": "<account name>",
        "type": "SERVICE_ACCOUNT",
        "IP": "<FQDN>",
        "actingAs": {
          "name": "<role_name>",
          "providerUniqueId": "<UniqID",
          "type": "SERVICE_ACCOUNT",
           "rawlog": {"addendum":null,"additionalEventData":null,"apiVersion":null,"awsRegion":"us-east-1","errorCode":null,"errorMessage":null,"eventCategory":"<event_category>","eventID":"<event_id>","eventName":"<event_name>","eventSource":"<amazon_link>","eventTime":"2024-06-12T03:01:18Z","eventType":"<type of event>","eventVersion":"<version number>","managementEvent":true,"readOnly":true,"recipientAccountId":"<account ID>","requestID":"<request_id>","requestParameters":{"DescribeVpcEndpointsRequest":{"VpcEndpointId":{"content":"<VPCENDPOINTID>","tag":1}}},"resources":null,"responseElements":null,"serviceEventDetails":null,"sessionCredentialFromConsole":null,"sharedEventID":null,"sourceIPAddress":"<source ip>","tlsDetails":null,"userAgent":"<user agent>","userIdentity":{"accountId":"<account ID>","arn":"<ARN>","invokedBy":"<USER>","principalId":"<principal ID>","sessionContext":{"attributes":{"creationDate":"2024-06-12T03:01:17Z","mfaAuthenticated":"false"},"sessionIssuer":{"accountId":"<account ID>","arn":"<ARN>","principalId":"<principal ID>","type":"Role","userName":"<role name>"}},"type":"AssumedRole"},"vpcEndpointId":null}
        }
      },
      "subjectResource": {
        "name": "",
        "type": "",
        "providerUniqueId": "",
        "externalId": "<external ID>",
        "region": "us-east-1",
        "kubernetesCluster": "",
        "kubernetesNamespace": "",
        "account": {"externalId":"<external ID>","id":"<ID>"}
      },
      "matchedRules": " ruleId: ; ruleName: <RULE NAME> "
    }
  }
}

To accomplish the sourcetype name change I cloned the current configuration for _json under app search which was as follows:

CHARSET = UTF-8

DATETIME_CONFIG=

INDEXED_EXTRACTIONS=json

KV_MODE=none

SHOULD_LINEMERGE=true

category=structured

disabled=false

pulldown_type=true

LINE_BREAKER=([\r\n]+)

NO_BINARY_CHECK=true

This cloned the config successfully but notably put it under app 000-self-service rather than search.

I then set the input to the new sourcetype wiz.

Following this change some events began breaking incorrectly at the first timestamp in the log, a behavior not previously observed on sourcetype _json which had the same config.

Sample broken event:

Event1:

{

  "event": {

    "trigger": {

      "source": "CLOUD_EVENTS",

      "type": "Created",

      "ruleId": "<rule_id>",

      "ruleName": "WIZ-Splunk Integration"

    },

    "event": {

      "name": "<eventname>",

      "eventURL": "<url>",

      "cloudPlatform": "AWS",

Event 2:

      "timestamp": "2024-06-12T03:01:18Z",

      "source": "<amazon source>",

      "category": "List",

      "path": null,

      "actor": {

        "name": "<account name>",

        "type": "SERVICE_ACCOUNT",

        "IP": "<FQDN>",

        "actingAs": {

          "name": "<role_name>",

          "providerUniqueId": "<UniqID",

          "type": "SERVICE_ACCOUNT",

           "rawlog": {"addendum":null,"additionalEventData":null,"apiVersion":null,"awsRegion":"us-east-1","errorCode":null,"errorMessage":null,"eventCategory":"<event_category>","eventID":"<event_id>","eventName":"<event_name>","eventSource":"<amazon_link>","eventTime":"2024-06-12T03:01:18Z","eventType":"<type of event>","eventVersion":"<version number>","managementEvent":true,"readOnly":true,"recipientAccountId":"<account ID>","requestID":"<request_id>","requestParameters":{"DescribeVpcEndpointsRequest":{"VpcEndpointId":{"content":"<VPCENDPOINTID>","tag":1}}},"resources":null,"responseElements":null,"serviceEventDetails":null,"sessionCredentialFromConsole":null,"sharedEventID":null,"sourceIPAddress":"<source ip>","tlsDetails":null,"userAgent":"<user agent>","userIdentity":{"accountId":"<account ID>","arn":"<ARN>","invokedBy":"<USER>","principalId":"<principal ID>","sessionContext":{"attributes":{"creationDate":"2024-06-12T03:01:17Z","mfaAuthenticated":"false"},"sessionIssuer":{"accountId":"<account ID>","arn":"<ARN>","principalId":"<principal ID>","type":"Role","userName":"<role name>"}},"type":"AssumedRole"},"vpcEndpointId":null}

        }

      },

      "subjectResource": {

        "name": "",

        "type": "",

        "providerUniqueId": "",

        "externalId": "<external ID>",

        "region": "us-east-1",

        "kubernetesCluster": "",

        "kubernetesNamespace": "",

        "account": {"externalId":"<external ID>","id":"<ID>"}

      },

      "matchedRules": " ruleId: ; ruleName: <RULE NAME> "

    }

  }

}

This was strange behavior but likely was caused by the default setting of BREAK_ONLY_BEFORE_DATE=true

To remedy this I edited the sourcetype config for wiz by adding the following:

BREAK_ONLY_BEFORE ={[\r\n]\s+\"event\"\: BREAK_ONLY_BEFORE_DATE = false

Note I left the value below as True

SHOULD_LINEMERGE = true

However after clicking save the following changes were made:

BREAK_ONLY_BEFORE ={[\r\n]\s+\"event\"\:

LINE_BREAKER = {[\r\n]\s+\"event

SHOULD_LINEMERGE = false

The configuration for BREAK_ONLY_BEFORE_DATE was unable to be saved and SHOULD_LINEMERGE was unable to be set to true while BREAK_ONLY_BEFORE was present.

I tried performing this change many times over hours and tried creating unrelated sourcetypes with BREAK_ONLY_BEFORE_DATE but was unable to set this setting on splunk cloud. 

In addition, any attempt to set SHOULD_LINEMERGE to true while BREAK_ONLY_BEFORE was present resulted in SHOULD_LINEMERGE being set to false and LINE_BREAKER being set to the same value as BREAK_ONLY_BEFORE

Other settings were able to be set as expected. 

A final note for information is timestamp was set to auto.

Are these configurations invalid in general or just unable to be set in settings > sourcetypes > advanced in splunk cloud?

As an additional note no settings applied were able to set the event breaking to earlier behavior and I was forced to revert the change on the input back to sourcetype _json where breaking worked as expected. 

Would appreciate any answers and happy to provide more info if needed

Apologies for the long read.