Hello team, For context this is a splunk cloud environment with an es and an ad hoc search head. Today I tried to change an http event collector input from sourcetype _json to wiz. The wiz events are json events with a date midway through the event. Sample event (heavily obfuscated as it is company data): { "event": { "trigger": { "source": "CLOUD_EVENTS", "type": "Created", "ruleId": "<rule_id>", "ruleName": "WIZ-Splunk Integration" }, "event": { "name": "<eventname>", "eventURL": "<url>", "cloudPlatform": "AWS", "timestamp": "2024-06-12T03:01:18Z", "source": "<amazon source>", "category": "List", "path": null, "actor": { "name": "<account name>", "type": "SERVICE_ACCOUNT", "IP": "<FQDN>", "actingAs": { "name": "<role_name>", "providerUniqueId": "<UniqID", "type": "SERVICE_ACCOUNT", "rawlog": {"addendum":null,"additionalEventData":null,"apiVersion":null,"awsRegion":"us-east-1","errorCode":null,"errorMessage":null,"eventCategory":"<event_category>","eventID":"<event_id>","eventName":"<event_name>","eventSource":"<amazon_link>","eventTime":"2024-06-12T03:01:18Z","eventType":"<type of event>","eventVersion":"<version number>","managementEvent":true,"readOnly":true,"recipientAccountId":"<account ID>","requestID":"<request_id>","requestParameters":{"DescribeVpcEndpointsRequest":{"VpcEndpointId":{"content":"<VPCENDPOINTID>","tag":1}}},"resources":null,"responseElements":null,"serviceEventDetails":null,"sessionCredentialFromConsole":null,"sharedEventID":null,"sourceIPAddress":"<source ip>","tlsDetails":null,"userAgent":"<user agent>","userIdentity":{"accountId":"<account ID>","arn":"<ARN>","invokedBy":"<USER>","principalId":"<principal ID>","sessionContext":{"attributes":{"creationDate":"2024-06-12T03:01:17Z","mfaAuthenticated":"false"},"sessionIssuer":{"accountId":"<account ID>","arn":"<ARN>","principalId":"<principal ID>","type":"Role","userName":"<role name>"}},"type":"AssumedRole"},"vpcEndpointId":null} } }, "subjectResource": { "name": "", "type": "", "providerUniqueId": "", "externalId": "<external ID>", "region": "us-east-1", "kubernetesCluster": "", "kubernetesNamespace": "", "account": {"externalId":"<external ID>","id":"<ID>"} }, "matchedRules": " ruleId: ; ruleName: <RULE NAME> " } } } To accomplish the sourcetype name change I cloned the current configuration for _json under app search which was as follows: CHARSET = UTF-8 DATETIME_CONFIG= INDEXED_EXTRACTIONS=json KV_MODE=none SHOULD_LINEMERGE=true category=structured disabled=false pulldown_type=true LINE_BREAKER=([\r\n]+) NO_BINARY_CHECK=true This cloned the config successfully but notably put it under app 000-self-service rather than search. I then set the input to the new sourcetype wiz. Following this change some events began breaking incorrectly at the first timestamp in the log, a behavior not previously observed on sourcetype _json which had the same config. Sample broken event: Event1: { "event": { "trigger": { "source": "CLOUD_EVENTS", "type": "Created", "ruleId": "<rule_id>", "ruleName": "WIZ-Splunk Integration" }, "event": { "name": "<eventname>", "eventURL": "<url>", "cloudPlatform": "AWS", Event 2: "timestamp": "2024-06-12T03:01:18Z", "source": "<amazon source>", "category": "List", "path": null, "actor": { "name": "<account name>", "type": "SERVICE_ACCOUNT", "IP": "<FQDN>", "actingAs": { "name": "<role_name>", "providerUniqueId": "<UniqID", "type": "SERVICE_ACCOUNT", "rawlog": {"addendum":null,"additionalEventData":null,"apiVersion":null,"awsRegion":"us-east-1","errorCode":null,"errorMessage":null,"eventCategory":"<event_category>","eventID":"<event_id>","eventName":"<event_name>","eventSource":"<amazon_link>","eventTime":"2024-06-12T03:01:18Z","eventType":"<type of event>","eventVersion":"<version number>","managementEvent":true,"readOnly":true,"recipientAccountId":"<account ID>","requestID":"<request_id>","requestParameters":{"DescribeVpcEndpointsRequest":{"VpcEndpointId":{"content":"<VPCENDPOINTID>","tag":1}}},"resources":null,"responseElements":null,"serviceEventDetails":null,"sessionCredentialFromConsole":null,"sharedEventID":null,"sourceIPAddress":"<source ip>","tlsDetails":null,"userAgent":"<user agent>","userIdentity":{"accountId":"<account ID>","arn":"<ARN>","invokedBy":"<USER>","principalId":"<principal ID>","sessionContext":{"attributes":{"creationDate":"2024-06-12T03:01:17Z","mfaAuthenticated":"false"},"sessionIssuer":{"accountId":"<account ID>","arn":"<ARN>","principalId":"<principal ID>","type":"Role","userName":"<role name>"}},"type":"AssumedRole"},"vpcEndpointId":null} } }, "subjectResource": { "name": "", "type": "", "providerUniqueId": "", "externalId": "<external ID>", "region": "us-east-1", "kubernetesCluster": "", "kubernetesNamespace": "", "account": {"externalId":"<external ID>","id":"<ID>"} }, "matchedRules": " ruleId: ; ruleName: <RULE NAME> " } } } This was strange behavior but likely was caused by the default setting of BREAK_ONLY_BEFORE_DATE=true To remedy this I edited the sourcetype config for wiz by adding the following: BREAK_ONLY_BEFORE ={[\r\n]\s+\"event\"\: BREAK_ONLY_BEFORE_DATE = false Note I left the value below as True SHOULD_LINEMERGE = true However after clicking save the following changes were made: BREAK_ONLY_BEFORE ={[\r\n]\s+\"event\"\: LINE_BREAKER = {[\r\n]\s+\"event SHOULD_LINEMERGE = false The configuration for BREAK_ONLY_BEFORE_DATE was unable to be saved and SHOULD_LINEMERGE was unable to be set to true while BREAK_ONLY_BEFORE was present. I tried performing this change many times over hours and tried creating unrelated sourcetypes with BREAK_ONLY_BEFORE_DATE but was unable to set this setting on splunk cloud. In addition, any attempt to set SHOULD_LINEMERGE to true while BREAK_ONLY_BEFORE was present resulted in SHOULD_LINEMERGE being set to false and LINE_BREAKER being set to the same value as BREAK_ONLY_BEFORE Other settings were able to be set as expected. A final note for information is timestamp was set to auto. Are these configurations invalid in general or just unable to be set in settings > sourcetypes > advanced in splunk cloud? As an additional note no settings applied were able to set the event breaking to earlier behavior and I was forced to revert the change on the input back to sourcetype _json where breaking worked as expected. Would appreciate any answers and happy to provide more info if needed Apologies for the long read.
... View more
