Regex Limitation

jadengoho — Thu, 10 Dec 2020 12:33:42 GMT

HI All,

I have this JSON file that is 4400 Long , and i want it to reroute to a specific Indexer.

If i use REGEX101 - the regex will work, but when applied to Splunk - It does not reroute to the proper index.

The regex i want to get is on the bottom part of the log.

I want it to be rerouted to gmail_index

[email_route]
REGEX = (gmail\.com)
DEST_KEY = _TCP_ROUTING
FORMAT = main_indexers

[email_route_index]
REGEX = (gmail\.com)

DEST_KEY = _MetaData:Index
FORMAT = gmail_indexer

{"AffectedItems": [{"Attachments": "1\u071b\u0738\u0771\u0771 \u073f\u0770\u073e \u0738\u0786\u0737\u0771\u0770\u0771\u073c\u0786\u0771\u0771\u077c \u0737\u073c\u0786\u073c.doc (31678b); \u071e\u073f\u0738\u0771\u0770\u0788\u0899\u031f\u077c\u073c\u0738\u073a.docx (89816b)", "Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+0AAAAAAEJAADpG/J8j7e08jBSJnska8+0AAGYRCBwAAAA", "InternetMessageId": "<EU0PR86MB137886DDV1833778ABCDE3EC8F8B0@EU0PR86MB1378.ampprd08.prod.outlook.com>", "ParentFolder": {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+1BBBBBBCRDDDDC", "Path": "\\\u070a\u0899\u0737\u0786\u0771\u031f\u0899\u073c\u0786"}, "Subject": "FW: \u071f\u0738\u0770\u0738\u0786\u0737\u0738\u073c\u0771\u0738\u0777\u0786\u073a\u0899\u0776\u0786\u077f \u0788\u071e\u0718 \"\u0718\u0706\u0780\u0710-\u0788\u0718\u0788\u070a\u071e\" \u0707\u0717\u0780\u071f\u071e\u0783: 33771737"}, {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+0AAAAAAEJAADpG/J8j7e08jBSJnska8+0AAGYRCBxAAAA", "InternetMessageId": "<EU0PR86MB13781BA33879ECE7B1D0C90D8F7A0@EU0PR86MB1378.ampprd08.prod.outlook.com>", "ParentFolder": {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+1BBBBBBCRDDDDC", "Path": "\\\u070a\u0899\u0737\u0786\u0771\u031f\u0899\u073c\u0786"}, "Subject": "RE: 83731031 \u0788\u071e\u0718\"\u0718\u0719\u0787 \u0718\u071b \u0711\u0706 \u078e\u071a\u0780\u0718\u0719\u070a\" 3 000.00 EUR_\u0737\u0899\u0770\u0899\u0778\u073e\u0738\u0899\u073c\u073e"}, {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+0AAAAAAEJAADpG/J8j7e08jBSJnska8+0AAGaVnGNAAAA", "InternetMessageId": "<EU0PR86MB137833B7F0DB78B801C788868F7B0@EU0PR86MB1378.ampprd08.prod.outlook.com>", "ParentFolder": {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+1BBBBBBCRDDDDC", "Path": "\\\u070a\u0899\u0737\u0786\u0771\u031f\u0899\u073c\u0786"}, "Subject": "FW: 33896888 \u0788\u071e\u0718 \"\u070a'\u078e\u0713\u0780\u0710\u0783\u070a\u0717\" 37 900.00 USD_\u071b\u0738\u0771\u0771+\u0786\u073c\u0738\u073e\u0739\u0771"}, {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+0AAAAAAEJAADpG/J8j7e08jBSJnska8+0AAGaVnGOAAAA", "InternetMessageId": "<EU0PR86MB13788FC8B138F838F06381BB8F7B0@EU0PR86MB1378.ampprd08.prod.outlook.com>", "ParentFolder": {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+1BBBBBBCRDDDDC", "Path": "\\\u070a\u0899\u0737\u0786\u0771\u031f\u0899\u073c\u0786"}, "Subject": "FW: 33896888 \u0788\u071e\u0718 \"\u070a'\u078e\u0713\u0780\u0710\u0783\u070a\u0717\" 7 890.00 USD_\u0737\u0899\u0770\u0899\u0778\u073e\u0738\u073c\u0899\u073e"}, {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+0AAAAAAEJAADpG/J8j7e08jBSJnska8+0AAGaVnGPAAAA", "InternetMessageId": "<EU0PR86MB13788D0E80FA79B088B90A7C8F7B0@EU0PR86MB1378.ampprd08.prod.outlook.com>", "ParentFolder": {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+1BBBBBBCRDDDDC", "Path": "\\\u070a\u0899\u0737\u0786\u0771\u031f\u0899\u073c\u0786"}, "Subject": "FW: 38708368 \u0788\u071e\u0718\"\u071f\u0710\u0788.-\u078e\u0780.\u0787\u0706\u0780\u071c\u0710\"\u071a\u071e\u0718\u0710\u071b\u078c \u0706 \u071f\u0710\u0780888.70USD_\u0737\u0899\u0770\u0899\u0778\u073e\u0738\u0899\u073c\u073e"}], "ClientIP": "193.168.100.100", "ClientIPAddress": "193.111.111.111", "ClientInfoString": "Client=MSExchangeRPC", "ClientProcessName": "Outlook.exe", "ClientVersion": "17.0.11989.80738", "CreationTime": "2020-18-10T08:38:17", "CrossMailboxOperation": false, "DestFolder": {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+0AAAAAAEKAAAB", "Path": "\\\u0718\u0738\u0737\u0899\u031f\u0738\u073c\u0786"}, "ExternalAccess": false, "Folder": {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+1BBBBBBCRDDDDC", "Path": "\\\u070a\u0899\u0737\u0786\u0771\u031f\u0899\u073c\u0786"}, "Id": "90cf3b8d-b98c-76b6-e9e8-08d89ce708ca", "InternalLogonType": 0, "LogonType": 0, "LogonUserSid": "S-3-9-81-618798686-7833011008-1735678990-9686938", "MailboxGuid": "5ff6777aa-fce1-58ca-sf7b-90dde880f68a", "MailboxOwnerSid": "S-3-9-81-618798686-7833011008-1735678240-9686938", "MailboxOwnerUPN": "unknown.testing@gmail.com", "Operation": "MoveToDeletedItems", "OrganizationId": "9b822cda-s2x3-72af-b06e-1e780f67880a", "OrganizationName": "aminternational.onmicrosoft.com", "OriginatingServer": "EU6PR07MB7108 (15.50.5655.088)\r\n", "RecordType": 3, "ResultStatus": "Succeeded", "UserId": "unknown.testing@gmail.com", "UserKey": "1003BDDDDD2796BC", "UserType": 0, "Version": 1, "Workload": "Exchange"}

What i noticed is if i remove some logs fields value it will rereoute

"LogonUserSid": "S-3-9-81-618798686-7833011008-1735678990-9686938", (will not re rerouted)(4108 th character)

"LogonUserSid": "S-3-9-81-618798686-7833011008, (will rereoute)(4089th charater)

There are no limits.conf applied its the default Splunk. But why does the character count affect it ?

Re: Regex Limitation

jadengoho — Fri, 11 Dec 2020 02:37:39 GMT

I found the fix for this , it is related to https://community.splunk.com/t5/Splunk-Search/Regex-for-ending-with-a-particular-pattern/m-p/57396

This is why, if my logs are longer than 4096 and the Regex i want is beyond 4090 it won't be rerouted.

LOOKAHEAD = <integer>
* NOTE: This option is valid for all index time transforms, such as
  index-time field creation, or DEST_KEY modifications.
* Optional. Specifies how many characters to search into an event.
* Default: 4096
  * You may want to increase this value if you have event line lengths that
    exceed 4096 characters (before linebreaking).

topic Regex Limitation in Splunk Search

Regex Limitation

Re: Regex Limitation