Splunk Search

Regex Limitation

jadengoho
Builder

HI All, 

I have this JSON file that is 4400 Long , and i want it to reroute to a specific Indexer.

If i use REGEX101 - the regex will work, but when applied to Splunk - It does not reroute to the proper index.

The regex i want to get is on the bottom part of the log.

I want it to  be rerouted to gmail_index

 

[email_route]
REGEX = (gmail\.com)
DEST_KEY = _TCP_ROUTING
FORMAT = main_indexers

[email_route_index]
REGEX = (gmail\.com)

DEST_KEY = _MetaData:Index
FORMAT = gmail_indexer

 

{"AffectedItems": [{"Attachments": "1\u071b\u0738\u0771\u0771 \u073f\u0770\u073e \u0738\u0786\u0737\u0771\u0770\u0771\u073c\u0786\u0771\u0771\u077c \u0737\u073c\u0786\u073c.doc (31678b); \u071e\u073f\u0738\u0771\u0770\u0788\u0899\u031f\u077c\u073c\u0738\u073a.docx (89816b)", "Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+0AAAAAAEJAADpG/J8j7e08jBSJnska8+0AAGYRCBwAAAA", "InternetMessageId": "<EU0PR86MB137886DDV1833778ABCDE3EC8F8B0@EU0PR86MB1378.ampprd08.prod.outlook.com>", "ParentFolder": {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+1BBBBBBCRDDDDC", "Path": "\\\u070a\u0899\u0737\u0786\u0771\u031f\u0899\u073c\u0786"}, "Subject": "FW: \u071f\u0738\u0770\u0738\u0786\u0737\u0738\u073c\u0771\u0738\u0777\u0786\u073a\u0899\u0776\u0786\u077f \u0788\u071e\u0718 \"\u0718\u0706\u0780\u0710-\u0788\u0718\u0788\u070a\u071e\" \u0707\u0717\u0780\u071f\u071e\u0783: 33771737"}, {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+0AAAAAAEJAADpG/J8j7e08jBSJnska8+0AAGYRCBxAAAA", "InternetMessageId": "<EU0PR86MB13781BA33879ECE7B1D0C90D8F7A0@EU0PR86MB1378.ampprd08.prod.outlook.com>", "ParentFolder": {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+1BBBBBBCRDDDDC", "Path": "\\\u070a\u0899\u0737\u0786\u0771\u031f\u0899\u073c\u0786"}, "Subject": "RE: 83731031 \u0788\u071e\u0718\"\u0718\u0719\u0787 \u0718\u071b \u0711\u0706 \u078e\u071a\u0780\u0718\u0719\u070a\" 3 000.00 EUR_\u0737\u0899\u0770\u0899\u0778\u073e\u0738\u0899\u073c\u073e"}, {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+0AAAAAAEJAADpG/J8j7e08jBSJnska8+0AAGaVnGNAAAA", "InternetMessageId": "<EU0PR86MB137833B7F0DB78B801C788868F7B0@EU0PR86MB1378.ampprd08.prod.outlook.com>", "ParentFolder": {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+1BBBBBBCRDDDDC", "Path": "\\\u070a\u0899\u0737\u0786\u0771\u031f\u0899\u073c\u0786"}, "Subject": "FW: 33896888 \u0788\u071e\u0718 \"\u070a'\u078e\u0713\u0780\u0710\u0783\u070a\u0717\" 37 900.00 USD_\u071b\u0738\u0771\u0771+\u0786\u073c\u0738\u073e\u0739\u0771"}, {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+0AAAAAAEJAADpG/J8j7e08jBSJnska8+0AAGaVnGOAAAA", "InternetMessageId": "<EU0PR86MB13788FC8B138F838F06381BB8F7B0@EU0PR86MB1378.ampprd08.prod.outlook.com>", "ParentFolder": {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+1BBBBBBCRDDDDC", "Path": "\\\u070a\u0899\u0737\u0786\u0771\u031f\u0899\u073c\u0786"}, "Subject": "FW: 33896888 \u0788\u071e\u0718 \"\u070a'\u078e\u0713\u0780\u0710\u0783\u070a\u0717\" 7 890.00 USD_\u0737\u0899\u0770\u0899\u0778\u073e\u0738\u073c\u0899\u073e"}, {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+0AAAAAAEJAADpG/J8j7e08jBSJnska8+0AAGaVnGPAAAA", "InternetMessageId": "<EU0PR86MB13788D0E80FA79B088B90A7C8F7B0@EU0PR86MB1378.ampprd08.prod.outlook.com>", "ParentFolder": {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+1BBBBBBCRDDDDC", "Path": "\\\u070a\u0899\u0737\u0786\u0771\u031f\u0899\u073c\u0786"}, "Subject": "FW: 38708368 \u0788\u071e\u0718\"\u071f\u0710\u0788.-\u078e\u0780.\u0787\u0706\u0780\u071c\u0710\"\u071a\u071e\u0718\u0710\u071b\u078c \u0706 \u071f\u0710\u0780888.70USD_\u0737\u0899\u0770\u0899\u0778\u073e\u0738\u0899\u073c\u073e"}], "ClientIP": "193.168.100.100", "ClientIPAddress": "193.111.111.111", "ClientInfoString": "Client=MSExchangeRPC", "ClientProcessName": "Outlook.exe", "ClientVersion": "17.0.11989.80738", "CreationTime": "2020-18-10T08:38:17", "CrossMailboxOperation": false, "DestFolder": {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+0AAAAAAEKAAAB", "Path": "\\\u0718\u0738\u0737\u0899\u031f\u0738\u073c\u0786"}, "ExternalAccess": false, "Folder": {"Id": "JCNAAAA18PlntFTRK9sdgawlMkwpMNkwL/J8j7e08jBSJnska8+1BBBBBBCRDDDDC", "Path": "\\\u070a\u0899\u0737\u0786\u0771\u031f\u0899\u073c\u0786"}, "Id": "90cf3b8d-b98c-76b6-e9e8-08d89ce708ca", "InternalLogonType": 0, "LogonType": 0, "LogonUserSid": "S-3-9-81-618798686-7833011008-1735678990-9686938", "MailboxGuid": "5ff6777aa-fce1-58ca-sf7b-90dde880f68a", "MailboxOwnerSid": "S-3-9-81-618798686-7833011008-1735678240-9686938", "MailboxOwnerUPN": "unknown.testing@gmail.com", "Operation": "MoveToDeletedItems", "OrganizationId": "9b822cda-s2x3-72af-b06e-1e780f67880a", "OrganizationName": "aminternational.onmicrosoft.com", "OriginatingServer": "EU6PR07MB7108 (15.50.5655.088)\r\n", "RecordType": 3, "ResultStatus": "Succeeded", "UserId": "unknown.testing@gmail.com", "UserKey": "1003BDDDDD2796BC", "UserType": 0, "Version": 1, "Workload": "Exchange"}

 

 

What i noticed is if i remove some logs fields value it will rereoute

"LogonUserSid": "S-3-9-81-618798686-7833011008-1735678990-9686938", (will not re rerouted)(4108 th character)

"LogonUserSid": "S-3-9-81-618798686-7833011008, (will rereoute)(4089th charater)

There are no limits.conf applied its the default Splunk. But why does the character count affect it ?

Labels (2)
Tags (1)
0 Karma
1 Solution

jadengoho
Builder

I found the fix for this , it is related to https://community.splunk.com/t5/Splunk-Search/Regex-for-ending-with-a-particular-pattern/m-p/57396

 

This is why, if my logs are longer than 4096 and the Regex i want is beyond 4090 it won't be rerouted.

LOOKAHEAD = <integer>
* NOTE: This option is valid for all index time transforms, such as
  index-time field creation, or DEST_KEY modifications.
* Optional. Specifies how many characters to search into an event.
* Default: 4096
  * You may want to increase this value if you have event line lengths that
    exceed 4096 characters (before linebreaking).

 

View solution in original post

0 Karma

jadengoho
Builder

I found the fix for this , it is related to https://community.splunk.com/t5/Splunk-Search/Regex-for-ending-with-a-particular-pattern/m-p/57396

 

This is why, if my logs are longer than 4096 and the Regex i want is beyond 4090 it won't be rerouted.

LOOKAHEAD = <integer>
* NOTE: This option is valid for all index time transforms, such as
  index-time field creation, or DEST_KEY modifications.
* Optional. Specifies how many characters to search into an event.
* Default: 4096
  * You may want to increase this value if you have event line lengths that
    exceed 4096 characters (before linebreaking).

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...