topic Re: Regex Help; exceeded configured match_limit in Splunk Search

Regex Help; exceeded configured match_limit

JoshuaJohn — Tue, 22 Jan 2019 16:22:47 GMT

I have this query that works in all regex assist sites but is too greedy for my Splunk Environment. I am unable to add it to props, and it must be in the query itself.

| rex "^(?:{\"publish_time\": )(?<PublishTime>[^,]+).+?(?:\"ver\\\" : \\\")(?<Ver>[^\\\]+).+?(?:\"storeId\\\" : \\\")(?<StoreNum>[^\\\]+).+?(?:\"MSRReach\\\" : \\\")(?<ScannerConnection>[^\\\]+).+?(?:\"isConnected\\\" : )(?<ConnectedStatus>[^,]+).+?(?:\"isProduction\\\" : )(?<Prod>[^,]+).+?(?:\"serialNumber\\\" : \\\")(?<ScannerSerialNum>[^\\\]+).+?(?:\"OSVersion\\\" : \\\")(?<ScannerOSVersion>[^\\\]+).+?(?:\"model\\\" : \\\")(?<ScannerModel>[^\\\]+).+?(?:\"Serial\\\" : \\\")(?<iPadSerial>[^\\\]+).+?(?:\"battery\\\" : \\\")(?<iPadBattery>[^\\\]+).+?(?:\"osVersion\\\" : \\\")(?<iPadOSVersion>[^\\\]+)$"

Here is what a log looks like

{"publish_time": 1548432475.596, "data": {"insertId": "xueeeeees", "receiveTimestamp": "2019-01-21T15:51:31.311111132Z", "logName": "projects/nitro-mobile/logs/boot", "labels": {"compute.awsapis.com/resource_name": "-------", "container.awsapis.com/pod_name": "-----------", "container.awsapis.com/stream": "----", "container.awsapis.com/namespace_name": "---------"}, "jsonPayload": {"thread": "http-nitro", "timestamp": "1548432475.596", "message": "payload=2019/01/01  StartEvent {\n  \"appDetails\" : {\n    \"version\" : \"1\",\n    \"number\" : \"1\",\n    \"ver\" : \"11.1.0\",\n    \"release\" : \"11.1.1\",\n    \"storeId\" : \"0120\",\n    \"terminalId\" : \"41\"\n  },\n  \"capTime\" : \"2019-01-01 10:59:48\",\n  \"MSR\" : {\n    \"MSRReach\" : \"Scanner not connected\",\n    \"version\" : \"Not available\",\n    \"isConnected\" : true,\n    \"SB\" : \"\"\n  },\n  \"isProduction\" : true,\n  \"pinpad\" : {\n    \"serialNumber\" : \"111-112-111-111\",\n    \"OSVersion\" : \"AWS12343\",\n    \"model\" : \"910X5\"\n  },\n  \"MDM\" : {\n    \"lastSeen\" : \"Not Available\"\n  },\n  \"device\" : {\n    \"Serial\" : \"BV12AXAJ013J\",\n    \"battery\" : \"100.0\",\n    \"network\" : \"ReachableViaWiFi\",\n    \"osVersion\" : \"11.0.1\",\n    \"memory\" : \"100\"\n  }\n} EndEvent\n"

Any ideas to improve efficiency?

Re: Regex Help; exceeded configured match_limit

JoshuaJohn — Tue, 22 Jan 2019 17:06:32 GMT

Well this is ugly, but it works.

| rex "^(?:{\"publish_time\": )(?<PublishTime>[^,]+)" 
| rex "(?:\\\"ver\\\\\\\" : \\\\\\\")(?<Ver>[^\\\\]+)" 
| rex "(?:\\\"storeId\\\\\\\" : \\\\\\\")(?P<StoreNum>[^\\\\]+)" 
| rex "(?:\\\"MSRReach\\\\\\\" : \\\\\\\")(?<ScannerConnection>[^\\\\]+)" 
| rex "(?:\\\"isConnected\\\\\\\" : )(?<ConnectedStatus>[^,]+)" 
| rex "(?:\\\"isProduction\\\\\\\" : )(?<Prod>[^,]+)" 
| rex "(?:\\\"serialNumber\\\\\\\" : \\\\\\\")(?<ScannerSerialNum>[^\\\\]+)" 
| rex "(?:\\\"OSVersion\\\\\\\" : \\\\\\\")(?<ScannerOSVersion>[^\\\\]+)" 
| rex "(?:\\\"model\\\\\\\" : \\\\\\\")(?<ScannerModel>[^\\\\]+)" 
| rex "(?:\\\"Serial\\\\\\\" : \\\\\\\")(?<iPadSerial>[^\\\\]+)" 
| rex "(?:\\\"battery\\\\\\\" : \\\\\\\")(?<iPadBattery>[^\\\\]+)" 
| rex "(?:\\\"osVersion\\\\\\\" : \\\\\\\")(?<iPadOSVersion>[^\\\\]+)"

Re: Regex Help; exceeded configured match_limit

woodcock — Tue, 22 Jan 2019 23:04:47 GMT

Have you tried piping your events through | kv or | spath? You will be surprised at how much gets done well enough!

Re: Regex Help; exceeded configured match_limit

JoshuaJohn — Wed, 23 Jan 2019 15:23:55 GMT

I haven't, but I will look into it. Looks promising especially for my data.