topic Re: Why is my JSON regex expression not working properly? in Splunk Search

Why is my JSON regex expression not working properly?

Dawson014 — Tue, 04 Sep 2018 15:52:02 GMT

I have a JSON file, which is being indexed by Splunk, the format is like -

{
   testdata : [
      {
          "testid" : 1234,
          "abc" : "def",
          "def" : "abc",
          "httpServer" : [
               {
                     "responseTime" : 300,
                     "responseCode" : 200,
                     "datetime": 0982894965
               },
               {
                    "responseTime": 312,
                    "responseCode": 200,
                    "datetime": 09230948509
                }
           ],
          "transactions" : [
                 {
                   ....
                 },
                 {
                   ....
                 }
            ]
       },
       {
           "testid": 1234,
           ....
       }
   ]
}

Can someone please suggest a regex which can give me relevant data for every "testid". Whatever regex I tried doesn't seem to work. I was using this stanza in my props.conf

[randomsourcetype]
[accountgroups]
TRUNCATE = 0
KV_MODE = json
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
LINE_BREAKER = ([\r\n]*)(?=\{)
DATETIME_CONFIG = CURRENT

Thanks in advance

Re: Why is my JSON regex expression not working properly?

felipesewaybric — Sun, 16 Sep 2018 21:37:26 GMT

have you try the with the default json sourcetype? Testing in local here, i can access the data:
testdata.testid

Re: Why is my JSON regex expression not working properly?

MonkeyK — Sun, 16 Sep 2018 22:30:48 GMT

I have also had success with the json sourcetype.
For more complex json hierarchies, spath works very well.

in addition to the Splunk doc link above, here is an answers refrence that may help
https://answers.splunk.com/answers/63368/how-to-handle-simple-json-array-with-spath.html

Re: Why is my JSON regex expression not working properly?

woodcock — Mon, 17 Sep 2018 01:40:41 GMT

Like this:

|makeresults|eval _raw="{
    testdata : [
       {
           \"testid\" : 1234,
           \"abc\" : \"def\",
           \"def\" : \"abc\",
           \"httpServer\" : [
                {
                      \"responseTime\" : 300,
                      \"responseCode\" : 200,
                      \"datetime\": 0982894965
                },
                {
                     \"responseTime\": 312,
                     \"responseCode\": 200,
                     \"datetime\": 09230948509
                 }
            ],
           \"transactions\" : [
                  {
                    ....
                  },
                  {
                    ....
                  }
             ]
        },
        {
            \"testid\": 1234,
            ....
        }
    ]
 }"
 | rex max_match=0 "\s+{[\r\n]+\s+\"testid\"\s*:\s*(?<testid>\d+)"

Re: Why is my JSON regex expression not working properly?

Noah_Woodcock — Mon, 17 Sep 2018 02:17:48 GMT

My favorite tool for problems like this is Regex101.com
At least take a look 🙂

Re: Why is my JSON regex expression not working properly?

tkopchak — Mon, 17 Sep 2018 12:35:08 GMT

I have had success using a configuration like this for handling json:

[sourcetype]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = json
KV_MODE = json
NO_BINARY_CHECK = true
category = Structured
disabled = false
TRUNCATE = 999999

Once this data is indexed you can use the mvexpand command to view all values for the testid field.

Re: Why is my JSON regex expression not working properly?

Dawson014 — Wed, 26 Sep 2018 14:58:16 GMT

turns out if you remove the LINE_BREAKER = it works. Thanks for your suggestion.

Re: Why is my JSON regex expression not working properly?

Dawson014 — Wed, 26 Sep 2018 14:59:45 GMT

For anyone having the same issue - Just done user LINE_BREAKER or leave it blank or remove it. That's how I got it to work. Thanks for the suggestions.