topic Re: Kafka regex: Why is the command not working in Splunk search? in Splunk Search

Kafka regex: Why is the command not working in Splunk search?

pswalia06 — Fri, 22 Jun 2018 14:54:33 GMT

{"topic": "amx", "total_lag": 2670, "partitions": [{"lag": 117, "partition_number": 0}, {"lag": 122, "partition_number": 1}, {"lag": 130, "partition_number": 2}, {"lag": 130, "partition_number": 3}, {"lag": 148, "partition_number": 4}, {"lag": 144, "partition_number": 5}, {"lag": 158, "partition_number": 6}, {"lag": 130, "partition_number": 7}, {"lag": 123, "partition_number": 8}, {"lag": 145, "partition_number": 9}, {"lag": 130, "partition_number": 10}, {"lag": 127, "partition_number": 11}, {"lag": 123, "partition_number": 12}, {"lag": 121, "partition_number": 13}, {"lag": 118, "partition_number": 14}, {"lag": 125, "partition_number": 15}, {"lag": 133, "partition_number": 16}, {"lag": 161, "partition_number": 17}, {"lag": 134, "partition_number": 18}, {"lag": 151, "partition_number": 19}]}


index=orion-platform  source="/opt/bda/logs/kafkalag.log" |spath output=AA path=counterList{1} | rex field=AA "\"lag\":\s(?.\w+)\,\s\"partition_number\"\:\s(?\d+)\}" max_match=100 | table State1,partition_number

Above command not working in splunk search.

Re: Kafka regex: Why is the command not working in Splunk search?

kamlesh_vaghela — Fri, 22 Jun 2018 15:52:22 GMT

Can you please try the following search? I haven't used any regular expression but it will give you all the data from JSON event.

YOUR_SEARCH |
| rename partitions{}.lag as lag, partitions{}.partition_number as partition_number
| eval temp = mvzip(lag,partition_number) | stats count by _time total_lag,topic,temp
| eval lag = mvindex(split(temp,","),0) ,partition_number=mvindex(split(temp,","),1) | table topic total_lag lag partition_number

My Sample Search:

| makeresults 
| eval _raw="{\"topic\": \"amx\", \"total_lag\": 2670, \"partitions\": [{\"lag\": 117, \"partition_number\": 0}, {\"lag\": 122, \"partition_number\": 1}, {\"lag\": 130, \"partition_number\": 2}, {\"lag\": 130, \"partition_number\": 3}, {\"lag\": 148, \"partition_number\": 4}, {\"lag\": 144, \"partition_number\": 5}, {\"lag\": 158, \"partition_number\": 6}, {\"lag\": 130, \"partition_number\": 7}, {\"lag\": 123, \"partition_number\": 8}, {\"lag\": 145, \"partition_number\": 9}, {\"lag\": 130, \"partition_number\": 10}, {\"lag\": 127, \"partition_number\": 11}, {\"lag\": 123, \"partition_number\": 12}, {\"lag\": 121, \"partition_number\": 13}, {\"lag\": 118, \"partition_number\": 14}, {\"lag\": 125, \"partition_number\": 15}, {\"lag\": 133, \"partition_number\": 16}, {\"lag\": 161, \"partition_number\": 17}, {\"lag\": 134, \"partition_number\": 18}, {\"lag\": 151, \"partition_number\": 19}]}" 
| kv
| rename partitions{}.lag as lag, partitions{}.partition_number as partition_number
| eval temp = mvzip(lag,partition_number) | stats count by _time total_lag,topic,temp
| eval lag = mvindex(split(temp,","),0) ,partition_number=mvindex(split(temp,","),1) | table topic total_lag lag partition_number

Please let me know if assistance required.

Thanks

Re: Kafka regex: Why is the command not working in Splunk search?

pswalia06 — Tue, 29 Sep 2020 20:08:25 GMT

The amx value is showing continuesly and the total_lag is showing the same repeated value for each lag and partition_name

I want this
Topic_name total_lag partition_number lag
amx 240. 0. 20
1. 30

Re: Kafka regex: Why is the command not working in Splunk search?

kamlesh_vaghela — Fri, 22 Jun 2018 16:58:42 GMT

@pswalia06

Are you looking for this?

| makeresults 
| eval _raw="{\"topic\": \"amx\", \"total_lag\": 2670, \"partitions\": [{\"lag\": 117, \"partition_number\": 0}, {\"lag\": 122, \"partition_number\": 1}, {\"lag\": 130, \"partition_number\": 2}, {\"lag\": 130, \"partition_number\": 3}, {\"lag\": 148, \"partition_number\": 4}, {\"lag\": 144, \"partition_number\": 5}, {\"lag\": 158, \"partition_number\": 6}, {\"lag\": 130, \"partition_number\": 7}, {\"lag\": 123, \"partition_number\": 8}, {\"lag\": 145, \"partition_number\": 9}, {\"lag\": 130, \"partition_number\": 10}, {\"lag\": 127, \"partition_number\": 11}, {\"lag\": 123, \"partition_number\": 12}, {\"lag\": 121, \"partition_number\": 13}, {\"lag\": 118, \"partition_number\": 14}, {\"lag\": 125, \"partition_number\": 15}, {\"lag\": 133, \"partition_number\": 16}, {\"lag\": 161, \"partition_number\": 17}, {\"lag\": 134, \"partition_number\": 18}, {\"lag\": 151, \"partition_number\": 19}]}" 
| kv
| rename partitions{}.lag as lag, partitions{}.partition_number as partition_number
| table topic total_lag  partition_number lag

Re: Kafka regex: Why is the command not working in Splunk search?

ddrillic — Fri, 22 Jun 2018 18:31:34 GMT

Is there a way to convert this feed to a json format? It's pretty close....

Re: Kafka regex: Why is the command not working in Splunk search?

pswalia06 — Sat, 23 Jun 2018 01:28:01 GMT

Here i have one more problem. If you see the below table topic name it is amx and amx1 but when i do line charts instead of showing two lines one for amx and one for amx1 it is showing only one line. How can we separate them?

Re: Kafka regex: Why is the command not working in Splunk search?

pswalia06 — Sun, 24 Jun 2018 05:00:27 GMT

it is json format only

Re: Kafka regex: Why is the command not working in Splunk search?

kamlesh_vaghela — Tue, 29 Sep 2020 20:08:52 GMT

HI @pswalia06,

Can you please try the following search?

YOUR_SEARCH
|kv
| rename partitions{}.lag as lag, partitions{}.partition_number as partition_number
| timechart latest(total_lag) as total_lag by topic