topic Re: How can I use rex to dedup and extract certain data from a field? in Splunk Search

How can I use rex to dedup and extract certain data from a field?

ameyapatil29 — Tue, 29 Sep 2020 21:03:00 GMT

Hello,

I am new to using rex and extract. I am trying to come up with a regex to extract certain data from a field only if that field exists.

Like in this query
[[0;37m2018-08-28 22:40:32.999[0m] [32mINFO [0m [pid:27567] [request_id:xxxxxxxx] [host:xxxxxxx] [remote_ip:xxxxx] [session_id:xxxxxxxx] [auth_id:] method=GET path=/questions/2044288 format=html controller=questions action=show status=410 duration=130.55 view=118.49 db=1.78 params={"id"=>"2044288"}

I am trying to extract the id number from params field and export it as article_number field. Can somebody help me how do I remove duplicates and use rex with extract?

So far I came up with index="cto-lc-app-prdidx" status=410 path="" params="" | dedup path,params | rex field=params ""

Thanks,
-Ameya

Re: How can I use rex to dedup and extract certain data from a field?

adonio — Wed, 29 Aug 2018 00:25:56 GMT

like that?
.... | rex field=params "\{"id"=>"(?<article_number>\d+)"
hope it helps

Re: How can I use rex to dedup and extract certain data from a field?

horsefez — Tue, 29 Sep 2020 21:03:17 GMT

Hi @ameyapatil29,

try to use the following query to do it. I've tested it and it works.

yoursearch | rex field=params "params={\"id\"=\>\"(?<article_number>[^\"]+)\"" | dedup article_number

First you are extracting the article_number from each logfile. After that you are removing duplicated article_numbers.

Re: How can I use rex to dedup and extract certain data from a field?

nawazns5038 — Fri, 31 Aug 2018 21:30:06 GMT

rex field=_raw "\"id\"\=\>\"(?<article_number>\d+)\""