How can I use rex to dedup and extract certain data from a field?

[ameyapatil29]

Hello,

I am new to using rex and extract. I am trying to come up with a regex to extract certain data from a field only if that field exists.

Like in this query
[2018-08-28 22:40:32.999] INFO  [pid:27567] [request_id:xxxxxxxx] [host:xxxxxxx] [remote_ip:xxxxx] [session_id:xxxxxxxx] [auth_id:] method=GET path=/questions/2044288 format=html controller=questions action=show status=410 duration=130.55 view=118.49 db=1.78 params={"id"=>"2044288"}

I am trying to extract the id number from params field and export it as article_number field. Can somebody help me how do I remove duplicates and use rex with extract?

So far I came up with index="cto-lc-app-prdidx" status=410 path="" params="" | dedup path,params | rex field=params ""

Thanks,
-Ameya

[nawazns5038]

rex field=_raw "\"id\"\=\>\"(?<article_number>\d+)\""

[horsefez · Accepted solution]

Hi @ameyapatil29,

try to use the following query to do it. I've tested it and it works.

yoursearch | rex field=params "params={\"id\"=\>\"(?<article_number>[^\"]+)\"" | dedup article_number

First you are extracting the article_number from each logfile. After that you are removing duplicated article_numbers.

[adonio]

like that?
.... | rex field=params "\{"id"=>"(?<article_number>\d+)"
hope it helps