Archive

Erex example

New Member

Hi, I have the below log and values for "days" field are 4, 10 , 15, 30. Could you please extract the "days" field using the "erex" command.

Log :

2017-11-21 04:55:34,060 tn="[14347886-7337]" ll=INFO cn="s.c.m" - id="57ef4442-aa9f-444d-bd80-e4fd3018f82", action="execute", operation="put", collection="Messages", "days" : "4", "Code" : "491833", useSecondary="false", retries="0"

Tags (1)
0 Karma
1 Solution

SplunkTrust
SplunkTrust

Following is a run anywhere search example for erex command to extract the field you are interested in (based on the sample provided). Please try out and confirm.

|  makeresults
|  eval _raw="2017-11-21 04:55:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f82\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"4\", \"Code\" : \"491833\", useSecondary=\"false\", retries=\"0\""
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:56:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f83\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"13\", \"Code\" : \"491834\", useSecondary=\"false\", retries=\"0\""]
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:57:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f84\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"24\", \"Code\" : \"491835\", useSecondary=\"false\", retries=\"0\""]
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:58:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f85\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"14\", \"Code\" : \"491836\", useSecondary=\"false\", retries=\"0\""]
|  erex days examples="4,13"

Ideally you should use rex command and once you have tested the same save your regular expression as Field Extraction for reusability and maintenance. Following is the regular expression:

|  makeresults
|  eval _raw="2017-11-21 04:55:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f82\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"4\", \"Code\" : \"491833\", useSecondary=\"false\", retries=\"0\""
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:56:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f83\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"13\", \"Code\" : \"491834\", useSecondary=\"false\", retries=\"0\""]
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:57:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f84\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"24\", \"Code\" : \"491835\", useSecondary=\"false\", retries=\"0\""]
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:58:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f85\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"14\", \"Code\" : \"491836\", useSecondary=\"false\", retries=\"0\""]
|  rex "\"days\"\s+:\s+\"(?<days>[^\"]+)\""

While the above examples use makeresults and append to mock some sample events as per question. You can try out the final pipe with erex or rex in your base search returning data as per your question:

Using rex command

<YourBaseSearch>
|  rex "\"days\"\s+:\s+\"(?<days>[^\"]+)\""

Using erex command

<YourBaseSearch>
|  erex days examples="4,13"

PS: erex might not be robust for field extraction in production data or else you will have to use a lot of sample examples and counterexamples.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

View solution in original post

Communicator

@amarish_vlabs, Here is your sample Event :

2017-11-21 04:55:34,060 tn="[14347886-7337]" ll=INFO cn="s.c.m" - id="57ef4442-aa9f-444d-bd80-e4fd3018f82", action="execute", operation="put", collection="Messages", "days" : "4", "Code" : "491833", useSecondary="false", retries="0"

using erex:

index=* source="amarish_vlabs.txt"  | erex newfield_days examples=4

Job says : Successfully learned regex. Consider using: | rex "(?i)\-e(?P<newfield_days>\d+)"

Now we can see new Search-time field "newfield_days" has been extracted on-the-fly (not persistent field) and moving forward better we use the rex command as that's efficient & faster.
Using rex :

index=* source="amarish_vlabs.txt"  | rex "(?i)\-e(?P<newfield_days>\d+)"

This answer is to start rolling the ball based on one given event. To generalize this on larger set of data and generate (possibly) precise regular expression using erex command, use the optional arguments like counterexamples, fromfield & maxtrainers.

here is syntax of erex :

erex [<field>] examples=<string> [counterexamples=<string>] [fromfield=<field>] [maxtrainers=<int>]

You may see more examples here.

0 Karma

Communicator

Hi @amarish_vlabs, does this answers your question or you have some query? Please feel free to ask.
If no query, please accept the answer so as to close this open question. 🙂 Thank you - Saurabh

0 Karma

SplunkTrust
SplunkTrust

Following is a run anywhere search example for erex command to extract the field you are interested in (based on the sample provided). Please try out and confirm.

|  makeresults
|  eval _raw="2017-11-21 04:55:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f82\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"4\", \"Code\" : \"491833\", useSecondary=\"false\", retries=\"0\""
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:56:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f83\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"13\", \"Code\" : \"491834\", useSecondary=\"false\", retries=\"0\""]
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:57:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f84\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"24\", \"Code\" : \"491835\", useSecondary=\"false\", retries=\"0\""]
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:58:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f85\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"14\", \"Code\" : \"491836\", useSecondary=\"false\", retries=\"0\""]
|  erex days examples="4,13"

Ideally you should use rex command and once you have tested the same save your regular expression as Field Extraction for reusability and maintenance. Following is the regular expression:

|  makeresults
|  eval _raw="2017-11-21 04:55:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f82\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"4\", \"Code\" : \"491833\", useSecondary=\"false\", retries=\"0\""
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:56:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f83\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"13\", \"Code\" : \"491834\", useSecondary=\"false\", retries=\"0\""]
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:57:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f84\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"24\", \"Code\" : \"491835\", useSecondary=\"false\", retries=\"0\""]
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:58:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f85\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"14\", \"Code\" : \"491836\", useSecondary=\"false\", retries=\"0\""]
|  rex "\"days\"\s+:\s+\"(?<days>[^\"]+)\""

While the above examples use makeresults and append to mock some sample events as per question. You can try out the final pipe with erex or rex in your base search returning data as per your question:

Using rex command

<YourBaseSearch>
|  rex "\"days\"\s+:\s+\"(?<days>[^\"]+)\""

Using erex command

<YourBaseSearch>
|  erex days examples="4,13"

PS: erex might not be robust for field extraction in production data or else you will have to use a lot of sample examples and counterexamples.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

View solution in original post

New Member

Thanks for your clear explanation. It is very useful.

0 Karma

SplunkTrust
SplunkTrust

@amarish_vlabs, glad you found it useful. Please accept/up vote the answer if it helped 🙂

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

SplunkTrust
SplunkTrust

@amarish_vlabs, any reason why you want to rely on erex and not rex or regular expression?

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

New Member

@niketnilay, I extracted the days field by using "rex" command. I just want to know how to extract with "erex". If this is possible with "erex", Please let me know. Otherwise just ignore. Thank you sir for your reply.

0 Karma

SplunkTrust
SplunkTrust

Hi

Can you please try this configuration?

props.conf

[my_sourcetype]
REPORT-mydays = mydays

transforms.conf

[mydays]
REGEX = \"days\"\s:\s\"(?<days>.+)\",\s\"
FORMAT = $1

Thanks

0 Karma

New Member

Thank you so much for your reply.

0 Karma