custom source column

[runiyal]

Hello,

I am trying to create a table output of events in logilfe. Here is the query -

index=myindex <my search>
| rex ".*source=(?<source>[^,]+).*col1=(?<Col1>[^,}]+).col2=(?<Col2>[^,}]+).col3=(?<Col3>[^,}]+)"
| kv
| table source Col1 Col2 Col3
| sort – source Col1 Col2

In my source column,  values can be like - Email, Scan, or Fax

But when I get the result then I get the value of source as the "source file" - "D:\App\tomcat\logs\applog.log" instead of values like "Email, Scan, or Fax"

How can I get the values of "source" fro the logfile event.

Thanks!

[richgalloway · Accepted solution]

That's better, but the fields in the events are not in the same order as in the regex.  I think you're better off using multiple rex commands.

| rex "source=(?<Source>[^,]+)."
| rex "col1=(?<Col1>[^,]+),"
| rex "col2=(?<Col2>[^,]+),"
| rex "col3=(?<Col3>[^}]+)"

If you really want a single rex then make sure the fields are extracted in the right order.

| rex "col1=(?<Col1>[^,]+), col2=(?<Col2>[^,]+), source=(?<Source>[^,]+), col3=(?<Col3>[^}]+)"

[richgalloway]

Splunk has a built-in field called "source" so you should avoid trying to create fields with that name.  Try this query:

index=myindex <my search>
| rex ".*source=(?<Source>[^,]+).*col1=(?<Col1>[^,}]+).col2=(?<Col2>[^,}]+).col3=(?<Col3>[^,}]+)"
| kv
| table Source Col1 Col2 Col3
| sort – Source Col1 Col2

[runiyal]

Hello Rich, Using "Source" is giving me null value. Field value is empty.

[richgalloway]

Double-check your query to make sure you've used the same field name throughout.

Make sure your regular expression is correct (use regex101.com to test it).

[runiyal]

Thanks for your inputs. Checked the code again but sees fine unless I am missing anything -

| rex " source=(?<Source>[^,]+).center=(?<center>[^,]+).accountNum=(?<accountNum>[^,]+).policyNum=(?<policyNum>[^,]+).clmNumber=(?<clmNumber>[^,]+).name=(?<name>[^,]+)"

 if I say <source> then it provide the source file name but not the value in the logfile for "source".

[richgalloway]

Please provide a sample event or two so we can verify the regex.

[runiyal]

Hello Rich, here are some of the events -

2020-11-25 18:23:47,432 INFO  [com.mycomp.docrepo.upload.FileUploadWebScript] [http-apr-8080-exec-3] The Upload Service /docrepo/service/comp/upload succeeded in 0.123000 seconds, Request: {col1=ab, col2=11252020_B1_DOC0_Fax_a4.HEIC, source=Fax, col3=application/octet-pdf}
2020-11-25 18:23:47,358 INFO  [com.mycomp.docrepo.upload.FileUploadWebScript] [http-apr-8080-exec-2] The Upload Service /docrepo/service/comp/upload succeeded in 0.255000 seconds, Request: {col1=cc, col2=11252020_B2_DOC0_Scan_a5.HEIC, source=Scan, col3=application/octet-pdf}
2020-11-25 18:23:43,066 INFO  [com.mycomp.docrepo.upload.FileUploadWebScript] [http-apr-8080-exec-4] The Upload Service /docrepo/service/comp/upload succeeded in 0.543000 seconds, Request: {col1=xy, col2=11252020_B3_DOC0_Email_a3.HEIC, source=Email, col3=application/octet-stream}
2020-11-25 18:23:42,789 INFO  [com.mycompe.docrepo.upload.FileUploadWebScript] [http-apr-8080-exec-2] The Upload Service /docrepo/service/comp/upload succeeded in 0.262000 seconds, Request: {col1=pc, col2=11252020_B3_DOC0_Email_a2.HEIC, source=Email, col3=application/octet-stream}
2020-11-25 18:23:42,344 INFO  [com.mycomp.docrepo.upload.FileUploadWebScript] [http-apr-8080-exec-8] The Upload Service /docrepo/service/comp/upload succeeded in 0.254000 seconds, Request: {col1=ab, col2=11252020_B3_DOC0_Email_a1.HEIC, source=Email, col3=application/octet-stream}
2020-11-25 16:04:19,342 INFO  [com.mycomp.docrepo.upload.FileUploadWebScript] [http-apr-8080-exec-7] The Upload Service /docrepo/service/comp/upload succeeded in 0.569000 seconds, Request: {col1=xy, col2=11252020_B4_DOC0_Email_a2.MOV, source=Email, col3=application/octet-stream}

[richgalloway]

Thanks for the sample events.  As it turns out, none of them come close to matching the regex in the rex command.

The rex is looking for the "source" keyword followed, in order, by "enter", "accountNum", "policyNum", "clmNumber", and "name". 

While all of the sample events contain "source", none contain "center" or "name".  I stopped looked for the other keywords at that point.

If all all keywords are expected in all events or if they may be in a different order then separate rex commands should be used for each keyword.

[runiyal]

Your suggestion of using multiple rex commands like below worked.

| rex "source=(?<Source>[^,]+)." | rex "col1=(?<Col1>[^,]+)," | rex "col2=(?<Col2>[^,]+)," | rex "col3=(?<Col3>[^}]+)"

Thanks!

[runiyal]

Sorry Rich, here is the actual regex for the events mentioned above that I am using -

rex " source=(?<Source>[^,]+).col1=(?<Col1>[^,]+).col2=(?<Col2>[^,]+).col3=(?<Col3>[^,]+)"