topic Re: custom source column in Splunk Search

custom source column

runiyal — Wed, 25 Nov 2020 20:18:01 GMT

Hello,

I am trying to create a table output of events in logilfe. Here is the query -

index=myindex <my search> | rex ".*source=(?<source>[^,]+).*col1=(?<Col1>[^,}]+).col2=(?<Col2>[^,}]+).col3=(?<Col3>[^,}]+)" | kv | table source Col1 Col2 Col3 | sort – source Col1 Col2

In my source column, values can be like - Email, Scan, or Fax

But when I get the result then I get the value of source as the "source file" - "D:\App\tomcat\logs\applog.log" instead of values like "Email, Scan, or Fax"

How can I get the values of "source" fro the logfile event.

Thanks!

Re: custom source column

richgalloway — Wed, 25 Nov 2020 20:31:57 GMT

Splunk has a built-in field called "source" so you should avoid trying to create fields with that name. Try this query:

index=myindex <my search> | rex ".*source=(?<Source>[^,]+).*col1=(?<Col1>[^,}]+).col2=(?<Col2>[^,}]+).col3=(?<Col3>[^,}]+)" | kv | table Source Col1 Col2 Col3 | sort – Source Col1 Col2

Re: custom source column

runiyal — Wed, 25 Nov 2020 21:17:15 GMT

Hello Rich, Using "Source" is giving me null value. Field value is empty.

Re: custom source column

richgalloway — Wed, 25 Nov 2020 22:27:49 GMT

Double-check your query to make sure you've used the same field name throughout.

Make sure your regular expression is correct (use regex101.com to test it).

Re: custom source column

runiyal — Thu, 26 Nov 2020 00:26:48 GMT

Thanks for your inputs. Checked the code again but sees fine unless I am missing anything -

| rex " source=(?<Source>[^,]+).center=(?<center>[^,]+).accountNum=(?<accountNum>[^,]+).policyNum=(?<policyNum>[^,]+).clmNumber=(?<clmNumber>[^,]+).name=(?<name>[^,]+)"

if I say <source> then it provide the source file name but not the value in the logfile for "source".

Re: custom source column

richgalloway — Thu, 26 Nov 2020 14:38:16 GMT

Please provide a sample event or two so we can verify the regex.

Re: custom source column

runiyal — Sat, 28 Nov 2020 02:27:57 GMT

Hello Rich, here are some of the events -

2020-11-25 18:23:47,432 INFO [com.mycomp.docrepo.upload.FileUploadWebScript] [http-apr-8080-exec-3] The Upload Service /docrepo/service/comp/upload succeeded in 0.123000 seconds, Request: {col1=ab, col2=11252020_B1_DOC0_Fax_a4.HEIC, source=Fax, col3=application/octet-pdf} 2020-11-25 18:23:47,358 INFO [com.mycomp.docrepo.upload.FileUploadWebScript] [http-apr-8080-exec-2] The Upload Service /docrepo/service/comp/upload succeeded in 0.255000 seconds, Request: {col1=cc, col2=11252020_B2_DOC0_Scan_a5.HEIC, source=Scan, col3=application/octet-pdf} 2020-11-25 18:23:43,066 INFO [com.mycomp.docrepo.upload.FileUploadWebScript] [http-apr-8080-exec-4] The Upload Service /docrepo/service/comp/upload succeeded in 0.543000 seconds, Request: {col1=xy, col2=11252020_B3_DOC0_Email_a3.HEIC, source=Email, col3=application/octet-stream} 2020-11-25 18:23:42,789 INFO [com.mycompe.docrepo.upload.FileUploadWebScript] [http-apr-8080-exec-2] The Upload Service /docrepo/service/comp/upload succeeded in 0.262000 seconds, Request: {col1=pc, col2=11252020_B3_DOC0_Email_a2.HEIC, source=Email, col3=application/octet-stream} 2020-11-25 18:23:42,344 INFO [com.mycomp.docrepo.upload.FileUploadWebScript] [http-apr-8080-exec-8] The Upload Service /docrepo/service/comp/upload succeeded in 0.254000 seconds, Request: {col1=ab, col2=11252020_B3_DOC0_Email_a1.HEIC, source=Email, col3=application/octet-stream} 2020-11-25 16:04:19,342 INFO [com.mycomp.docrepo.upload.FileUploadWebScript] [http-apr-8080-exec-7] The Upload Service /docrepo/service/comp/upload succeeded in 0.569000 seconds, Request: {col1=xy, col2=11252020_B4_DOC0_Email_a2.MOV, source=Email, col3=application/octet-stream}

Re: custom source column

richgalloway — Sat, 28 Nov 2020 21:34:07 GMT

Thanks for the sample events. As it turns out, none of them come close to matching the regex in the rex command.

The rex is looking for the "source" keyword followed, in order, by "enter", "accountNum", "policyNum", "clmNumber", and "name".

While all of the sample events contain "source", none contain "center" or "name". I stopped looked for the other keywords at that point.

If all all keywords are expected in all events or if they may be in a different order then separate rex commands should be used for each keyword.

Re: custom source column

runiyal — Sat, 28 Nov 2020 22:14:37 GMT

Sorry Rich, here is the actual regex for the events mentioned above that I am using -

rex " source=(?<Source>[^,]+).col1=(?<Col1>[^,]+).col2=(?<Col2>[^,]+).col3=(?<Col3>[^,]+)"

Re: custom source column

richgalloway — Sun, 29 Nov 2020 01:51:25 GMT

That's better, but the fields in the events are not in the same order as in the regex. I think you're better off using multiple rex commands.

| rex "source=(?<Source>[^,]+)." | rex "col1=(?<Col1>[^,]+)," | rex "col2=(?<Col2>[^,]+)," | rex "col3=(?<Col3>[^}]+)"

If you really want a single rex then make sure the fields are extracted in the right order.

| rex "col1=(?<Col1>[^,]+), col2=(?<Col2>[^,]+), source=(?<Source>[^,]+), col3=(?<Col3>[^}]+)"

Re: custom source column

runiyal — Mon, 30 Nov 2020 19:40:25 GMT

Your suggestion of using multiple rex commands like below worked.

| rex "source=(?<Source>[^,]+)." | rex "col1=(?<Col1>[^,]+)," | rex "col2=(?<Col2>[^,]+)," | rex "col3=(?<Col3>[^}]+)"

Thanks!