Splunk Search

custom source column

runiyal
Path Finder

Hello,

I am trying to create a table output of events in logilfe. Here is the query -

 

index=myindex <my search>
| rex ".*source=(?<source>[^,]+).*col1=(?<Col1>[^,}]+).col2=(?<Col2>[^,}]+).col3=(?<Col3>[^,}]+)"
| kv
| table source Col1 Col2 Col3
| sort – source Col1 Col2

 

In my source column,  values can be like - Email, Scan, or Fax

But when I get the result then I get the value of source as the "source file" - "D:\App\tomcat\logs\applog.log" instead of values like "Email, Scan, or Fax"

How can I get the values of "source" fro the logfile event.

Thanks!

Labels (2)
1 Solution

richgalloway
SplunkTrust
SplunkTrust

That's better, but the fields in the events are not in the same order as in the regex.  I think you're better off using multiple rex commands.

| rex "source=(?<Source>[^,]+)."
| rex "col1=(?<Col1>[^,]+),"
| rex "col2=(?<Col2>[^,]+),"
| rex "col3=(?<Col3>[^}]+)"

If you really want a single rex then make sure the fields are extracted in the right order.

| rex "col1=(?<Col1>[^,]+), col2=(?<Col2>[^,]+), source=(?<Source>[^,]+), col3=(?<Col3>[^}]+)"

 

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

That's better, but the fields in the events are not in the same order as in the regex.  I think you're better off using multiple rex commands.

| rex "source=(?<Source>[^,]+)."
| rex "col1=(?<Col1>[^,]+),"
| rex "col2=(?<Col2>[^,]+),"
| rex "col3=(?<Col3>[^}]+)"

If you really want a single rex then make sure the fields are extracted in the right order.

| rex "col1=(?<Col1>[^,]+), col2=(?<Col2>[^,]+), source=(?<Source>[^,]+), col3=(?<Col3>[^}]+)"

 

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Splunk has a built-in field called "source" so you should avoid trying to create fields with that name.  Try this query:

index=myindex <my search>
| rex ".*source=(?<Source>[^,]+).*col1=(?<Col1>[^,}]+).col2=(?<Col2>[^,}]+).col3=(?<Col3>[^,}]+)"
| kv
| table Source Col1 Col2 Col3
| sort – Source Col1 Col2
---
If this reply helps you, an upvote would be appreciated.

runiyal
Path Finder

Hello Rich, Using "Source" is giving me null value. Field value is empty.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Double-check your query to make sure you've used the same field name throughout.

Make sure your regular expression is correct (use regex101.com to test it).

---
If this reply helps you, an upvote would be appreciated.

runiyal
Path Finder

Thanks for your inputs. Checked the code again but sees fine unless I am missing anything -

| rex " source=(?<Source>[^,]+).center=(?<center>[^,]+).accountNum=(?<accountNum>[^,]+).policyNum=(?<policyNum>[^,]+).clmNumber=(?<clmNumber>[^,]+).name=(?<name>[^,]+)"

 if I say <source> then it provide the source file name but not the value in the logfile for "source".

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Please provide a sample event or two so we can verify the regex.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

runiyal
Path Finder

Hello Rich, here are some of the events -

2020-11-25 18:23:47,432 INFO  [com.mycomp.docrepo.upload.FileUploadWebScript] [http-apr-8080-exec-3] The Upload Service /docrepo/service/comp/upload succeeded in 0.123000 seconds, Request: {col1=ab, col2=11252020_B1_DOC0_Fax_a4.HEIC, source=Fax, col3=application/octet-pdf}
2020-11-25 18:23:47,358 INFO  [com.mycomp.docrepo.upload.FileUploadWebScript] [http-apr-8080-exec-2] The Upload Service /docrepo/service/comp/upload succeeded in 0.255000 seconds, Request: {col1=cc, col2=11252020_B2_DOC0_Scan_a5.HEIC, source=Scan, col3=application/octet-pdf}
2020-11-25 18:23:43,066 INFO  [com.mycomp.docrepo.upload.FileUploadWebScript] [http-apr-8080-exec-4] The Upload Service /docrepo/service/comp/upload succeeded in 0.543000 seconds, Request: {col1=xy, col2=11252020_B3_DOC0_Email_a3.HEIC, source=Email, col3=application/octet-stream}
2020-11-25 18:23:42,789 INFO  [com.mycompe.docrepo.upload.FileUploadWebScript] [http-apr-8080-exec-2] The Upload Service /docrepo/service/comp/upload succeeded in 0.262000 seconds, Request: {col1=pc, col2=11252020_B3_DOC0_Email_a2.HEIC, source=Email, col3=application/octet-stream}
2020-11-25 18:23:42,344 INFO  [com.mycomp.docrepo.upload.FileUploadWebScript] [http-apr-8080-exec-8] The Upload Service /docrepo/service/comp/upload succeeded in 0.254000 seconds, Request: {col1=ab, col2=11252020_B3_DOC0_Email_a1.HEIC, source=Email, col3=application/octet-stream}
2020-11-25 16:04:19,342 INFO  [com.mycomp.docrepo.upload.FileUploadWebScript] [http-apr-8080-exec-7] The Upload Service /docrepo/service/comp/upload succeeded in 0.569000 seconds, Request: {col1=xy, col2=11252020_B4_DOC0_Email_a2.MOV, source=Email, col3=application/octet-stream}
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Thanks for the sample events.  As it turns out, none of them come close to matching the regex in the rex command.

The rex is looking for the "source" keyword followed, in order, by "enter", "accountNum", "policyNum", "clmNumber", and "name". 

While all of the sample events contain "source", none contain "center" or "name".  I stopped looked for the other keywords at that point.

If all all keywords are expected in all events or if they may be in a different order then separate rex commands should be used for each keyword.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

runiyal
Path Finder

Your suggestion of using multiple rex commands like below worked.

| rex "source=(?<Source>[^,]+)." | rex "col1=(?<Col1>[^,]+)," | rex "col2=(?<Col2>[^,]+)," | rex "col3=(?<Col3>[^}]+)"

Thanks!

0 Karma

runiyal
Path Finder

Sorry Rich, here is the actual regex for the events mentioned above that I am using -

rex " source=(?<Source>[^,]+).col1=(?<Col1>[^,]+).col2=(?<Col2>[^,]+).col3=(?<Col3>[^,]+)"

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!