Splunk Search

How to create a search with conditional field extraction logic? If sourcetype=x, use rex extraction 1, but if source=y, use rex extraction 2

Motivator

Hi,

I have the following search:

host="*beta*" index=wls OR index=main sourcetype=wls_managedserver OR source="/etc/httpd/logs/portal-access_log*" |rex field=message_text "UCE-(?< UCE_Code >[^\s\:;]+)"|rex "UCE-(?< UCE_Code1 >[^\"]+)"|table UCE_Code UCE_Code1

The search works if I break it apart, meaning using:

Index=wls and sourcetype=wls_managedserver with rex UCE-(?< UCE_Code >[^\s\:;]+)" as query1

index=main source="/etc/httpd/logs/portal-accesslog*" with rex "UCE-(?< UCECode1 >[^\"]+)" as query2
but of course I'd like to combine the two searches. How can I have the rex definitions associate to a sourcetype/index/source?

0 Karma
1 Solution

Legend

Try this run-anywhere sample. (use everything after the | table x segment)

| gentimes start=-1 | eval x="< Warning > < ucontrol > < betamax-cpe2 > < managedServer2 > < pool-5-thread-4 > << anonymous >> < > < > < 1471557920294 > < BEA-000000 > < fn.xmpp.v2.IQSMAPHandler - UCE-24100 - Server exception while processing SMAP message '/event/lighting' from '1000275@xmpp/9c972687947f'|

 - - [18/Aug/2016:17:07:22 -0500] \"GET /rest/icontrol/sites/1001226/network/instances/1004a1514a4f98.0/functions/getUserCodes HTTP/1.1\" 404 43 0 10418 \"-\" \"Mozilla/5.0 (Linux; Android 6.0.1; SAMSUNG-SM-G930A Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/52.0.2743.98 Mobile Safari/537.36\" \"UCE-16001\"|

 - - [18/Aug/2016:17:13:28 -0500] \"GET /rest/icontrol/users/1000251/preferences/scenes HTTP/1.1\" 404 46 0 22809 \"-\" \"Apache-HttpClient/4.3.6 (java 1.5)\" \"UCE-16000\""| makemv x delim="|" | mvexpand x | table x | rex field=x "UCE-(?<code>(\d+\")|(\d+\s.*))" | table x code | rex mode=sed field=code "s/(\d+)\"/\1/g"

View solution in original post

Legend

Try this run-anywhere sample. (use everything after the | table x segment)

| gentimes start=-1 | eval x="< Warning > < ucontrol > < betamax-cpe2 > < managedServer2 > < pool-5-thread-4 > << anonymous >> < > < > < 1471557920294 > < BEA-000000 > < fn.xmpp.v2.IQSMAPHandler - UCE-24100 - Server exception while processing SMAP message '/event/lighting' from '1000275@xmpp/9c972687947f'|

 - - [18/Aug/2016:17:07:22 -0500] \"GET /rest/icontrol/sites/1001226/network/instances/1004a1514a4f98.0/functions/getUserCodes HTTP/1.1\" 404 43 0 10418 \"-\" \"Mozilla/5.0 (Linux; Android 6.0.1; SAMSUNG-SM-G930A Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/52.0.2743.98 Mobile Safari/537.36\" \"UCE-16001\"|

 - - [18/Aug/2016:17:13:28 -0500] \"GET /rest/icontrol/users/1000251/preferences/scenes HTTP/1.1\" 404 46 0 22809 \"-\" \"Apache-HttpClient/4.3.6 (java 1.5)\" \"UCE-16000\""| makemv x delim="|" | mvexpand x | table x | rex field=x "UCE-(?<code>(\d+\")|(\d+\s.*))" | table x code | rex mode=sed field=code "s/(\d+)\"/\1/g"

View solution in original post

Motivator

Hi Sundareshr,

I think you are very close. The table does have the UCE code (numeric digits) but it also has the error description which is pretty technical and won't be understood by the users. I've made a lookup table that replaces the description with a more user friendly one. How can the rex be modified so the UCE code is only the digits?

0 Karma

Motivator

Thanks (again) Sundareshr!!

0 Karma

Motivator

Think I figured it out UCE-(?< code >(\d+))

0 Karma

Legend

If all you need is the numbers, you just need this. You don't need the sed

... | rex field=x "UCE-(?<code>\d+) | ...
0 Karma

Legend

How about something like this.

host="*beta*" index=wls OR index=main sourcetype=wls_managedserver OR source="/etc/httpd/logs/portal-access_log*" | rex field=message_text "UCE-(?< UCE_Code >[^\s\:;]+)"| rex "UCE-(?< UCE_Code1 >[^\"]+)" | eval UCE_Code=if(source="/etc/httpd/logs/portal-access_log*", UCE_CODE1, UCE_CODE) | table UCE_Code
0 Karma

Motivator

Hi sundareshr, thats close! One problem though, the rex extraction for UCE_code1 is pulling the wrong data because it is looking at a different log than intended.

0 Karma

Motivator

a bit more detail.....

rex field=messagetext "UCE-(?< UCECode >[^\s:;]+)" gets the right UCE Code if it is looking at index=wls sourcetype=wls_managedserver events

similarly rex "UCE-(?< UCE_Code1 >[^\"]+)" gets the right UCE Code if it is looking at index=main source="/etc/httpd/logs/portal-accesslog" events

but when the sourcetype/index/sources are cobbled together the rex gets confused because the formats don't match if you switch them.

0 Karma

Legend

I understand. I am wondering if there is a different rex that could be applied to make this work without two searches. Can you share couple of samples, you can obfuscate any sensitive info.

0 Karma

Motivator

rex field=messagetext "UCE-(?< UCECode >[^\s:;]+)" index=wls sourcetype=wls_managedserver events

< Aug 18, 2016 5:05:20 PM CDT > < Warning > < ucontrol > < betamax-cpe2 > < managedServer2 > < pool-5-thread-4 > << anonymous >> < > < > < 1471557920294 > < BEA-000000 > < fn.xmpp.v2.IQSMAPHandler - UCE-24100 - Server exception while processing SMAP message '/event/lighting' from '1000275@xmpp/9c972687947f'

rex "UCE-(?< UCE_Code1 >[^\"]+)" index=main source="/etc/httpd/logs/portal-accesslog" events

"beta.icontrol.com" 99.98.192.121 "99.98.192.121" - - [18/Aug/2016:17:07:22 -0500] "GET /rest/icontrol/sites/1001226/network/instances/1004a1514a4f98.0/functions/getUserCodes HTTP/1.1" 404 43 0 10418 "-" "Mozilla/5.0 (Linux; Android 6.0.1; SAMSUNG-SM-G930A Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/52.0.2743.98 Mobile Safari/537.36" "UCE-16001"

"beta.icontrol.com" 54.174.106.18 "54.174.106.18" - - [18/Aug/2016:17:13:28 -0500] "GET /rest/icontrol/users/1000251/preferences/scenes HTTP/1.1" 404 46 0 22809 "-" "Apache-HttpClient/4.3.6 (java 1.5)" "UCE-16000"

0 Karma

Legend

Try this regex `"UCE-(?(\d+\")|(\d+\s.*))"

0 Karma

Legend

can you share a sample for each type (source="/etc/httpd/logs/portal-access_log*" AND the other)

0 Karma