Splunk Dev

Mutliple field extractions in props.conf of one sourcetype

andresito123
Communicator

Hello all,

I am ingesting NetScaler Logs on Splunk with sourcetype "citrix:netscaler:syslog". For the set I receive, I am interested in WAF events that include either one of the following words:

  • APPFW_CONTENT_TYPE
  • APPFW_XSS
  • APPFW_BUFFEROVERFLOW_URL
  • APPFW_SQL

The issue is that I need four different regexes for all of these four events. I have set in my props.conf the following declarations, but unfortunately not all fields are extracted:

# APPFW_CONTENT_TYPE
EXTRACT-function,src,dest_host,dest,signature_desc,action = .*(?<function>SSLVPN|SSLLOG|APPFW|EVENT)\sAPPFW_CONTENT_TYPE[0-9 \:]{15}(?<src>((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)).*\s(?<dest_host>www.unisystems.gr)\s(?<dest>https?:\/\/(?:www\.|(?!www))[a-zA-Z0-9][a-zA-Z0-9-]+[a-zA-Z0-9]\.[^\s]{2,}|www\.[a-zA-Z0-9][a-zA-Z0-9-]+[a-zA-Z0-9]\.[^\s]{2,}|https?:\/\/(?:www\.|(?!www))[a-zA-Z0-9]\.[^\s]{2,}|www\.[a-zA-Z0-9]\.[^\s]{2,})(?<signature_desc>.*)\<(?<action>.*)\>
# APPFW_XSS
EXTRACT-src,dest_host,dest,signature_desc,action = .*(?<function>SSLVPN|SSLLOG|APPFW|EVENT)\sAPPFW_XSS.*\s[0-9 \:]{14}(?<src>((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)).*\s(?<dest_host>www.unisystems.gr)\s(?<dest>https?:\/\/(?:www\.|(?!www))[a-zA-Z0-9][a-zA-Z0-9-]+[a-zA-Z0-9]\.[^\s]{2,}|www\.[a-zA-Z0-9][a-zA-Z0-9-]+[a-zA-Z0-9]\.[^\s]{2,}|https?:\/\/(?:www\.|(?!www))[a-zA-Z0-9]\.[^\s]{2,}|www\.[a-zA-Z0-9]\.[^\s]{2,})(?<signature_desc>.*)\<(?<action>.*)\>
# APPFW_BUFFEROVERFLOW_URL
EXTRACT-src,dest_host,signature_desc,action = .*(?<function>SSLVPN|SSLLOG|APPFW|EVENT)\sAPPFW_BUFFEROVERFLOW_URL.*\s[0-9 \:]{14}(?<src>((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)).*\s(?<dest_host>www.unisystems.gr)\s(?<signature_desc>.*)\:\s(?<dest>https?:\/\/(?:www\.|(?!www))[a-zA-Z0-9][a-zA-Z0-9-]+[a-zA-Z0-9]\.[^\s]{2,}|www\.[a-zA-Z0-9][a-zA-Z0-9-]+[a-zA-Z0-9]\.[^\s]{2,}|https?:\/\/(?:www\.|(?!www))[a-zA-Z0-9]\.[^\s]{2,}|www\.[a-zA-Z0-9]\.[^\s]{2,})\s\<(?<action>.*)\>
# APPFW_SQL
EXTRACT-src,dest_host,dest,signature_desc,action = .*(?<function>SSLVPN|SSLLOG|APPFW|EVENT)\sAPPFW_SQL.*\s[0-9 \:]{14}(?<src>((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)).*\s(?<dest_host>www.unisystems.gr)\s(?<dest>https?:\/\/(?:www\.|(?!www))[a-zA-Z0-9][a-zA-Z0-9-]+[a-zA-Z0-9]\.[^\s]{2,}|www\.[a-zA-Z0-9][a-zA-Z0-9-]+[a-zA-Z0-9]\.[^\s]{2,}|https?:\/\/(?:www\.|(?!www))[a-zA-Z0-9]\.[^\s]{2,}|www\.[a-zA-Z0-9]\.[^\s]{2,})\s(?<signature_desc>.*)\s\<(?<action>.*)\>

Do you have any clue why this is not working?

Thanks,
Andreas

Edit: The interesting part is when I go to "extract fields" option below a log line, I get all the fields extracted but I can't search them.
alt text

Tags (1)
0 Karma

FrankVl
Ultra Champion

I would start by giving your EXTRACT-... items proper names, instead of putting the field names, comma separated into the class name of the extract. Specifically, because now your second and 4th extract have the same classname, which means one of them will get ignored if I'm not mistaken.

So just use names like EXTRACT-APPFW_CONTENT_TYPE for instance.

PS: any reason you're writing all this from scratch, rather than using the Netscaler Add-on from splunkbase? https://splunkbase.splunk.com/app/2770/

0 Karma

andresito123
Communicator

I used the pattern you mention in EXTRACT but it didn't work. I read that it might be better to use the fields in the "EXTRACT-".

I didn't use the Splunk addons because the data I have are different from the data SPLUNK addons process...

0 Karma

FrankVl
Ultra Champion

Looks like the citrix:netscaler:syslog sourcetype from the TA I linked doesn't match your data. However, the ns_log sourcetype, from the netscaler TA bundled with the App for Citrix NetScaler with AppFlow does seem to work quite well with your sample data. So you might want to use that as the starting point.

https://splunkbase.splunk.com/app/370/

0 Karma

xpac
SplunkTrust
SplunkTrust

Could you provide some example events?

0 Karma

andresito123
Communicator
 05/14/2018:14:26:07  netscaler 0-PPE-0 : default APPFW APPFW_CONTENT_TYPE 17026121 0 :  192.99.7.203 37165443-PPE0 Mq0djxWpCR9ogxW7d5ZwQnjQ2Mg0000 www.example.com http://www.example.com/ Unknown content-type header value="text/html" <not blocked>
 05/14/2018:03:40:20  netscaler 0-PPE-0 : default APPFW APPFW_XSS 16932959 0 :  5.101.140.66 36996274-PPE0 ph0QaNs5L/DXhZyM3aUBnI3WkZU0000 www.example.com http://example.com/xmlrpc.php Cross-site script check failed for field &lt
 05/14/2018:03:40:20  netscaler 0-PPE-0 : default APPFW APPFW_XSS 16932955 0 :  5.101.140.66 36996273-PPE0 EoljFu6x/+qIovZRliKof87LjMA0000 www.example.com http://example.com/blog/xmlrpc.php Cross-site script check failed for field &lt
 05/14/2018:03:40:20  netscaler 0-PPE-0 : default APPFW APPFW_XSS 16932951 0 :  5.101.140.66 36996272-PPE0 HB0tfWSePbvs10p6a1pNUL5RoII0000 www.example.com http://example.com/blog/xmlrpc.php Cross-site script check failed for field &lt
 05/14/2018:03:40:20  netscaler 0-PPE-0 : default APPFW APPFW_XSS 16932947 0 :  5.101.140.66 36996271-PPE0 L9rwzkwsTiZFP3EyePpEtLsADIY0000 www.example.com http://example.com/xmlrpc.php Cross-site script check failed for field &lt
 05/13/2018:13:21:39  netscaler 0-PPE-0 : default APPFW APPFW_CONTENT_TYPE 16806275 0 :  208.100.26.230 36705424-PPE0 30iII3l3UinctCDeqFxjcMDBJrk0000 www.example.com http://193.92.173.66/sdk Unknown content-type header value="" <not blocked>
 05/13/2018:13:14:30  netscaler 0-PPE-0 : default APPFW APPFW_XSS 16805341 0 :  158.69.101.124 36700340-PPE0 0WUjjIz42lI/5Qg59uscoruCGW00000 www.example.com http://www.example.com/?q=user%2Fpassword&name%5B%23post_render%5D%5B%5D=passthru&name%5B%23type%5D=... Cross-site script check failed for field name[#markup]="Bad tag: ?php" <not blocked>
 05/13/2018:13:14:30  netscaler 0-PPE-0 : default APPFW APPFW_XSS 16805331 0 :  158.69.101.124 36700333-PPE0 FLG4AXecx5WjhLwVUeheHV3Dsg80000 www.example.com http://www.example.com/?q=user%2Fpassword&name%5B%23post_render%5D%5B%5D=passthru&name%5B%23type%5D=... Cross-site script check failed for field name[#markup]="Bad tag: ?php" <not blocked>
 05/13/2018:12:58:17  netscaler 0-PPE-0 : default APPFW APPFW_BUFFEROVERFLOW_URL 16803625 0 :  69.12.66.213 36687138-PPE0 NsGHMkHJdizkUdSykXw71ku9BUs0000 www.example.com URL length(1407) is greater than maximum allowed(1024): http://www.example.com/?q=user/password&name[%23post_render][]=passthru&name[%23type]=markup&name[%23markup]=echo+PD9waHAgaWYoaXNzZXQoJF9GSUxFU1sidXNlcmZpbGUiXVsibmFtZSJdKSl7ICR1cGxvYWRkaXIgPSBnZXRjd2QoKSAuICIvIjsgJHVwbG9hZGZpbGUgPSAkdXBsb2FkZGlyIC4gYmFzZW5hbWUoJF9GSUxFU1sidXNlcmZpbGUiXVsibmFtZSJdKTsgZWNobyAiPHA+IjsgaWYgKG1vdmVfdXBsb2FkZWRfZmlsZSgkX0ZJTEVTWyJ1c2VyZmlsZSJdWyJ0bX <blocked>
 05/13/2018:08:56:00  netscaler 0-PPE-0 : default APPFW APPFW_BUFFEROVERFLOW_URL 16752492 0 :  69.12.66.213 36546864-PPE0 V6GTdjzy5ES5jxR2obCXoROwWRs0000 www.example.com URL length(1407) is greater than maximum allowed(1024): https://www.example.com/?q=user/password&name[%23post_render][]=passthru&name[%23type]=markup&name[%23markup]=echo+PD9waHAgaWYoaXNzZXQoJF9GSUxFU1sidXNlcmZpbGUiXVsibmFtZSJdKSl7ICR1cGxvYWRkaXIgPSBnZXRjd2QoKSAuICIvIjsgJHVwbG9hZGZpbGUgPSAkdXBsb2FkZGlyIC4gYmFzZW5hbWUoJF9GSUxFU1sidXNlcmZpbGUiXVsibmFtZSJdKTsgZWNobyAiPHA+IjsgaWYgKG1vdmVfdXBsb2FkZWRfZmlsZSgkX0ZJTEVTWyJ1c2VyZmlsZSJdWyJ0bX <blocked>
 05/12/2018:23:01:17  netscaler 0-PPE-0 : default APPFW APPFW_BUFFEROVERFLOW_URL 16705334 0 :  77.111.175.75 36471517-PPE0 oIst98GdXzo+8lFQ0kCdRGUoEIc0000 www.example.com URL length(1124) is greater than maximum allowed(1024): http://example.com/user/password?name[%23post_render][0]=system&name%5B%23markup%5D=echo+PD9waHAgZWNobyA3NDU3NzM3KzczNjcyMzskcmFQb19yWmx1b0U9YmFzZTY0X2RlY29kZSgiWSIuY2hyKDEwOSkuIkYiLmNocigxMjIpLmNocig5MCkuIlQiLmNocig4OSkuY2hyKDQ4KS5jaHIoODgpLiIyIi4iUiIuImwiLiJZIi5jaHIoNTApLiI5Ii5jaHIoMTA3KS4iWiIuY2hyKDgxKS4iPSIuIj0iKTskeWRTSlB0bndyU3Y9YmFzZTY0X2RlY29kZShjaHIoODkpLiIyIi5jaHIoNTcp <blocked>
 05/12/2018:20:27:48  netscaler 0-PPE-0 : default APPFW APPFW_CONTENT_TYPE 16685255 0 :  46.105.57.86 36439997-PPE0 BLpRSN+vcL9JddTXNccS3+HIY+40000 www.example.com http://www.example.com/user/ Unknown content-type header value="text/html" <not blocked>
 05/12/2018:20:27:46  netscaler 0-PPE-0 : default APPFW APPFW_CONTENT_TYPE 16685246 0 :  46.105.57.86 36439991-PPE0 BLpRSN+vcL9JddTXNccS3+HIY+40000 www.example.com http://www.example.com/admin/ Unknown content-type header value="text/html" <not blocked>
 05/12/2018:20:27:44  netscaler 0-PPE-0 : default APPFW APPFW_CONTENT_TYPE 16685241 0 :  46.105.57.86 36439986-PPE0 BLpRSN+vcL9JddTXNccS3+HIY+40000 www.example.com http://www.example.com/admin/login.php Unknown content-type header value="text/html" <not blocked>
 05/12/2018:20:27:42  netscaler 0-PPE-0 : default APPFW APPFW_CONTENT_TYPE 16685234 0 :  46.105.57.86 36439983-PPE0 BLpRSN+vcL9JddTXNccS3+HIY+40000 www.example.com http://www.example.com/bitrix/admin/index.php?lang=en Unknown content-type header value="text/html" <not blocked>
 05/12/2018:20:27:40  netscaler 0-PPE-0 : default APPFW APPFW_CONTENT_TYPE 16685229 0 :  46.105.57.86 36439979-PPE0 BLpRSN+vcL9JddTXNccS3+HIY+40000 www.example.com http://www.example.com/admin.php Unknown content-type header value="text/html" <not blocked>
 05/12/2018:20:27:38  netscaler 0-PPE-0 : default APPFW APPFW_CONTENT_TYPE 16685224 0 :  46.105.57.86 36439975-PPE0 BLpRSN+vcL9JddTXNccS3+HIY+40000 www.example.com http://www.example.com/administrator/index.php Unknown content-type header value="text/html" <not blocked>
 05/12/2018:20:27:38  netscaler 0-PPE-0 : default APPFW APPFW_CONTENT_TYPE 16685222 0 :  46.105.57.86 36439974-PPE0 BLpRSN+vcL9JddTXNccS3+HIY+40000 www.example.com http://www.example.com/wp-login.php Unknown content-type header value="text/html" <not blocked>
 05/12/2018:17:01:30  netscaler 0-PPE-0 : default APPFW APPFW_BUFFEROVERFLOW_URL 16660727 0 :  69.12.66.213 36394656-PPE0 nHuDCzYtywEqXZZfl5udU1Zzbeg0000 www.example.com URL length(1407) is greater than maximum allowed(1024): https://www.example.com/?q=user/password&name[%23post_render][]=passthru&name[%23type]=markup&name[%23markup]=echo+PD9waHAgaWYoaXNzZXQoJF9GSUxFU1sidXNlcmZpbGUiXVsibmFtZSJdKSl7ICR1cGxvYWRkaXIgPSBnZXRjd2QoKSAuICIvIjsgJHVwbG9hZGZpbGUgPSAkdXBsb2FkZGlyIC4gYmFzZW5hbWUoJF9GSUxFU1sidXNlcmZpbGUiXVsibmFtZSJdKTsgZWNobyAiPHA+IjsgaWYgKG1vdmVfdXBsb2FkZWRfZmlsZSgkX0ZJTEVTWyJ1c2VyZmlsZSJdWyJ0bX <blocked>
 05/12/2018:14:34:56  netscaler 0-PPE-0 : default APPFW APPFW_CONTENT_TYPE 16646737 0 :  191.96.249.235 36371007-PPE0 sQ8NN6HOKm+y4wMRTjqibKNWTs00000 www.example.com http://www.example.com/ Unknown content-type header value="text/html" <not blocked>
 05/12/2018:14:34:56  netscaler 0-PPE-0 : default APPFW APPFW_CONTENT_TYPE 16646735 0 :  191.96.249.235 36371006-PPE0 mjXIaLYQRzKGui3ZJYJL6oFVt/M0000 www.example.com http://example.com/ Unknown content-type header value="text/html" <not blocked>
 05/12/2018:14:34:56  netscaler 0-PPE-0 : default APPFW APPFW_CONTENT_TYPE 16646733 0 :  191.96.249.235 36371005-PPE0 flcJrCPYxSqXV0E3X3EMPIdhbZo0000 www.example.com http://www.example.com/ Unknown content-type header value="text/html" <not blocked>
 05/12/2018:13:17:33  netscaler 0-PPE-0 : default APPFW APPFW_XSS 16639123 0 :  80.211.40.18 36358356-PPE0 HQlSdwdZiXdODfcRIDphbKGUB8g0000 www.example.com http://example.com/xmlrpc.php Cross-site script check failed for field &lt
 05/12/2018:13:17:33  netscaler 0-PPE-0 : default APPFW APPFW_XSS 16639119 0 :  80.211.40.18 36358355-PPE0 iP1wV/rmOUYLSIdG7TGvAGQgdN40000 www.example.com http://example.com/blog/xmlrpc.php Cross-site script check failed for field &lt
 05/12/2018:12:11:06  netscaler 0-PPE-0 : default APPFW APPFW_CONTENT_TYPE 16631721 0 :  89.248.168.171 36345725-PPE0 LLlnVL/4fgKkPOtkjFOeXXqGXKI0000 www.example.com http://193.92.173.66/GponForm/diag_Form?images/ Unknown content-type header value="gzip, deflate" <not blocked>
 05/11/2018:23:19:16  netscaler 0-PPE-0 : default APPFW APPFW_SQL 16557325 0 :  195.181.172.173 36151762-PPE0 tyqgqCvM2o1CF5QNAsy2VuxH5i80000 www.example.com http://example.com/user/login/ SQL Keyword check failed for field name[0
 05/11/2018:22:36:54  netscaler 0-PPE-0 : default APPFW APPFW_BUFFEROVERFLOW_URL 16552886 0 :  95.110.235.19 36144571-PPE0 GhgwMyNEPf8V17u8zb/zXEsCgMo0000 www.example.com URL length(1025) is greater than maximum allowed(1024): http://www.example.ro/sites/default/files/attachment/index.php/admin/Cms_Wysiwyg/directive/?forwarde... <blocked>
 05/11/2018:22:34:34  netscaler 0-PPE-0 : default APPFW APPFW_BUFFEROVERFLOW_URL 16552574 0 :  95.110.235.19 36144164-PPE0 Nvn7vh4JJ0Yu1gjLP2C1LeCenjE0000 www.example.com URL length(1140) is greater than maximum allowed(1024): http://www.example.ro/sites/default/files/attachment/FData2015ENG.pdf&amp
 05/11/2018:21:03:50  netscaler 0-PPE-0 : default APPFW APPFW_SQL 16540147 0 :  195.181.172.173 36125803-PPE0 zgF4WR6LAwVcRUYk+wdXOa4yL7A0000 www.example.com http://example.com/user/login/ SQL Keyword check failed for field name[0
 05/11/2018:13:57:36  netscaler 0-PPE-0 : default APPFW APPFW_CONTENT_TYPE 16477695 0 :  163.172.249.131 36026582-PPE0 Qv0fl7m+pdCb1aIVfsWspJi8qjE0000 www.example.com https://www.example.com/company/our-people'A=0 Unknown content-type header value="text/html" <not blocked>
0 Karma

xpac
SplunkTrust
SplunkTrust

I guess that your regex looks for unisystems.gr, but the events contain example.com is due to anonymization, right?

0 Karma

andresito123
Communicator

Correct. I have tested the regex with "rex field=_raw" command and it extracts the data, however the configuration in props.conf does not work out.

0 Karma

andresito123
Communicator

Does it matter in which props.conf those definitions are?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...