Hi Micahkemp, as we are unable to over write host" field for logs to use to value for the "dvc" field . Kindly guide me on this issue. ... View more

Hi Micahkemp, could you please guide me on this issue. thanks in advance. ... View more

Hi Micahkemp, After implementing this changes in splunk HF instance where the parsing is happening , the events stopped ingesting into splunk, when tried to search for the output , I got zero result. Props.conf: [opsec] TRANSFORMS-orig_as_host Transforms.conf: [orig_as_host] REGEX = orig=([^|]+)| DEST_KEY = MetaData:Host FORMAT = host::$1 index=firwall sourcetype="opsec*" Zero result. Similarly before applying the changes, I tried to execute the query with the regex which you had mentioned in your comment and it was throwing an error. index=firewall sourcetype="opsec*" | rex field=_raw "orig=([^|]+)\|" | table host dvc Error in 'rex' command: The regex 'orig=([^|]+)\|' does not extract anything. It should specify at least one named group. Format: (?<name>...). The search job has failed due to an error. You may be able view the job in the Job Inspector. Kindly guide me on this please.. ... View more

Hi Micahkemp, Based on the below query I could see there are four sourcetype which are fetching the information from the destination. index=firewall sourcetype="opsec*" "orig" | rex field=_raw "orig=(?<host>[^(\.|\|)]+)" Sourcetype : [opsec] [opsec:smartdefense] [opsec:vpn] [opsec:audit] Props.conf: [opsec] TRANSFORMS-orig_as_host = orig_as_host transforms.conf: [orig_as_host] REGEX = orig=([^|]+)\| DEST_KEY = MetaData:Host Props.conf [opsec:vpn] TRANSFORMS-orig_as_host = orig_as_host transforms.conf: [orig_as_host] REGEX = orig=([^|]+)\| DEST_KEY = MetaData:Host FORMAT = host::$1 Kindly guide me whether the above props.conf and transforms are correct. ... View more

Hi rich, thanks for your effort on this, hey I am not familiar with the IIS settings so can you please guide me on this or share me the doc link to validate the configuration details. Another point though - IIRC IIS logs HAVE A HEADER LINE and I believe the IIS add-on uses this to make the fields correct, then just aliases things to make them match in certain cases. It may be useful to install and configure the app, and use that as the sourcetype and see if this doesn't get resolved from that. I did not understand this comment, when checked the splunk add-on for iis, could not come across IIRC IIS logs, so can you help me on this too. Hey can I paste the props.conf and transforms.conf detail in the comment, so that you can guide me where to correct the stanza to fix the issue, will this be better idea. thanks in advance. ... View more

hi its not autoextracted, as i could see Report, Extract stanza along with Field alias has been used in props.conf. ... View more

Hi rich, Please find the Event data Event Details: Which is working fine. 2018-02-01 11:28:18 W3SVC1 test01 10.x.x.7 POST /XMII/Illuminator QueryTemplate=AIM%2FCommon%2FGetLineShiftXacute&Content-Type=text%2Fxml&X-ARR-CACHE-HIT=0&X-ARR-LOG-ID=4d2ca237-5c38-4436-bd34-d74c8c345cda 80 - 10.x.x.157 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/7.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+.NET4.0C;+.NET4.0E) ACTSESSION=aHR0cDovL3N0dWFydHNkcmxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx1hNSUkvQ00vQUlNL1BhY2thZ2luZy9Db21wbGV0ZVF1YWxpdHkuaXJwdA%3D%3D;+CTSESSION=AAAAAgABAGBVh14WQ8UkIIg54kAqRirk8rjD2c9%2Ffu%2BpeYSWtWpsd7kYF5NX6s6EE58Atx9yk6xRLY3je6i8c3Xav5LmGAqnupZhuremKw8XWJDDGuYLFv86vSVVkLulZA9V%2BeshH%2Bw%3D;+ARRAffinity=8a9bfbe1cddd7747c4cf1be435c17290c2c92477166719b6265b2b5d8634f683;+saplb_*=(J2EE7745720)7745751;+JSESSIONID=wt4-0D-5TBFLpqthMEyB4OlXoRIoYQHXMHYA_SAP5BJ1xzFg6IdsGx1xGGXXNk7r;+JSESSIONMARKID=IEy6wwbT5obwlmyqCHvHEur-beZ_QrsSK05NcwdgA;+MYSAPSSO2=AjExMDAgABJwb3J0YWw6U1RVTDQ3T1BSMDKIAAdkZWZhdWx0AQALU1RVTDQ3T1BSMDICAAMwMDADAANNMDcEAAwyMDE4MDIwMTA0MDAFAAQAAAAICgALU1RVTDQ3T1BSMDL%2FAQUwggEBBgkqhkiG9w0BBwKggfMwgfACAQExCzAJBgUrDgMCGgUAMAsGCSqGSIb3DQEHATGB0DCBzQIBATAiMB0xDDAKBgNVBAMTA00wNzENMAsGA1UECxMESjJFRQIBADAJBgUrDgMCGgUAoF0wGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTgwMjAxMDQwMDE1WjAjBgkqhkiG9w0BCQQxFgQUpXQoBd7SqP%2FgEyCwD!w5T3NAGKMwCQYHKoZIzjgEAwQvMC0CFDRfXhDnQmcZYDpDVcOrbHZZ5WZSAhUAxQXo2IZtxEGicAdDy3gmCMmm9G8%3D http://stard.aim.xxxx.com/XMII/CM/AIM/Packaging/CompleteQuality.irpt star.aim.xxxx.com 200 0 0 712 1716 109 Event Detail: Which is not working correctly 2018-01-31 13:45:07 10.x.x.x GET /areas/ehs/book_style/erp_learning_book/textbox/textbox.css - 80 - 10.x.x.245 Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/63.0.3239.132+Safari/537.36 http://erphelp.xxxx.com/areas/ehs/trainer/10_x_x_1406/slide.htm 304 0 0 218 Kindly guide me how/where to trouble shoot to fix this issue. ... View more

hey regex got applied to the field dvc and i have already pasted the output in my previous comment. ... View more

Hi Micahkemp, thanks for your effort on this, yes i need change the value of the host. currently host=test01 is the source from where the data is being ingested in to splunk and other host details are found in the event data. With the help of the forum, I had executed the below query to overwrite the "host" field for logs to use to value for the "dvc" field and also to remove ".xxxx.com " and write to the "host" field without xxxx.com. But it did not work as expected, it included the other host details test02,test03 etc under the host field along with the host=test01. index=firwall sourcetype="opsec:vpn" | rex field=dvc "(?<host>[^\.]+)" | table host dvc host dvc test01 test02 test02 test01 test03 test03 Event details: time=1517397957|loc=10718231|fileid=1517392517|action=decrypt|**orig=test02.xxxx.com**|i/f_dir=inbound|i/f_name=bond0.470|has_accounting=0|uuid=<5a71a7c5,00000005,30f08e0a,c0000000>|product=VPN-1 & FireWall-1|inzone=External|outzone=Internal|rule=250|rule_uid={DAE6-F7DD-4167-BCAC-1DE4B472}|rule_name=DNS|service_id=domain-udp|src=10.x.x.x|s_port=577|dst=dip02.xxxx.com|service=domain-udp|proto=udp|scheme:=IKE|methods:=ESP: AES-128 + SHA1 + PFS (group 2)|peer gateway=VPN_AWS_Gateway_Prod2|community=vpn-xxxxe799|fw_subproduct=VPN-1|vpn_feature_name=VPN|__policy_id_tag=product=VPN-1 & FireWall-1[db_tag={0181B41F-6A86-E04C-8E1E-38146FBFD921};**mgmt=test01**;date=1517158905;policy_name=Global-2]|origin_sic_name=CN=test02.xxxx.com,O=dron01.xxxx.com.evv25 Getting the other host details under selected fields, along with the host=test01 Note: test01 is the source from where the data is ingested into splunk and other hosts values are found in the events. Need to over ride the host=test01 when the data is getting indexed itself and in the host field we should get only the host values orig=test02.xxxx.com from the event data. Props.conf Details: [opsec:vpn] KV_MODE = none REPORT-0policy_id_tag_for_opsec = policy_id_tag_for_opsec,mgmt_for_opsec, REPORT-action_as_threat_emulation_action = action_as_threat_emulation_action REPORT-auto_kv_for_opsec = auto_kv_for_opsec FIELDALIAS-dvc_for_opsec = orig as dvc There are lot many EVAL and FIELDALIAS in the props for this sourcetype. Transforms.conf: [opsec_sourcetype_vpn] REGEX = fw_subproduct\=VPN-1 DEST_KEY = MetaData:Sourcetype FORMAT = sourcetype::opsec:vpn [action_as_threat_emulation_action] REGEX = action\=([^|]+) FORMAT = te_action::$1 [auto_kv_for_opsec] REGEX = ([^|=]*)\b:?=([^|]*) FORMAT = $1::$2 [policy_id_tag_for_opsec] REGEX = __policy_id_tag\=([^|]+) FORMAT = policy_id_tag::$1 Kindly guide me how to overwrite host value = test01 with dvc filed=test02. ... View more

Hi thanks for your effort on this, but it did not fetch the result as expected. After executing the above query I had got the below output. index=firwall | rex field=dvc "(?<host>\w+)" | table host dvc host dvc test01 test02 test02 test01 test03 test03 Event details: time=1517397957|loc=10718231|fileid=1517392517|action=decrypt|**orig=test02.xxxx.com**|i/f_dir=inbound|i/f_name=bond0.470|has_accounting=0|uuid=<5a71a7c5,00000005,30f08e0a,c0000000>|product=VPN-1 & FireWall-1|inzone=External|outzone=Internal|rule=250|rule_uid={DAE6-F7DD-4167-BCAC-1DE4B472}|rule_name=DNS|service_id=domain-udp|src=10.x.x.x|s_port=577|dst=dip02.xxxx.com|service=domain-udp|proto=udp|scheme:=IKE|methods:=ESP: AES-128 + SHA1 + PFS (group 2)|peer gateway=VPN_AWS_Gateway_Prod2|community=vpn-xxxxe799|fw_subproduct=VPN-1|vpn_feature_name=VPN|__policy_id_tag=product=VPN-1 & FireWall-1[db_tag={0181B41F-6A86-E04C-8E1E-38146FBFD921};**mgmt=test01**;date=1517158905;policy_name=Global-2]|origin_sic_name=CN=test02.xxxx.com,O=dron01.xxxx.com.evv25 And in the selected field host=test01,test02, test03 etc Note: test01 is the source from where the data is ingested into splunk and other hosts values are found in the events. Need to over ride the host=test01 when the data is getting indexed itself. So kindly guide me how to fix this issue. ... View more

Hi rich, could you please guide me on how to check the IIS logging configuration details, as I don't have much knowledge on IIS configuration. Yes we are using UF to collect the IIS log data and this is parsed in HF and then get indexed. There are 42 remote IIS node machine ingesting the iis log data in splunk and all are sharing the same inputs.conf detail, as the configuration is managed and push centrally via deployment server. Inputs.conf details: [monitor://C:\Windows\inetpub\logs\LogFiles\W3SVC*\*.log] ignoreOlderThan = 3d index=web sourcetype=iis:web:common [monitor://D:\LogFiles\W3SVC*\*.log] ignoreOlderThan = 3d index=web sourcetype=iis:web:common [monitor://E:\inetpub\logs\LogFiles\W3SVC*\*.log] ignoreOlderThan = 3d index=web sourcetype=iis:web:common Kindly let me know how to fix this issue. ... View more

Hi Nickhill, Yes you are right but its not reading the wtmpx file (binary file) from this location /var/adm/wtmpx. and I hope this is not built in the lastlog.sh script. So we decided to write a script that can read this binary file and write it to a normal txt file but at the same time it script should be in such away that it is not re-indexing the same file again and again. So could please guide me on this request to create a script which can read and write a binary file into a normal txt file. thanks in advance. ... View more

Hi Nickhill, I am not getting any Error message, getting only the INFO, when executing the below Search query index="_internal" host="test01" source=*web_service.log sourcetype=splunk_web_service 1/18/18 1:07:24.957 AM 2018-01-18 01:07:24,957 INFO [5a60399ca07ff1ac66c910] view:1053 - bypass module system fast path kindly guide me how to fix this issue. ... View more

hey we are getting permission denied issue while executing the ./who.sh on the remote node. /opt/splunkforwarder bash-3.2$ cd etc/apps/test-ta-nix/bin bash: cd: etc/apps/test-ta-nix/bin: Permission denied I could see this error in splunkd.log 01-17-2018 11:55:19.134 -0500 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/test-ta-nix/bin/who.sh" /bin/sh: /opt/splunkforwarder/etc/apps/test-ta-nix/bin/who.sh: cannot execute ... View more

Hey I dn't have an access to the remote test node, i had just copy pasted who.sh and lastlog.sh from the splunk add-on directly not changed their content. So could please let me know whether the below inputs.conf stanza are right. # Shows current user sessions [script://./bin/who.sh] sourcetype = who source = who interval = 150 index = nix disabled = 0 # Shows last login time for users who have ever logged in [script://./bin/lastlog.sh] sourcetype = lastlog source = lastlog interval = 300 index = nix disabled = 0 [monitor:///var/adm/wtmpx] index = nix disabled = 0 I am checking with unix admin to test the same. meanwhile could you please let me know whether the above inputs.conf is correct. ... View more

okay let me try to execute the ./who.sh on the remote machine under this path /opt/splunkforwarder/etc/test-ia-unix/bin ./who.sh it should fetch some out put is that correct! ... View more

No, I haven't tried executing the script in the remote Solaris node, so you mean to execute the ./who.sh but from which path or a directory we need to execute the script. ... View more

Hi All, Can any one guide me on the scripted inputs to pull the who and last data in to splunk. thanks in advance. ... View more

Hi Nickhill, I tried the below inputs.conf stanza in a test machine to pull the binary data from the Solaris machine but it did not fetching any data from that node. # Shows current user sessions [script://./bin/who.sh] sourcetype = who source = who interval = 150 index = nix disabled = 0 # Shows last login time for users who have ever logged in [script://./bin/lastlog.sh] sourcetype = lastlog source = lastlog interval = 300 index = nix disabled = 0 [monitor:///var/adm/wtmpx] index = nix disabled = 0 Kindly guide me on this. ... View more

Hey I have downloaded the Splunk_TA_nix add on from the splunk base and I have copy/pasted the who.sh and lastlogin.sh in to bin folder of my own app called test-ta-nix and also copy pasted the scripted inputs details in the inputs.conf in the same app. # Shows current user sessions [script://./bin/who.sh] sourcetype = who source = who interval = 150 index = nix disabled = 0 # Shows last login time for users who have ever logged in [script://./bin/lastlog.sh] sourcetype = lastlog source = lastlog interval = 300 index = nix disabled = 0 [monitor:///var/adm] whitelist=(\.log|log$|wtmpx|utmpx|message) index=nix disable=0 [monitor:///var/log] whitelist=(\.log|log$|secure|message|auth|wtmp|btmp|cron$|\.out) blacklist=(lastlog) index=nix disable=0 Kindly let me know whether I can use the above stanza to fetch the required binary details in to splunk. thanks in advance. ... View more

Hi Mayurr, I tried by creating a test app and pushed the below configuration for the test machine and it was fetching the some data but the data was not in a readable format. Inputs.conf [monitor:///var/adm/wtmpx] index = unix sourcetype = unix:host:wtmpx Props.conf [unix:host:wtmpx] CHARSET = AUTO NO_BINARY_CHECK = true And the output in splunk console : \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00ts/1pts/1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00No\x00\x00\x00\x00\x00\x00\x00Z^>Q\x00P Kindly let me know how to correct this. ... View more