About sansri7680

sansri7680 · ‎01-09-2014

I am trying to read log files from Hadoop cluster. These are unstructured files which otherwise can be filtered after indexing using Regex searches. But my input data is huge and the throughput requirement is also very high. The result is only a small portion of the input. Hence is it possible to filter the input data before being indexed by Hunk so that I can avoid searching unnecessary data

sansri7680 · ‎12-27-2013

thanks very much. this info will be very useful for me

sansri7680 · ‎12-20-2013

Thank you very much. I have another question. When you say Hunk license is based on size of the hadoop cluster, is there any limit to the amount of data that hunk can handle per day as per the licenses?

sansri7680 · ‎12-19-2013

Hi I am doing an application in Splunk that processes that processes 200K records per second fetched from Hadoop. What is the sizing that I need to look at for the licensing. I could see in Hunk that virtual indexes are created for Hadoop data to be processed. Does the indexing of data using these virtual indexes count on the per day data limit in Hunk license. Can someone help me on this Thanks in Advance Regards Subbu

sansri7680 · ‎05-22-2013

I found out the solution. We need to specify the BREAK_ONLY_BEFORE=^(\s+)Session-ID and SHOULD_LINEMERGE=true and the events split correctly

sansri7680 · ‎05-21-2013

hi, i tried this but again it is doing the same thing. is there any work around for this

sansri7680 · ‎05-17-2013

Hi Sorry I am a newbie to Splunk and the question may sound silly but the splunk regex that I used to split events in the file doesn't work props.conf [3GPP] BREAK_ONLY_BEFORE = ^Session-ID:\s NO_BINARY_CHECK = 1 SHOULD_LINEMERGE = true The file looks like below. The Event split has to happen before the Session-ID field. Can someone help me please. I tried various combinations but all gives the same result and the split happens in a place where it shouldn't Session-ID: 8:885 Username: 240010000099923@3glab.com Callid: 02166055 IMSI/MSID: 240010000099923 ACSMgr Instance: 8 ACSMgr Card/Cpu: 16/0 SessMgr Instance: 8 Client-IP: 10.127.1.63 NAS-IP: 10.10.10.253 Access-NAS-IP(FA): NAS-PORT: 0 NSAPI: 5 Acct-Session-ID: 0A0A0BFC02555693 NAS-ID: n/a Access-NAS-ID(FA): n/a 3GPP2-BSID: n/a Access-Correlation-ID(FA): n/a 3GPP2-Correlation-ID: n/a MEID: n/a Carrier-ID: n/a ESN: n/a Uplink Bytes: 19570539 Downlink Bytes: 405939823 Uplink Packets: 360119 Downlink Packets: 709687 Injected Uplink Bytes: 0 Injected Downlink Bytes: 0 Injected Uplink Packets: 0 Injected Downlink Packets: 0 Buffered Uplink Packets: 0 Buffered Downlink Packets: 0 Buffered Uplink Bytes: 0 Buffered Downlink Bytes: 0 Uplink Packets in Buffer: 0 Uplink Bytes in Buffer: 0 Downlink Packets in Buffer: 0 Downlink Bytes in Buffer: 0 Buff Over-limit Uplink Pkts: 0 Buff Over-limit Uplink Bytes: 0 Buff Over-limit Downlink Pkts: 0 Buff Over-limit Downlink Bytes: 0 Processed Uplink Packets: 0 Processed Downlink Packets: 0 Dropped Uplink Packets: 0 Dropped Downlink Packets: 0 Uplink Out of Order Packets: 3 Downlink Out of Order Packets: 3865 Dyn FUI Redirected Flows: 0 Dyn FUI Discarded Pkts: 0 ITC Terminated Flows: 0 ITC Redirected Flows: 0 ITC Dropped Packets: 0 ITC ToS Remarked Packets: 0 Flow action Terminated Flows: 0 PP Flow action Terminated Flows: 0 CC Dropped Uplink Packets: 0 CC Dropped Uplink Bytes: 0 CC Dropped Downlink Packets: 0 CC Dropped Downlink Bytes: 0 NRUPC Req Made: 1 NRUPC Req Success: 1 NRUPC Req Failed: 0 NRUPC Req Time Out: 0 Current Readdressed Sessions: 0 Total Readdressed Uplink Pkts: 0 Total Readdressed Uplink Bytes: 0 Total Readdressed Downlink Pkts: 0 Total Readdressed Downlink Bytes: 0 Total Readdressing Failure: 0 Creation Time: Tuesday May 14 04:14:44 GMT 2013 Last Pkt Time: Tuesday May 14 05:40:04 GMT 2013 Duration: 01h:25m:20s Active Charging Service name: ecs Rule Base name: Rapids_Silver_Rule Bandwidth Policy: n/a FW-and-NAT Policy: n/a NAT Policy: Not-required TPO Policy: n/a CF Policy ID: n/a Old CF Policy ID: n/a Dynamic Charging: Enabled Dynamic Chrg Msg Received: 1 Rule Definitions Received: 5 Installs Received: 5 Removes Received: 0 Installs Succeeded: 5 Installs Failed: 0 Removes Succeeded: 0 Removes Failed: 0 Uplink Dynamic Rule Packets: 0 Uplink Dynamic Rule Bytes: 0 Downlink Dynamic Rule Packets: 0 Downlink Dynamic Rule Bytes: 0 Dynamic Charging Packet Drop statistics: Bearer BW Limit Upl Pkts: 0 Bearer BW Limit Dnl Pkts: 0 Bearer BW Limit Upl Bytes: 0 Bearer BW Limit Dnl Bytes: 0 PCC Rule BW Limit Upl Pkts: 0 PCC Rule BW Limit Dnl Pkts: 0 PCC Rule BW Limit Upl Bytes: 0 PCC Rule BW Limit Dnl Bytes: 0 PCC Rule Gating Upl Pkts: 0 PCC Rule Gating Dnl Pkts: 0 PCC Rule Gating Upl Bytes: 0 PCC Rule Gating Dnl Bytes: 0 RuleMatch Fail Upl Pkts: 0 RuleMatch Fail Dnl Pkts: 0 RuleMatch Fail Upl Bytes: 0 RuleMatch Fail Dnl Bytes: 0 Credit-Control: Off QoS Renegotiate Up: 0 QoS Renegotiate Dn: 0 Current TCP Proxy Flows: 0 Total TCP Proxy Flows: 0 TCP-proxy reset for non-SYN flows: 0 Current IP Flows: 128 Current ICMP Flows: 1 Current IPv6 Flows: 0 Current ICMPv6 Flows: 0 Current TCP Flows: 44 Current UDP Flows: 83 Current HTTP Flows: 0 Current HTTPS Flows: 0 Current FTP Flows: 0 Current POP3 Flows: 0 Current SMTP Flows: 0 Current SIP Flows: 0 Current RTSP Flows: 0 Current RTP Flows: 0 Current RTCP Flows: 0 Current IMAP Flows: 0 Current WSP-CO Flows: 0 Current WSP-CL Flows: 0 Current MMS Flows: 0 Current DNS Flows: 0 Current PPTP-GRE Flows: 0 Current PPTP Flows: 0 Current P2P Flows: 0 Current H323 Flows: 0 Current TFTP Flows: 0 Current UNKNOWN Flows: 127 CAE-Readdressing: GET Requests redirected: 0 POST Requests redirected: 0 Other Requests redirected: 0 HTTP Responses redirected: 0 Requests having xheader inserted: 0 Total connect failed to video server: 0 Total Uplink Bytes: 0 Total Uplink Packets: 0 Total Downlink Bytes: 0 Total Downlink Bytes: 0 Transrating: Total Transrated Video Connections: 0 Transrated Sorenson H263 Connections: 0 Transrated H264 Connections: 0 Failed Sorenson H263 Connections: 0 Failed H264 Connections: 0 Total Input Video Data Bytes: 0 SH263 Input Video Data Bytes: 0 H264 Input Video Data Bytes: 0 Total Output Video Data Bytes: 0 SH263 Output Video Data Bytes: 0 H264 Output Video Data Bytes: 0 Average Input Video Bit Rate: 0 SH263 Input Video Bit Rate: 0 H264 Input Video Bit Rate: 0 Average Output Video Bit Rate: 0 SH263 Output Video Bit Rate: 0 H264 Output Video Bit Rate: 0 Average Bit Rate Reduction: 0 SH263 Bit Rate Reduction: 0 H264 Bit Rate Reduction: 0 TCP-Proxy Session Stats: n/a WiMAX Hotlining Status: n/a Link Monitoring Average Throughput: 0 kbps Link Monitoring Average RTT: 0 ms Ruledef Name Pkts-Down Bytes-Down Pkts-Up Bytes-Up Hits ip_any 709687 405939823 360119 19570539 1069806 Dynamic Charging Rule Name Statistics: n/a Total Dynamic Rules: 5 Total Predefined Rules: 0 Total Firewall Predefined Rules: 0 Dynamic Charging Rule Definition(s) Configured: Name Prior Content-Id Chrg-Type Rule Parameters Gold_time1_3577508100 1 0 None Gate Status: Discard All QoS Class Identifier: 5 ARP Priority Level: 1 Reporting Level: Rating Grp Metering Method: Volume Uplink MBR: 100000000 Downlink MBR: 200000000 Uplink GBR: 10000001 Downlink GBR: 20000001 Rule Activation Time: Tuesday May 14 08:15:00 GMT 2013 Rule De-activation Time: Tuesday May 14 09:15:00 GMT 2013 Filter 1: Direction: Uplink Dst Addr 0.0.0.0/0 Filter 2: Direction: Downlink Src Addr 0.0.0.0/0 Gold_time1_3577515300 1 0 None Gate Status: Discard All QoS Class Identifier: 5 ARP Priority Level: 1 Reporting Level: Rating Grp Metering Method: Volume Uplink MBR: 100000000 Downlink MBR: 200000000 Uplink GBR: 10000001 Downlink GBR: 20000001 Rule Activation Time: Tuesday May 14 10:15:00 GMT 2013 Rule De-activation Time: Tuesday May 14 11:15:00 GMT 2013 Filter 1: Direction: Uplink Dst Addr 0.0.0.0/0 Filter 2: Direction: Downlink Src Addr 0.0.0.0/0 Gold_time1_3577522500 1 0 None Gate Status: Discard All QoS Class Identifier: 5 ARP Priority Level: 1 Reporting Level: Rating Grp Metering Method: Volume Uplink MBR: 100000000 Downlink MBR: 200000000 Uplink GBR: 10000001 Downlink GBR: 20000001 Rule Activation Time: Tuesday May 14 12:15:00 GMT 2013 Rule De-activation Time: Tuesday May 14 13:15:00 GMT 2013 Filter 1: Direction: Uplink Dst Addr 0.0.0.0/0 Filter 2: Direction: Downlink Src Addr 0.0.0.0/0 Gold_time1_3577533300 1 0 None Gate Status: Discard All QoS Class Identifier: 5 ARP Priority Level: 1 Reporting Level: Rating Grp Metering Method: Volume Uplink MBR: 100000000 Downlink MBR: 200000000 Uplink GBR: 10000001 Downlink GBR: 20000001 Rule Activation Time: Tuesday May 14 15:15:00 GMT 2013 Rule De-activation Time: Tuesday May 14 16:15:00 GMT 2013 Filter 1: Direction: Uplink Dst Addr 0.0.0.0/0 Filter 2: Direction: Downlink Src Addr 0.0.0.0/0 Gold_time1_3577540500 1 0 None Gate Status: Discard All QoS Class Identifier: 5 ARP Priority Level: 1 Reporting Level: Rating Grp Metering Method: Volume Uplink MBR: 100000000 Downlink MBR: 200000000 Uplink GBR: 10000001 Downlink GBR: 20000001 Rule Activation Time: Tuesday May 14 17:15:00 GMT 2013 Rule De-activation Time: Tuesday May 14 18:15:00 GMT 2013 Filter 1: Direction: Uplink Dst Addr 0.0.0.0/0 Filter 2: Direction: Downlink Src Addr 0.0.0.0/0 Predefined Rules Enabled List: n/a Predefined Firewall Rules Enabled List: n/a Total acs sessions matching specified criteria: 1 Session-ID: 8:886 Username: 240010000099924@3glab.com Callid: 02166055 IMSI/MSID: 240010000099924 ACSMgr Instance: 8 ACSMgr Card/Cpu: 16/0 SessMgr Instance: 8 Client-IP: 10.127.1.64 NAS-IP: 10.10.10.254 Access-NAS-IP(FA): NAS-PORT: 0 NSAPI: 5 Acct-Session-ID: 0A0A0BFC02555694 NAS-ID: n/a Access-NAS-ID(FA): n/a 3GPP2-BSID: n/a Access-Correlation-ID(FA): n/a 3GPP2-Correlation-ID: n/a MEID: n/a Carrier-ID: n/a ESN: n/a Uplink Bytes: 19570540 Downlink Bytes: 405939824 Uplink Packets: 360120 Downlink Packets: 709688 Injected Uplink Bytes: 0 Injected Downlink Bytes: 0 Injected Uplink Packets: 0 Injected Downlink Packets: 0 Buffered Uplink Packets: 0 Buffered Downlink Packets: 0 Buffered Uplink Bytes: 0 Buffered Downlink Bytes: 0 Uplink Packets in Buffer: 0 Uplink Bytes in Buffer: 0 Downlink Packets in Buffer: 0 Downlink Bytes in Buffer: 0 Buff Over-limit Uplink Pkts: 0 Buff Over-limit Uplink Bytes: 0 Buff Over-limit Downlink Pkts: 0 Buff Over-limit Downlink Bytes: 0 Processed Uplink Packets: 0 Processed Downlink Packets: 0 Dropped Uplink Packets: 0 Dropped Downlink Packets: 0 Uplink Out of Order Packets: 3 Downlink Out of Order Packets: 3861 Dyn FUI Redirected Flows: 0 Dyn FUI Discarded Pkts: 0 ITC Terminated Flows: 0 ITC Redirected Flows: 0 ITC Dropped Packets: 0 ITC ToS Remarked Packets: 0 Flow action Terminated Flows: 0 PP Flow action Terminated Flows: 0 CC Dropped Uplink Packets: 0 CC Dropped Uplink Bytes: 0 CC Dropped Downlink Packets: 0 CC Dropped Downlink Bytes: 0 NRUPC Req Made: 1 NRUPC Req Success: 1 NRUPC Req Failed: 0 NRUPC Req Time Out: 0 Current Readdressed Sessions: 0 Total Readdressed Uplink Pkts: 0 Total Readdressed Uplink Bytes: 0 Total Readdressed Downlink Pkts: 0 Total Readdressed Downlink Bytes: 0 Total Readdressing Failure: 0 Creation Time: Tuesday May 14 05:15:44 GMT 2013 Last Pkt Time: Tuesday May 14 05:46:04 GMT 2013 Duration: 00h:41m:20s Active Charging Service name: ecs Rule Base name: Rapids_Silver_Rule Bandwidth Policy: n/a FW-and-NAT Policy: n/a NAT Policy: Not-required TPO Policy: n/a CF Policy ID: n/a Old CF Policy ID: n/a Dynamic Charging: Enabled Dynamic Chrg Msg Received: 1 Rule Definitions Received: 5 Installs Received: 5 Removes Received: 0 Installs Succeeded: 5 Installs Failed: 0 Removes Succeeded: 0 Removes Failed: 0 Uplink Dynamic Rule Packets: 0 Uplink Dynamic Rule Bytes: 0 Downlink Dynamic Rule Packets: 0 Downlink Dynamic Rule Bytes: 0 Dynamic Charging Packet Drop statistics: Bearer BW Limit Upl Pkts: 0 Bearer BW Limit Dnl Pkts: 0 Bearer BW Limit Upl Bytes: 0 Bearer BW Limit Dnl Bytes: 0 PCC Rule BW Limit Upl Pkts: 0 PCC Rule BW Limit Dnl Pkts: 0 PCC Rule BW Limit Upl Bytes: 0 PCC Rule BW Limit Dnl Bytes: 0 PCC Rule Gating Upl Pkts: 0 PCC Rule Gating Dnl Pkts: 0 PCC Rule Gating Upl Bytes: 0 PCC Rule Gating Dnl Bytes: 0 RuleMatch Fail Upl Pkts: 0 RuleMatch Fail Dnl Pkts: 0 RuleMatch Fail Upl Bytes: 0 RuleMatch Fail Dnl Bytes: 0 Credit-Control: Off QoS Renegotiate Up: 0 QoS Renegotiate Dn: 0 Current TCP Proxy Flows: 0 Total TCP Proxy Flows: 0 TCP-proxy reset for non-SYN flows: 0 Current IP Flows: 128 Current ICMP Flows: 1 Current IPv6 Flows: 0 Current ICMPv6 Flows: 0 Current TCP Flows: 44 Current UDP Flows: 83 Current HTTP Flows: 0 Current HTTPS Flows: 0 Current FTP Flows: 0 Current POP3 Flows: 0 Current SMTP Flows: 0 Current SIP Flows: 0 Current RTSP Flows: 0 Current RTP Flows: 0 Current RTCP Flows: 0 Current IMAP Flows: 0 Current WSP-CO Flows: 0 Current WSP-CL Flows: 0 Current MMS Flows: 0 Current DNS Flows: 0 Current PPTP-GRE Flows: 0 Current PPTP Flows: 0 Current P2P Flows: 0 Current H323 Flows: 0 Current TFTP Flows: 0 Current UNKNOWN Flows: 127 CAE-Readdressing: GET Requests redirected: 0 POST Requests redirected: 0 Other Requests redirected: 0 HTTP Responses redirected: 0 Requests having xheader inserted: 0 Total connect failed to video server: 0 Total Uplink Bytes: 0 Total Uplink Packets: 0 Total Downlink Bytes: 0 Total Downlink Bytes: 0 Transrating: Total Transrated Video Connections: 0 Transrated Sorenson H263 Connections: 0 Transrated H264 Connections: 0 Failed Sorenson H263 Connections: 0 Failed H264 Connections: 0 Total Input Video Data Bytes: 0 SH263 Input Video Data Bytes: 0 H264 Input Video Data Bytes: 0 Total Output Video Data Bytes: 0 SH263 Output Video Data Bytes: 0 H264 Output Video Data Bytes: 0 Average Input Video Bit Rate: 0 SH263 Input Video Bit Rate: 0 H264 Input Video Bit Rate: 0 Average Output Video Bit Rate: 0 SH263 Output Video Bit Rate: 0 H264 Output Video Bit Rate: 0 Average Bit Rate Reduction: 0 SH263 Bit Rate Reduction: 0 H264 Bit Rate Reduction: 0 TCP-Proxy Session Stats: n/a WiMAX Hotlining Status: n/a Link Monitoring Average Throughput: 0 kbps Link Monitoring Average RTT: 0 ms Ruledef Name Pkts-Down Bytes-Down Pkts-Up Bytes-Up Hits ip_any 709687 405939823 360119 19570539 1069806 Dynamic Charging Rule Name Statistics: n/a Total Dynamic Rules: 5 Total Predefined Rules: 0 Total Firewall Predefined Rules: 0 Dynamic Charging Rule Definition(s) Configured: Name Prior Content-Id Chrg-Type Rule Parameters Gold_time1_3577508100 1 0 None Gate Status: Discard All QoS Class Identifier: 5 ARP Priority Level: 1 Reporting Level: Rating Grp Metering Method: Volume Uplink MBR: 100000000 Downlink MBR: 200000000 Uplink GBR: 10000001 Downlink GBR: 20000001 Rule Activation Time: Tuesday May 14 08:15:00 GMT 2013 Rule De-activation Time: Tuesday May 14 09:15:00 GMT 2013 Filter 1: Direction: Uplink Dst Addr 0.0.0.0/0 Filter 2: Direction: Downlink Src Addr 0.0.0.0/0 Gold_time1_3577515300 1 0 None Gate Status: Discard All QoS Class Identifier: 5 ARP Priority Level: 1 Reporting Level: Rating Grp Metering Method: Volume Uplink MBR: 100000000 Downlink MBR: 200000000 Uplink GBR: 10000001 Downlink GBR: 20000001 Rule Activation Time: Tuesday May 14 10:15:00 GMT 2013 Rule De-activation Time: Tuesday May 14 11:15:00 GMT 2013 Filter 1: Direction: Uplink Dst Addr 0.0.0.0/0 Filter 2: Direction: Downlink Src Addr 0.0.0.0/0 Gold_time1_3577522500 1 0 None Gate Status: Discard All QoS Class Identifier: 5 ARP Priority Level: 1 Reporting Level: Rating Grp Metering Method: Volume Uplink MBR: 100000000 Downlink MBR: 200000000 Uplink GBR: 10000001 Downlink GBR: 20000001 Rule Activation Time: Tuesday May 14 12:15:00 GMT 2013 Rule De-activation Time: Tuesday May 14 13:15:00 GMT 2013 Filter 1: Direction: Uplink Dst Addr 0.0.0.0/0 Filter 2: Direction: Downlink Src Addr 0.0.0.0/0 Gold_time1_3577533300 1 0 None Gate Status: Discard All QoS Class Identifier: 5 ARP Priority Level: 1 Reporting Level: Rating Grp Metering Method: Volume Uplink MBR: 100000000 Downlink MBR: 200000000 Uplink GBR: 10000001 Downlink GBR: 20000001 Rule Activation Time: Tuesday May 14 15:15:00 GMT 2013 Rule De-activation Time: Tuesday May 14 16:15:00 GMT 2013 Filter 1: Direction: Uplink Dst Addr 0.0.0.0/0 Filter 2: Direction: Downlink Src Addr 0.0.0.0/0 Gold_time1_3577540500 1 0 None Gate Status: Discard All QoS Class Identifier: 5 ARP Priority Level: 1 Reporting Level: Rating Grp Metering Method: Volume Uplink MBR: 100000000 Downlink MBR: 200000000 Uplink GBR: 10000001 Downlink GBR: 20000001 Rule Activation Time: Tuesday May 14 17:15:00 GMT 2013 Rule De-activation Time: Tuesday May 14 18:15:00 GMT 2013 Filter 1: Direction: Uplink Dst Addr 0.0.0.0/0 Filter 2: Direction: Downlink Src Addr 0.0.0.0/0 Predefined Rules Enabled List: n/a Predefined Firewall Rules Enabled List: n/a Total acs sessions matching specified criteria: 1

sansri7680 · ‎03-28-2013

Found it out. The dispatch.ttl was the culprit. I changed it to 1 second and everything is fine now

sansri7680 · ‎03-28-2013

Hi I created a scheduled search with the below search string sourcetype="MME_Reject-3" | eval indextime=_indextime | eval tnow=now() | eval diff=tnow-indextime | sort + diff | where diff<60 | reverse | table _raw | outputcsv 4G_REJECT_LOG.txt The source is fed by a forwarder that forwards data from windows machine The search brings results whenever it is force run but when it runs using the schedule of 1 minute it doesn't bring any result Can someone throw light on what the problem could be. splunkd.log doesn't display any error

sansri7680 · ‎03-27-2013

but it will require a separate full licensing if deployed in production environment right? My requirement is that these kind of logs will be in various machines and I will have light forwarder installed in all the machines to forward log data to one Splunk instance which will be indexed there. I even tried to reindex the data after it enters the splunk instance using "windows eventlog timefix" app but even that doesn't work

sansri7680 · ‎03-27-2013

The config doesn't seem to have any effect. Also is there anything to do with the Splunk version that I am using? Does this forwarding and indexing thing work on the free version that is available for download in the Splunk site?

sansri7680 · ‎03-27-2013

I created a new index in my splunk instance. I want to set some custom settings for event splitting in props.conf for the new index. which path should this props.conf go. Is it $SPLUNK_HOME/etc/apps/search/local or $SPLUNK_HOME/etc/system/local Thanks in advance

sansri7680 · ‎03-27-2013

Hi Kristian The props.conf file is very much in the indexer side. I have put it in the $SPLUNK_HOME/etc/apps/search/local folder in the main splunk instance. I have juggled with moving the NO_BINARY_CHECK to the forwarder section too but no luck. Can you suggest if I am going wrong anywhere in the config. Whether I am putting the props.conf in the wrong folder of the splunk instance

sansri7680 · ‎03-26-2013

Hi I have a Universal forwarder forwarding data from a monitored file on Windows. This file contains multiline events. The splunk instance is available in the Linux machine and is able to receive the data on port 9997. But the indexer doesn't seem to split this data into multiline events. Rather it is being split at all the lines. I have the regex configured in the props.conf in the $SPLUNK_HOME/etc/apps/search/local folder as well as $SPLINK_HOME/etc/system/local/props.conf. But it doesn't seem to have any effect on the data. Also the same regex works if I create an input on a local file on the linux box but doesn't seem to work for forwarded data. The data is coming into _main index without any problems but not splitting properly. Any inputs on this is highly appreciated. I am posting the regex as below [4GCDR] BREAK_ONLY_BEFORE = (.*)(Inbound>>>>>|<<<<Outbound) NO_BINARY_CHECK = 1 SHOULD_LINEMERGE = True TRUNCATE=0 The events look like below INBOUND>>>>> 19:00:45:775 Eventid:155212(3) S1AP Rx PDU, from 10.10.11.36:36412 to 10.10.11.226:36412 (21) S1 Application Part (S1AP) (21 bytes) | 0... .... | Ext bit : 0 | .01. .... | Choice index : Successful Outcome (1) Procedure Code : UE CONTEXT RELEASE (23) Criticality | 00.. .... | Reject (0) UE CONTEXT RELEASE Value : | .001 0001 | Length Determinant : 17 Value : | 0... .... | Ext bit : 0 IEs Count : 2 IE : 1 Protocol IE ID : MME_UE_S1AP_ID (0) Criticality | 01.. .... | Ignore (1) MME_UE_S1AP_ID Value : | .000 0100 | Length Determinant : 4 Value : | 10.. .... | Length Determinant : 3 10485764 (0xa00004) IE : 2 Protocol IE ID : eNB_UE_S1AP_ID (8) Criticality | 01.. .... | Ignore (1) eNB_UE_S1AP_ID Value : | .000 0010 | Length Determinant : 2 Value : | 00.. .... | Length Determinant : 1 14 (0x0e) INBOUND>>>>> 19:00:45:577 Eventid:141004(3) [MME-S11]GTPv2C Rx PDU, from 10.10.10.246:2123 to 10.10.10.245:30160 (18) TEID: 0x8003400A, Message type: EGTP_DELETE_SESSION_RESPONSE (0x25) Sequence Number: 0x00800A (32778) GTP HEADER Version number: 2 TEID flag: Present Piggybacking flag: Not present Message Length: 0x000E (14) INFORMATION ELEMENTS CAUSE: Type: 2 Length: 2 Inst: 0 Value: Cause: EGTP_CAUSE_REQ_ACCEPTED (0x10) OI: 0 Hex: 0200 0200 1000

sansri7680 · ‎03-25-2013

I tried that also. But no events are coming into splunk now

sansri7680 · ‎03-25-2013

Hi Ayn Can you tell me how to configure the settings in the indexer

sansri7680 · ‎03-25-2013

what I am trying to do is that I am forwarding a 4G network log from the windows machine to the splunk instance on the linux server and trying to index the data there. I have only a universal forwarder installed in the windows end but a full splunk instance on the linux end. Even after doing the above config the events are not getting split whatever changes done to the log is coming in as a single record. The Regex logic is fine if tested in isolation

sansri7680 · ‎03-25-2013

Here are the configurations that I did Universal forwarder output.conf ($SPLUNK_HOME/etc/system/local) on windows [tcpout] defaultGroup = 10.200.221.158_9997 disabled = false [tcpout:10.200.221.158_9997] server = 10.200.221.158:9997 [tcpout-server://10.200.221.158:9997] Heavy forwarder Input.conf ($SPLUNK_HOME/etc/system/local) on linux machine [splunktcp://9997] Indexer Inputs.conf ($SPLUNK_HOME/etc/apps/search/local [splunktcp://9998] Indexer props.conf [4GCDR] BREAK_ONLY_BEFORE = (.*)(INBOUND>>>>>|<<<<OUTBOUND) NO_BINARY_CHECK = 1 SHOULD_LINEMERGE = true pulldown_type = 1 TRUNCATE=0

sansri7680 · ‎03-25-2013

can you give me an example configuration for receiving a light forwarder data by a heavy forwarder and forwarding it to an indexer

sansri7680 · ‎03-22-2013

Hi I have a Universal forwarder forwarding data from windows machine and I have created a receiver in the Splunk instance which resides in the Linux server. Instead of creating an input that takes data from the receiver, I want to forward the raw data received by the receiver to the indexer and get it indexed properly as we know Universal forwarder doesn't split the events in the data and just forwards the data to a receiver. I need to split the forwarded raw data into events and index it. Can someone throw some light on this Thanks in advance Subbu

sansri7680 · ‎03-22-2013

Hi The props.conf settings that I used was fine if I upload a single file. But it is not working with the Universal forwarder

sansri7680 · ‎03-22-2013

Which is the path where the configuration for heavy forwarder needs to be stored. whether it is the usual path $SPLUNK_HOME/etc/system/local or is it a different path

sansri7680 · ‎03-22-2013

I have a file which is monitored by the Universal forwarder in Windows box. I installed the forwarder on windows using the msi installer. I have data coming into the file regularly and the format is as below INBOUND>>>>> 19:00:17:308 Eventid:153001(3) NAS Rx PDU, from 10.10.11.36:36412 to 10.10.11.226:36412 (42) Non Access Stratum (NAS) (42 bytes) EPS Mobility Management Protocol Discriminator EPS MOBILITY MANAGEMENT MESSAGES(0x7) Security Header Type NAS_MSG_SECURITY_HDR_PLAIN_NAS_MSG(0x0) Message Type ATTACH_REQUEST(0x41) Attach Type EPS ATTACH(0x1) Key Set Identifier NO KEY AVAILABLE(0x7) Mobile Identity IMSI (240010000099935) UE n/w capability (0xe0e0) ESM CONTAINER EPS Session Management Protocol Discriminator EPS SESSION MANAGEMENT MESSAGES(0x2) EPS Bearer Id (0x0) Transaction Id (0x1) Message Type PDN_CONNECTIVITY_REQUEST(0xd0) Request Type INITIAL REQUEST(0x1) PDN Type IPv4(0x1) Protocol Config Options Configuration Protocol: PPP Proto/Container ID: IPCP Contents:0x0100000A810600000000 MS n/w capability (0xc540f4) <<<<OUTBOUND 19:00:17:730 Eventid:153002(3) NAS Tx PDU, from 10.10.11.226:36412 to 10.10.11.36:36412 (36) Non Access Stratum (NAS) (36 bytes) EPS Mobility Management Protocol Discriminator EPS MOBILITY MANAGEMENT MESSAGES(0x7) Security Header Type NAS_MSG_SECURITY_HDR_PLAIN_NAS_MSG(0x0) Message Type AUTHENTICATION_REQUEST(0x52) Key Set Identifier Security Context Type: Native (0x0) Key Set Index: (0x6) Spare-Half (0x0) RAND (0x81d97b15a0aa040081d97b15a0aa0400) AUTN (0x18f97648c158fffe445a366fe14f1160) <<<<OUTBOUND 19:00:17:730 Eventid:155213(3) S1AP Tx PDU, from 10.10.11.226:36412 to 10.10.11.36:36412 (62) S1 Application Part (S1AP) (62 bytes) | 0... .... | Ext bit : 0 | .00. .... | Choice index : Initiating Message (0) Procedure Code : DOWNLINK NAS TRANSPORT (11) Criticality | 01.. .... | Ignore (1) DOWNLINK NAS TRANSPORT Value : | .011 1010 | Length Determinant : 58 Value : | 0... .... | Ext bit : 0 IEs Count : 3 IE : 1 Protocol IE ID : MME_UE_S1AP_ID (0) Criticality | 00.. .... | Reject (0) MME_UE_S1AP_ID Value : | .000 0100 | Length Determinant : 4 Value : | 10.. .... | Length Determinant : 3 12582917 (0xc00005) IE : 2 Protocol IE ID : eNB_UE_S1AP_ID (8) Criticality | 00.. .... | Reject (0) eNB_UE_S1AP_ID Value : | .000 0010 | Length Determinant : 2 Value : | 00.. .... | Length Determinant : 1 13 (0x0d) IE : 3 Protocol IE ID : NAS_PDU (26) Criticality | 00.. .... | Reject (0) NAS_PDU Value : | .010 0101 | Length Determinant : 37 Value : | .010 0100 | Length Determinant : 36 0x07520681d97b15a0aa040081d97b15a0aa04001018f97648c158fffe445a366fe14f1160 EPS Mobility Management Protocol Discriminator EPS MOBILITY MANAGEMENT MESSAGES(0x7) Security Header Type NAS_MSG_SECURITY_HDR_PLAIN_NAS_MSG(0x0) Message Type AUTHENTICATION_REQUEST(0x52) Key Set Identifier Security Context Type: Native (0x0) Key Set Index: (0x6) Spare-Half (0x0) RAND (0x81d97b15a0aa040081d97b15a0aa0400) AUTN (0x18f97648c158fffe445a366fe14f1160) My props.conf on my UNIX box is as below [4GCDR] BREAK_ONLY_BEFORE = (.*)(INBOUND>>>>>|<<<<OUTBOUND) NO_BINARY_CHECK = 1 SHOULD_LINEMERGE = true pulldown_type = 1 TRUNCATE=0 I created a UDP input to monitor the port where the windows forwarder sends the data But whenever more than one event occurs in the monitored file all the events are merged into a single event. If events are updated one by one there are no problems. Can someone please help

sansri7680 · ‎03-06-2013

Hi, Thanks for your response. I didn't give any modifiers after the first 1500 events. But I found the problem with special characters in the file. I did the below to resolve the problem. I just opened the file in a word processor and trimmed all trailing tabs and spaces which was correctly indexed by splunk. But that doesn't solve my exact problem. In my app, the file is a constantly growing file and it is not expected for someone to open it up in word processor and trim trailing spaces everytime. Is there any permanent fix for this

sansri7680 · ‎03-05-2013

I have a file with multiline events. Though there is no structured data in the events, the events themselves can be identified by proper splits. Below is an example Frame 1: 110 bytes on wire (880 bits), 110 bytes captured (880 bits) WTAP_ENCAP: 1 Arrival Time: Dec 20, 2007 14:01:56.000165000 India Standard Time [Time shift for this packet: 0.000000000 seconds] Epoch Time: 1198139516.000165000 seconds [Time delta from previous captured frame: 0.000000000 seconds] [Time delta from previous displayed frame: 0.000000000 seconds] [Time since reference or first frame: 0.000000000 seconds] Frame Number: 1 Frame Length: 110 bytes (880 bits) Capture Length: 110 bytes (880 bits) [Frame is marked: False] [Frame is ignored: False] [Protocols in frame: eth:ip:ospf] Ethernet II, Src: Cisco_f7:97:c2 (00:1b:8f:f7:97:c2), Dst: IPv4mcast_00:00:05 (01:00:5e:00:00:05) Destination: IPv4mcast_00:00:05 (01:00:5e:00:00:05) Address: IPv4mcast_00:00:05 (01:00:5e:00:00:05) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast) Source: Cisco_f7:97:c2 (00:1b:8f:f7:97:c2) Address: Cisco_f7:97:c2 (00:1b:8f:f7:97:c2) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Type: IP (0x0800) Internet Protocol Version 4, Src: 10.112.28.1 (10.112.28.1), Dst: 224.0.0.5 (224.0.0.5) Version: 4 Header length: 20 bytes Differentiated Services Field: 0xc0 (DSCP 0x30: Class Selector 6; ECN: 0x00: Not-ECT (Not ECN-Capable Transport)) 1100 00.. = Differentiated Services Codepoint: Class Selector 6 (0x30) .... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00) Total Length: 96 Identification: 0xd719 (55065) Flags: 0x00 0... .... = Reserved bit: Not set .0.. .... = Don't fragment: Not set ..0. .... = More fragments: Not set Fragment offset: 0 Time to live: 1 Protocol: OSPF IGP (89) Header checksum: 0xdaf5 [correct] [Good: True] [Bad: False] Source: 10.112.28.1 (10.112.28.1) Destination: 224.0.0.5 (224.0.0.5) [Source GeoIP: Unknown] [Destination GeoIP: Unknown] Open Shortest Path First OSPF Header OSPF Version: 2 Message Type: Hello Packet (1) Packet Length: 64 Source OSPF Router: 195.156.12.222 (195.156.12.222) Area ID: 0.0.0.0 (Backbone) Packet Checksum: 0x077d [correct] Auth Type: Null Auth Data (none) OSPF Hello Packet Network Mask: 255.255.255.128 Hello Interval: 10 seconds Options: 0x12 (L, E) 0... .... = DN: DN-bit is NOT set .0.. .... = O: O-bit is NOT set ..0. .... = DC: Demand Circuits are NOT supported ...1 .... = L: The packet contains LLS data block .... 0... = NP: NSSA is NOT supported .... .0.. = MC: NOT Multicast Capable .... ..1. = E: External Routing Capability .... ...0 = MT: NO Multi-Topology Routing Router Priority: 10 Router Dead Interval: 40 seconds Designated Router: 10.112.28.1 Backup Designated Router: 10.112.28.14 Active Neighbor: 10.112.29.19 Active Neighbor: 10.112.29.20 Active Neighbor: 10.112.29.66 Active Neighbor: 10.112.29.131 Active Neighbor: 10.112.29.254 OSPF LLS Data Block Checksum: 0xfff6 LLS Data Length: 12 bytes Extended options TLV Type: 1 Length: 4 Options: 0x00000001 (LR) .... .... .... .... .... .... .... ..0. = RS: Restart Signal (RS-bit) is NOT set .... .... .... .... .... .... .... ...1 = LR: LSDB Resynchronization (LR-bit) is SET Frame 2: 110 bytes on wire (880 bits), 110 bytes captured (880 bits) WTAP_ENCAP: 1 Arrival Time: Dec 20, 2007 14:01:56.000173000 India Standard Time [Time shift for this packet: 0.000000000 seconds] Epoch Time: 1198139516.000173000 seconds [Time delta from previous captured frame: 0.000008000 seconds] [Time delta from previous displayed frame: 0.000008000 seconds] [Time since reference or first frame: 0.000008000 seconds] Frame Number: 2 Frame Length: 110 bytes (880 bits) Capture Length: 110 bytes (880 bits) [Frame is marked: False] [Frame is ignored: False] [Protocols in frame: eth:ip:ospf] Ethernet II, Src: Cisco_f7:97:c2 (00:1b:8f:f7:97:c2), Dst: IPv4mcast_00:00:05 (01:00:5e:00:00:05) Destination: IPv4mcast_00:00:05 (01:00:5e:00:00:05) Address: IPv4mcast_00:00:05 (01:00:5e:00:00:05) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast) Source: Cisco_f7:97:c2 (00:1b:8f:f7:97:c2) Address: Cisco_f7:97:c2 (00:1b:8f:f7:97:c2) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Type: IP (0x0800) Internet Protocol Version 4, Src: 10.112.28.1 (10.112.28.1), Dst: 224.0.0.5 (224.0.0.5) Version: 4 Header length: 20 bytes Differentiated Services Field: 0xc0 (DSCP 0x30: Class Selector 6; ECN: 0x00: Not-ECT (Not ECN-Capable Transport)) 1100 00.. = Differentiated Services Codepoint: Class Selector 6 (0x30) .... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00) Total Length: 96 Identification: 0xd719 (55065) Flags: 0x00 0... .... = Reserved bit: Not set .0.. .... = Don't fragment: Not set ..0. .... = More fragments: Not set Fragment offset: 0 Time to live: 1 Protocol: OSPF IGP (89) Header checksum: 0xdaf5 [correct] [Good: True] [Bad: False] Source: 10.112.28.1 (10.112.28.1) Destination: 224.0.0.5 (224.0.0.5) [Source GeoIP: Unknown] [Destination GeoIP: Unknown] Open Shortest Path First OSPF Header OSPF Version: 2 Message Type: Hello Packet (1) Packet Length: 64 Source OSPF Router: 195.156.12.222 (195.156.12.222) Area ID: 0.0.0.0 (Backbone) Packet Checksum: 0x077d [correct] Auth Type: Null Auth Data (none) OSPF Hello Packet Network Mask: 255.255.255.128 Hello Interval: 10 seconds Options: 0x12 (L, E) 0... .... = DN: DN-bit is NOT set .0.. .... = O: O-bit is NOT set ..0. .... = DC: Demand Circuits are NOT supported ...1 .... = L: The packet contains LLS data block .... 0... = NP: NSSA is NOT supported .... .0.. = MC: NOT Multicast Capable .... ..1. = E: External Routing Capability .... ...0 = MT: NO Multi-Topology Routing Router Priority: 10 Router Dead Interval: 40 seconds Designated Router: 10.112.28.1 Backup Designated Router: 10.112.28.14 Active Neighbor: 10.112.29.19 Active Neighbor: 10.112.29.20 Active Neighbor: 10.112.29.66 Active Neighbor: 10.112.29.131 Active Neighbor: 10.112.29.254 OSPF LLS Data Block Checksum: 0xfff6 LLS Data Length: 12 bytes Extended options TLV Type: 1 Length: 4 Options: 0x00000001 (LR) .... .... .... .... .... .... .... ..0. = RS: Restart Signal (RS-bit) is NOT set .... .... .... .... .... .... .... ...1 = LR: LSDB Resynchronization (LR-bit) is SET Frame 3: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) WTAP_ENCAP: 1 Arrival Time: Dec 20, 2007 14:01:56.474107000 India Standard Time [Time shift for this packet: 0.000000000 seconds] Epoch Time: 1198139516.474107000 seconds [Time delta from previous captured frame: 0.473934000 seconds] [Time delta from previous displayed frame: 0.473934000 seconds] [Time since reference or first frame: 0.473942000 seconds] Frame Number: 3 Frame Length: 60 bytes (480 bits) Capture Length: 60 bytes (480 bits) [Frame is marked: False] [Frame is ignored: False] [Protocols in frame: eth:llc:stp] IEEE 802.3 Ethernet Destination: Spanning-tree-(for-bridges)_00 (01:80:c2:00:00:00) Address: Spanning-tree-(for-bridges)_00 (01:80:c2:00:00:00) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast) Source: Cisco_f7:97:8a (00:1b:8f:f7:97:8a) Address: Cisco_f7:97:8a (00:1b:8f:f7:97:8a) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Length: 38 Padding: 0000000000000000 Logical-Link Control DSAP: Spanning Tree BPDU (0x42) IG Bit: Individual SSAP: Spanning Tree BPDU (0x42) CR Bit: Command Control field: U, func=UI (0x03) 000. 00.. = Command: Unnumbered Information (0x00) .... ..11 = Frame type: Unnumbered frame (0x03) Spanning Tree Protocol Protocol Identifier: Spanning Tree Protocol (0x0000) Protocol Version Identifier: Spanning Tree (0) BPDU Type: Configuration (0x00) BPDU flags: 0x00 0... .... = Topology Change Acknowledgment: No .... ...0 = Topology Change: No Root Identifier: 32768 / 10 / 00:1b:8f:f7:97:80 Root Bridge Priority: 32768 Root Bridge System ID Extension: 10 Root Bridge System ID: 00:1b:8f:f7:97:80 Root Path Cost: 0 Bridge Identifier: 32768 / 10 / 00:1b:8f:f7:97:80 Bridge Priority: 32768 Bridge System ID Extension: 10 Bridge System ID: 00:1b:8f:f7:97:80 Port identifier: 0x800a Message Age: 0 Max Age: 20 Hello Time: 2 Forward Delay: 15 Each event can be separated using the word Frame followed by a incremental number and a colon. I tried the below regex in the props.conf file [4G] BREAK_ONLY_BEFORE = (?m)Frame ([0-9]+): SHOULD_LINEMERGE = true After doing the above, the events are not split up properly. The first 1500 events are appearing in a random manner and split at improper positions. But the events after 1500 events are split up properly. Can someone help me in finding what is wrong in the way I would have defined the regex

Posts	25
Solutions	2
Karma Given	11
Karma Received	1
Member Since	‎03-05-2013

Online Status	Offline
Date Last Visited	‎06-05-2020 02:04 AM

Filtering events from Hadoop unstructured data

Hunk sizing

Regex not working for event splitting

scheduled search doesn't get any events

indexer configuration

Splunk indexer splits multiline events on all line...

forward data from a receiver

heavy forwarder configuration path

Universal Forwarder merges multiple events into on...

Regex for multiline events

Filtering events from Hadoop unstructured data

Re: Hunk sizing

Re: Hunk sizing

Hunk sizing

Re: Regex not working for event splitting

Re: Regex not working for event splitting

Regex not working for event splitting

Re: scheduled search doesn't get any events

scheduled search doesn't get any events

Re: Splunk indexer splits multiline events on all ...

Re: Splunk indexer splits multiline events on all ...

indexer configuration

Re: Splunk indexer splits multiline events on all ...

Splunk indexer splits multiline events on all line...

Re: Universal Forwarder merges multiple events int...

Re: Universal Forwarder merges multiple events int...

Re: forward data from a receiver

Re: forward data from a receiver

Re: heavy forwarder configuration path

forward data from a receiver

Re: Universal Forwarder merges multiple events int...

heavy forwarder configuration path

Universal Forwarder merges multiple events into on...

Re: Regex for multiline events

Regex for multiline events