Splunk Search

Line breaking - regex and capturing group

armonsal
Explorer

Hello,
Need some help on regex here, am sure i maybe making mistake here but..
I don't undesrtand the problem in splunkd.log said:

(I had this line >11,000)

03-14-2014 17:11:49.108 -0300 ERROR LineBreakingProcessor - Line breaking regex has no capturing groups: \n - data_source="eStreamer", data_host="splunk.infocorp.cl", data_sourcetype="estreamer"

sample output of my events (This is a "estreamer" from Sourcefire) :

I capture this log whit app "estreamer" and i need to use this with ESS Splunk app

rec_type=112 rec_type_simple=POLICY event_sec=1394827933 policy_sensor="Defense Center" policy_event_id=122385 corr_policy="SNMP Y SYSLOG" corr_rule="DUMP TO SYSLOG" priority=0 description="" event_type="Intrusion Event" sensor=192.168.9.42 msg="" sid=26583 gid=1 tv_sec=1394827933 tv_usec=447702 event_id=67506 defined_mask=34059 impact=1 impact_bits=64 ip_proto=UDP net_protocol=0 src_ip=10.150.1.40 src_host_type=Host src_vlan_id=0 src_os_name=Unknown src_os_vendor=Unknown src_os_ver=Unknown src_criticality=None src_user=Unknown src_port=53005 src_app_proto=Unknown dest_ip=192.42.93.30 dest_host_type=Host dest_vlan_id=0 dest_os_name=Unknown dest_os_vendor=Unknown dest_os_ver=Unknown dest_criticality=None dest_user=Unknown dest_port=53 dest_app_proto=Unknown blocked=Yes iface_ingress=s1p5 iface_egress=s1p6 sec_zone_ingress="AS - Lan_Int" sec_zone_egress="AS - Lan_Ext"

rec_type=112 rec_type_simple=POLICY event_sec=1394827933 policy_sensor="Defense Center" policy_event_id=122386 corr_policy="SNMP Y SYSLOG" corr_rule="DUMP TO SYSLOG" priority=0 description="" event_type="Intrusion Event" sensor=192.168.9.42 msg="" sid=26583 gid=1 tv_sec=1394827932 tv_usec=974524 event_id=67505 defined_mask=34059 impact=1 impact_bits=65 ip_proto=UDP net_protocol=0 src_ip=192.168.9.190 src_host_type=Host src_vlan_id=0 src_os_name=Unknown src_os_vendor=Unknown src_os_ver=Unknown src_criticality=None src_user=Unknown src_port=62165 src_app_proto=Unknown dest_ip=192.58.128.30 dest_host_type=Host dest_vlan_id=0 dest_os_name=Unknown dest_os_vendor=Unknown dest_os_ver=Unknown dest_criticality=None dest_user=Unknown dest_port=53 dest_app_proto=Unknown blocked=Yes iface_ingress=s1p5 iface_egress=s1p6 sec_zone_ingress="AS - Lan_Int" sec_zone_egress="AS - Lan_Ext"

rec_type=400 rec_type_simple="IPS EVENT" event_sec=1394827935 event_usec=447631 sensor=192.168.9.42 event_id=59190 msg="BLACKLIST DNS request for known malware domain msnsolution.nicaze.net - Genome Trojan" sid=26583 gid=1 rev=1 class_desc="A Network Trojan was Detected" class=trojan-activity priority=high src_ip=10.150.1.40 dest_ip=192.35.51.30 src_port=58993 dest_port=53 ip_proto=UDP impact_bits=64 impact=1 blocked=Yes mpls_label=0 vlan_id=0 ids_policy=Agrosuper user=Unknown web_app=Unknown client_app=Unknown app_proto=Unknown fw_rule="Default Action" fw_policy=Agrosuper iface_ingress=s1p5 iface_egress=s1p6 sec_zone_ingress="AS - Lan_Int" sec_zone_egress="AS - Lan_Ext" connection_second=1394827935 connection_instance_id=1 connection_counter=36231 src_ip_country=unknown dest_ip_country="united states"

rec_type=2 rec_type_simple=PACKET event_sec=1394827935 sensor=192.168.9.42 event_id=59190 packet_sec=1394827935 packet_usec=447631 link_type=1 packet_len=82 packet=9f6223538fd4060052000000520000000010dbff207000181965a1bf0800450000444b6d00007a11f63c0a960128c023331ee6710035003035bb88dc000000010000000000000b6d736e736f6c7574696f6e066e6963617a65036e65740000010001

rec_type=112 rec_type_simple=POLICY event_sec=1394827935 policy_sensor="Defense Center" policy_event_id=122387 corr_policy="SNMP Y SYSLOG" corr_rule="DUMP TO SYSLOG" priority=0 description="" event_type="Intrusion Event" sensor=192.168.9.42 msg="" sid=26583 gid=1 tv_sec=1394827935 tv_usec=447631 event_id=59190 defined_mask=34059 impact=1 impact_bits=64 ip_proto=UDP net_protocol=0 src_ip=10.150.1.40 src_host_type=Host src_vlan_id=0 src_os_name=Unknown src_os_vendor=Unknown src_os_ver=Unknown src_criticality=None src_user=Unknown src_port=58993 src_app_proto=Unknown dest_ip=192.35.51.30 dest_host_type=Host dest_vlan_id=0 dest_os_name=Unknown dest_os_vendor=Unknown dest_os_ver=Unknown dest_criticality=None dest_user=Unknown dest_port=53 dest_app_proto=Unknown blocked=Yes iface_ingress=s1p5 iface_egress=s1p6 sec_zone_ingress="AS - Lan_Int" sec_zone_egress="AS - Lan_Ext"

rec_type=400 rec_type_simple="IPS EVENT" event_sec=1394827936 event_usec=38701 sensor=192.168.9.42 event_id=185263 msg=HI_CLIENT_DOUBLE_DECODE sid=2 gid=119 rev=1 class_desc="Not Suspicious Traffic" class=not-suspicious priority=low src_ip=192.168.1.229 dest_ip=200.143.16.5 src_port=58387 dest_port=80 ip_proto=TCP impact_bits=0 impact=0 blocked=No mpls_label=0 vlan_id=0 ids_policy=Agrosuper user=Unknown web_app=Unknown client_app=Unknown app_proto=Unknown fw_rule="Default Action" fw_policy=Agrosuper iface_ingress=s1p5 iface_egress=N/A sec_zone_ingress="AS - Lan_Int" sec_zone_egress=N/A connection_second=1394827930 connection_instance_id=3 connection_counter=1946 src_ip_country=unknown dest_ip_country=brazil

rec_type=2 rec_type_simple=PACKET event_sec=1394827936 sensor=192.168.9.42 event_id=185263 packet_sec=1394827936 packet_usec=38701 link_type=1 packet_len=986 packet=a06223532d970000da030000da0300000010dbff207000181965a1bf0800450003cc425440007f061ab6c0a801e5c88f1005e4130050ce86300e351b5db950184029e4040000474554202f7363726970742f3f64633d353535303030303231353b6f72643d3831363036393334393b737263747970653d6966726d3b636c69636b323d687474703a2f2f6164636c69636b2e672e646f75626c65636c69636b2e6e65742f61636c6b2532353346736125323533444c253235323661692532353344424b4574506e32496a55387a4a4d4d48666c6766517859434141746a5f2d714145414141414541456741446741574d6a6b6d7553614157435a38632d42684157434152646a59533177645749744e4445314e5451354e7a4d784e4455334e5441354d37494244486433647935305a584a795953356a624c6f424357646d634639706257466e5a63674243646f4248476830644841364c7939336433637564475679636d457559327776634739796447466b59532d5941764369424d414341754143414f6f43477a45794d5445765932777564475679636d4575614739745a5842685a325576614739745a666743676449656b414f63424a674434414f6f417748674241476742685925323532366e756d25323533443025323532367369672532353344414f4436345f334c2d586d427453724b347a497a36554539372d57646e4d705061672532353236636c69656e74253235334463612d7075622d343135353439373331343537353039332532353236616475726c253235334420485454502f312e310d0a4163636570743a20746578742f68746d6c2c206170706c69636174696f6e2f7868746d6c2b786d6c2c202a2f2a0d0a526566657265723a20687474703a2f2f7777772e74657272612e636c2f676c6f62616c5354415449432f61746d2f332f636f72652f5f74706c2f61646d616e616765722e68746d6c0d0a4163636570742d4c616e67756167653a2065732d434c0d0a557365722d4167656e743a204d6f7a696c6c612f352e302028636f6d70617469626c653b204d5349452031302e303b2057696e646f7773204e5420362e313b2054726964656e742f362e30290d0a4163636570742d456e636f64696e673a20677a69702c206465666c6174650d0a486f73743a20742e64796e61642e6e65740d0a444e543a20310d0a436f6e6e656374696f6e3a204b6565702d416c6976650d0a436f6f6b69653a207569643d313339323033343730353137362e64667474636b627230322e313339313533363437373233363b206463723d31322d353535303030303231352d343030303030313234342e302d353030303030303330322d312d313339333530323939303536360d0a0d0a

rec_type=112 rec_type_simple=POLICY event_sec=1394827936 policy_sensor="Defense Center" policy_event_id=122388 corr_policy="SNMP Y SYSLOG" corr_rule="DUMP TO SYSLOG" priority=0 description="" event_type="Intrusion Event" sensor=192.168.9.42 msg="" sid=2 gid=119 tv_sec=1394827936 tv_usec=38701 event_id=185263 defined_mask=34059 impact=0 impact_bits=0 ip_proto=TCP net_protocol=0 src_ip=192.168.1.229 src_host_type=Host src_vlan_id=0 src_os_name=Unknown src_os_vendor=Unknown src_os_ver=Unknown src_criticality=None src_user=Unknown src_port=58387 src_app_proto=Unknown dest_ip=200.143.16.5 dest_host_type=Host dest_vlan_id=0 dest_os_name=Unknown dest_os_vendor=Unknown dest_os_ver=Unknown dest_criticality=None dest_user=Unknown dest_port=80 dest_app_proto=Unknown blocked=No iface_ingress=s1p5 iface_egress=N/A sec_zone_ingress="AS - Lan_Int" sec_zone_egress=N/A

rec_type=400 rec_type_simple="IPS EVENT" event_sec=1394827939 event_usec=436017 sensor=192.168.9.42 event_id=59192 msg="BLACKLIST DNS request for known malware domain msnsolution.nicaze.net - Genome Trojan" sid=26583 gid=1 rev=1 class_desc="A Network Trojan was Detected" class=trojan-activity priority=high src_ip=192.168.9.190 dest_ip=192.5.5.241 src_port=60461 dest_port=53 ip_proto=UDP impact_bits=65 impact=1 blocked=Yes mpls_label=0 vlan_id=0 ids_policy=Agrosuper user=Unknown web_app=Unknown client_app=Unknown app_proto=Unknown fw_rule="Default Action" fw_policy=Agrosuper iface_ingress=s1p5 iface_egress=s1p6 sec_zone_ingress="AS - Lan_Int" sec_zone_egress="AS - Lan_Ext" connection_second=1394827939 connection_instance_id=1 connection_counter=36579 src_ip_country=unknown dest_ip_country="united states"

rec_type=2 rec_type_simple=PACKET event_sec=1394827939 sensor=192.168.9.42 event_id=59192 packet_sec=1394827939 packet_usec=436017 link_type=1 packet_len=82 packet=a362235331a7060052000000520000000010dbff207000181965a1bf08004500004413f000007f11975cc0a809bec00505f1ec2d003500301f050879000000010000000000000b6d736e736f6c7574696f6e066e6963617a65036e65740000010001

my props from "estreamer" app:

[source::eStreamer]
SHOULD_LINEMERGE = false
LINE_BREAKER = \n
TRUNCATE = 0
TIME_PREFIX = event_sec=

THANKS YOU!!!!

Tags (1)
0 Karma

lguinn2
Legend

Splunk is complaining because of this line in your props.conf

LINE_BREAKER = \n

LINE_BREAKER must have a capture group as defined in the documentation "Index multi-line events" and props.conf.spec.

I suggest either

LINE_BREAKER=(\n+)

or

LINE_BREAKER=([\r\n]+)  # this is the default

Actually, if your data should be parsed as "one line per event", you can leave out the LINE_BREAKER in your props.conf. Then Splunk will use the default.

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...