topic Re: Universal Forwarder merges multiple events into one in Getting Data In

Universal Forwarder merges multiple events into one

sansri7680 — Mon, 28 Sep 2020 13:35:00 GMT

I have a file which is monitored by the Universal forwarder in Windows box. I installed the forwarder on windows using the msi installer. I have data coming into the file regularly and the format is as below

INBOUND>>>>> 19:00:17:308 Eventid:153001(3)
NAS Rx PDU, from 10.10.11.36:36412 to 10.10.11.226:36412 (42)

Non Access Stratum (NAS) (42 bytes)
EPS Mobility Management
Protocol Discriminator
EPS MOBILITY MANAGEMENT MESSAGES(0x7)
Security Header Type
NAS_MSG_SECURITY_HDR_PLAIN_NAS_MSG(0x0)
Message Type
ATTACH_REQUEST(0x41)
Attach Type
EPS ATTACH(0x1)
Key Set Identifier
NO KEY AVAILABLE(0x7)
Mobile Identity
IMSI (240010000099935)
UE n/w capability
(0xe0e0)
ESM CONTAINER
EPS Session Management
Protocol Discriminator
EPS SESSION MANAGEMENT MESSAGES(0x2)
EPS Bearer Id
(0x0)
Transaction Id
(0x1)
Message Type
PDN_CONNECTIVITY_REQUEST(0xd0)
Request Type
INITIAL REQUEST(0x1)
PDN Type
IPv4(0x1)
Protocol Config Options
Configuration Protocol:
PPP
Proto/Container ID:
IPCP
Contents:0x0100000A810600000000
MS n/w capability
(0xc540f4)

<<<<OUTBOUND 19:00:17:730 Eventid:153002(3)
NAS Tx PDU, from 10.10.11.226:36412 to 10.10.11.36:36412 (36)

Non Access Stratum (NAS) (36 bytes)
EPS Mobility Management
Protocol Discriminator
EPS MOBILITY MANAGEMENT MESSAGES(0x7)
Security Header Type
NAS_MSG_SECURITY_HDR_PLAIN_NAS_MSG(0x0)
Message Type
AUTHENTICATION_REQUEST(0x52)
Key Set Identifier
Security Context Type: Native (0x0)
Key Set Index: (0x6)
Spare-Half
(0x0)
RAND
(0x81d97b15a0aa040081d97b15a0aa0400)
AUTN
(0x18f97648c158fffe445a366fe14f1160)

<<<<OUTBOUND 19:00:17:730 Eventid:155213(3)
S1AP Tx PDU, from 10.10.11.226:36412 to 10.10.11.36:36412 (62)

S1 Application Part (S1AP) (62 bytes)
| 0... .... | Ext bit : 0
| .00. .... | Choice index : Initiating Message (0)
Procedure Code : DOWNLINK NAS TRANSPORT (11)
Criticality
| 01.. .... | Ignore (1)
DOWNLINK NAS TRANSPORT Value :
| .011 1010 | Length Determinant : 58
Value :
| 0... .... | Ext bit : 0
IEs Count : 3
IE : 1
Protocol IE ID : MME_UE_S1AP_ID (0)
Criticality
| 00.. .... | Reject (0)
MME_UE_S1AP_ID Value :
| .000 0100 | Length Determinant : 4
Value :
| 10.. .... | Length Determinant : 3
12582917 (0xc00005)
IE : 2
Protocol IE ID : eNB_UE_S1AP_ID (8)
Criticality
| 00.. .... | Reject (0)
eNB_UE_S1AP_ID Value :
| .000 0010 | Length Determinant : 2
Value :
| 00.. .... | Length Determinant : 1
13 (0x0d)
IE : 3
Protocol IE ID : NAS_PDU (26)
Criticality
| 00.. .... | Reject (0)
NAS_PDU Value :
| .010 0101 | Length Determinant : 37
Value :
| .010 0100 | Length Determinant : 36
0x07520681d97b15a0aa040081d97b15a0aa04001018f97648c158fffe445a366fe14f1160
EPS Mobility Management
Protocol Discriminator
EPS MOBILITY MANAGEMENT MESSAGES(0x7)
Security Header Type
NAS_MSG_SECURITY_HDR_PLAIN_NAS_MSG(0x0)
Message Type
AUTHENTICATION_REQUEST(0x52)
Key Set Identifier
Security Context Type: Native (0x0)
Key Set Index: (0x6)
Spare-Half
(0x0)
RAND
(0x81d97b15a0aa040081d97b15a0aa0400)
AUTN
(0x18f97648c158fffe445a366fe14f1160)

My props.conf on my UNIX box is as below
[4GCDR]
BREAK_ONLY_BEFORE = (.*)(INBOUND>>>>>|<<<<OUTBOUND)
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
pulldown_type = 1
TRUNCATE=0

I created a UDP input to monitor the port where the windows forwarder sends the data

But whenever more than one event occurs in the monitored file all the events are merged into a single event. If events are updated one by one there are no problems. Can someone please help

Re: Universal Forwarder merges multiple events into one

aholzer — Fri, 22 Mar 2013 13:57:08 GMT

Your break only before param should look like this:

BREAK_ONLY_BEFORE = ^(INBOUND>>>>>|<<<<OUTBOUND)

Also for future reference you should probably use the data preview to generate the props.conf with test data. Go to Manager > Data inputs > Files & directories > new, upload the file and go into advanced settings. Make sure the events are being parsed out the way you want them to and then you can copy the parameters you have created into your existing props.conf.

Re: Universal Forwarder merges multiple events into one

sansri7680 — Fri, 22 Mar 2013 14:05:50 GMT

The props.conf settings that I used was fine if I upload a single file. But it is not working with the Universal forwarder

Re: Universal Forwarder merges multiple events into one

Ayn — Fri, 22 Mar 2013 14:08:55 GMT

Universal Forwarders do not perform event breaking, and so you should put these settings on the indexer, not the forwarder.

Re: Universal Forwarder merges multiple events into one

sansri7680 — Mon, 25 Mar 2013 13:47:03 GMT

Hi Ayn

Can you tell me how to configure the settings in the indexer

Re: Universal Forwarder merges multiple events into one

aholzer — Mon, 25 Mar 2013 15:47:09 GMT

Just copy the props.conf that you posted into your indexer instead of your forwarder. Then restart your indexer

Re: Universal Forwarder merges multiple events into one

sansri7680 — Tue, 26 Mar 2013 06:51:12 GMT

I tried that also. But no events are coming into splunk now

Re: Universal Forwarder merges multiple events into one

aholzer — Tue, 26 Mar 2013 14:15:02 GMT

We are going to need a bit more information if you want us to help you.

Check your splunkd.log under $SPLUNK_HOME/var/log/splunk. Are there any error messages? Where did you save your props.conf? Have you defined your inputs.conf on both your forwarder and indexer properly? Have you defined your outputs.conf on your forwarder?

Make sure you use the same sourcetype in your inputs.conf on your forwarder, and on the stanza header of your props.conf. Your sourcetype should be "4GCDR" based on what you put in the props.conf example you gave.