Getting Data In

Splunk indexer splits multiline events on all lines for forwarded data

Path Finder

Hi

I have a Universal forwarder forwarding data from a monitored file on Windows. This file contains multiline events. The splunk instance is available in the Linux machine and is able to receive the data on port 9997. But the indexer doesn't seem to split this data into multiline events. Rather it is being split at all the lines. I have the regex configured in the props.conf in the $SPLUNKHOME/etc/apps/search/local folder as well as $SPLINKHOME/etc/system/local/props.conf. But it doesn't seem to have any effect on the data. Also the same regex works if I create an input on a local file on the linux box but doesn't seem to work for forwarded data. The data is coming into _main index without any problems but not splitting properly. Any inputs on this is highly appreciated. I am posting the regex as below

[4GCDR]
BREAKONLYBEFORE = (.*)(Inbound>>>>>|<<<<Outbound)
NOBINARYCHECK = 1
SHOULD_LINEMERGE = True
TRUNCATE=0

The events look like below
INBOUND>>>>> 19:00:45:775 Eventid:155212(3)
S1AP Rx PDU, from 10.10.11.36:36412 to 10.10.11.226:36412 (21)

S1 Application Part (S1AP) (21 bytes)
| 0... .... | Ext bit : 0
| .01. .... | Choice index : Successful Outcome (1)
Procedure Code : UE CONTEXT RELEASE (23)
Criticality
| 00.. .... | Reject (0)
UE CONTEXT RELEASE Value :
| .001 0001 | Length Determinant : 17
Value :
| 0... .... | Ext bit : 0
IEs Count : 2
IE : 1
Protocol IE ID : MMEUES1APID (0)
Criticality
| 01.. .... | Ignore (1)
MME
UES1APID Value :
| .000 0100 | Length Determinant : 4
Value :
| 10.. .... | Length Determinant : 3
10485764 (0xa00004)
IE : 2
Protocol IE ID : eNBUES1APID (8)
Criticality
| 01.. .... | Ignore (1)
eNB
UES1APID Value :
| .000 0010 | Length Determinant : 2
Value :
| 00.. .... | Length Determinant : 1
14 (0x0e)

INBOUND>>>>> 19:00:45:577 Eventid:141004(3)
[MME-S11]GTPv2C Rx PDU, from 10.10.10.246:2123 to 10.10.10.245:30160 (18)
TEID: 0x8003400A, Message type: EGTPDELETESESSION_RESPONSE (0x25)
Sequence Number: 0x00800A (32778)
GTP HEADER
Version number: 2
TEID flag: Present
Piggybacking flag: Not present
Message Length: 0x000E (14)

INFORMATION ELEMENTS
CAUSE:
Type: 2 Length: 2 Inst: 0
Value:
Cause: EGTPCAUSEREQ_ACCEPTED (0x10)
OI: 0
Hex: 0200 0200 1000

0 Karma

Ultra Champion

You know that the props.conf settings should go on the indexer, right? Not on the forwarder.
Or at least in part. NO_BINARY_CHECK should be on the forwarder, not the indexer.

props.conf especially is one of those files which could/should be present in several parts of the chain, but depending on which phase the data is in, only some props parameters will be read by that Splunk instance. See the following;

http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings


UPDATE:
You do realize that Inbound != INBOUND, right (difference between regex and the event data)

If that was just a typo, I'd say it's interesting. I'd probably do/try two things - even though I'm not sure they would work differently (since you say this works for locally indexed files). These two would be to a) Break the events differently and b) specify the timestamp better;

props.conf

[your_sourcetype]
SHOULD_LINEMERGE=false
TRUNCATE=0
LINE_BREAKER = ([\r\n]+)(?:INBOUND>>>>>|<<<<OUTBOUND)
TIME_FORMAT = %H:%M:%S:%3N
TIME_PREFIX = \s
MAX_TIMESTAMP_LOOKAHEAD = 15

Hope this helps,

Kristian

Ultra Champion

As for reindexing events, it depends a bit on how you got it in in the first place, but you should look into clearing out the fishbucket.

That is where Splunk stores the information on which file it has already read, and how far into the files they have come.

See the docs for more info. Search for "what is the fishbucket"

/K

0 Karma

Ultra Champion

Not entirely sure what you mean, but the only operation that requires a (big) license is indexing. So if you have a full splunk instance working as a HF, it does not do any indexing, and thus will not need a 'real' license.

There is (used to be?) a forwarder license that you might need/want to copy/install on the HF instead of the Download_Trial license. That license should be included in the download package. See the docs for more info.

Also, should probably mention that Universal Forwarder is preferred over configuring a full Splunk as a Lightweight Forwarder.

Good luck,

K

0 Karma

Path Finder

but it will require a separate full licensing if deployed in production environment right? My requirement is that these kind of logs will be in various machines and I will have light forwarder installed in all the machines to forward log data to one Splunk instance which will be indexed there. I even tried to reindex the data after it enters the splunk instance using "windows eventlog timefix" app but even that doesn't work

0 Karma

Ultra Champion

If all else fails, you could always install a Heavy Forwarder (HF) instead of a Universal Forwarder (UF) on the machine that generates the log. Since you say that locally indexed files with this event format work well in terms of event breaking.

A HF is just a full Splunk instance, which forwards events instead of indexing them locally (this can be configured in the GUI on the HF). But prior to forwarding, a HF will take the data through the Parsing Phase, which includes breaking the data stream into separate events.

You'd need to transfer the props.conf settings to the HF

/k

Ultra Champion

The splunk version should not matter, and the free download works fine for forwarding/receiving.

Path Finder

The config doesn't seem to have any effect. Also is there anything to do with the Splunk version that I am using? Does this forwarding and indexing thing work on the free version that is available for download in the Splunk site?

0 Karma

Ultra Champion

Choose one of the folders etc/system/local or /etc/apps/search/local. Either should be fine.

Also see the update above.

/K

Path Finder

Hi Kristian

The props.conf file is very much in the indexer side. I have put it in the $SPLUNKHOME/etc/apps/search/local folder in the main splunk instance. I have juggled with moving the NOBINARY_CHECK to the forwarder section too but no luck. Can you suggest if I am going wrong anywhere in the config. Whether I am putting the props.conf in the wrong folder of the splunk instance

0 Karma