topic Re: Linebreaking input in Getting Data In

Linebreaking input

timmalos — Tue, 30 Jul 2013 07:10:00 GMT

Hi
Im sorry to disturb you but cant manage to solve my problem. Got Inputs like that :

Titlis,NetBackup Client Service,0,Auto,OK,0 
Titlis,NetBackup Compatibility Service,0,Auto,OK,0 
Titlis,NetBackup Remote Manager and Monitor Service,0,Auto,OK,0 
Titlis,NetBackup Service Layer,0,Auto,OK,0 
Weisshorn,NetBackup Service Layer,0,Auto,OK,0 
Weisshorn,NetBackup Service Monitor,0,Auto,OK,0 
Weisshorn,NetBackup Volume Manager,0,Auto,OK,0

I want each line to be an event, with the timestamp of the modified file.
Here is my props.conf

[NbService]
DATETIME_CONFIG = NONE
SHOULD_LINEMERGE=false
EXTRACT-NbService =^(?P<server>[^,]+),(?P<serviceName>[^,]+),,?(?P<serviceState>[^,]+),(?P<servicePolicy>[^,]+),(?P<serviceStatus>[^,]+),(?P<service>[^,]+)$

[\\matterhorn\Netbackup4Splunk\OUT_services.txt]
CHECK_METHOD = modtime

And my inputs.conf

[monitor://\\matterhorn\Netbackup4Splunk\OUT_services.txt]
disabled = 0
followTail = 0
sourcetype = NbService
index = Infra_NB
host = Matterhorn

Tried other params but every time i got only one event with all the lines merged. Other inputs who got Timestamp work perfectly.
Thks for ur help

Re: Linebreaking input

mloven_splunk — Tue, 30 Jul 2013 14:42:42 GMT

ok, a couple things I'm seeing here. Not sure if any of them will actually fix your problem.

In your inputs.conf, your monitor statement looks weird. Is this a Windows system? If so, I'd expect to see something like:

[monitor://C:\blah]
Also, in your props.conf, I'd again expect to see a drive letter, but also, I think you meant to put:

[source::C:\blah]
Finally, and this definitely doesn't have anything to do with the issue, in your EXTRACT statement, you have:

...(?P[^,]+),,?...

but I think you should probably have:

...(?P<serviceName>[^,]+),?...

You had an extra comma in there. That may have just been a typo in your writeup though.

Otherwise, I don't really see why what you have wouldn't work.

I suppose you could also try specifying a line breaker. Something like:

LINE_BREAKER = ([\r\n]+)(?\w+,)

Re: Linebreaking input

Gilberto_Castil — Tue, 30 Jul 2013 14:52:33 GMT

There are two parts to the answer to your question:

List breaker
Date of Event

Line Breaker:

The primordial, inherent line breaker in Splunk is a time stamp. If the events in your data do not have a time stamp, then you must tell Splunk how to break the events. There are multiple methods for this and the following works well in your case.

#inputs.conf
[monitor:///tests/answers/7-30-2013/1/data]
disabled = false
sourcetype = answers-1375192607
index = test

#props.conf
[answers-1375192607]
SHOULD_LINEMERGE = False
LINE_BREAKER = ([\r\n]+)[A-Z][a-z]+,

This will break the events when a line return is found and a full word capitalized, followed by a comma.

Date of Event

Notice that the date and time for all of the events is reflected by the file modification time. That is: the time when the file was last updated.

[gcastill0@sandbox 1]# pwd
/media/answers/7-30-2013/1
[gcastill0@sandbox 1]# ls -ltr
total 4
-rw-r--r-- 1 root root 350 Jul 30 09:54 data

All of the events, therefore, inherit this time stamp.

Any subsequent event additions to the file will reflect the file modification time. For instance, we append an additional entry, like this one

Gcastill0,NetBackup Volume Manager,0,Auto,OK,0

to the end of your data, you see the following:

... Which reflects the file modification time.

[gcastill0@sandbox 1]# pwd
/media/answers/7-30-2013/1
[gcastill0@sandbox 1]# ls -ltr
total 4
-rw-r--r-- 1 root root 396 Jul 30 10:36 data

There is a school of thought about being able to extract the date of an event using datetime.xml -where you look at the file name and extract the data. Before you consider that, please note that the time of day piece is not inherited from the field extractions. Time of day is obtained from the event (index time) and/or from the file modification time.

In other words, the suggestions above are your best option to obtain a precise date and time for the events going forward. Anything historical will inherit the date and time of the first-time index process.

I hope this helps,

--gc

Re: Linebreaking input

timmalos — Tue, 30 Jul 2013 15:10:32 GMT

Its a Windows server, the file is on a distant server so my \server is working pretty well
I forgot the source:: , thks ! Working now (Another problem i didnt mention solved)
Actually this was good cause there was a mistake in the first logs i had where there was 2 commas. But they fixed it, so the ,? said that there could be or not a comma here , depending on the version of the log. With ^and$ was ok. To finish. its working now, dont know why cause i didnt change nothing. Maybe a lag on my server. Thks for your help !

Re: Linebreaking input

timmalos — Mon, 28 Sep 2020 14:27:42 GMT

I delete the file each time before new datas, no probleé with the modification time. Actually its working now, i dont really know what i modified but its ok... Thks for your help.

props.conf

[source::NbServices.txt]
CHECK_METHOD = modtime
DATETIME_CONFIG = NONE
SHOULD_LINEMERGE = false
[NbServices]
EXTRACT-NbService = ^(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+),(?P[^,]+)$

inputs.conf

[default]
initCrcLength = 2048
NO_BINARY_CHECK = true

Will put ur answer as correct for the time u spent 🙂