Archive
Highlighted

Limit the index of a txt file to only last xxx lines

New Member

I want to forward the data of a Alarm.txt file using splunk forwarder but limit the index to only last xxx lines of the file.

The problem is my txt file has Day Month date and time and it does not have year. It will just repeat with 5 years of repeat of the months on this file. Only the end of the file is the most current year information.

Tue Jan 25 11:53:02, Set Alarm
Tue Jan 25 11:53:15, Set Alarm,
Tue Jan 25 12:02:54, Set Alarm,
Wed Feb 02 08:51:07, Set Alarm,
.....
Thu Dec 04 05:59:13, Set Alarm
Tue Jan 25 12:02:54, Set Alarm,

Can someone guide me on what I need to do so that it only extract the last part of the file?

Or if someone has a way for me to assign the correct year on each part of this file that will also work, so that I can index all the data with the correct year information.

Right now Splunk index everything as 2017 on this file.

Tags (1)
0 Karma
Highlighted

Re: Limit the index of a txt file to only last xxx lines

Esteemed Legend

Splunk will automatically assume the year is the current year as long as the month/day is not in the future. As far as only indexing some of the lines, is this a one-time-only forwarding or do you need to start with the last 4 lines now and then forward every new line as it comes in?

0 Karma
Highlighted

Re: Limit the index of a txt file to only last xxx lines

New Member

Start with last few lines now and then forward every new line would be good, this is an alarm log for an equipment and I can work without the history.

0 Karma
Highlighted

Re: Limit the index of a txt file to only last xxx lines

Builder

I don't know if this is what you're looking for, but you could perhaps set the HEADERFIELDLINE_NUMBER in props.conf.

HEADER_FIELD_LINE_NUMBER = integer
- Tells Splunk the line number of the line within the file that contains the header fields.  If set to 0, Splunk attempts to locate the header fields within the file automatically.
- The default value is set to 0.
Highlighted

Re: Limit the index of a txt file to only last xxx lines

New Member

my text file don't have a header, but this is good information to know about. Thanks!

0 Karma
Highlighted

Re: Limit the index of a txt file to only last xxx lines

Builder

I was thinking, if you set HEADERFIELDLINE_NUMBER so that Splunk will start reading the file from the line you want Splunk to read from, reading only the last "xxx" lines, wouldn't that work?

0 Karma
Highlighted

Re: Limit the index of a txt file to only last xxx lines

New Member

Is there a command that is similar to End of File EOF? or do I have to define a line number?

0 Karma
Highlighted

Re: Limit the index of a txt file to only last xxx lines

Esteemed Legend

Initially I thought that this was brilliant but this will only work if he is using INDEXED_EXTRACTIONS , which he probably is not.

0 Karma
Highlighted

Re: Limit the index of a txt file to only last xxx lines

Builder

INDEXEDEXTRACTIONS, why is that? Though, when I look at the props.conf documentation again, I understand that the HEADERFIELDLINENUMBER will have to be configured in props.conf on the instance reading the log file (probably an UF).

0 Karma
Highlighted

Re: Limit the index of a txt file to only last xxx lines

Esteemed Legend

Look again. The HEADER_FIELD_LINE_NUMBER setting is in the INDEXED_EXTRACTIONS section.

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.