topic Re: Avoid header at Index time and extract field during search in Getting Data In

Avoid header at Index time and extract field during search

emaccaferri — Tue, 15 Oct 2013 15:54:28 GMT

Hi,

I'm trying to not have in my events the header of a txt file separated by tab (I suppose it's at index time), after that I extract field at search time. The problem is that everything I tried left the header in the event.
Any key?

Here my code in props.conf and transfomrs.conf:

props.conf

[mysourcetype]
NO_BINARY_CHECK = 1
pulldown_type = 1
TRANSFORMS-NoHeader = NoHeader
REPORT -mysourcetypeextract = extract_mysourcetype

transforms.conf

 [extract_mysourcetype]
 DELIMS = "\t"
 FIELDS = personid, email,  createdate, time_window_web

[NoHeader]
REGEX = ^"Personid".*
DEST_KEY = queue
FORMAT = nullQueue

and the first lines of my log file

"Personid"  "Email" "Createdate"    "Time Window Web"   
"0000000"   "xxxxxx@xxxx.xxx"   "00/00/1900"    "00:00"

I really don't know what to do, I've tried all the answer on the site 🙂

Thank you for the help
Erica

Re: Avoid header at Index time and extract field during search

_d_ — Tue, 15 Oct 2013 17:48:31 GMT

This seems to work:

[NoHeader] REGEX = ^"Personid" DEST_KEY = queue FORMAT = nullQueue

and also your original:

[NoHeader] REGEX = ^"Personid".* DEST_KEY = queue FORMAT = nullQueue

Make sure you're doing this on an indexer (or heavy forwarder) and restart after change.

Re: Avoid header at Index time and extract field during search

emaccaferri — Fri, 18 Oct 2013 07:25:38 GMT

Yes, it was working, but testing on the same file I had to cancel all the index and the data

Re: Avoid header at Index time and extract field during search

ogdin — Thu, 13 Feb 2014 16:11:18 GMT

When you use Header-based Index-time field extractions, the header doesn't get indexed but we use it to map the field:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileheadersatindextime

Try in props.conf:

FIELD_DELIMITER=space FIELD_QUOTE=" HEADER_FIELD_DELIMITER=space HEADER_FIELD_QUOTE="

You will need to play with the settings a bit if tabs separate the header or events but this is way better than stripping the header as it skips the need for doing search-time field extraction.

I should note this also only works on Splunk 6.