Getting Data In

Avoid header at Index time and extract field during search

emaccaferri
Communicator

Hi,

I'm trying to not have in my events the header of a txt file separated by tab (I suppose it's at index time), after that I extract field at search time. The problem is that everything I tried left the header in the event.
Any key?

Here my code in props.conf and transfomrs.conf:

props.conf

[mysourcetype]
NO_BINARY_CHECK = 1
pulldown_type = 1
TRANSFORMS-NoHeader = NoHeader
REPORT -mysourcetypeextract = extract_mysourcetype

transforms.conf

 [extract_mysourcetype]
 DELIMS = "\t"
 FIELDS = personid, email,  createdate, time_window_web

[NoHeader]
REGEX = ^"Personid".*
DEST_KEY = queue
FORMAT = nullQueue

and the first lines of my log file

"Personid"  "Email" "Createdate"    "Time Window Web"   
"0000000"   "xxxxxx@xxxx.xxx"   "00/00/1900"    "00:00"

I really don't know what to do, I've tried all the answer on the site 🙂

Thank you for the help
Erica

0 Karma
1 Solution

_d_
Splunk Employee
Splunk Employee

This seems to work:

[NoHeader]
REGEX = ^"Personid"
DEST_KEY = queue
FORMAT = nullQueue

and also your original:

[NoHeader]
REGEX = ^"Personid".*
DEST_KEY = queue
FORMAT = nullQueue

Make sure you're doing this on an indexer (or heavy forwarder) and restart after change.

View solution in original post

ogdin
Splunk Employee
Splunk Employee

When you use Header-based Index-time field extractions, the header doesn't get indexed but we use it to map the field:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileheadersatindextime

Try in props.conf:


FIELD_DELIMITER=space
FIELD_QUOTE="
HEADER_FIELD_DELIMITER=space
HEADER_FIELD_QUOTE="

You will need to play with the settings a bit if tabs separate the header or events but this is way better than stripping the header as it skips the need for doing search-time field extraction.

I should note this also only works on Splunk 6.

0 Karma

_d_
Splunk Employee
Splunk Employee

This seems to work:

[NoHeader]
REGEX = ^"Personid"
DEST_KEY = queue
FORMAT = nullQueue

and also your original:

[NoHeader]
REGEX = ^"Personid".*
DEST_KEY = queue
FORMAT = nullQueue

Make sure you're doing this on an indexer (or heavy forwarder) and restart after change.

View solution in original post

emaccaferri
Communicator

Yes, it was working, but testing on the same file I had to cancel all the index and the data

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.