Getting Data In

Avoid header at Index time and extract field during search

emaccaferri
Communicator

Hi,

I'm trying to not have in my events the header of a txt file separated by tab (I suppose it's at index time), after that I extract field at search time. The problem is that everything I tried left the header in the event.
Any key?

Here my code in props.conf and transfomrs.conf:

props.conf

[mysourcetype]
NO_BINARY_CHECK = 1
pulldown_type = 1
TRANSFORMS-NoHeader = NoHeader
REPORT -mysourcetypeextract = extract_mysourcetype

transforms.conf

 [extract_mysourcetype]
 DELIMS = "\t"
 FIELDS = personid, email,  createdate, time_window_web

[NoHeader]
REGEX = ^"Personid".*
DEST_KEY = queue
FORMAT = nullQueue

and the first lines of my log file

"Personid"  "Email" "Createdate"    "Time Window Web"   
"0000000"   "xxxxxx@xxxx.xxx"   "00/00/1900"    "00:00"

I really don't know what to do, I've tried all the answer on the site 🙂

Thank you for the help
Erica

0 Karma
1 Solution

_d_
Splunk Employee
Splunk Employee

This seems to work:

[NoHeader]
REGEX = ^"Personid"
DEST_KEY = queue
FORMAT = nullQueue

and also your original:

[NoHeader]
REGEX = ^"Personid".*
DEST_KEY = queue
FORMAT = nullQueue

Make sure you're doing this on an indexer (or heavy forwarder) and restart after change.

View solution in original post

ogdin
Splunk Employee
Splunk Employee

When you use Header-based Index-time field extractions, the header doesn't get indexed but we use it to map the field:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileheadersatindextime

Try in props.conf:


FIELD_DELIMITER=space
FIELD_QUOTE="
HEADER_FIELD_DELIMITER=space
HEADER_FIELD_QUOTE="

You will need to play with the settings a bit if tabs separate the header or events but this is way better than stripping the header as it skips the need for doing search-time field extraction.

I should note this also only works on Splunk 6.

0 Karma

_d_
Splunk Employee
Splunk Employee

This seems to work:

[NoHeader]
REGEX = ^"Personid"
DEST_KEY = queue
FORMAT = nullQueue

and also your original:

[NoHeader]
REGEX = ^"Personid".*
DEST_KEY = queue
FORMAT = nullQueue

Make sure you're doing this on an indexer (or heavy forwarder) and restart after change.

emaccaferri
Communicator

Yes, it was working, but testing on the same file I had to cancel all the index and the data

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...