Splunk Search

Why is my configuration for index-time field extractions of CSV data not working?

New Member

I have a csv file , which is delimited by ~ character .I am trying to do an index time field extraction so that My fields are extracted and ready to go by the time ingestion completes .

The CSV file does not contain a header with the field layout details , so a header based extraction is not feasible .

Here is what I have tried so far .

UF - Inputs.conf
[monitor:///splunk/SFT_Test/scrubbed_test/informatica.disclosure.data.infile*]

index = ifind_test

sourcetype = tr_disclosure_table

On my Indexer :

Props.conf

[tr_disclosure_table]
TIME_PREFIX = ^

MAX_TIMESTAMP_LOOKAHEAD = 22

TIME_FORMAT = %m/%d/%Y %I:%M:%S %p

TRUNCATE = 99999

SHOULD_LINEMERGE = false

LINE_BREAKER = ([\r\n]+)(?=\d{2}/\d{2}/\d{4}\s\d{1,2}:\d{2}:\d{2}\s[A-Z]{2})

TRANSFORMS-csvextract = disclosure_file_extract

I tried REPORT also instead of TRANSFORMS in the above stanza to try and get it to work .

[disclosure_file_extract]
DELIMS="~"
FIELDS=DATE~RECORD_INDICATOR~TRI_INTR_ACC_NO~PLSTC_CARD_NO~DISCLO_CD~NTFY_ID~ACCT_CYC_DAY_CD~DCSN_EVENT_DT~GNRC_EXTNL_NO~DISCLO_DS~CM_ANNIV_YEAR~DISCLO_PACK_CD~DISCLO_PACK_DS_TX~DISCLO_MODL_CD~DISCLO_TYPE_CD~DISCLO_ATTR_SUB_TYPE_VAL_CD~DISCLO_STRT_DT~DISCLO_END_DT~NTFY_DT~EFF_DT~RESP_BY_DT~REJCT_IN~AXIOM_KEY_ID~REQ_COMMUN_ID~ACCT_STA_CD~PLYR_ACCNT_NO~PROD_NO_CD~IA_PROD_ID_CD~TERM_VAL_CHNG_FROM_TX~TERM_VAL_CHNG_TO_TX~COMMUN_CHAN_TYPE_ID~ACCT_OPEN_DT~INIT_MAIL_DT~RSLT_ACT_IN~RSLT_ACT_DS~FEATR_SRVC_CD~FEATR_SRVC_CD_DS~APLCBL_TO_CANCELED_ACCT_IN~APLCBL_TO_ALL_PROD_IN~APLCBL_TO_PYMT_PROG_IN~APLCBL_TO_SRVC_MBR_CR_ENRL_IN~APLCBL_TO_DSTR_RELF_IN~SYS_TOKEN_CD~SYS_OF_REC_NM~TERM_CHNG_STA_CD~TERM_CHNG_STA_RSN_CD~TERM_CHNG_RSCND_IN~LAST_TERM_STA_CHNG_DT~RCPT_DT~ADD_TS~LST_UPDT_TS

What am I doing wrong and how can I get the index time extraction to work . (Since I don't want to do a search time extraction using the configuration files on SH ) .

0 Karma

SplunkTrust
SplunkTrust

DELIMS is only valid for search-time field extraction, see docs.splunk.com/Documentation/Splunk/6.2.0/Admin/transformsconf for reference. Additionally, I'm pretty sure you should comma-separate the FIELDS setting even if your DELIMS are not commas.

What's the motivation for not using search-time field extraction?