I have issue with index field which contain comma. Below is my csv input
"28650096","2013-12-02 20:30:30","blocked","porn\, sexual content","email@example.com","18.104.22.168"
"28650093","2013-12-02 20:30:30","allow","search site","firstname.lastname@example.org","22.214.171.124"
FIELDDELIMITER = ,
INDEXEDEXTRACTIONS = csv
KVMODE = none
NOBINARYCHECK = 1
REPORT-audit = temp-audit-csv
SHOULDLINEMERGE = false
pulldown_type = 1
When add data using A file or directory of files it can see three events without problem. But after done adding data when in search when I do "search *" it only return 2 events it seem the first one didn't make it to the search.
I could see you have a "," in the extracted field value. So it's better not to confuse splunk and keep it simple. So just let splunk decide all the things. Below config should work for you.
[temp-audit] KV_MODE = none NO_BINARY_CHECK = 1 REPORT-audit = temp-audit-csv SHOULD_LINEMERGE = false pulldown_type = 1 [temp-audit-csv] DELIMS="," FIELDS="id","timeStamp","Type","Reason","email","SourceIP"
I tried the above solution and it does not work. I believe transforms.conf is for csv column mapping. This is at indexing time which happen before transforms.csv. Anyhow here is what I did:
splunk clean all
add data source and index it. Of course at this time the transforms.conf does not exist yet. After index only two events showed up.
//add transforms.conf and modified props.conf with recommended above solutions
Perform search and still only two events showed up.
Do I need to re-index, if yes how do i do that
yes you need to re-index it. You need to clear out the fishbucket from your universal forwarder. Please follow the link
or you can just change the file name it will re-index it.
INDEXED_EXTRACTIONS is attempting to use the first line as the header (the column/field names). That is why you are only seeing 3 events. You can use DELIMS here but in Splunk 6 we introduced:
INDEXED_EXTRACTIONS = csv
specifically so you don't have to define the fields in DELIMS. We attempt to automatically read the first line of the CSV (usually the header) and create index-time fields.
So, if the file has no header, you can use INDEXED_EXTRACTIONS = csv with the
OR use props/transforms with DELIMS but you cannot mix the two.