indexed only 1 CSV file

[thinguy]

Trying to index some radius accounting (.act) files that are really CSV files with a header

"Date","Time","RAS-Client","Record-Type","Full-Name","Auth-Type","User-Name","NAS-IP-Address","NAS-Port","Service-Type","Framed-Protocol","Framed-IP-Address","Framed-IP-Netmask","Framed-Routing","Filter-ID","Framed-MTU","Framed-Compression","Login-IP-Host","Login-Service","Login-TCP-Port","Callback-Number","Callback-ID","Framed-Route","Framed-IPX-Network","Class","Session-Timeout","Idle-Timeout","Termination-Action","Called-Station-ID","Calling-Station-ID","NAS-Identifier","Proxy-State","Acct-Status-Type","Acct-Delay-Time","Acct-Input-Octets","Acct-Output-Octets","Acct-Session-Id","Acct-Authentic","Acct-Session-Time","Acct-Input-Packets","Acct-Output-Packets","Acct-Termination-Cause","Acct-Multi-Session-Id","Acct-Link-Count","NAS-Port-Type","Port-Limit","Tunnel-Type","Tunnel-Medium-Type","Tunnel-Client-Endpoint","Tunnel-Server-Endpoint","Acct-Tunnel-Connection","Tunnel-Private-Group-ID","Tunnel-Assignment-ID","Acct-Tunnel-Packets-Lost","Acct-Input-Gigawords","Acct-Output-Gigawords","Connect-Info","MS-Acct-Auth-Type","MS-Acct-EAP-Type","Event-Timestamp","NAS-Port-ID","ACC-Err-Message","Annex-Product-Name","Annex-SW-Version","Annex-System-Disc-Reason","Annex-Modem-Disc-Reason","Annex-Disconnect-Reason","Annex-Transmit-Speed","Annex-Receive-Speed","Ascend-Modem-Port-Number","Ascend-Modem-Slot-Number","Ascend-Modem-Shelf-Number","Ascend-Xmit-Rate","Nautica-Acct-SessionId","Nautica-Acct-Direction","Nautica-Acct-CauseProtocol","Nautica-Acct-CauseSource","Telebit-Accounting-Info","Last-Number-Dialed-Out","Last-Number-Dialed-In-DNIS","Last-Callers-Number-ANI","Channel","Event-Id","Event-Date-Time","Call-Start-Date-Time","Call-End-Date-Time","Default-DTE-Data-Rate","Initial-Rx-Link-Data-Rate","Final-Rx-Link-Data-Rate","Initial-Tx-Link-Data-Rate","Final-Tx-Link-Data-Rate","Sync-Async-Mode","Originate-Answer-Mode","Modulation-Type","Equalization-Type","Fallback-Enabled","Characters-Sent","Characters-Received","Blocks-Sent","Blocks-Received","Blocks-Resent","Retrains-Requested","Retrains-Granted","Line-Reversals","Number-Of-Characters-Lost","Number-of-Blers","Number-of-Link-Timeouts","Number-of-Fallbacks","Number-of-Upshifts","Number-of-Link-NAKs","Back-Channel-Data-Rate","Simplified-MNP-Levels","Simplified-V42bis-Usage","PW_VPN_ID"
"06/10/2009","08:36:13","CISCO 3000 VPN","Start","jsmith","200","jsmith","10.12.44.33","1922","2","1","10.19.12.13",,,,,,,,,,,,,"0x53425232434ce3d796b1dadd9dd5b98011802501800481998c868002800781b0d8cdc68b8dd612800e81e3d796b1dadd9dd5b98082edaa98",,,,,"74.133.61.240","CISCO 3000 VPN",,"1","0",,,"BF70ACEA","1",,,,,,,"5",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

No matter how many files I put in my directory only the first file is indexed. I noticed that if I delete the header from another file it will get indexed but it shows as a second sourcetype. If I add another file with a deleted header it will appear in the second sourcetype.

I've tried setting as automatic and as CSV. Doing fresh installs on each test.

How do I index all files without having to delete the header? And how do I get the header fields recognized?

Thanks for any help you can throw my way.

[ziegfried · Accepted solution]

Seems to be the same problem as here: http://answers.splunk.com/questions/4629/splunks-mechanism-to-detect-files-with-the-same-content

You can work around that by specifying a crcSalt in your monitor configuration:

inputs.conf

[monitor:///path/to/directory]
crcSalt = <SOURCE>
host = your_host
index = your_index
sourcetype = your_sourcetype

[thinguy]

Thanks that did it.

[ziegfried]

Obviously it doesn't look at the last 256 bytes. Have you added your new index (sbrras) to the default indexes of one of your roles? If not, you won't see it on the summary page.

[thinguy]

Also wondering like the link you posted.
If Splunk uses the first AND last 256bytes of the file, it should be seeing my files as unique. Since only the first part is duplicated.

[thinguy]

Thanks for the fast response. I've done something wrong.
I did a fresh install created an index "sbrras" a data input of csv pointing to my new index
and updated my f:\splunks\etc\apps\search\local\inputs.conf file with the info below

I added one file into my Dir and nothing shows up.
I added 2 more files and still now, nothing appears on my search page under Source|Sourcetype|Hosts

[monitor://F:\Splunk\var\raslogs]
disabled = false
followTail = 0
host = RAS
crcSalt =
index = sbrras
sourcetype = csv