indexed only 1 CSV file

thinguy — Fri, 03 Dec 2010 01:32:53 GMT

Trying to index some radius accounting (.act) files that are really CSV files with a header

"Date","Time","RAS-Client","Record-Type","Full-Name","Auth-Type","User-Name","NAS-IP-Address","NAS-Port","Service-Type","Framed-Protocol","Framed-IP-Address","Framed-IP-Netmask","Framed-Routing","Filter-ID","Framed-MTU","Framed-Compression","Login-IP-Host","Login-Service","Login-TCP-Port","Callback-Number","Callback-ID","Framed-Route","Framed-IPX-Network","Class","Session-Timeout","Idle-Timeout","Termination-Action","Called-Station-ID","Calling-Station-ID","NAS-Identifier","Proxy-State","Acct-Status-Type","Acct-Delay-Time","Acct-Input-Octets","Acct-Output-Octets","Acct-Session-Id","Acct-Authentic","Acct-Session-Time","Acct-Input-Packets","Acct-Output-Packets","Acct-Termination-Cause","Acct-Multi-Session-Id","Acct-Link-Count","NAS-Port-Type","Port-Limit","Tunnel-Type","Tunnel-Medium-Type","Tunnel-Client-Endpoint","Tunnel-Server-Endpoint","Acct-Tunnel-Connection","Tunnel-Private-Group-ID","Tunnel-Assignment-ID","Acct-Tunnel-Packets-Lost","Acct-Input-Gigawords","Acct-Output-Gigawords","Connect-Info","MS-Acct-Auth-Type","MS-Acct-EAP-Type","Event-Timestamp","NAS-Port-ID","ACC-Err-Message","Annex-Product-Name","Annex-SW-Version","Annex-System-Disc-Reason","Annex-Modem-Disc-Reason","Annex-Disconnect-Reason","Annex-Transmit-Speed","Annex-Receive-Speed","Ascend-Modem-Port-Number","Ascend-Modem-Slot-Number","Ascend-Modem-Shelf-Number","Ascend-Xmit-Rate","Nautica-Acct-SessionId","Nautica-Acct-Direction","Nautica-Acct-CauseProtocol","Nautica-Acct-CauseSource","Telebit-Accounting-Info","Last-Number-Dialed-Out","Last-Number-Dialed-In-DNIS","Last-Callers-Number-ANI","Channel","Event-Id","Event-Date-Time","Call-Start-Date-Time","Call-End-Date-Time","Default-DTE-Data-Rate","Initial-Rx-Link-Data-Rate","Final-Rx-Link-Data-Rate","Initial-Tx-Link-Data-Rate","Final-Tx-Link-Data-Rate","Sync-Async-Mode","Originate-Answer-Mode","Modulation-Type","Equalization-Type","Fallback-Enabled","Characters-Sent","Characters-Received","Blocks-Sent","Blocks-Received","Blocks-Resent","Retrains-Requested","Retrains-Granted","Line-Reversals","Number-Of-Characters-Lost","Number-of-Blers","Number-of-Link-Timeouts","Number-of-Fallbacks","Number-of-Upshifts","Number-of-Link-NAKs","Back-Channel-Data-Rate","Simplified-MNP-Levels","Simplified-V42bis-Usage","PW_VPN_ID"
"06/10/2009","08:36:13","CISCO 3000 VPN","Start","jsmith","200","jsmith","10.12.44.33","1922","2","1","10.19.12.13",,,,,,,,,,,,,"0x53425232434ce3d796b1dadd9dd5b98011802501800481998c868002800781b0d8cdc68b8dd612800e81e3d796b1dadd9dd5b98082edaa98",,,,,"74.133.61.240","CISCO 3000 VPN",,"1","0",,,"BF70ACEA","1",,,,,,,"5",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

No matter how many files I put in my directory only the first file is indexed. I noticed that if I delete the header from another file it will get indexed but it shows as a second sourcetype. If I add another file with a deleted header it will appear in the second sourcetype.

I've tried setting as automatic and as CSV. Doing fresh installs on each test.

How do I index all files without having to delete the header? And how do I get the header fields recognized?

Thanks for any help you can throw my way.

Re: indexed only 1 CSV file

ziegfried — Fri, 03 Dec 2010 01:57:30 GMT

Seems to be the same problem as here: http://answers.splunk.com/questions/4629/splunks-mechanism-to-detect-files-with-the-same-content

You can work around that by specifying a crcSalt in your monitor configuration:

inputs.conf

[monitor:///path/to/directory]
crcSalt = <SOURCE>
host = your_host
index = your_index
sourcetype = your_sourcetype

Re: indexed only 1 CSV file

thinguy — Fri, 03 Dec 2010 03:50:29 GMT

Thanks for the fast response. I've done something wrong.
I did a fresh install created an index "sbrras" a data input of csv pointing to my new index
and updated my f:\splunks\etc\apps\search\local\inputs.conf file with the info below

I added one file into my Dir and nothing shows up.
I added 2 more files and still now, nothing appears on my search page under Source|Sourcetype|Hosts

[monitor://F:\Splunk\var\raslogs]
disabled = false
followTail = 0
host = RAS
crcSalt =
index = sbrras
sourcetype = csv

Re: indexed only 1 CSV file

thinguy — Fri, 03 Dec 2010 05:16:24 GMT

Also wondering like the link you posted.
If Splunk uses the first AND last 256bytes of the file, it should be seeing my files as unique. Since only the first part is duplicated.

Re: indexed only 1 CSV file

ziegfried — Fri, 03 Dec 2010 16:38:51 GMT

Obviously it doesn't look at the last 256 bytes. Have you added your new index (sbrras) to the default indexes of one of your roles? If not, you won't see it on the summary page.

Re: indexed only 1 CSV file

thinguy — Fri, 03 Dec 2010 22:48:58 GMT

Thanks that did it.

topic Re: indexed only 1 CSV file in Getting Data In

indexed only 1 CSV file

Re: indexed only 1 CSV file

Re: indexed only 1 CSV file

Re: indexed only 1 CSV file

Re: indexed only 1 CSV file

Re: indexed only 1 CSV file