CSV Field extractions with one field has extra commas with in single quotes

[som3guy]

Here is an example of one log:

20170309 10:41:16,hostname.vagrantup.com,username,localhost,155,9823,QUERY,database_name,'select min(t.nextcheck) from httptest t,hosts h where t.hostid=h.hostid and mod(t.httptestid,5)=4 and t.status=0 and h.proxy_hostid is null and h.status=0 and (h.maintenance_status=0 or h.maintenance_type=0)',0

Here is an example of another log:

20170309 10:40:44,hostname,username,localhost,222,0,CONNECT,database,,0

Here is the format:

[timestamp],[serverhost],[username],[host],[connectionid],[queryid],[operation],[database],[object],[retcode]

So I am stuck trying to get this correctly setup to have each field configured correctly. As you can see the logs differ based on what type of operation is taking place.
When the operation is a query they will likely have several commas inside of single quotes. Any help will be much appreciated.

Thanks

[acruise_splunk · Accepted solution]

I was able to ingest your data successfully using the Add Data Wizard, using the following changes...

1. In the "Set source type" stage, you should choose CSV from the "Source type" dropdown
2. Under "Delimited settings", which appears after you choose CSV, you should choose single quote as your quote character.
3. At the end of the process, you should pick a different source type name, so you don't confuse future, more general CSV users.

Accepting all the other defaults worked for me!

[som3guy]

Thanks this worked out great.

[richgalloway]

I've done this a while ago using a REGEX statement in my props.conf file. This regex string should get you started if you want to try that approach.

(?<ts>[^,]+),(?<serverhost>[^,]+),(?<username>[^,]+),(?<host>[^,]+),(?<connectionid>[^,]+),(?<queryid>[^,]+),(?<operation>[^,]+),(?<database>[^,]+),(?<object>'?.*'?),(?<retcode>\d+)