Getting Data In

After creating defining a new sourcetype in Splunk Cloud, why are the specified fields not being extracted for a PostgreSQL CSV log type?

Explorer

Splunk Cloud user here. I have defined a new sourcetype with the following properties:

Name: postgresql_csv
Destination app: Distributed Management Console
Category: Database
Description: PostgreSQL CSV log
Indexed Extractions: csv
Timestamp:
  Extraction: Auto
Delimited settings:
  Field delimiter: ,
  Quote character: "
  File preamble: (empty)
  Field names: Custom
    Comma separated field names: log_time,user_name,database_name,process_id,connection_from,session_id,session_line_num,command_tag,session_start_time,virtual_transaction_id,transaction_id,error_severity,sql_state_code,message,detail,hint,internal_query,internal_query_pos,con,query,query_pos,location,application_name

Then I conduct a search for sourcetype=postgresql_csv, but it doesn't look like any of the fields I defined above appear in the fields bar on the left. After a bit of searching, 40+ new fields pop up, all with approximately 0% coverage, fields such as "database" or "average" or "AND_classid", which are AFAICT just guessed extractions from the actual log text, but aren't particularly useful.

Sample CSV below:

2015-08-30 00:50:31.539 UTC,"classroom_api","classroom_dev",17910,"127.0.0.1:48594",55e25357.45f6,2,"authentication",2015-08-30 00:50:31 UTC,2/1713,0,LOG,00000,"connection authorized: user=classroom_api database=classroom_dev",,,,,,,,,""
2015-08-30 00:50:31.541 UTC,"classroom_api","classroom_dev",17910,"127.0.0.1:48594",55e25357.45f6,3,"PARSE",2015-08-30 00:50:31 UTC,2/1714,0,LOG,00000,"duration: 0.106 ms  parse <unnamed>: SET extra_float_digits = 3",,,,,,,,,""
2015-08-30 00:50:31.541 UTC,"classroom_api","classroom_dev",17910,"127.0.0.1:48594",55e25357.45f6,4,"BIND",2015-08-30 00:50:31 UTC,2/1714,0,LOG,00000,"duration: 0.011 ms  bind <unnamed>: SET extra_float_digits = 3",,,,,,,,,""
2015-08-30 00:50:31.541 UTC,"classroom_api","classroom_dev",17910,"127.0.0.1:48594",55e25357.45f6,5,"SET",2015-08-30 00:50:31 UTC,2/1714,0,LOG,00000,"duration: 0.011 ms  execute <unnamed>: SET extra_float_digits = 3",,,,,,,,,""
2

What am I doing wrong here? I read in a couple of posts that it's not possible to disable CSV auto-learning without digging into props.conf, which I don't have access to in Splunk Cloud. Please advise.

0 Karma
1 Solution

Splunk Employee
Splunk Employee

INDEXED_EXTRACTIONS only works when ingesting local files.
So you want to setup your props.conf with the sourcetype definition on the forwarder.

View solution in original post

Splunk Employee
Splunk Employee

INDEXED_EXTRACTIONS only works when ingesting local files.
So you want to setup your props.conf with the sourcetype definition on the forwarder.

View solution in original post

Explorer

Looks like your suggestion works, I added a props.conf with a custom source type to system/local.

On that note, if I want to add a few custom field extractions to this type that only exists in props.conf at this point, do I create a sourcetype with matching name on my Splunk Cloud instance and add those extractions there?

0 Karma

SplunkTrust
SplunkTrust

Does your CSV have a header line? CSV parsing works much better if there is a header.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

Explorer

I don't think postgres ever leaves the header in the file itself, so that wouldn't be a possibility. I thought the system was able to use column ordering for identifying which column it's working with.

0 Karma

SplunkTrust
SplunkTrust

Could you add a line or two of the log file you are applying this to? Thanks.

0 Karma

Explorer

Sure thing, done!

0 Karma