topic Source type is not setup properly and field extractions are not working in Getting Data In

Source type is not setup properly and field extractions are not working

desi-indian — Mon, 28 Nov 2011 21:47:38 GMT

I need to monitor/splunk multiple files in a directory "/app/products/xxxx/yyy/logs/"

This directory has files like aaa-queries.log.yyyymmdd AND aaa-engineInfo.log.yyyymmdd .

The issue I have is thesefiles will stay unzipped for 3 days ..so today I will have following 3 files for onelog

aaa-queries.log.20111128
aaa-queries.log.20111127
aaa-queries.log.20111126

My inputs.conf and props.conf entries on app/ agent box are as follows

[monitor:///app/products/xxx/yyy/logs/]

index=xxxx

disabled = false

followTail = 1

blacklist = .(gz|zip|bkz|arch|etc)$

[root@ local]# more props.conf

[source::.../aaa-queries.log.d+]

sourcetype = queries_log

[source::.../aaa-engineInfo.log.d+]

sourcetype = engineInfo_log

Becasue I get multiple logs i.e all 3 logs for each source as a part of my source regex ..My sourcetype is not being set properly i.e Is shows as "suggestions-queries.log-too_small" etc . I think because my sourcetype is not set properly I am not able to do field extractions on my search head servers . Is there a way to uniqely set sourcetype in this case . Zipping the old file / changing application is not a option as the logs are huge and other app components need those files to stay for 3 days .

Re: Source type is not setup properly and field extractions are not working

tgow — Mon, 28 Nov 2011 21:56:57 GMT

I am wondering if the "d+" is causing the problem. What happens if you change the source stanza in the props.conf to the following:

[source::.../aaa-queries.log...]
sourcetype = queries_log

source::.../aaa-engineInfo.log...]
sourcetype = engineInfo_log

Re: Source type is not setup properly and field extractions are not working

desi-indian — Mon, 28 Nov 2011 22:15:25 GMT

Thank you ! that seems to work ..If you do not mind what is the trailing "..." doing ? I added d+ cos each log had a date at the end

Re: Source type is not setup properly and field extractions are not working

tgow — Mon, 28 Nov 2011 22:21:38 GMT

The trailing "..." is a catchall that you can use your configuration files. Here are links to more information:

http://docs.splunk.com/Documentation/Splunk/4.2.4/Data/Specifyinputpathswithwildcards

http://docs.splunk.com/Documentation/Splunk/4.2.4/admin/Propsconf