Props.conf settings are not working

[mnarmada]

Hello All,

Hope you all are doing good!!

I am trying to send some data to Splunk using UF. Below are my settings but I am getting data to Splunk without breaking the lines as I specified in my stanza. I want to break my events whenever there is messages tag.

Kindly help me. I am just getting started my journey as admin but getting all issues. If possible please help with points using which we can trouble shoot all the issues 

My Settings:

inuts.conf:

[monitor:///usr/narmada/props_test.log]
index=narmada
sourcetype=logs_format

 outputs.conf:

[tcpout:abc]
server = 65.2.122.16:9997

props.conf:

[logs_format]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]*)<messages>
BREAK_ONLY_BEFORE=<messages>

raw data:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><logs schemaVersion="0"><messages><timestamp>2021-04-22T11:55:13.766-07:00</timestamp><level>PROGRESS</level><thread>backup4 ee5fa1cb0c31a3e56f4fed2c99ff7745</thread>location>com.netapp.common.flow.tasks.Log</location><msgKeyClass>com.netapp.smvi.SMMsgKey</msgKeyClass><msgKeyValue>PROGRESS_TASK_BACKUP_STARTING</msgKeyValue><parameters xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/><message>Starting backup request</message>/messages>messages<timestamp>2021-04-22T11:55:14.156-07:00</timestamp><level>INFO</level><thread>backup4 aaaaaaaaaajksbcjkbud7yh8y83eh38</thread><location>com.netapp.smvi.task.validation.BackupValidation</location><msgKeyClass>com.netapp.smvi.SMMsgKey</msgKeyClass><msgKeyValue>BACKUP_VALIDATION_INTERNAL_BACKUP_NAME_FOR_SCHEDULE_JOB</msgKeyValue><parameters><parameter>66fc1387-594c-48cb-b35d-94ca319a4a3c</parameter><parameter>backup_PM cDOT Datastore_20210422115514</parameter></parameters><message>Generating backupName for the scheduleJob 66fc1387-594c-48cb-b35d-94ca319a4a3c is backup_PM cDOT Datastore_20210422115514</message></messages>

[mnarmada · Accepted solution]

Hello All, @kamlesh_vaghela,

My props settings are working now from both UI and backend. The reason why my props.con file not reading was, I have placed that file in forwarder i.e., /opt/splunkforwarder/etc/system/local where we basically keep inputs.conf and outputs.conf files. But we actually have to put it in indexer (i.e., /opt/splunkforwarder/etc/system/local).  I was confused as everywhere in Splunk documentation it is mentioned as $SPLUNK_HOME.

Thanks for all your support!! I really appreciate.

[kamlesh_vaghela]

Glad to help you @mnarmada .

If any of my reply helps you to solve the problem Or gain knowledge, an upvote would be appreciated. 🙂 

[mnarmada]

Hello All,

I dont see my props settings when I debug it like below.

./splunk btool props list --debug

can you please clarify one thing for me?. Will these settings work when we are using trail version of Splunk enterprise and UF? 

I am using trail version of Splunk enterprise and UF and trying to send data from Unix environment to Splunk using UF.  Will it break events according to props settings?

Thanks for your response.

[kamlesh_vaghela]

@mnarmada 

Your Splunk version should parse the event.

Just want to know. At what components you put this props/conf changes?

Ideally I suggest to keep in both UF and Indexer.

For more understanding I would like you to check below links which will help you how UF and Indexer works for event parsing.

https://wiki.splunk.com/Community:HowIndexingWorks

https://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

https://docs.splunk.com/Documentation/Splunk/7.0.0/Admin/Configurationparametersandthedatapipeline

KV

[mnarmada]

@kamlesh_vaghela ,

I kept my inputs.conf , outputs.conf and props.conf in Universal forwarder.

I will check the reference links, which you have provided. 

Also could you please tell me if any props.conf or any file settings are not detected when we debug those using btool, what are the actions we could take next? or how to resolve the issue.

Thanks!!

[kamlesh_vaghela]

@mnarmada 

btool should work. Can you please try this

./splunk btool props list --debug | grep YOUR_SOURCETYPE

Or 

this in /etc/apps folder.

grep YOUR_SOURCETYPE `find . -name "props.conf"`

KV

[kamlesh_vaghela]

@mnarmada 

Your provided _raw xml looks invalid. Can you please share sample valid XML and expected OP from that sample?

KV

[mnarmada]

Hello @kamlesh_vaghela ,

Thanks for the response!

The XML data is valid only. As the data is huge, I have shared only 20% of the data here.  Also I tried keeping the data in .txt, .xml and even .log format to test the props settings. But still same issue persists in all the ways. 

Could you please help? 

[kamlesh_vaghela]

@mnarmada 

Can you please try this?

[YOUR_SOURCETYPE]
SHOULD_LINEMERGE=true
LINE_BREAKER=<\/messages>(.*)<messages>
NO_BINARY_CHECK=true
SEDCMD-a=s/.*<logs schemaVersion=\"0\">//g
SEDCMD-b=s/<\/logs>$//g

My Sample Data :

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><logs schemaVersion="0"><messages><timestamp>2021-04-22T11:55:13.766-07:00</timestamp><level>PROGRESS</level><thread>backup4 ee5fa1cb0c31a3e56f4fed2c99ff7745</thread><location>com.netapp.common.flow.tasks.Log</location><msgKeyClass>com.netapp.smvi.SMMsgKey</msgKeyClass><msgKeyValue>PROGRESS_TASK_BACKUP_STARTING</msgKeyValue><parameters xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true" /><message>Starting backup request</message></messages><messages><timestamp>2021-04-22T11:55:14.156-07:00</timestamp><level>INFO</level><thread>backup4 aaaaaaaaaajksbcjkbud7yh8y83eh38</thread><location>com.netapp.smvi.task.validation.BackupValidation</location><msgKeyClass>com.netapp.smvi.SMMsgKey</msgKeyClass><msgKeyValue>BACKUP_VALIDATION_INTERNAL_BACKUP_NAME_FOR_SCHEDULE_JOB</msgKeyValue><parameters><parameter>66fc1387-594c-48cb-b35d-94ca319a4a3c</parameter><parameter>backup_PM cDOT Datastore_20210422115514</parameter></parameters><message>Generating backupName for the scheduleJob 66fc1387-594c-48cb-b35d-94ca319a4a3c is backup_PM cDOT Datastore_20210422115514</message></messages></logs>

Thanks
KV
▄︻̷̿┻̿═━一

If any of my reply helps you to solve the problem Or gain knowledge, an upvote would be appreciated. 

[mnarmada]

Hello @kamlesh_vaghela,

Thanks for the trial!!

I have kept the same settings which you have given, but still not working for me. Not sure why. I could see my settings and yours are working when I manually try to upload the file from Splunk UI/front end. Only props.conf file settings are not working. 

Someone please help me to check what exactly went wrong. I really appreciate your help!

[kamlesh_vaghela]

@mnarmada 

This setting will work on new data only. This will not work with data which is already index. 

Please share valid sample events so we can work on it.

KV

[mnarmada]

Sure @kamlesh_vaghela ,

But meanwhile can you please help me to understand why the settings worked for you, are not working for me. Also why the props settings are working from UI but not from backend.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><logs schemaVersion="0"><messages><timestamp>2021-05-20T11:55:13.766-07:00</timestamp><level>PROGRESS</level><thread>backup4 ee5fa1cb0c31a3e56f4fed2c99ff7745</thread><location>com.netapp.common.flow.tasks.Log</location><msgKeyClass>com.netapp.smvi.SMMsgKey</msgKeyClass><msgKeyValue>PROGRESS_TASK_BACKUP_STARTING</msgKeyValue><parameters xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true" /><message>Starting backup request</message></messages><messages><timestamp>2021-05-20T11:55:14.156-07:00</timestamp><level>INFO</level><thread>backup4 aaaaaaaaaajksbcjkbud7yh8y83eh38</thread><location>com.netapp.smvi.task.validation.BackupValidation</location><msgKeyClass>com.netapp.smvi.SMMsgKey</msgKeyClass><msgKeyValue>BACKUP_VALIDATION_INTERNAL_BACKUP_NAME_FOR_SCHEDULE_JOB</msgKeyValue><parameters><parameter>66fc1387-594c-48cb-b35d-94ca319a4a3c</parameter><parameter>backup_PM cDOT Datastore_20210520115514</parameter></parameters><message>Generating backupName for the scheduleJob 66fc1387-594c-48cb-b35d-94ca319a4a3c is backup_PM cDOT Datastore_20210520115514</message></messages></logs>

[kamlesh_vaghela]

@mnarmada 

Your sample data also working for me.

Can you please try by uploading your sample data? 

https://docs.splunk.com/Documentation/Splunk/8.2.1/SearchTutorial/GetthetutorialdataintoSplunk

[mnarmada]

@kamlesh_vaghela ,

I am getting the events broke as per my props settings from Splunk front end ( i.e., when I upload file manually using "Upload" option in Splunk UI) as I said earlier. But when I try to use props.conf file in linux environment and send the file using Universal forwarder, those settings are not working and data is breaking as only one event. That is my problem.

Also when I create index in UI, it is taking the data to that index. But not taking data to any index when I create that from conf files. Not Sure why. Something seriously wrong!!

Can someone please help??