topic Re: Datanow props/transforms not working properly in Getting Data In

Datanow props/transforms not working properly

a548506 — Wed, 21 Jun 2017 16:43:27 GMT

I have some Datanow syslog data coming into my environment and i have setup a transforms.conf file to extract some specific fields for me.

Unfortunately, it is not pulling these fields. I am following the instructions on this site: https://community.ivanti.com/docs/DOC-43745
Pretty straight forward stuff. I am not sure if i am using the "Report" extraction correctly, never used it before.

My props and transforms are the same as the one on the site.

Any guidance would help.

Thanks.

Re: Datanow props/transforms not working properly

somesoni2 — Wed, 21 Jun 2017 16:59:33 GMT

The blog specifies configs for single server deployment (single Splunk Enterprise instance doing work of forwarder, indexer and search head). Do you've single server deployment OR distributed deployment?

In distributed deployment, the inputs.conf will go to forwarder (universal/Splunk Enterprise/heavy forwarder), the props.conf file (except the REPORT- line) and transforms.conf (with stanza referenced in TRANSFORM- in props.conf) will go to heavy forwarder/indexer. The props.conf with REPORT- and transforms.conf with stanza referenced in REPORT- will go to Search Head. Did you place the file on correct servers (and restarted splunk on them)?

Re: Datanow props/transforms not working properly

woodcock — Wed, 21 Jun 2017 17:21:24 GMT

You need to make sure that your stanza header in props.conf is correct so that the configurations are engaged against your event data (case usually matters). You need to deploy the configuration files to the indexing tier (HF/Indexers). You need to restart all splunk instances on that tier. When testing, you need to evaluate ONLY events that have been indexed after your restarts (you can use _index_earliest and _index_latest search parameters for this).

Re: Datanow props/transforms not working properly

sloshburch — Thu, 22 Jun 2017 21:25:33 GMT

Building on this, I would suggest putting the config in a new app (rather than in the system/local folder). For example, create a folder called datanow with a local folder for the config detailed in link you referenced. That way you can distribute the config to the appropriate places as articulated by @woodcock.

Re: Datanow props/transforms not working properly

a548506 — Fri, 23 Jun 2017 17:18:13 GMT

We have a distributed environment. I have placed the files in our deployment server and am pushing them out by reloading the serverclass. The configs then are pushed out to our HFW which I looked at and see my apps(inputs/props/transforms) in there.

I have the apps set to restart which are restarting the associated HFW. I am still not seeing the data though.

@SloshBurch these are in its own app

Re: Datanow props/transforms not working properly

sloshburch — Mon, 26 Jun 2017 12:28:19 GMT

@a548506 - It sounds like you highlighted all the config you are pushing to the HFW but what about the config sent to the Search Head? As you could surmise, the search-time field extractions (report command) must be on the search heads.

If it still doesn't work, I'd suggest standing up a new (clean) standalone install in a lab and validating it works before adding in the distributed elements. Then troubleshoot as you distribute.

Re: Datanow props/transforms not working properly

a548506 — Mon, 26 Jun 2017 23:41:53 GMT

@SloshBurch, I created a custom app called it DataNow and added my inputs/props/transforms in the datanow/local dir. Rebooted the search heads.

Is this what you meant?

Thanks

Re: Datanow props/transforms not working properly

sloshburch — Tue, 27 Jun 2017 13:08:33 GMT

Kinda...For typo sake, I hope that's the same app that lives on the heavy forwarders. It might not hurt to have it on the indexers as well - at least until it starts working.

Did the SH part fix it?

Re: Datanow props/transforms not working properly

a548506 — Wed, 28 Jun 2017 14:55:47 GMT

@SloshBurch, the SH part did not fix it. I must be missing something or not adding the inputs/props/transforms in the right place.

Yes, the same inputs/props/transforms that are in the HFW are the ones that i placed in the newly created app that i put in the SH's.

Re: Datanow props/transforms not working properly

sloshburch — Wed, 28 Jun 2017 15:24:20 GMT

btool it all so we can validate your config is loading as you desired?

Re: Datanow props/transforms not working properly

a548506 — Sat, 01 Jul 2017 23:06:48 GMT

It's working now. The creating the custom app and placing the knowledge objs in the SH worked.

Thanks @SloshBurch , @somesoni2 & @woodcock