Splunk Deployment App Filtering SharePoint Logs

[tbarn005]

Hi Splunk Community,

I’m trying to reduce disk space usage on my Splunk Universal Forwarder by filtering out unnecessary SharePoint logs and only forwarding those with a severity of High, error, or warning in the message

I created a deployment app named SharePoint. here is what's in that folder:

I attempted to create a props and transforms.conf files to filter out the data that was unnecessary. i only need to see the log files in the dir that have certain key words not all of those logs here is what i wrote in the files. I didn't write the regex myself i found something similar to it online somewhere and tried to make it work for my environment

After deploying this i now do not see any of my SharePoint logs indexed at all for this specific server even the ones with high. As you can see from the logs i even pointed them at a test index that i made so i should be seeing them I'm not sure what's going on. 

[livehybrid]

Hi @tbarn005 

Can I just check, you want to reduce your storage usage on your Universal Forwarder, but the UF isnt storing your data ingested, its only sending it on. 

UFs are typically not used for parsing the data. Did you apply the screenshotted configuration to your UF or a different (HF/IDX) instance?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[tbarn005]

I may have misspoken i want to reduce the storage usage on my indexer.  I have a SharePoint server that has Splunk UF on it and its ingesting unnecessary data that is eating a lot of storage on my indexer. The screen shots come from my indexer. Im doing a bit of research now and it looks as if i can use the ingest actions to possibly filter out some of that unnecessary data from that sharepoint UF? 

[Prewin27]

@tbarn005 

Your props and transform looks ok. Make sure you are applying this to HF or Indexer not on the UF. Also add one more transform to filter out other noise.

props.conf

[source::E:\\SPLogs\\CLGDEVSPAPPSO1*]
TRANSFORMS-debug = route_high_to_debug,drop_noise

In transforms.conf
[drop_noise]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

Restart Splunk and check again. Also make sure you have new high category logs from this server.

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[tbarn005]

Sorry about the week late reply but that does not seem to work. I am still getting logs that i dont need i just disabled ingestion from that folder location. Does splunk have any app that would filter data easier than creating the transforms and props.conf files? 

[richgalloway · Accepted solution]

Try Ingest Actions.  They're easy to use and even have a preview GUI so you know they'll work before they're implemented.

[richgalloway]

There appear to be a few problems here.

1) The SharePoint app should have a single folder called 'default'.  The default folder should contain the files shown in the first screenshot.

2) Universal Forwarders do not consume disk space so filtering will not save any there.  Caveat: if you use persistent queuing then the UF will use disk space, but the space will be returned once the queue is drained.

3) Universal Forwarders do not process transforms so they cannot filter events this way.  Put the props and transforms on the first full instance that touches the data (indexer or heavy forwarder).