topic Recommended way to ingest files from remote server into clustered indexers? in Getting Data In

Recommended way to ingest files from remote server into clustered indexers?

jwalzerpitt — Fri, 18 Oct 2019 17:56:36 GMT

We have a clustered search head and indexer environment with 16 indexers and a Deployment server

On a remote Windows server we have a PS script that runs a Microsoft API call every hour to pull alerts from Azure and then dumps the output into a .csv file on that Windows server. This server is not running a UF.

I’m not seeing any of the four/five Azure add-ons that pull the Azure AD related alerts so with that I would like to assistance in pulling those .csv files into an index on Splunk.

Is the best way to get the files from the remote Windows server via a UF that is set to monitor the ,csv files in the directory?

Thx

Re: Recommended way to ingest files from remote server into clustered indexers?

wmyersas — Fri, 18 Oct 2019 19:08:18 GMT

Yes - the best way to get data into Splunk is via the UF

Or using HEC - which might be simpler in this case (would need to know more about your use cases to know for sure)

Re: Recommended way to ingest files from remote server into clustered indexers?

jwalzerpitt — Fri, 18 Oct 2019 19:29:24 GMT

Thx for the reply.

Our use case is ingesting Azure alerts. I've combed the various Azure add-ons and even referenced this solid doc - https://docs.google.com/spreadsheets/d/139tuKUqqXqPp1yzCvH60LZfDThz067rLzYSAdHzomxo/edit#gid=0 - and there is no add-on that specifically lists Azure alerts as a sourcetype.

We have a PowerShell script that makes an API call every hour and pulls the events down and dumps them into a .csv file. I can probably easily install a UF on the server.

I plan on taking a look at Azure Event Hub to see if I can centralize all relevant Azure logs there and then pull the events from Event Hub into Splunk, but that's a bit in the future.

Thx

Re: Recommended way to ingest files from remote server into clustered indexers?

wmyersas — Sat, 19 Oct 2019 13:30:54 GMT

You could probably just have your PS script push to the HEC