How to Ingest Data From Host Machine Into Splunk App?

[akulg]

Hi,

I am a bit new to the Splunk community and interested in building a Splunk app that can process host-level log data (particularly logs produced by audit D).  My end goal is to provide some analysis of the host log and report that back to the user in the Splunk dashboard. I am unsure how to do the first step of ingesting data from the host machine into the app.

[gcusello]

Hi @akulg,

At first I hint to follow some training (first courses are free) and some videos on the YouTube Splunk Channel, e.g.:

https://www.splunk.com/en_us/resources/videos/getting-data-into-splunk.html

https://www.splunk.com/en_us/resources/videos/getting-data-in-to-splunk-enterprise-windows.html

https://www.youtube.com/watch?v=gHzUW9oOvKA

https://www.youtube.com/watch?v=t02Y0uj38is

Ib addition, searching with Google, you can find all the needed documentation, e.g.:

https://docs.splunk.com/Documentation/SplunkCloud/9.0.2305/Admin/IntroGDI

https://dev.splunk.com/view/SP-CAAAEE6

https://docs.splunk.com/Documentation/Splunk/9.1.0/Data/Usingforwardingagents 

I suppose that you have a Splunk infrastructure (at least one stand-alone server) ready to receive data, otherwise the job is longer that this.

Anyway, the first step is to identify the perimeter to monitor: create an Excel File with the list of hostnames that should send logs identifying for each of them the way to ingest data: es. agent (Splunk Universal Forwarder) from Windows and Linux servers, syslog from appliances,etc...

In this way you have a map of the ingestion systems and a list of data sources to monitor.

If you have to use the UF, you have to install it on the target systems and configure them to send logs to indexers, for more infos see at

Then you have to identify, for each system in the perimeter, the logs to ingest: e.g. for Windows, wineventlog, performance monitor, etc...

When you have these information, you can choose the Add-ons to use: e.g. Splunk_TA_Windows for windows systems, etc...

Then you have to enable the inputs in the Add-Ons that you defined and install the Add-Ons on the target systems.

At this point you should have the logs in yur Splunk system and you should create the searches to find the logs you want to display in the dashboard; in Splunk every dashboard or report or alert is one or more searches, so you have to learn how to create searches: https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchTutorial/WelcometotheSearchTutorial

When you'll have data and you'll learn to search data you'll be able to create your own dashboards.

Ciao.

Giuseppe

[akulg]

@gcusello 

Thank you for such a detailed reply. I understand how to forward host-level data to Splunk if I manage the entire infrastructure. However, I am interested in how to get host-level data if I am creating a Splunk app that someone else will install. Should I assume such data already exists? And if so, how do I access it from the app?

[gcusello]

Hi @akulg,

as I said, if you already ingested data, you have only to create your searches and saving them as dashboard's panels.

You could also have an help if there's an app in Splunkbase (splunkbase.splunk.com) regarding the technology you have to monitor; in this way you can use those dashboards or use them as a starting point to create your own;

Otherwise you have to create them from scratch: remember that the main difficulty in Splunk isn't the search creating but to understand what to search, in other words, you have to well know the logs to monitor, to extract fields understaning values and choices.

Ciao.

Giuseppe