topic Re: add new data source via UDP to indexer in Getting Data In

add new data source via UDP to indexer

glenngermiathen — Thu, 14 Nov 2013 15:35:51 GMT

I have a Splunk indexer running on Ubuntu that forwards to the Splunk web on a Windows box and I want to add a new data source to the Ubuntu system. How do I go about doing this from the command line? I am fairly new to both Splunk and Ubuntu, so the more detailed answer the better.

Re: add new data source via UDP to indexer

jtrucks — Thu, 14 Nov 2013 15:44:59 GMT

As shown in the manual section on getting data into Splunk via CLI, the most basic method is simply to run:

$SPLUNK_HOME/bin/splunk add monitor /path/to/log/directory

$SPLUNK_HOME/bin/splunk add monitor /path/to/log/filename

Run this as the user Splunk runs as on the system.

If you want to specify the sourcetype and/or index, then it would look like:

$SPLUNK_HOME/bin/splunk add monitor /path/to/log/filename -sourcetype mysourcetypehere -index myotherindexhere

If you use a different index, it should already exist on the indexer. You can omit either or both of the -sourcetype or -index flags above.

Run $SPLUNK_HOME/bin/splunk help monitor to see more complete details.

Re: add new data source via UDP to indexer

glenngermiathen — Thu, 14 Nov 2013 16:21:25 GMT

Thanks, how do I determine where $SPLUNK_HOME is?

Re: add new data source via UDP to indexer

jtrucks — Thu, 14 Nov 2013 16:28:53 GMT

$SPLUNK_HOME is the installation location of Splunk on that system. Usually on *nix based machines, this is /opt/splunk/ and I think on Windows is C:\Program Files\Splunk\

Re: add new data source via UDP to indexer

glenngermiathen — Thu, 14 Nov 2013 17:34:31 GMT

All great information. I know our firewall is feeding data into the indexer, but how do I determine where the /path/to/log/directory or /path/to/log/logfile is so that I can add the data source?

Re: add new data source via UDP to indexer

jtrucks — Thu, 14 Nov 2013 17:56:46 GMT

If the indexer is getting the data via syslog, look at /etc/syslog.conf, /etc/syslog-ng.conf, or /etc/rsyslog.conf depending on what syslog daemon you use.

Re: add new data source via UDP to indexer

glenngermiathen — Thu, 14 Nov 2013 18:11:01 GMT

I guess that is where Im confused, so Im taking over an existing build for the former admin that left. Mutiple sources are feeding into the indexer and I want to add a new firewall into it too. Im trying to determine where each data stream is going, but when I look at both /etc/syslog-ng.conf and /etc/rsyslog.conf I dont see anything indicating this information.

Re: add new data source via UDP to indexer

jtrucks — Thu, 14 Nov 2013 18:30:23 GMT

Ahh! In your indexer's splunkweb UI, go to Manager » Data inputs and look in those sections to see if you can find something obviously labeled that corresponds to your syslog based sources. This will start showing you where these things come from to get to your indexer.

Re: add new data source via UDP to indexer

glenngermiathen — Thu, 14 Nov 2013 18:46:34 GMT

Im not sure if this is right or not, but I dont think my indexer is running a splunk web UI. From my understanding the indexer is just a command line ubuntu build which feeds into the splunk web head on a Windows build. I checked the data inputs from the web interface on the Windows box, but I couldnt see anything that helped me there. Any other ideas?

Re: add new data source via UDP to indexer

jtrucks — Thu, 14 Nov 2013 18:53:46 GMT

That means that your ubuntu system is a Splunk Forwarder client and the Windows system is the Splunk Indexer server. Go look at the "source" field for any event in your indexer listed as "sourcetype=syslog" like:

sourcetype=syslog | head 1 | table source

That should get you a list of the source, which is most likely a file path to the files on the syslog machine. It could be a TCP connection, which means the data goes directly to the indexer, which makes it a rather different situation.

Re: add new data source via UDP to indexer

glenngermiathen — Thu, 14 Nov 2013 19:14:21 GMT

It appears to be a straight UDP feed into the forwarder.

Re: add new data source via UDP to indexer

jtrucks — Thu, 14 Nov 2013 19:17:17 GMT

To clarify, it is a UDP feed into the Ubuntu machine?

Re: add new data source via UDP to indexer

glenngermiathen — Thu, 14 Nov 2013 19:25:21 GMT

Yes, UDP into the ubuntu forwarder

Re: add new data source via UDP to indexer

glenngermiathen — Thu, 14 Nov 2013 20:06:06 GMT

The firewall information is getting to the Splunk web. I can see the raw data, but it isnt being parsed like the old firewall we were using. I cant create searches by src or dest IP port etc.

Re: add new data source via UDP to indexer

jtrucks — Thu, 14 Nov 2013 20:41:59 GMT

I suspect the new firewall might be showing as a different source and the extracts are per source, not sourcetype. Go to Manager » Fields » Field extractions to see if the firewall extracts are listed with "source::" in the front. If so, you may need to create a new one using sourcetype instead, or copy the existing one and create it using your new firewall source.

Re: add new data source via UDP to indexer

glenngermiathen — Thu, 14 Nov 2013 21:07:40 GMT

That doesnt seem to be it. The firewall doesnt have the source:: before it.

Re: add new data source via UDP to indexer

jtrucks — Thu, 14 Nov 2013 21:45:34 GMT

I think you need to read http://docs.splunk.com/Documentation/Splunk/6.0/Knowledge/Addfieldsatsearchtime and check out the http://docs.splunk.com/Documentation/Splunk/6.0/Knowledge/Addfieldsatsearchtime#Use_interactive_field_extraction_to_create_new_fields section specifically. Try to extract some of the fields with your new source and then it will suggest what extracts it sorta looks like. Also, you need to have a much stronger understanding of how Splunk field extractions, sources, and sourcetypes work to get much further managing it.

Re: add new data source via UDP to indexer

jtrucks — Thu, 14 Nov 2013 21:46:25 GMT

I think the issue you posted on here is answered, which is to get your firewall data into Splunk.

Perhaps go read the above, maybe experiment, and then create a NEW answers post with the extract question to get better visibility into it. Meanwhile, this issue of getting the data should likely be marked answered...

Re: add new data source via UDP to indexer

jtrucks — Thu, 14 Nov 2013 21:49:13 GMT

Further reading for you: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Howtousethismanual

Re: add new data source via UDP to indexer

jtrucks — Thu, 14 Nov 2013 21:50:12 GMT

Also, check this out to understand forwarders: http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Aboutforwardingandreceivingdata

And to understand ingesting data: http://docs.splunk.com/Documentation/Splunk/latest/Data/WhatSplunkcanmonitor