UDP logs going to messages file but not indexing -- why?

[pfabrizi]

I have a Cisco ASA sending to my universal forwarder. However I noticed that as of May the events stopped indexing.

Looking at my /var/log/messages file I noticed that all the messages are going there, how do I correct this?

Cisco is send UDP.

[tmarlette · Accepted solution]

welp.... if you have a network appliance sending syslog on port 514 to a universal forwarder, there may an issue right there. While a UF can listen for data, it can't do anything to redirect it to indexers cooked properly.

I'd check a couple of things.

1. is rsyslog on this machine running, and receiving this data as well and dumping it into /var/log/messages?
2. install a HF on this machine instead, and then use inputs/props/transforms to redirect all data from that specific IP address to a different index / sourcetype.

[pfabrizi]

so because I am using a UF I can't forward anything? What if I can get the CISCO UDP port changed?

can I send directly the 1 of my 2 indexes instead?

Thanks!

[pfabrizi]

I resolved this issue by adding an entry to the rsyslogd to forward any events from my specified IP (ASA appliance) to a cisco log. I knew this would work as we forward other UDP (514) events to SPLUNK this way.

[tmarlette]

You can forward local logs to your indexer from a UF, but if you want to listen on other ports, you'll need a heavy forwarder. It's always good to have a heavy forwarder in an environment for these types of things, or any other modular data inputs that you may need to use, like AWS or something. That way, you can assure the incoming data stream is following your standardized outputs.conf for you whole environment.

If you want, you can open up the UDP port on one of your indexers, or even on a search head, yes, however direct to the indexer is generally not used.

If you have a cluster, then all of your data is in one spot, and it lacks any cluster mappings for it's buckets.

if you added an outputs.conf to your search head, and opened the listening port there, you would be able to do it, though the resources of the box would take a slight hit for doing the extra data forwarding.

[pfabrizi]

so if those ASA logs are posting in the messages log file on my universal forwarder can those now be considered local logs and be forwarded ?

if so would it be a configured as syslog?

Thanks!

[tmarlette]

Yes they can. It would be a standard [monitor:///var/log/messages] stanza.

You'd have the seperate the data at the indexers using data routing / filtering techniques if you wanted standard host level /var/log/messages data on a different index.

http://docs.splunk.com/Documentation/Splunk/6.6.2/Forwarding/Routeandfilterdatad