How to create a logger to send Syslog data into Splunk

[POR160893]

Hi,

I am currently working on developing a logger  in Python that would send Syslog data into Splunk.

So, I want to create a logging interface or an abstract class that would create a logging class for the Syslog data, e.g. log -> write_log.

The log->write_log is then overrides the abstract method with the concrete method for syslog->write_log.

How can I start this task? What Python library would I need? Like splunklib or splunk_handler?


Thanks!

[POR160893]

Hey,

At the moment, I have been trying to connect to Splunk but appears I am getting a connection issue:

The error message is as follows:

Any advice on how to approach this please?

[sloshburch]

I think everyone here is encouraging you to use existing supported methods to do this rather than write your own. The outcome is that if there is a problem you will have to support your custom solution and Splunk Support will not be able to help you. If you use one of the existing approaches then you have the help of Splunk Support and eng there as well as a large community of users experienced with the already-existing approaches.

Nonetheless, if you are trying to send network communication then there must be a listener open on that port. In Splunk those are called TCP/UDP/HEC Inputs. https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Monitornetworkports

[PickleRick]

So you decided to write a syslog-based logger.

Ok, should more or less work but it's not splunk-specific and you have to prepare for it properly on splunk's side.

I assume that you didn't. That's why you're getting your errors. Do you have any input (either sc4s or a direct splunk input on port 514)?

[PickleRick]

I'm  a bit confused - are you trying to log from your own application or want to build application to receive syslog and write to splunk?

If the former - why you mention syslog? If the latter - there are already working solutions for that.

[sloshburch]

I agree with these questions for more information. I think a good logical flow to assume is that the private software produces syslog and sends it to Splunk. Splunk will then receive it using something like https://splunk.github.io/splunk-connect-for-syslog. The point I'm making is that you don't want to be natively trying to write into Splunk to create the log events - rather, use the very mature interfaces that are set up for this. Additionally, if you are creating the log events yourself then I would recommend some of the less lossy approaches like log files or HTTP events. https://dev.splunk.com/enterprise/docs/devtools/httpeventcollector

[POR160893]

So, the logs would be coming from a Python Django REST API. We want to log what was received from this Django API onto Splunk, I just need now to ingest the Syslog into Splunk using Python but I don't know how.

[PickleRick]

Again - syslog is a separate layer.

From application perspective you should just use a common logging framework (if I remember correctly, Python has one built-in). Then you - depending on the target logging method - should just create proper handler either emiting syslog messages (then you can use rsyslog/SC4S or even splunk's built-in tcp/udp listener) or a handler sending events to HEC endpoint. Or one writing to files (then you can set your UF to read from files).

[sloshburch]

I'm not familiar with using that but if the logs are already available through a REST API then creating a modular input would still be a solid option. Again, this is a bit out of my domain but it sounds like if you can push the data to Splunk, then HEC would be best. If you must pull the data into Splunk then a Modular Input (easily built with the Splunk Add-on Builder). If you can write to a log file then a Splunk Universal Forwarder can help. Finally, syslog with Splunk Connect 4 Syslog, assuming you can push the data out and communicate over standard syslog. All of the proper names I mentioned area easily found online with a web search.

I hope that helps!

Also, if it's relevant: https://community.splunk.com/t5/Developing-for-Splunk-Enterprise/Writing-own-REST-API-in-Splunk-app/...