Getting Data In

Unable to ingest the data in to splunk from syslogs servers ?

Hemnaath
Motivator

Hi All, Currently got a request from the client to ingest the mguard data from newly set plant into splunk. I could see that data from the mguard are reaching the syslogs servers under the path /opt/syslogs/generic/10.x.x.x/mguard.log, the same server is used as Heavy forwarder server to read the syslog data directly from the path /opt/syslogs/mguard/.../mguard.log* in to splunk. We have five individual HF instances all have same set of configurations. Other mguard devices data are being ingested into splunk from all the Five HF instances.

Problem : Only newly added mguard data are not being ingested into splunk.

Below are the configuration details followed to get the data from mguard to syslog/HF instances.

syslog-ng.conf
filter f_mguardx { host("10.x.x*" type(glob) ); };
filter f_mguardx { host("10.x.x*" type(glob) ); };
filter f_mguardx { host("10.x.x*" type(glob) ); };
filter f_mguardx { host("10.x.x*" type(glob) ); };
filter f_mguardx { host("10.x.*" type(glob) ); }; -- > Newly added mguard details

COMBINATION FILTERS

filter f_mguard { filter(f_mguard1) or filter(f_mguard2) or filter(f_mguard3) or filter(f_mguard4) or filter(f_mguard5) ; };

DESTINATIONS

destination d_mguard { file( "/opt/syslogs/mguard/${HOST}/mguard.log" owner(splunk) group(splunk) create_dirs(yes) dir-group(splunk) dir-owner(splunk)); };

log { source(s_net); filter(f_mguard); destination(d_mguard); };

After configuring the above details in syslog-ng.conf, I have restarted the syslogs-ng services and checked the status of the syslog-ng using the below command.
systemctl restart syslog-ng.service
systemctl status syslog-ng.service

Inputs.conf stanza details :

[monitor:///opt/syslogs/mguard/.../mguard.log*]
index=firewall
sourcetype=mguard:network:firewall
host_segment = 4

We have props.conf and transforms.conf configured per- event host name settings.

Kindly let me know how to troubleshoot and fix this issue. What are the log files should I need to check to figure out the issue.

Tags (2)
0 Karma

rpquinlan
Path Finder

I think your issue is that the * wildcard isn't working as you want.

ref: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/regu...

"10.x.x*" works in your already configured IP addresses because "in regular expressions, the asterisk (*) character means 0, 1 or any number of the previous expression."

Can you use a CIDR expression or a netmask to specify a range of IP addresses, such as: netmask(10.124.203.0/24) (Ref: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/refe... )

0 Karma

Hemnaath
Motivator

Hi All, thanks for your efforts on this issue, we have figured the issue is with the firewall is down in that site and its not sending the data properly, network team is working on it.

thanks all.

0 Karma

sbbadri
Motivator

@Hemnaath

Please check below stuff,

1) check that file got written on this /opt/syslogs/mguard/.../mguard.log - If file is not written check the syslog-ng configuration revisit your filter settings. filter f_mguard { match("10.x.*" value("HOST_FROM")) };

2) If file got written check the access permission of the folder or files to splunk id/group.

3) If above two steps are satisfying, then execute this command
$SPLUNK_HOME$/bin/splunk btool list inputs --debug ---> check any issue with inputs
$SPLUNK_HOME$/bin/splunk show inputstatus ---> check the stanza for your inputs. File should have been read.

4) Go to splunkd.log to check for errors.

0 Karma

Hemnaath
Motivator

Hi Sbbadri, thanks for your effort on this, As mentioned in the above comment, I could not see the data being written in the /opt/syslogs/mguard/10.x.x.x/mguard.log but the file are reaching the /opt/syslogs/generic/10.x.x.x/mguard.log from the source, both the path are in the same instances (syslogs & Heavy Forwarder instance)

Do you want me to change the syslog-ng configuration like this instead of filter f_mguardx { host("10.x.*" type(glob) ); };

And also how to checked the error in splunkd.log for all the HF instances but could not find anything related to this mguard.

index="_internal" host="test*" sourcetype=splunkd log_level=ERROR component!=ExecProcessor component!=JsonLineBreaker

Kindly guide me on this.

0 Karma

sbbadri
Motivator

i still didn't understand your problem. Does mguard logs writing in two locations.

0 Karma

Hemnaath
Motivator

Hi Sbbadri, Good Morning.

My exact problem is that my newly configured mguard device are not getting written in the following path from where the splunk inputs.conf reads the file and ingest into index
Path : /opt/syslogs/mguard/10.X.X.X/mguard.log.

Inputs.conf stanza
[monitor:///opt/syslogs/mguard/.../mguard.log*]
index=firewall
sourcetype=mguard:network:firewall
host_segment = 4

At the same time the newly configured devices data are reaching the syslog server and writes the data under this location.
path: /opt/syslogs/generic/10.X.X.X/mguard.log.

We have five individual server which acts as syslogs server cum Heavy forwarder instances.

I am not sure where is the problem now? Is it with the syslog-ng.conf configuration ? If is it with syslog-ng.conf issue then how come other mguard device data are able to send the data from different location to syslog /HF servers correctly.

syslog-ng.conf
filter f_mguardx { host("10.x.x*" type(glob) ); };
filter f_mguardx { host("10.x.x*" type(glob) ); };
filter f_mguardx { host("10.x.x*" type(glob) ); };
filter f_mguardx { host("10.x.x*" type(glob) ); };
filter f_mguardx { host("10.x.*" type(glob) ); }; -- > Newly added mguard details

COMBINATION FILTERS

filter f_mguard { filter(f_mguard1) or filter(f_mguard2) or filter(f_mguard3) or filter(f_mguard4) or filter(f_mguard5) ; };

DESTINATIONS

destination d_mguard { file( "/opt/syslogs/mguard/${HOST}/mguard.log" owner(splunk) group(splunk) create_dirs(yes) dir-group(splunk) dir-owner(splunk)); };

log { source(s_net); filter(f_mguard); destination(d_mguard); };

Kindly guide me on this.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...