Onboardering Data from Syslog-ng server

[jorob]

Hello,

I recently started with a company that has a syslog-ng server saving logs to /mnt/syslog/$year/$month/<filename> and onboarded to splunk as sourcetype syslog and index main. 

I need to begin on-boarding data by the actual sourcetype

The current monitor stanza is [monitor:\\\mnt\syslog].  The $year and $date are part of log rotation with from the previous month being archived and deleted and only logs in the current year and month are updated and pulled into Splunk.

We are starting with Juniper for onboarding by sourcetype.  I updated syslog-ng.conf to write juniper logs into their own folder  and setup the following a monitor in inputs.conf

[monitor: ///mnt/storage/.../juniper]. 
sourcetype = juniper
host_segment = 5

However, I am unable to get Splunk to see anything in the juniper folder.   When I list monitor from the CLI, it only shows /mnt/storage/$year/$month.   Also, due to network restrictions,  we are currently not are not sending via udp/tcp.  Everything has to be onboarded via file (for now).  

I have tried different variations for the monitor stanza to include
///mnt/syslog/*/juniper,  ///mnt/syslog/.../.../juniper, ///mnt/syslog/*/juniper/ to list a few.  

I have also used the CLI to "add monitor /mnt/syslog/.../juniper -sourcetype juniper, with the same results.

Any recommendations? Or is more information needed?

Thanks

[thambisetty]

• Did you see new folder juniper created as per your syslog-ng.conf rules.

[monitor: ///mnt/storage/.../juniper]. 
sourcetype = juniper
host_segment = 5

you mentioned storage in monitor stanza but your syslog-ng directory is /mnt/syslog.

[jorob]

Hi,

Yes, the juniper folder was created.  And storage was a typo/error, it is supposed to be syslog.

Thanks

jorobb

[isoutamo]

Hi

if I understand your file system hierarchy is

/mnt/storage/YEAR/MONTH/juniper or is there node name between MONTH and juniper?

If it is first then it's easiest to use

host = juniper 

and if it's .../MONTH/<host>/juniper 

then your inputs.conf is ok.

If possible you should also add index=<juniper index> there.

You could check what your UF (or HF) see by command:

..../splunk btool inputs list --debug

If you have separate files for different log types on .../juniper directory then you should add own monitoring stanzas to those with correct source types. Don't use same sourcetype for those logs if those content are different!

Is the splunk user capable of reading those files?

r. Ismo

[jorob]

Thank you for the reply,

I ended up taking a different approach with props and transforms and using regex to identify juniper logs and assign the sourcetype., which worked (although now the challenge is getting it to work with the Juniper Addon and CIM).   That said, I'm relatively new to Splunk and wondering if that's the best solution, or should the solution be more inline with what you outlined.  

For the juniper logs, (all the same format), file system hierarchy is  /mnt/syslog/YEAR/MONTH/juniper/<file>,  however the primary syslog monitor is /mnt/syslog/YEAR/MONTH/<file>, which is where everything is dumped, except for what is identified as juniper, which now goes to the /juniper/<file>.   

Juniper logs are just the first ones we looked to properly identify, we still have to go back and break out the remaining source types such as apache, linux_secure, asterisk_*, access_combined,  operating systems, etc...   It's a bit of a mess to clean up without disruption current work flows.

Thanks again!

jrobb

[venkatasri]

The monitor stanza looks fine if the syslog-ng is running in linux machine. 

• Option 1: Check your splunkd.log for Access Denied error for /.../juniper directory. Make sure the Splunk user running Splunkd process having read permissions to  /.../juniper directory.
• Option 2: You could run following command from /bin directory of Splunk. It will output an xml file having list of directories and files Splunk is monitoring at that instance. Check your files under /juniper directory visible in this xml file/ any Access Denied messages.

./splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus

[jorob]

Thank you for your reply,

I was able to get the sourcetype working using props and transformed.  However, information you provided will definitely benefit me in future troubleshooting.   I have a long way to go before everything is fixed and am sure to run into similar problems as we move forward.

Thanks again!

Jorobb