All Apps and Add-ons

App Wanted: LSF log parser

pixelgrunt
New Member

I'm new to splunk and python, but I've been managing a ROCKS-based HPCC for over 3 years now and I'm looking for an easier way to analyze the LSF scheduler accounting logs. They are plain-text, but relatively voluminous (3-5 GB/year). The management here likes graphs and pictures too.

Anyone else have a need like this? I'd love to take it on myself, but as I stated earlier, I'm still quite green with this stuff.

Tags (1)
0 Karma

mkimler
New Member

I was just curious if anyone has an update to this thread. Has anyone been working with LSF parsing and is willing to share their information for an app?

Thanks.

0 Karma

MillerTime
Splunk Employee
Splunk Employee

HeyBigBen: Would you be willing to share some of your work with us here on SplunkBase? If you don't feel like posting an App, I'd be happy to review your work and bundle it as an App here (I'm a SE at Splunk). If you're willing to share, it would be a great resource for everyone -- feel free to send me what you've got at jmiller (at) splunk (dot) com.

Cheers,
Jesse

0 Karma

heybigben
Explorer

The LSF daemon (lim, mbatchd, mbschd, sbatchd) logs are pretty simple and can be done via standard file input/monitor. Another option is to configure LSF to log the daemon logs to syslog.

The LSF accounting logs (lsb.acct.*) can be quite tricky since the number of fields in each record is variable depending upon the type of LSF job and the specific LSF job options. For example, LSF jobs which specify -n slots (numAskedHosts >0) will then add n additional fields for each askedHost. The lsb.acct man page all has all details on the specific format of the LSF accounting file. We've developed a perl regular expression to extract the LSF accounting file fields into something meaningful, but ran into some splunk limitation on long complex perl regular expressions (it works fine outside splunk when we run it directly, but for some reason fails to parse when configured in props.conf

I've also written some wrapper scripts to extract the LSF cluster file and LSF hostgroups out of lsb.hosts so splunk can use them as well.

0 Karma

John_Mark
Splunk Employee
Splunk Employee

Alex - could you let me know who this is working on an app? I'd like to add them to my pipeline.

Thanks,
JM

0 Karma

araitz
Splunk Employee
Splunk Employee

I recently talked to some folks that are splunking their LSF scheduler logs across their grid with some success and have interest in posting an app to Splunkbase. I will do my best to update this question when I have more information.

0 Karma

sideview
SplunkTrust
SplunkTrust

Hmm. If you could send me a sample of the log (nick at sideviewapps.com) I could take a look at it. Often it takes only a couple hours to build the interactive skeleton for an app. The difficulty level largely depends on the data though.

Splunk even has a tool to help you anonymize it if you need to. http://www.splunk.com/base/Documentation/latest/Admin/Anonymizedatasamplestosendtosupport

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...