All Apps and Add-ons

How to customize splunk Add-on for F5 BIG-IP based on the data available in Splunk?

Hemnaath
Motivator

Hi All, I need to customize the splunk Add-on for F5 BIG-IP based on the data ingested into splunk.

F5 LTM data are being ingested into splunk Environment from syslogs servers. We have 5 Heavy forwarder instances configured to fetch this syslogs data's and forwards it to the 5 individual indexer instances.
Splunk F5 Add-on is uploaded in search head cluster master and we want to customize the Technology add on, based on the data available in splunk from F5.

Currently we could see the F5 data being ingested into splunk from 17 hosts and could see some 104 fields under interesting field details. Below is the inputs.conf detail.

inputs.conf detail:

F5 LTM

[monitor:///opt/syslogs/web_access/.../*.log]
index = web_app
sourcetype = f5:network:loadbalancer
host_segment = 4

From the splunk Add-on for F5 BIG-IP, I could see the below details:

Appserver -- Not sure what is the purpose of it and whether we need to use this for our data.
Default folder contains following Technology add on configuration file like
props.conf, transforms.conf,web.conf, eventtypes.conf, tags.conf and these configuration files contains lots of predefined information but not sure how/where to start of with customization.

Kindly guide me on how to customize the Add-on app based on F5 LTM data available in splunk.

0 Karma

alemarzu
Motivator

Hi there @Hemnaath

From where it comes from the sourcetype you are using? Because if I'm not mistaken, according to the type of input (syslog) you should use this sourcetype "f5:bigip:syslog". The add-on will recognize certain patterns on your data through regexes and will categorize the events into new sourcetypes, but in order for this "categorization/renaming" to work, the previews sourcetype is needed on your input.

By doing this all the intelligence thats comes from this add-on will occur.

Hope it helps.

0 Karma

Hemnaath
Motivator

Hi Alemarzu, Yes data are being ingested from syslog and we have customized app to monitor the F5 data from syslog. This is the query index= web sourcetype = f5:network:loadbalancer is used to fetch the data in splunk console. Do you want me to change the sourcetype to "f5:bigip:syslog" ?

Not sure what are the stanza in props,transforms,eventtypes,tags should be customized in this app based on the data available in splunk.

Kindly guide me on this please.

0 Karma

alemarzu
Motivator

Do you want me to change the sourcetype to "f5:bigip:syslog" ?
Yes.

0 Karma

Hemnaath
Motivator

thanks Alemaruz, after changing the sourcetype, do i need to change anything in props.conf, transforms.conf, eventtype.conf and tags.conf.

In props.conf for the sourcetype=f5:big:syslog I could see below details and along with this stanza there are other stanza related to other sourcetypes. Should we need to remove them as i our case we have only one sourcetype is used to fetch the F5 LTM data into splunk.

  Rename   

General

[f5_bigip:syslog]
rename = f5:bigip:syslog

General

[f5:bigip:syslog]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TRANSFORMS-sourcetype=f5_bigip-irule-default, f5_bigip-irule-http, f5_bigip-irule-dns-request, f5_bigip-irule-dns-response, f5_bigip-irule-lb-failed, f5_bigip-syslog-asm, f5-bigip-apm-syslog, f5_bigip-irule-exclude-audit

Kindly guide me on this.

0 Karma

alemarzu
Motivator

You do not need to change anything else. Props & Transforms config files holds the parsing/extraction knowledge for your data.

The stanzas that you mention are the ones that recognize certain patterns on your F5 logs and then rename the sourcetypes according to the type of event.

0 Karma

Hemnaath
Motivator

Hi Alemarzu, thanks for your effort on this. So it mean we need to change only the sourcetype in inputs.conf and upload the splunk Add-on for F5 BIG-IP in the search head cluster master, without altering the other config file present in the add-on.

I have doubt before proceeding to change the sourcetype inputs.conf, since the data which is currently being ingested in splunk is with the sourcetype=f5:network:loadbalancer, so by changing the source type will that have any impact. I mean will data ingest into splunk properly. I am going to push this app in search head cluster member.

Kindly advise me.

0 Karma

Richfez
SplunkTrust
SplunkTrust

Hi, folks. I'm trying to follow this train of thought and it sounds like we're at the edge of an answer. If I may throw a few supporting ideas out here to help ...

Also, if we're about at an answer, I would suggest @alemarzu that you convert your answer into an actual answer so that it can be accepted - although, it might be useful to summarize the problem for the answer itself.

Anyway.

Hemnaath, there's a few things you might want to review about how Splunk works with respect to configuration file precedence. The files in the app/default folders shouldn't be changed - you should copy the relevant pieces to your app/local folder to "override" the settings.

It's also a bit more reading, but the getting data in manual has a discussion on sourcetypes and why they matter, too.

The F5 app documents a list of sourcetypes. I know little about this app, but I can see that your sourcetype for incoming syslog data from the F5's should be set to f5:bigip:syslog. I can't be sure about that having little experience with this particular app, but it makes sense; it fits the docs (probably, unless I"m just reading them wrong) and matches what @alemarzu indicates.

So here's the crux of the matter - if you are ingesting the data right now with a sourcetype that isn't right, the data is in Splunk but probably doesn't quite 'work' right. It doesn't populate panels on the F5 dashboards, fields aren't all there and so on. In that case, there's not much harm in changing the sourcetype to something else - it's won't make it much worse, and may fix it all. And if it DOESN'T fix it all, just remove that overridden piece of configuration and it'll all be back to your originals again. This process will only take a few minutes to test, just keep track of what you are doing so you can easily change it back if necessary.

And do let us know if that works out for you!

0 Karma

Hemnaath
Motivator

Hi rich/Alemarzu, thanks for your effort on this,

Exact Requirement:

Actually we want to just add this app in search head cluster members in our environment. And we want to implement the appropriate configurations from the F5 app to the data that is available in Splunk from F5 LTM devices.

As stated in my earlier comment, that F5 LTM data are being ingested into splunk from syslogs servers via Heavy forwarder instances. In our Environment HF instance act as syslogs server.

Below inputs.conf stanza are configure to monitor the data from this location into splunk.

[monitor:///opt/syslogs/web_access/.../*.log]
index = web_app
sourcetype = f5:network:loadbalancer
host_segment = 4

We are not configuring, moduler inputs, iRule For LTM, F5 for syslogs, logging level for ASM /APM. As mentioned in the document from the app.

Question :

So in this case by changing the sourcetype in the inputs.conf and extracting the add-on and adding into search head cluster, will fetch the output . I mean we want match the data available in splunk.

0 Karma

Hemnaath
Motivator

Hi rich/Alemarzu, can you please guide me on this.
thanks in advance.

0 Karma

alemarzu
Motivator

Hi there mate, sry I've lost track of this thread.

Just edit your /local/inputs.conf and change the sourcetype name, then re-deploy for changes to take effect.

0 Karma

Hemnaath
Motivator

Hi alemarzu, thanks for your effort on this, Hey got the approval to change the sourcetype from sourcetype = f5:network:loadbalancer to sourcetype = f5:bigip:syslog as per the Splunk Add on.

We have inputs.conf configured in a customized app "Test-IA-F5" as per our environment and placed in the HF instances to monitor the data. Only change to this app is the sourcetype.

I have final question to you before pushing the changes in Prod Environment.

In our case the splunk Add-on for F5 BIG-IP - Is going to be pushed into the search head cluster member and as per the document we need to remove the inputs.conf , eventgen.conf files and all files in the Samples folder.

1) appserver, lookups and statics folder should be pushed along with the app or we can remove it from the package as we are not going to use the irule for LTM, modular inputs and lookups related to this.

2) Similarly under default folder, props.conf, transforms.conf , tags.conf and events.conf contains lots of stanza details which is not going to applicable for us, so in that case, can we remove or ## it out the other stanzas, Keeping only the stanza related to f5_bigip:syslog

General
[f5_bigip:syslog]
rename = f5:bigip:syslog

3) I am not sure what is purpose of the rename = ?

Kindly advise me on this please.

Thanks in advance.

0 Karma

Hemnaath
Motivator

Hi Alemarzu, Good Morning, could you please guide me on the above mentioned comments.

thanks in advance.

0 Karma

Hemnaath
Motivator

Hi Alemarzu, Good Morning, I have changed the sourcetype to f5:bigip:syslog and we could see the data getting into splunk. But could please guide me on the below statement. Before pushing the app to the search head cluster member.

In our case the splunk Add-on for F5 BIG-IP - Is going to be pushed into the search head cluster member and as per the document we need to remove the inputs.conf , eventgen.conf files and all files in the Samples folder.

1) appserver, lookups and statics folder should be pushed along with the app or we can remove it from the package as we are not going to use the irule for LTM, modular inputs and lookups related to this.

2) Similarly under default folder, props.conf, transforms.conf , tags.conf and events.conf contains lots of stanza details which is not going to applicable for us, so in that case, can we remove or ## it out the other stanzas, Keeping only the stanza related to f5_bigip:syslog

General
[f5_bigip:syslog]
rename = f5:bigip:syslog

3) I am not sure what is purpose of the rename = ?

Thanks in advance.

0 Karma

alemarzu
Motivator

I don´t have a SHC on my arch but it's working okey and no changes were needed on the configuration files besides the sourcetypes in /local/inputs.conf according to the type of input.

Edit: http://docs.splunk.com/Documentation/AddOns/released/F5BIGIP/Install#Distributed_deployment_feature_...

0 Karma

Hemnaath
Motivator

Hi All, Can any one guide me to get started with this ...
thanks in advance.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!