All Apps and Add-ons

Splunk Add-on for Unix and Linux - What to keep/discard in Search head field extraction?

fatsug
Contributor

Hello community

I am looking at TA/apps and trying to figure out what to use, where to use it and how to use it optimally. Potentially there are some best practices for this?

Just to set the stage, let’s take as an example. Wanting to collect only some logs from hosts using a universal forwarder, using the “Splunk_TA_nix” and setting up a “/local/inputs.conf”, cherry-picking a few sources/folders from default/inputs.conf seems reasonable. Though pushing the “entire” app to a UF seems a bit “overkill”. There should be lots of things not being used in a setup like this. Not all scripts will be needed etc.

Then on the search heads, field extraction should be performed for these sources. I assume you need a/the app installed on the these as well for search time extraction. However, things like “/bin”, inputs.conf, outputs.conf, etc. seem unnecessary. Generally it seems like keeping excerpts from props.conf and transforms.conf could suffice?

I should formulate a question which can be answered. Say I have a few logs/sources being indexed and searchable. On the search heads, assuming I am only interested in field extraction for the affected source types, no dashboards, reports, log collection on SH etc.

Should I just install the entire application/TA regardless of how much or little of it will actually be used?

Or, can I remove things which fills no purpose for function outlined?

If so, what is the most efficient way to sort out which things to keep and what to discard (like will it always be enough to keep props.conf/transforms.conf)?

Looking forward to your feedback regarding this, all the best

Labels (2)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @fatsug,

yes, there surely a misunderstanding: my hint is to use the TA as is, modifying only the inputs.conf to enable the inputs you need without touching anything else.

At the same time I hint to maintain the same version of the TA in all your Splunk machines (UFs, HFs, SHs, IDXs and DS) even if they use different parts of the TA: UFs use only inputs.conf and scripts, the other ones uses props.conf, transforms,conf, eventtypes.conf tags. conf, etc....

Ciao.

Giuseppe

View solution in original post

gcusello
SplunkTrust
SplunkTrust

Hi @fatsug,

You idea isn't a good idea!

I hint to use the full TA-nix app, and generally use an entire app and not take a part of them for many reasons: the first is maintainability: taking the full app, even if enabling only few inputs, it's easier to update.

Then remember that the TA-nix is used for input phase (on Forwarders and on all Splunk servers) but also for parsing and merging phases (on Indexers) and for parsing on Search Heads.

In all these phases there are many actions to do: field extractions, aliases, normalization, etc...

If you take only a part of the TA you have to remember each customization and you have the risk to forget something.

In other words, forget to cut a part of the TA, use the full released TA, customizing only the inputs, and when you do this, remember to copy inputs.conf in local folder before customizing.

Ciao.

Giuseppe

fatsug
Contributor

Hi @gcusello 

Thank you for the feeback! OK, so even if monitoring only a single folder ar single log source using a custom "local/input.conf" one should deploy the entire TA on SH and IX?

// Gustaf

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @fatsug,

as I said, you use TA not only for input, but also for parsing, to better understand what a TA does, see the props.conf and transforms.conf files, so you can see all the transformation and elaboration that a TA does.

Then it's easier to maintain.

Ciao.

Giuseppe

fatsug
Contributor

Hello again @gcusello 

Thank you for clarifying.

I get the point, there’s a lot going on “under the hood” and by picking the TA apart I might mess something up.

I was looking at the two files you mention, props and transforms, realizing that with the setup planned would only make use of a fraction of the content of these two files, the TA in general. There would, for instance, never be any use of the UI, no scripts would be executed and so forth. It seemed like a lot of unnecessary configuration to be throwing around.

Though you’ve presented good argument for not poking around at it, I’ll take a clean copy of the TA and add to SH and potentially IX and see what happens.

Best regards

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @fatsug,

I disagree your choice for the reasons I explained, but it's your.

See next time.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉

fatsug
Contributor

Sorry @gcusello, we may have a misunderstanding? 😊

I interpreted your advice as 'use the TA "as is", without modification' and this was what I was planning to do, "take a clean copy of the TA and add to SH and potentially IX".

Did I understand you correctly, or are you recommending I should modify the TA before deploying?

Best regards

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @fatsug,

yes, there surely a misunderstanding: my hint is to use the TA as is, modifying only the inputs.conf to enable the inputs you need without touching anything else.

At the same time I hint to maintain the same version of the TA in all your Splunk machines (UFs, HFs, SHs, IDXs and DS) even if they use different parts of the TA: UFs use only inputs.conf and scripts, the other ones uses props.conf, transforms,conf, eventtypes.conf tags. conf, etc....

Ciao.

Giuseppe

fatsug
Contributor

@gcusello OK, then we're on the same page again 😊

So use the same version, though not the same configuration?

UF only make use of the "input.conf" (and some scripts). Though if I place the same "App" (the same configuration) on any other machine acting as a forwarder, say an HF, that machine will  start forwarding according to the "input.conf"?

What I'm asking is if best practice is to deploy "the same" TA-app/configuration to all hosts. The problem being that some machiens where the TA need to be deployed (HF/SH/IX) should not forward the same logs/need no/different inputs.conf.

Best regards

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @fatsug,

you installa TA-nix only in Linux machines, to have Linux logs.

In my opinion is relevant to have Linux logs from alla Linux machines, also Splunk Servers.

For this reason I hint to use the same TA-nix in all machines: inputs.conf is used in all machines, the other ones only in SHs and eventually on HFs and IDXs.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...