topic Re: props deployed on SHCluster but no extractions in Splunk Search

props deployed on SHCluster but no extractions

cdstealer — Thu, 03 Sep 2020 17:01:36 GMT

Hi, A bit of a strange one that I can't workout. I have a deployer server and a search head in one DC and 2 searchheads in another DC. They are all part of the searchhead cluster and all share the same configs. My problem is that the searchhead app has been deployed to all the searchheads. The 2 searchheads located in the same DC have the app and correct configs, but don't perform any field extractions. Interestingly, if I open an event and select extract field, the parser sees the fields? The searchhead on its own performs as expected. I can see no errors. Running btool confirms the file is also correct.

It's the first time I've ever come across this.

TIA

Steve

Re: props deployed on SHCluster but no extractions

isoutamo — Thu, 03 Sep 2020 17:08:07 GMT

Have you also multisite cluster or how your peers have arranged?

R. Ismo

Re: props deployed on SHCluster but no extractions

Nisha18789 — Thu, 03 Sep 2020 17:17:32 GMT

Hi @cdstealer , could you please share your props? Also, can you check using below query on SH that your extractions shows up ?

Re: props deployed on SHCluster but no extractions

cdstealer — Mon, 07 Sep 2020 08:35:08 GMT

Hi Nisha, Thanks for the query.. this is the output on both a working and non-working searchhead.

id = https://127.0.0.1:8089/servicesNS/nobody/app_name/configs/conf-props/sourcetype title = sourcetype updated = 1970-01-01T01:00:00+01:00 disabled = false ADD_EXTRA_TIME_FIELDS = true ANNOTATE_PUNCT = true AUTO_KV_JSON = true BREAK_ONLY_BEFORE_DATE = true CHARSET = UTF-8 DATETIME_CONFIG = /etc/datetime.xml DEPTH_LIMIT = 1000 EVAL-duration = hour * 3600 + minute * 60 + second + precise/10000000 EXTRACT-fields = \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} \[\d+\] \[(?P<ClientIP>[^ ]*) $\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$\] INFO\s+(?P<GingerClass>[^ ]*) - Start: (?P<JobStartTime>[^;]*);Reseller:\s(?P<ResellerName>[^;]*);ResellerUnit: (?P<ResellerUnit>[^;]*);(?P<JobGUID>[^;]*);(?P<UserGUID>[^;]*)?;(?P<ApiMethod>[^;]*);Duration: (?P<hour>[^:]\d+):(?P<minute>[^:]\d+):(?P<second>[^\.]\d+)\.(?P<precise>[^;]\d+);(?P<RegOutcome>[^;]*)?;(?P<FailReason>[^$]*)? LEARN_MODEL = true LEARN_SOURCETYPE = true LINE_BREAKER_LOOKBEHIND = 100 MATCH_LIMIT = 100000 MAX_DAYS_AGO = 2000 MAX_DAYS_HENCE = 2 MAX_DIFF_SECS_AGO = 3600 MAX_DIFF_SECS_HENCE = 604800 MAX_EVENTS = 256 MAX_TIMESTAMP_LOOKAHEAD = 128 SHOULD_LINEMERGE = true TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N TIME_PREFIX = ^ TRUNCATE = 10000 app = app_name author = nobody detect_trailing_nulls = false maxDist = 100 splunk_server = searchhead1

The actual props.conf is:

[sourcetype] TIME_PREFIX = ^ TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N EXTRACT-fields = \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} \[\d+\] \[(?P<ClientIP>[^ ]*) $\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$\] INFO\s+(?P<GingerClass>[^ ]*) - Start: (?P<JobStartTime>[^;]*);Reseller:\s(?P<ResellerName>[^;]*);ResellerUnit: (?P<ResellerUnit>[^;]*);(?P<JobGUID>[^;]*);(?P<UserGUID>[^;]*)?;(?P<ApiMethod>[^;]*);Duration: (?P<hour>[^:]\d+):(?P<minute>[^:]\d+):(?P<second>[^\.]\d+)\.(?P<precise>[^;]\d+);(?P<RegOutcome>[^;]*)?;(?P<FailReason>[^$]*)? EVAL-duration = hour * 3600 + minute * 60 + second + precise/10000000

Re: props deployed on SHCluster but no extractions

cdstealer — Mon, 07 Sep 2020 08:50:44 GMT

Hi Soutamo, We don't run a multisite due to its scale and function. So I have the deployer and a search head in one location and then 2 search heads in the other location. These are then setup behind a load balancer which will fail to the single searchhead should the primary location fail.

Re: props deployed on SHCluster but no extractions

cdstealer — Tue, 08 Sep 2020 10:07:45 GMT

OK, I fixed the field extractions by converting to indextime rather than searchtime. However, any searchtime functions eg calculated fields are still being ignored by the 2 searchheads. I'll keep digging.

Re: props deployed on SHCluster but no extractions

isoutamo — Tue, 08 Sep 2020 10:54:53 GMT

I propose that you should create ticket to Splunk support, if you haven't done it yet.

But I'm really interested to hear what was the issue when you have fixed it 😉

Re: props deployed on SHCluster but no extractions

cdstealer — Tue, 08 Sep 2020 11:09:56 GMT

Will do 🙂

Re: props deployed on SHCluster but no extractions

cdstealer — Tue, 08 Sep 2020 13:51:24 GMT

Hi, It looks like some weird file permission issue. Although everything in /opt/splunk is readable by splunk, it still couldn't see props.conf. Which of course makes no sense when the working searchhead has the exact same perms and works without issue. On one of the affected search heads, I did a chmod -R 755 on /opt/splunk/etc/apps/* and it started working immediately on both affected servers??? Very strange.

I'll mark this as resolved 🙂

Thanks

Steve