Splunk Search

props deployed on SHCluster but no extractions

cdstealer
Contributor

Hi,  A bit of a strange one that I can't workout.  I have a deployer server and a search head in one DC and 2 searchheads in another DC.  They are all part of the searchhead cluster and all share the same configs.  My problem is that the searchhead app has been deployed to all the searchheads.  The 2 searchheads located  in the same DC have the app and correct configs, but don't perform any field extractions.  Interestingly, if I open an event and select extract field, the parser sees the fields?  The searchhead on its own performs as expected.  I can see no errors.  Running btool confirms the file is also correct.

It's the first time I've ever come across this.

 

TIA

Steve

Labels (1)
0 Karma
1 Solution

cdstealer
Contributor

Hi,  It looks like some weird file permission issue.  Although everything in /opt/splunk is readable by splunk, it still couldn't see props.conf.  Which of course makes no sense when the working searchhead has the exact same perms and works without issue.  On one of the affected search heads, I did a chmod -R 755 on /opt/splunk/etc/apps/* and it started working immediately on both affected servers???  Very strange.

I'll mark this as resolved 🙂

Thanks

Steve

View solution in original post

Nisha18789
Builder

Hi @cdstealer , could you please share your props? Also, can you check using below query on SH that your extractions shows up ?

| rest splunk_server=local servicesNS/nobody/<your-app_name>/configs/conf-props/
| rex field=id "servicesNS\/nobody\/(?<app>.+?)\/"
| where app="<your-app_name>"
| fields - eai:* SEGMENTATION*
| fields id,title, sourcetype, updated, disabled *, FIELDALIAS-*
| search title=<sourcetype name>
| transpose 1 column_name=setting
| rename "row 1" as value
| search value != ""
| eval value=case(value=1,"true", value=0,"false",1=1,value)
| eval stanzas = setting +" = "+value
| stats LIST(stanzas) as Stanza

cdstealer
Contributor

Hi Nisha,  Thanks for the query.. this is the output on both a working and non-working searchhead.

id = https://127.0.0.1:8089/servicesNS/nobody/app_name/configs/conf-props/sourcetype
title = sourcetype
updated = 1970-01-01T01:00:00+01:00
disabled = false
ADD_EXTRA_TIME_FIELDS = true
ANNOTATE_PUNCT = true
AUTO_KV_JSON = true
BREAK_ONLY_BEFORE_DATE = true
CHARSET = UTF-8
DATETIME_CONFIG = /etc/datetime.xml
DEPTH_LIMIT = 1000
EVAL-duration = hour * 3600 + minute * 60 + second + precise/10000000
EXTRACT-fields = \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} \[\d+\] \[(?P<ClientIP>[^ ]*) \(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\)\] INFO\s+(?P<GingerClass>[^ ]*) - Start: (?P<JobStartTime>[^;]*);Reseller:\s(?P<ResellerName>[^;]*);ResellerUnit: (?P<ResellerUnit>[^;]*);(?P<JobGUID>[^;]*);(?P<UserGUID>[^;]*)?;(?P<ApiMethod>[^;]*);Duration: (?P<hour>[^:]\d+):(?P<minute>[^:]\d+):(?P<second>[^\.]\d+)\.(?P<precise>[^;]\d+);(?P<RegOutcome>[^;]*)?;(?P<FailReason>[^$]*)?
LEARN_MODEL = true
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
MATCH_LIMIT = 100000
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 256
MAX_TIMESTAMP_LOOKAHEAD = 128
SHOULD_LINEMERGE = true
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
TIME_PREFIX = ^
TRUNCATE = 10000
app = app_name
author = nobody
detect_trailing_nulls = false
maxDist = 100
splunk_server = searchhead1

 The actual props.conf is:

[sourcetype]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
EXTRACT-fields = \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} \[\d+\] \[(?P<ClientIP>[^ ]*) \(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\)\] INFO\s+(?P<GingerClass>[^ ]*) - Start: (?P<JobStartTime>[^;]*);Reseller:\s(?P<ResellerName>[^;]*);ResellerUnit: (?P<ResellerUnit>[^;]*);(?P<JobGUID>[^;]*);(?P<UserGUID>[^;]*)?;(?P<ApiMethod>[^;]*);Duration: (?P<hour>[^:]\d+):(?P<minute>[^:]\d+):(?P<second>[^\.]\d+)\.(?P<precise>[^;]\d+);(?P<RegOutcome>[^;]*)?;(?P<FailReason>[^$]*)?
EVAL-duration = hour * 3600 + minute * 60 + second + precise/10000000
0 Karma

cdstealer
Contributor

OK, I fixed the field extractions by converting to indextime rather than searchtime.  However, any searchtime functions eg calculated fields are still being ignored by the 2 searchheads.  I'll keep digging.

0 Karma

soutamo
SplunkTrust
SplunkTrust

I propose that you should create ticket to Splunk support, if you haven't done it yet.

But I'm really interested to hear what was the issue when you have fixed it 😉

0 Karma

cdstealer
Contributor

Will do 🙂  

0 Karma

cdstealer
Contributor

Hi,  It looks like some weird file permission issue.  Although everything in /opt/splunk is readable by splunk, it still couldn't see props.conf.  Which of course makes no sense when the working searchhead has the exact same perms and works without issue.  On one of the affected search heads, I did a chmod -R 755 on /opt/splunk/etc/apps/* and it started working immediately on both affected servers???  Very strange.

I'll mark this as resolved 🙂

Thanks

Steve

View solution in original post

soutamo
SplunkTrust
SplunkTrust
Have you also multisite cluster or how your peers have arranged?

R. Ismo
0 Karma

cdstealer
Contributor

Hi Soutamo,  We don't run a multisite due to its scale and function.  So I have the deployer and a search head in one location and then 2 search heads in the other location.  These are then setup behind a load balancer which will fail to the single searchhead should the primary location fail.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!