Deployment Architecture

After setting the cron to schedule a PDF report, why am I getting error "Your network connection may have been lost or Splunk may be down." and all search heads restart?

Raghav2384
Motivator

Experts,

When I try to schedule PDF for a report, I get the following error after I set the cron

"Your network connection may have been lost or Splunk may be down." and all my search heads start to restart.

Checked in _internal and this is what i found

[build 237341] 2015-09-07 17:43:44
Received fatal signal 6 (Aborted).
 Cause:
   Signal sent by PID 7962 running under UID 5132.
 Crashing thread: TcpChannelThread
 Registers:
    RIP:  [0x00007F936EBC2625] gsignal + 53 (/lib64/libc.so.6)
    RDI:  [0x0000000000001F1A]
    RSI:  [0x0000000000002044]
    RBP:  [0x00000000016CA900]
    RSP:  [0x00007F93445F8588]
    RAX:  [0x0000000000000000]
    RBX:  [0x00007F93343FF000]
    RCX:  [0xFFFFFFFFFFFFFFFF]
    RDX:  [0x0000000000000006]
    R8:  [0xFEFEFEFEFEFEFEFF]
    R9:  [0x00007F936FF25F60]
    R10:  [0x0000000000000008]
    R11:  [0x0000000000000202]
    R12:  [0x00000000016CB050]
    R13:  [0x00000000016CD640]
    R14:  [0x0000000000000000]
    R15:  [0x00007F93574C0AC0]
    EFL:  [0x0000000000000202]
    TRAPNO:  [0x0000000000000000]
    ERR:  [0x0000000000000000]
    CSGSFS:  [0x0000000000000033]
    OLDMASK:  [0x0000000000000000]
 OS: Linux
 Arch: x86-64
 Backtrace:
  [0x00007F936EBC2625] gsignal + 53 (/lib64/libc.so.6)
  [0x00007F936EBC3E05] abort + 373 (/lib64/libc.so.6)
  [0x00007F936EBBB74E] ? (/lib64/libc.so.6)
  [0x00007F936EBBB810] __assert_perror_fail + 0 (/lib64/libc.so.6)
  [0x0000000000ED555E] _ZN21HttpServerTransaction15postFinalResultERK9AppendStr + 158 (splunkd)
  [0x0000000000DF8EA1] _ZN15shpProxyRequest16proxyReqToMemberEP33ServicesEndpointReplyDataProviderRK14SchemeHostPortRK9AppendStrRbS8_PS5_ + 913 (splunkd)
  [0x0000000000DF9321] _ZN15shpProxyRequest17proxyReqToCaptainEP33ServicesEndpointReplyDataProviderRbS2_ + 193 (splunkd)
  [0x0000000000BCB637] _ZN14MConfigHandler24proxyShpRequestToCaptainERb + 215 (splunkd)
  [0x0000000000B9A415] _ZN23SavedSearchAdminHandler12handleCreateER10ConfigInfo + 85 (splunkd)
  [0x0000000000B9E405] _ZN25ScheduledViewAdminHandler19createScheduledViewER10ConfigInfo + 2101 (splunkd)
  [0x0000000000B9E949] _ZN25ScheduledViewAdminHandler10handleListER10ConfigInfo + 153 (splunkd)
  [0x0000000000BDA701] _ZN14MConfigHandler16handleListFilterER10ConfigInfo + 65 (splunkd)
  [0x0000000000BDA7DB] _ZN14MConfigHandler11getItemByIdERK3StrR10ConfigInfo + 43 (splunkd)
  [0x0000000000B9EA67] _ZN25ScheduledViewAdminHandler10handleEditER10ConfigInfo + 103 (splunkd)
  [0x0000000000BCF494] _ZN14MConfigHandler14executeHandlerER10ConfigInfo + 596 (splunkd)
  [0x0000000000BDC054] _ZN14MConfigHandler2goER10ConfigInfo + 212 (splunkd)
  [0x0000000000BDDED5] _ZN29AdminManagerReplyDataProvider2goEv + 693 (splunkd)
  [0x0000000000C5CBB2] _ZN33ServicesEndpointReplyDataProvider9rawHandleEv + 50 (splunkd)
  [0x0000000000C539B7] _ZN18RawRestHttpHandler10getPreBodyEP21HttpServerTransaction + 39 (splunkd)
  [0x0000000000ED8C90] _ZN32HttpThreadedCommunicationHandler11communicateER17TcpSyncDataBuffer + 320 (splunkd)
  [0x0000000000F416D9] _ZN16TcpChannelThread4mainEv + 473 (splunkd)
  [0x0000000000F4526E] _ZN6Thread8callMainEPv + 62 (splunkd)
  [0x00007F936EF2B9D1] ? (/lib64/libpthread.so.0)
  [0x00007F936EC788FD] clone + 109 (/lib64/libc.so.6)
 Linux / host.x.com / 2.6.32-504.30.3.el6.x86_64 / #1 SMP Thu Jul 9 15:20:47 EDT 2015 / x86_64
 Last few lines of stderr (may contain info on assertion failure, but also could be old):
    2015-08-31 20:23:42.193 -0400 splunkd started (build 237341)
    2015-09-02 01:01:11.355 -0400 Interrupt signal received
    2015-09-02 01:01:46.782 -0400 splunkd started (build 237341)
    2015-09-07 16:14:04.429 -0400 Interrupt signal received
    2015-09-07 16:14:37.336 -0400 splunkd started (build 237341)
    splunkd: /home/build/build-src/6.2.0/src/util/HttpListener.cpp:2691: void HttpServerTransaction::postFinalResult(const AppendStr&): Assertion `_toSendState == TOSEND_VIRGIN || _toSendState == TOSEND_UNSET' failed.
    2015-09-07 17:42:44.298 -0400 splunkd started (build 237341)
    splunkd: /home/build/build-src/6.2.0/src/util/HttpListener.cpp:2691: void HttpServerTransaction::postFinalResult(const AppendStr&): Assertion `_toSendState == TOSEND_VIRGIN || _toSendState == TOSEND_UNSET' failed.
 /etc/redhat-release: Red Hat Enterprise Linux Server release 6.6 (Santiago)
 glibc version: 2.12
 glibc release: stable
Last errno: 111
Threads running: 70
argv: [splunkd -p 8089 restart]
Thread: "TcpChannelThread", did_join=0, ready_to_run=Y, main_thread=N
First 8 bytes of Thread token @0x7f9357607010:
00000000  00 a7 5f 44 93 7f 00 00                           |.._D....|
00000008
commandForThread=0, nextIdle=0x7f93575b3000, requestAfterThread=0, _tpfd=0x7f93574e1000, writeCorkCount=0, terminateCallback=(nil), ioError=No error, lastError=No error, terminateError=No error
giveCmd @0x7f9357607128: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, cmd=0, ok=Y, chan=0x7f9357547000
writeDataAvail @0x7f9357607188: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, chan=0x7f93575c5000
wbuf: ptr=0x7f9357607228, size=0x8000, rptr=0x0, wptr=0x0
HttpListeningConnection: _transactionActive=Y, _haveHadTransaction=Y, _alreadyLoggedTimeout=N
HttpTcpConnection: peer=10.62.60.37, _desiredCompressionLevel=6
RestHttpServerTransaction: _restPath="scheduled/views/_ScheduledView__apigee", namespaced=Y, context=admin/search, session=[user=admin, refcnt=6, touched=1441662223, refreshEligible=1441662403, removed=N, id=944b7a4e715021fc02b21016e247b95e, created=1441647731, refreshed=1441662178, expires=1441665778, initialLife=3600, createdBy=055D1A79-AE4D-4C4C-95F8-5728656120E6, portable, ip=10.62.60.37, csrf=14807573426342708366]
HttpServerTransaction: _state=6, _shouldLog=Y, _startTime=1441662223.766261
REQUEST: POST /en-US/splunkd/__raw/servicesNS/admin/search/scheduled/views/_ScheduledView__apigee HTTP/1.1
    Host: host:8443
    Connection: keep-alive
    Content-Length: 539
    Accept: text/javascript, text/html, application/xml, text/xml, */*
    Origin: host:8443
    X-Requested-With: XMLHttpRequest
    X-Splunk-Form-Key: 14807573426342708366
    User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36
    Content-Type: application/x-www-form-urlencoded; charset=UTF-8
    Referer: https://host:8443/en-US/app/search/dashboards
    Accept-Encoding: gzip, deflate
    Accept-Language: en-US,en;q=0.8
    Cookie: location-zipcode-invalid-noservice=36116; company-locale=Wj0zNjExNl58XlQ9Ul58Xkw9XnxeSD1odHRwczovL3d3dy5jb3guY29tL3Jlc2lkZW50aWFsLXN0b3JlL2hvbWUuY294; TandC=6; mbox=PC#1439421966867-537435.17_50#1449235297|check#true#1441459357|session#1441459296616-488211#1441461157; s_vi=[CS]v1|2A2409F105078C43-4000010E4005D86B[CE]; s_fid=75AFCB4365A850EB-0A12B884A1202DAA; company-current-zipcode=36116; AMCV_8C6767C25245AD1A0A490D4C%40AdobeOrg=-2017484664%7CMCAID%7C2A2409F105078C43-4000010E4005D86B%7CMCMID%7C74855126067035151220716829742186752874%7CMCAAMLH-1442064162%7C7%7CMCAAMB-1442064162%7Chmk_Lq6TPIBMW925SPhw3Q; SMIDENTITY=S786vxgAWtHbFZrJkZvk0GdoOXh7L2et9e9YCgHB18A1e1XNkuKbuquvmSyACzZebubmzbSUOA3q2RsFaryaZ68UCnM1cziKihbdkEtbulIt4ieXg+zBMgd+6qmHCq58q1BMGR/r+yrrvSPHwOzYMYewhym4V5nwGMezZhdr47XS2V3If+et87esl31Mu4B1KPH00L8isXx64ToWzy6tWzOYR0kjXIGzmtRj9CXvtdhCJaiTpOoLzl2HsAC3cmE+RrP8vPFX/Fd2ZNkVX/sguOts6+VeMk8fEBevVnoJv3JREec3PhOjQJ7jzDcomZFj0CWBh1uKxLXycoBFeysKTljvrfluH/OKCHCOd5t4qdHwDzwA+v/CGOT62Q3C2q3ir83wjCFqj3zqnaypNqo5vsz/SsUeIyq35ELV9NU2BRSWqWaUUwKfP2E8RtwKTv7Rfdc3WE18BlT1oei40TLmyfajgITriLEvoxEwxMBagFKT1RDlrt8KoCKpbYyOrBEozHm0b153egIMxs2hKXLWPMr6JyRa+lKo7gQGvEtkTFCUhNX6iCrb2QkIqX6u2bW29vNEFF/FmFNp4AaPWkk1+z0+T3ce/GKBzLIUOrhKl159Z7Rd2hj5cwPPZFhPtBR7EcCrMPp/WYiQtBLlxO301IflaISFSaJuaw5R6yAbu0fnAb0GczpnYNHdKXLLS+DQ; utag_main=_st:1441461181848$ses_id:1441460050249%3Bexp-session$offer_origin:companycom%3Bexp-session$_prevpage:corp%3Aresaccount%3Asign%20in%3Bexp-1441462983894$companyVisitorType:noncustomer%3Aunknown%3Aanonymous%3Aunknown$scid_referrer:0%3Bexp-session; s_pers=%20s_ev41%3D%255B%255B%27shop_all_setupnewservice_offer3_3%27%252C%271414009885379%27%255D%252C%255B%27shop_all_setupnewservice_offer3_3%27%252C%271414009902584%27%255D%252C%255B%27shop_all_setupnewservice_offer3_3%27%252C%271414009915559%27%255D%252C%255B%27shop_all_setupnewservice_offer3_3%27%252C%271414009937171%27%255D%252C%255B%27shop_all_setupnewservice_offer3_3%27%252C%271414009938795%27%255D%255D%7C1571776338795%3B%20pn%3D4%7C1444051305062%3B%20gpv_v14%3Dcompany%253Abus%253Ahome%7C1441461105065%3B%20s_lv%3D1441459383949%7C1536067383949%3B%20s_lv_s%3DMore%2520than%25207%2520days%7C1441461183949%3B%20s_invisit%3Dtrue%7C1441461183956%3B%20s_vnum%3D1449235383956%2526vn%253D2%7C1449235383956%3B; aam_sc=seg%3DGeneric%20Company.com%20Visitor; aam_uuid=74629000044245982600685355110506814572; session_id_8443=7176c88212a2ba5cdb6b10bf05a79be90fb35dd6; splunkd_8443=TO9e_LnLnFsbohCmdjWhvfkUdSk1ffBlKhPLTfWRDTEbF4QSAc9IdR7NHfbLVVsMXNPWyASNGJBDY6r3wTjrVOizBlRhaqjquk^QKGJKmNejuNEkLwFhifOTuceoDWNKeLBQ2I^4NF; splunkweb_csrf_token_8443=14807573426342708366
  _bytesReceived=539, _maximumRequestDataSize=524288000, _totalBytesExpectedOfRequestData=539
  _bytesLeftInRequestDataChunk=0, _requestTransferEncodingIsChunked=N, _receivingRequestDataForever=N
  _needToSetupRequestGunzip=N, _owedConsume=0, _wantSavedRequestData=N
  _100continue=0, _expectDisconnect=N, _overrideSourceState=0
POST arguments: {["cron_schedule"] = "0 6 * * 1", ["description"] = "scheduled search for view name=apigee", ["disabled"] = "0", ["is_scheduled"] = "1", ["action.email.to"] = "Raghav.Gomatham@company.com", ["action.email"] = "1", ["action.email.maxtime"] = "60m", ["action.email.pdfview"] = "apigee", ["action.email.sendpdf"] = "1", ["action.email.sendresults"] = "0", ["action.email.ttl"] = "10", ["action.email.subject.view"] = "Splunk Dashboard: '$name$'", ["action.email.message.view"] = "A PDF was generated for $name$", ["action.email.useNSSubject"] = "1", ["action.email.papersize"] = "letter", ["action.email.paperorientation"] = "portrait", ["action.email.priority"] = "3"}
REPLY: 502 Error connecting: Connection refused
    Set-Cookie: splunkd_8443=TO9e_LnLnFsbohCmdjWhvfkUdSk1ffBlKhPLTfWRDTEbF4QSAc9IdR7NHfbLVVsMXNPWyASNGJBDY6r3wTjrVOizBlRhaqjquk^QKGJKmNejuNEkLwFhifOTuceoDWNKeLBQ2I^4NF; Path=/; Secure; HttpOnly; Max-Age=3600; Expires=Mon, 07 Sep 2015 22:43:43 GMT
    Set-Cookie: splunkweb_csrf_token_8443=14807573426342708366; Path=/; Secure; Max-Age=157680000; Expires=Sat, 05 Sep 2020 21:43:43 GMT
admin_handler="scheduledviews"
MConfigHandler: name=scheduledviews, _atomFormat=2, _customAction=
  caller args: id="_ScheduledView__apigee": { action.email -> { _dataType=string _isMultiValue==N, _values: ["1"] }, action.email.maxtime -> { _dataType=string _isMultiValue==N, _values: ["60m"] }, action.email.pdfview -> { _dataType=string _isMultiValue==N, _values: ["apigee"] }, action.email.sendpdf -> { _dataType=string _isMultiValue==N, _values: ["1"] }, action.email.sendresults -> { _dataType=string _isMultiValue==N, _values: ["0"] }, action.email.ttl -> { _dataType=string _isMultiValue==N, _values: ["10"] }, alert_type -> { _dataType=string _isMultiValue==N, _values: ["always"] }, cron_schedule -> { _dataType=string _isMultiValue==N, _values: [""] }, description -> { _dataType=string _isMultiValue==N, _values: ["scheduled search for view name=apigee"] }, disabled -> { _dataType=string _isMultiValue==N, _values: ["0"] }, dispatch.earliest_time -> { _dataType=string _isMultiValue==N, _values: ["1"] }, dispatch.latest_time -> { _dataType=string _isMultiValue==N, _values: ["2"] }, isEventType -> { _dataType=string _isMultiValue==N, _values: ["0"] }, is_scheduled -> { _dataType=string _isMultiValue==N, _values: ["0"] }, is_visible -> { _dataType=string _isMultiValue==N, _values: ["0"] }, search -> { _dataType=string _isMultiValue==N, _values: ["| noop"] } }
  _docShowEntry=Y, _didFilter=N, _didPaginate=N
  _maxCount=30, posOffset=0, _requestedAction=4
  _shouldFilter=N, _shouldReload=N, _shouldAutoList=N, _sortSpecified=N
  _strict_mode=Y, _list_new=N, _force_stanza_overwite=N, _force_app_context_on_write=Y
  sort keys: ["name"]
  sort modes: ["auto"]
  supported args: ["action.email*" type=0, "action.email.to" type=0 (required), "cron_schedule" type=0 (required), "description" type=0, "disabled" type=0, "isProxyRequest" type=0, "is_scheduled" type=0 (required), "next_scheduled_time" type=0, "noProxy" type=0, action.email*:/(?s)action\.email.*/]
  Paginator: offset=0, count=30
  _customStatusCode=0, _supportedActions=0x3AE, hasSession=Y
  _forceBoolNormalization=Y, _contextMode=2, _didCapCheck=Y
  _ranSetup=Y, _restartRequired=N, _listingOne=N
  _userName=admin, _appName=search
ServicesEndpointReplyDataProvider: _setupState=0, _outputMode=2, _explicitOutputMode=Y
GET args: {}
  _allowedMethods={GET,POST,PUT,DELETE,HEAD,OPTIONS}, _preconditionState=0
  _wantsSeparateThread=N, _alreadyBuiltHeaders=N, _needToSendBody=Y
  _bodyBytesWritten=0, _chunkedState=0, _isLastTransaction=N
  _varyBy=0x8, _redirectUrl="", _downloadFilename="", _totalScheduledLength=0
  _willSendDataLater=N, _toSendState=2, _toSendSafe=Y
  _knowCompleteLength=Y, content-type="application/json; charset=UTF-8", _desiredCompressionLevel=6
  _replyIsGzipCompressed=N, _cacheControl=0x0, _maxCacheSeconds=4294967295, _dontIncludeFrameOptions==N
In TcpChannel 0x7f93574e1000, _tcloop=0x7f935b45dae0, no async write data, _data._shouldKill=N, r/w_timeouts=5000/300000, timeout_count=0
SSL: version="TLSv1.2", state="SSL negotiation finished successfully", cipher="AES256-SHA", compression="none"
rbuf: ptr=0x7f93574e10a0, size=0x2000, rptr=0x0, wptr=0x0
TcpChannelAcceptor: , tcloop=0x7f935b45dae0, _disabledReasons=0, _activeCount=5
HttpListener: ssl=Y, _maxActiveConnections=1365, _wellBelowConnectionLimit=Y, _maxThreads=1365
SplunkdHttpListener: PORT: _allowGzip=Y, bind=https://:8443
  conf: _sslVersions=SSL3,TLS1.0,TLS1.1,TLS1.2, _requireClientCert=N, _allowSslCompression=N, _allowSslRenegotiation=Y, _frameOptionsSameOrigin=Y, _strictTransportSecurityHeader=N, _allowBasicAuth=N, _allowCookieAuth=N
  conf: _streamInWriteTimeoutMsec=5000, _maxContentLength=524288000, _maxThreads=1365, _maxSockets=1365, _forceHttp10=0
_thread=0x7f9357607000: commandForThread=0, nextIdle=0x7f93575b3000, requestAfterThread=0, _tpfd=0x7f93574e1000, writeCorkCount=0, terminateCallback=(nil), ioError=No error, lastError=No error, terminateError=No error
giveCmd @0x7f9357607128: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, cmd=0, ok=Y, chan=0x7f9357547000
writeDataAvail @0x7f9357607188: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, chan=0x7f93575c5000
wbuf: ptr=0x7f9357607228, size=0x8000, rptr=0x0, wptr=0x0
x86 CPUID registers:
         0: 0000000B 756E6547 6C65746E 49656E69
         1: 000106A4 04080800 80982201 1FEBFBFF
         2: 76035A01 00F0B2FF 00000000 00CA0000
         3: 00000000 00000000 00000000 00000000
         4: 00000000 00000000 00000000 00000000
         5: 00001000 00001000 00000003 00000120
         6: 00000077 00000002 00000001 00000000
         7: 00000000 00000000 00000000 00000000
         8: 00000000 00000000 00000000 00000000
         9: 00000000 00000000 00000000 00000000
         A: 07300403 0000007F 00000000 00000000
         B: 00000000 00000000 6EF326FD 00000004
  80000000: 80000008 00000000 00000000 00000000
  80000001: 00000000 00000000 00000001 28100800
  80000002: 20202020 49202020 6C65746E 20295228
  80000003: 6E6F6558 20295228 20555043 322D3545
  80000004: 20303436 20402030 30352E32 007A4847
  80000005: 00000000 00000000 00000000 00000000
  80000006: 00000000 00000000 01006040 00000000
  80000007: 00000000 00000000 00000000 00000100
  80000008: 00003028 00000000 00000000 00000000
terminating...

Has anyone see this happening before? It restarts all my search heads and it's a strict no no.

Thanks in advance.

R2384

0 Karma
1 Solution

Giada8989
Engager

I had the same problem with 6.2.1 version
I found that it is solved with a newer release, like 6.2.2 or more

View solution in original post

0 Karma

Giada8989
Engager

I had the same problem with 6.2.1 version
I found that it is solved with a newer release, like 6.2.2 or more

0 Karma

Raghav2384
Motivator

It's a bug and I have been asked to upgrade to 6.2.5 latest at that time. That fixed the issue for me

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...