After setting the cron to schedule a PDF report, why am I getting error "Your network connection may have been lost or Splunk may be down." and all search heads restart?

[Raghav2384]

Experts,

When I try to schedule PDF for a report, I get the following error after I set the cron

"Your network connection may have been lost or Splunk may be down." and all my search heads start to restart.

Checked in _internal and this is what i found

[build 237341] 2015-09-07 17:43:44
Received fatal signal 6 (Aborted).
 Cause:
   Signal sent by PID 7962 running under UID 5132.
 Crashing thread: TcpChannelThread
 Registers:
    RIP:  [0x00007F936EBC2625] gsignal + 53 (/lib64/libc.so.6)
    RDI:  [0x0000000000001F1A]
    RSI:  [0x0000000000002044]
    RBP:  [0x00000000016CA900]
    RSP:  [0x00007F93445F8588]
    RAX:  [0x0000000000000000]
    RBX:  [0x00007F93343FF000]
    RCX:  [0xFFFFFFFFFFFFFFFF]
    RDX:  [0x0000000000000006]
    R8:  [0xFEFEFEFEFEFEFEFF]
    R9:  [0x00007F936FF25F60]
    R10:  [0x0000000000000008]
    R11:  [0x0000000000000202]
    R12:  [0x00000000016CB050]
    R13:  [0x00000000016CD640]
    R14:  [0x0000000000000000]
    R15:  [0x00007F93574C0AC0]
    EFL:  [0x0000000000000202]
    TRAPNO:  [0x0000000000000000]
    ERR:  [0x0000000000000000]
    CSGSFS:  [0x0000000000000033]
    OLDMASK:  [0x0000000000000000]
 OS: Linux
 Arch: x86-64
 Backtrace:
  [0x00007F936EBC2625] gsignal + 53 (/lib64/libc.so.6)
  [0x00007F936EBC3E05] abort + 373 (/lib64/libc.so.6)
  [0x00007F936EBBB74E] ? (/lib64/libc.so.6)
  [0x00007F936EBBB810] __assert_perror_fail + 0 (/lib64/libc.so.6)
  [0x0000000000ED555E] _ZN21HttpServerTransaction15postFinalResultERK9AppendStr + 158 (splunkd)
  [0x0000000000DF8EA1] _ZN15shpProxyRequest16proxyReqToMemberEP33ServicesEndpointReplyDataProviderRK14SchemeHostPortRK9AppendStrRbS8_PS5_ + 913 (splunkd)
  [0x0000000000DF9321] _ZN15shpProxyRequest17proxyReqToCaptainEP33ServicesEndpointReplyDataProviderRbS2_ + 193 (splunkd)
  [0x0000000000BCB637] _ZN14MConfigHandler24proxyShpRequestToCaptainERb + 215 (splunkd)
  [0x0000000000B9A415] _ZN23SavedSearchAdminHandler12handleCreateER10ConfigInfo + 85 (splunkd)
  [0x0000000000B9E405] _ZN25ScheduledViewAdminHandler19createScheduledViewER10ConfigInfo + 2101 (splunkd)
  [0x0000000000B9E949] _ZN25ScheduledViewAdminHandler10handleListER10ConfigInfo + 153 (splunkd)
  [0x0000000000BDA701] _ZN14MConfigHandler16handleListFilterER10ConfigInfo + 65 (splunkd)
  [0x0000000000BDA7DB] _ZN14MConfigHandler11getItemByIdERK3StrR10ConfigInfo + 43 (splunkd)
  [0x0000000000B9EA67] _ZN25ScheduledViewAdminHandler10handleEditER10ConfigInfo + 103 (splunkd)
  [0x0000000000BCF494] _ZN14MConfigHandler14executeHandlerER10ConfigInfo + 596 (splunkd)
  [0x0000000000BDC054] _ZN14MConfigHandler2goER10ConfigInfo + 212 (splunkd)
  [0x0000000000BDDED5] _ZN29AdminManagerReplyDataProvider2goEv + 693 (splunkd)
  [0x0000000000C5CBB2] _ZN33ServicesEndpointReplyDataProvider9rawHandleEv + 50 (splunkd)
  [0x0000000000C539B7] _ZN18RawRestHttpHandler10getPreBodyEP21HttpServerTransaction + 39 (splunkd)
  [0x0000000000ED8C90] _ZN32HttpThreadedCommunicationHandler11communicateER17TcpSyncDataBuffer + 320 (splunkd)
  [0x0000000000F416D9] _ZN16TcpChannelThread4mainEv + 473 (splunkd)
  [0x0000000000F4526E] _ZN6Thread8callMainEPv + 62 (splunkd)
  [0x00007F936EF2B9D1] ? (/lib64/libpthread.so.0)
  [0x00007F936EC788FD] clone + 109 (/lib64/libc.so.6)
 Linux / host.x.com / 2.6.32-504.30.3.el6.x86_64 / #1 SMP Thu Jul 9 15:20:47 EDT 2015 / x86_64
 Last few lines of stderr (may contain info on assertion failure, but also could be old):
    2015-08-31 20:23:42.193 -0400 splunkd started (build 237341)
    2015-09-02 01:01:11.355 -0400 Interrupt signal received
    2015-09-02 01:01:46.782 -0400 splunkd started (build 237341)
    2015-09-07 16:14:04.429 -0400 Interrupt signal received
    2015-09-07 16:14:37.336 -0400 splunkd started (build 237341)
    splunkd: /home/build/build-src/6.2.0/src/util/HttpListener.cpp:2691: void HttpServerTransaction::postFinalResult(const AppendStr&): Assertion `_toSendState == TOSEND_VIRGIN || _toSendState == TOSEND_UNSET' failed.
    2015-09-07 17:42:44.298 -0400 splunkd started (build 237341)
    splunkd: /home/build/build-src/6.2.0/src/util/HttpListener.cpp:2691: void HttpServerTransaction::postFinalResult(const AppendStr&): Assertion `_toSendState == TOSEND_VIRGIN || _toSendState == TOSEND_UNSET' failed.
 /etc/redhat-release: Red Hat Enterprise Linux Server release 6.6 (Santiago)
 glibc version: 2.12
 glibc release: stable
Last errno: 111
Threads running: 70
argv: [splunkd -p 8089 restart]
Thread: "TcpChannelThread", did_join=0, ready_to_run=Y, main_thread=N
First 8 bytes of Thread token @0x7f9357607010:
00000000  00 a7 5f 44 93 7f 00 00                           |.._D....|
00000008
commandForThread=0, nextIdle=0x7f93575b3000, requestAfterThread=0, _tpfd=0x7f93574e1000, writeCorkCount=0, terminateCallback=(nil), ioError=No error, lastError=No error, terminateError=No error
giveCmd @0x7f9357607128: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, cmd=0, ok=Y, chan=0x7f9357547000
writeDataAvail @0x7f9357607188: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, chan=0x7f93575c5000
wbuf: ptr=0x7f9357607228, size=0x8000, rptr=0x0, wptr=0x0
HttpListeningConnection: _transactionActive=Y, _haveHadTransaction=Y, _alreadyLoggedTimeout=N
HttpTcpConnection: peer=10.62.60.37, _desiredCompressionLevel=6
RestHttpServerTransaction: _restPath="scheduled/views/_ScheduledView__apigee", namespaced=Y, context=admin/search, session=[user=admin, refcnt=6, touched=1441662223, refreshEligible=1441662403, removed=N, id=944b7a4e715021fc02b21016e247b95e, created=1441647731, refreshed=1441662178, expires=1441665778, initialLife=3600, createdBy=055D1A79-AE4D-4C4C-95F8-5728656120E6, portable, ip=10.62.60.37, csrf=14807573426342708366]
HttpServerTransaction: _state=6, _shouldLog=Y, _startTime=1441662223.766261
REQUEST: POST /en-US/splunkd/__raw/servicesNS/admin/search/scheduled/views/_ScheduledView__apigee HTTP/1.1
    Host: host:8443
    Connection: keep-alive
    Content-Length: 539
    Accept: text/javascript, text/html, application/xml, text/xml, */*
    Origin: host:8443
    X-Requested-With: XMLHttpRequest
    X-Splunk-Form-Key: 14807573426342708366
    User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36
    Content-Type: application/x-www-form-urlencoded; charset=UTF-8
    Referer: https://host:8443/en-US/app/search/dashboards
    Accept-Encoding: gzip, deflate
    Accept-Language: en-US,en;q=0.8
    Cookie: location-zipcode-invalid-noservice=36116; company-locale=Wj0zNjExNl58XlQ9Ul58Xkw9XnxeSD1odHRwczovL3d3dy5jb3guY29tL3Jlc2lkZW50aWFsLXN0b3JlL2hvbWUuY294; TandC=6; mbox=PC#1439421966867-537435.17_50#1449235297|check#true#1441459357|session#1441459296616-488211#1441461157; s_vi=[CS]v1|2A2409F105078C43-4000010E4005D86B[CE]; s_fid=75AFCB4365A850EB-0A12B884A1202DAA; company-current-zipcode=36116; AMCV_8C6767C25245AD1A0A490D4C%40AdobeOrg=-2017484664%7CMCAID%7C2A2409F105078C43-4000010E4005D86B%7CMCMID%7C74855126067035151220716829742186752874%7CMCAAMLH-1442064162%7C7%7CMCAAMB-1442064162%7Chmk_Lq6TPIBMW925SPhw3Q; SMIDENTITY=S786vxgAWtHbFZrJkZvk0GdoOXh7L2et9e9YCgHB18A1e1XNkuKbuquvmSyACzZebubmzbSUOA3q2RsFaryaZ68UCnM1cziKihbdkEtbulIt4ieXg+zBMgd+6qmHCq58q1BMGR/r+yrrvSPHwOzYMYewhym4V5nwGMezZhdr47XS2V3If+et87esl31Mu4B1KPH00L8isXx64ToWzy6tWzOYR0kjXIGzmtRj9CXvtdhCJaiTpOoLzl2HsAC3cmE+RrP8vPFX/Fd2ZNkVX/sguOts6+VeMk8fEBevVnoJv3JREec3PhOjQJ7jzDcomZFj0CWBh1uKxLXycoBFeysKTljvrfluH/OKCHCOd5t4qdHwDzwA+v/CGOT62Q3C2q3ir83wjCFqj3zqnaypNqo5vsz/SsUeIyq35ELV9NU2BRSWqWaUUwKfP2E8RtwKTv7Rfdc3WE18BlT1oei40TLmyfajgITriLEvoxEwxMBagFKT1RDlrt8KoCKpbYyOrBEozHm0b153egIMxs2hKXLWPMr6JyRa+lKo7gQGvEtkTFCUhNX6iCrb2QkIqX6u2bW29vNEFF/FmFNp4AaPWkk1+z0+T3ce/GKBzLIUOrhKl159Z7Rd2hj5cwPPZFhPtBR7EcCrMPp/WYiQtBLlxO301IflaISFSaJuaw5R6yAbu0fnAb0GczpnYNHdKXLLS+DQ; utag_main=_st:1441461181848$ses_id:1441460050249%3Bexp-session$offer_origin:companycom%3Bexp-session$_prevpage:corp%3Aresaccount%3Asign%20in%3Bexp-1441462983894$companyVisitorType:noncustomer%3Aunknown%3Aanonymous%3Aunknown$scid_referrer:0%3Bexp-session; s_pers=%20s_ev41%3D%255B%255B%27shop_all_setupnewservice_offer3_3%27%252C%271414009885379%27%255D%252C%255B%27shop_all_setupnewservice_offer3_3%27%252C%271414009902584%27%255D%252C%255B%27shop_all_setupnewservice_offer3_3%27%252C%271414009915559%27%255D%252C%255B%27shop_all_setupnewservice_offer3_3%27%252C%271414009937171%27%255D%252C%255B%27shop_all_setupnewservice_offer3_3%27%252C%271414009938795%27%255D%255D%7C1571776338795%3B%20pn%3D4%7C1444051305062%3B%20gpv_v14%3Dcompany%253Abus%253Ahome%7C1441461105065%3B%20s_lv%3D1441459383949%7C1536067383949%3B%20s_lv_s%3DMore%2520than%25207%2520days%7C1441461183949%3B%20s_invisit%3Dtrue%7C1441461183956%3B%20s_vnum%3D1449235383956%2526vn%253D2%7C1449235383956%3B; aam_sc=seg%3DGeneric%20Company.com%20Visitor; aam_uuid=74629000044245982600685355110506814572; session_id_8443=7176c88212a2ba5cdb6b10bf05a79be90fb35dd6; splunkd_8443=TO9e_LnLnFsbohCmdjWhvfkUdSk1ffBlKhPLTfWRDTEbF4QSAc9IdR7NHfbLVVsMXNPWyASNGJBDY6r3wTjrVOizBlRhaqjquk^QKGJKmNejuNEkLwFhifOTuceoDWNKeLBQ2I^4NF; splunkweb_csrf_token_8443=14807573426342708366
  _bytesReceived=539, _maximumRequestDataSize=524288000, _totalBytesExpectedOfRequestData=539
  _bytesLeftInRequestDataChunk=0, _requestTransferEncodingIsChunked=N, _receivingRequestDataForever=N
  _needToSetupRequestGunzip=N, _owedConsume=0, _wantSavedRequestData=N
  _100continue=0, _expectDisconnect=N, _overrideSourceState=0
POST arguments: {["cron_schedule"] = "0 6 * * 1", ["description"] = "scheduled search for view name=apigee", ["disabled"] = "0", ["is_scheduled"] = "1", ["action.email.to"] = "Raghav.Gomatham@company.com", ["action.email"] = "1", ["action.email.maxtime"] = "60m", ["action.email.pdfview"] = "apigee", ["action.email.sendpdf"] = "1", ["action.email.sendresults"] = "0", ["action.email.ttl"] = "10", ["action.email.subject.view"] = "Splunk Dashboard: '$name$'", ["action.email.message.view"] = "A PDF was generated for $name$", ["action.email.useNSSubject"] = "1", ["action.email.papersize"] = "letter", ["action.email.paperorientation"] = "portrait", ["action.email.priority"] = "3"}
REPLY: 502 Error connecting: Connection refused
    Set-Cookie: splunkd_8443=TO9e_LnLnFsbohCmdjWhvfkUdSk1ffBlKhPLTfWRDTEbF4QSAc9IdR7NHfbLVVsMXNPWyASNGJBDY6r3wTjrVOizBlRhaqjquk^QKGJKmNejuNEkLwFhifOTuceoDWNKeLBQ2I^4NF; Path=/; Secure; HttpOnly; Max-Age=3600; Expires=Mon, 07 Sep 2015 22:43:43 GMT
    Set-Cookie: splunkweb_csrf_token_8443=14807573426342708366; Path=/; Secure; Max-Age=157680000; Expires=Sat, 05 Sep 2020 21:43:43 GMT
admin_handler="scheduledviews"
MConfigHandler: name=scheduledviews, _atomFormat=2, _customAction=
  caller args: id="_ScheduledView__apigee": { action.email -> { _dataType=string _isMultiValue==N, _values: ["1"] }, action.email.maxtime -> { _dataType=string _isMultiValue==N, _values: ["60m"] }, action.email.pdfview -> { _dataType=string _isMultiValue==N, _values: ["apigee"] }, action.email.sendpdf -> { _dataType=string _isMultiValue==N, _values: ["1"] }, action.email.sendresults -> { _dataType=string _isMultiValue==N, _values: ["0"] }, action.email.ttl -> { _dataType=string _isMultiValue==N, _values: ["10"] }, alert_type -> { _dataType=string _isMultiValue==N, _values: ["always"] }, cron_schedule -> { _dataType=string _isMultiValue==N, _values: [""] }, description -> { _dataType=string _isMultiValue==N, _values: ["scheduled search for view name=apigee"] }, disabled -> { _dataType=string _isMultiValue==N, _values: ["0"] }, dispatch.earliest_time -> { _dataType=string _isMultiValue==N, _values: ["1"] }, dispatch.latest_time -> { _dataType=string _isMultiValue==N, _values: ["2"] }, isEventType -> { _dataType=string _isMultiValue==N, _values: ["0"] }, is_scheduled -> { _dataType=string _isMultiValue==N, _values: ["0"] }, is_visible -> { _dataType=string _isMultiValue==N, _values: ["0"] }, search -> { _dataType=string _isMultiValue==N, _values: ["| noop"] } }
  _docShowEntry=Y, _didFilter=N, _didPaginate=N
  _maxCount=30, posOffset=0, _requestedAction=4
  _shouldFilter=N, _shouldReload=N, _shouldAutoList=N, _sortSpecified=N
  _strict_mode=Y, _list_new=N, _force_stanza_overwite=N, _force_app_context_on_write=Y
  sort keys: ["name"]
  sort modes: ["auto"]
  supported args: ["action.email*" type=0, "action.email.to" type=0 (required), "cron_schedule" type=0 (required), "description" type=0, "disabled" type=0, "isProxyRequest" type=0, "is_scheduled" type=0 (required), "next_scheduled_time" type=0, "noProxy" type=0, action.email*:/(?s)action\.email.*/]
  Paginator: offset=0, count=30
  _customStatusCode=0, _supportedActions=0x3AE, hasSession=Y
  _forceBoolNormalization=Y, _contextMode=2, _didCapCheck=Y
  _ranSetup=Y, _restartRequired=N, _listingOne=N
  _userName=admin, _appName=search
ServicesEndpointReplyDataProvider: _setupState=0, _outputMode=2, _explicitOutputMode=Y
GET args: {}
  _allowedMethods={GET,POST,PUT,DELETE,HEAD,OPTIONS}, _preconditionState=0
  _wantsSeparateThread=N, _alreadyBuiltHeaders=N, _needToSendBody=Y
  _bodyBytesWritten=0, _chunkedState=0, _isLastTransaction=N
  _varyBy=0x8, _redirectUrl="", _downloadFilename="", _totalScheduledLength=0
  _willSendDataLater=N, _toSendState=2, _toSendSafe=Y
  _knowCompleteLength=Y, content-type="application/json; charset=UTF-8", _desiredCompressionLevel=6
  _replyIsGzipCompressed=N, _cacheControl=0x0, _maxCacheSeconds=4294967295, _dontIncludeFrameOptions==N
In TcpChannel 0x7f93574e1000, _tcloop=0x7f935b45dae0, no async write data, _data._shouldKill=N, r/w_timeouts=5000/300000, timeout_count=0
SSL: version="TLSv1.2", state="SSL negotiation finished successfully", cipher="AES256-SHA", compression="none"
rbuf: ptr=0x7f93574e10a0, size=0x2000, rptr=0x0, wptr=0x0
TcpChannelAcceptor: , tcloop=0x7f935b45dae0, _disabledReasons=0, _activeCount=5
HttpListener: ssl=Y, _maxActiveConnections=1365, _wellBelowConnectionLimit=Y, _maxThreads=1365
SplunkdHttpListener: PORT: _allowGzip=Y, bind=https://:8443
  conf: _sslVersions=SSL3,TLS1.0,TLS1.1,TLS1.2, _requireClientCert=N, _allowSslCompression=N, _allowSslRenegotiation=Y, _frameOptionsSameOrigin=Y, _strictTransportSecurityHeader=N, _allowBasicAuth=N, _allowCookieAuth=N
  conf: _streamInWriteTimeoutMsec=5000, _maxContentLength=524288000, _maxThreads=1365, _maxSockets=1365, _forceHttp10=0
_thread=0x7f9357607000: commandForThread=0, nextIdle=0x7f93575b3000, requestAfterThread=0, _tpfd=0x7f93574e1000, writeCorkCount=0, terminateCallback=(nil), ioError=No error, lastError=No error, terminateError=No error
giveCmd @0x7f9357607128: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, cmd=0, ok=Y, chan=0x7f9357547000
writeDataAvail @0x7f9357607188: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, chan=0x7f93575c5000
wbuf: ptr=0x7f9357607228, size=0x8000, rptr=0x0, wptr=0x0
x86 CPUID registers:
         0: 0000000B 756E6547 6C65746E 49656E69
         1: 000106A4 04080800 80982201 1FEBFBFF
         2: 76035A01 00F0B2FF 00000000 00CA0000
         3: 00000000 00000000 00000000 00000000
         4: 00000000 00000000 00000000 00000000
         5: 00001000 00001000 00000003 00000120
         6: 00000077 00000002 00000001 00000000
         7: 00000000 00000000 00000000 00000000
         8: 00000000 00000000 00000000 00000000
         9: 00000000 00000000 00000000 00000000
         A: 07300403 0000007F 00000000 00000000
         B: 00000000 00000000 6EF326FD 00000004
  80000000: 80000008 00000000 00000000 00000000
  80000001: 00000000 00000000 00000001 28100800
  80000002: 20202020 49202020 6C65746E 20295228
  80000003: 6E6F6558 20295228 20555043 322D3545
  80000004: 20303436 20402030 30352E32 007A4847
  80000005: 00000000 00000000 00000000 00000000
  80000006: 00000000 00000000 01006040 00000000
  80000007: 00000000 00000000 00000000 00000100
  80000008: 00003028 00000000 00000000 00000000
terminating...

Has anyone see this happening before? It restarts all my search heads and it's a strict no no.

Thanks in advance.

R2384

[Giada8989 · Accepted solution]

I had the same problem with 6.2.1 version
I found that it is solved with a newer release, like 6.2.2 or more

[Raghav2384]

It's a bug and I have been asked to upgrade to 6.2.5 latest at that time. That fixed the issue for me