here is sample event from log file. Issue is, in below event there is a timestamp. That and splunk time not matching. "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","xx.xxx.xxx.xx","XXXXXXXXXX","200","","","XXXX","xxxxxxxxxxx-xxx.xx.com","","xxxx xxxx xxxxx","xxxxx","http://xxxx.xxx.com/xxxxxxxxxxx/xxxxx/service/bac/v1","xxxxxxxxxxx","xxxxxx","xxxxxxxx","XXXXXXX","xxxxxx","","","ID","37","1","6809","12928",***"2020-02-20 10:14:59.363"***,"2","0","88","97","0","0","6","0","1","0","0","0","0","SOAP","false","40000","false" ... View more

So are you suggesting, we add another volume /Data2 And on any net new indexes send them to /Data2? Do you know the procedure to move existing index data to /Data2? ... View more

I enabled HEC manually on each indexers. When there are multiple indexers behind a load balancer, we will be specifying splunk-url=ourladobancerurl But what about --log-opt splunk-token how can we specify all the indexers token there? ... View more

Thank you for responding. I think we want to go with http://dev.splunk.com/view/event-collector/SP-CAAAE73#scen1 We are at 6.3.3 and referring http://docs.splunk.com/Documentation/Splunk/6.3.3/Data/UsetheHTTPEventCollector when checking "select allowed indexes" field, we do not see all our indexes from the index cluster listed there. Do you know what are we missing? ... View more

last question, we came to search head clustering from single search head. So we had copied user items manually from that single search head to clustered search heads. So will those items also get replicated to newly added server? ... View more

No i am not talking about DMC, but indexmaster:8080/en-US/manager/system/clustering?tab=searchheads ... View more

so when added the new search head, I did below on the new search head to get it added to index master dashboard (Search Heads tab) splunk edit cluster-config -mode searchhead -master_uri https:// what would be the command to remove our old search head from the index master dashboard? ... View more

that worked nicely. user saved items got synced to new search head. Now to permanently remove the old one, i got to run below from the old search head to be removed? splunk remove shcluster-member splunk stop ... View more

So basically i got to run below on the new member? splunk init shcluster-config -auth admin:<password> -mgmt_uri https://<newservername>:8089 -replication_port 34567 -replication_factor 4 -conf_deploy_fetch_url https://<ourshcdeployername>:8089 -secret <password> -shcluster_label <ourlabel> splunk restart splunk add shcluster-member -current_member_uri https://<anyexistingmembername>:8089 ... View more

yes i had, i couldn't find anything wrong there. 2017-05-04 09:11:29.474 dbx123:INFO:TailDatabaseMonitor - Executing database monitor=[dbmon-tail://xxxxx/xxxxx1_new_new] 2017-05-04 09:11:24.472 dbx9400:INFO:TailDatabaseMonitor - Database monitor=[dbmon-tail://xxxxx/xxxxx1_new_new] finished with status=true resultCount=294 in duration=1473 ms 2017-05-04 09:11:22.999 dbx9400:INFO:TailDatabaseMonitor - Executing database monitor=[dbmon-tail://xxxxx/xxxxx1_new_new] 2017-05-04 09:11:17.997 dbx9861:INFO:TailDatabaseMonitor - Database monitor=[dbmon-tail://xxxxx/xxxxx1_new_new] finished with status=true resultCount=380 in duration=2091 ms 2017-05-04 09:11:15.905 dbx9861:INFO:TailDatabaseMonitor - Executing database monitor=[dbmon-tail://xxxxx/xxxxx1_new_new] 2017-05-04 09:11:10.904 dbx6352:INFO:TailDatabaseMonitor - Database monitor=[dbmon-tail://xxxxx/xxxxx1_new_new] finished with status=true resultCount=327 in duration=2018 ms 2017-05-04 09:11:08.886 dbx6352:INFO:TailDatabaseMonitor - Executing database monitor=[dbmon-tail://xxxxx/xxxxx1_new_new] 2017-05-04 09:11:03.885 dbx9475:INFO:TailDatabaseMonitor - Database monitor=[dbmon-tail://xxxxx/xxxxx1_new_new] finished with status=true resultCount=293 in duration=1618 ms ... View more

this resolved. ... View more

Hello Burch, I must have said wrong on the call. The config is on the forwarder and on indexers. Below is a sample. It has more lines similar. Around 8000. We want that whole 8000 lines as one event. <?xml version="1.0" encoding="UTF-8" ?> <?xml-stylesheet type='text/xsl' href='freb.xsl'?> <!-- saved from url=(0014)about:internet --> <failedRequest url="https://xxxxx.fmr.com:443/xxxxxx/xxxxxxx.aspx" siteId="21" appPoolId="xxx_xx" processId="xxxxx" verb="POST" remoteUserName="" userName="" tokenUserName="NT AUTHORITY\IUSR" authenticationType="anonymous" activityId="{00000000-0000-0000-D07B-0080010000DB}" failureReason="STATUS_CODE" statusCode="500" triggerStatusCode="500" timeTaken="765" xmlns:freb="http://schemas.microsoft.com/win/2006/06/iis/freb" > <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="WWW Server" Guid="{3A2A4E84-4C21-4981-AE10-3FDA0D9B0F83}"/> <EventID>0</EventID> <Version>1</Version> <Level>5</Level> <Opcode>2</Opcode> <Keywords>0x100</Keywords> <TimeCreated SystemTime="2017-02-08T18:02:20.952Z"/> <Correlation ActivityID="{00000000-0000-0000-D07B-0080010000DB}"/> <Execution ProcessID="12812" ThreadID="16488"/> <Computer>xxxxxxx</Computer> </System> <EventData> <Data Name="ContextId">{00000000-0000-0000-D07B-0080010000DB}</Data> <Data Name="ModuleName">FailedRequestsTracingModule</Data> <Data Name="Notification">1</Data> <Data Name="fIsPostNotificationEvent">false</Data> <Data Name="NotificationStatus">0</Data> </EventData> <RenderingInfo Culture="en-US"> <Opcode>NOTIFY_MODULE_END</Opcode> <Keywords> <Keyword>RequestNotifications</Keyword> </Keywords> <freb:Description Data="Notification">BEGIN_REQUEST</freb:Description> <freb:Description Data="NotificationStatus">NOTIFICATION_CONTINUE</freb:Description> </RenderingInfo> <ExtendedTracingInfo xmlns="http://schemas.microsoft.com/win/2004/08/events/trace"> <EventGuid>{002E91E3-E7AE-44AB-8E07-99230FFA6ADE}</EventGuid> </ExtendedTracingInfo> </Event> <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="WWW Server" Guid="{3A2A4E84-4C21-4981-AE10-3FDA0D9B0F83}"/> <EventID>0</EventID> <Version>1</Version> <Level>5</Level> <Opcode>1</Opcode> <Keywords>0x100</Keywords> <TimeCreated SystemTime="2017-02-08T18:02:20.952Z"/> <Correlation ActivityID="{00000000-0000-0000-D07B-0080010000DB}"/> <Execution ProcessID="12812" ThreadID="16488"/> <Computer>xxxxxxx</Computer> </System> <EventData> <Data Name="ContextId">{00000000-0000-0000-D07B-0080010000DB}</Data> <Data Name="ModuleName">ConfigurationValidationModule</Data> <Data Name="Notification">1</Data> <Data Name="fIsPostNotification">false</Data> </EventData> <RenderingInfo Culture="en-US"> <Opcode>NOTIFY_MODULE_START</Opcode> <Keywords> <Keyword>RequestNotifications</Keyword> </Keywords> <freb:Description Data="Notification">BEGIN_REQUEST</freb:Description> </RenderingInfo> <ExtendedTracingInfo xmlns="http://schemas.microsoft.com/win/2004/08/events/trace"> <EventGuid>{002E91E3-E7AE-44AB-8E07-99230FFA6ADE}</EventGuid> </ExtendedTracingInfo> </Event> </failedRequest> ... View more

I tried to do this, but sill i am not getting it as one full event, i put below in my forwarder and index props.conf [ name ] SHOULD_LINEMERGE=true NO_BINARY_CHECK=true CHARSET=UTF-8 disabled=false BREAK_ONLY_BEFORE=GobblyGook MAX_EVENTS=100000 ... View more

that did not work either. ... View more

"Sunday 2016-10-09 12:02:46,047 [tomcat-http--9]" is the beginning of the event I tried below and did not work [ecapi_log4j_2] TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N MAX_TIMESTAMP_LOOKAHEAD = 26 **SHOULD_LINEMERGE = true** LINE_BREAKER = ([\r\n]+)(?:\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}) TRUNCATE = 200000 ... View more

That did not work. still multiple events getting clubbed together as one event in splunk. These events are so large. our splunkd.log has below many times, 10-13-2016 06:43:14.649 -0400 WARN LineBreakingProcessor - Truncating line because limit of 100000 bytes has been exceeded with a line length >= 131072 - data_source="<source masked>", data_host="<host masked>", data_sourcetype="ecapi_log4j_2" ... View more