Splunk Search

dispatch.finalizeRemoteTimeline taking a long time?

dshpritz
SplunkTrust
SplunkTrust

When I run a search in Splunk 6.x, the results come back quickly, but it seems like a lot of time is spent on "Finalize Job". When I look at the search inspector, it shows that a lot of time is spend on "dispatch.finalizeRemoteTimeline". Is there anything I can do about that?

Thanks,

Dave

Tags (1)
1 Solution

abhijitmishra
Explorer

These were turned on by default in Splunk 6 for some reason. In the limits.conf on local folder, add these two configurations.

fetch_remote_search_log = false

remote_timeline_fetchall = false

View solution in original post

koshyk
Super Champion

Just to make the answer more clear for those who want to test out. (It had quite significant change for us at 6.3.x version)

  • Create an app eg: MY_sh_limits and put these settings into "limits.conf"
    Ultmiately it will be present in $SPLUNK_HOME/etc/apps/MY_sh_limits/default/limits.conf of your SH member.

  • Deploy it to SH cluster (via deployer) or manually if its a standalone. Please note, the values are "disabled" in newer versions.

    [search]
    fetch_remote_search_log = disabled
    remote_timeline_fetchall = disabled

  • Check the difference in speed after implementing it. If its bad, just remove MY_sh_limits app. if cluster, just redeploy from deployer.

sibbsnb
Path Finder

What is the impact of turning this off?

anupkpal
New Member

Can someone tell me the full path name of limits.conf file? Where is this .conf file located?

Thanks

Anup pal

0 Karma

anupkpal
New Member

Can someone tell me the full path name of limits.conf file? Where is this .conf file located?

Thanks

Anup pal

0 Karma

koshyk
Super Champion

it depends it you have cluster or standalone system. What we do is to put limits.conf within an app and push it to all Search heads.
So if you create an app & put in MY_sh_limits/local/limits.conf and these settings under

# $SPLUNK_HOME/etc/apps/MY_sh_limits/local/limits.conf 
[search]
fetch_remote_search_log = disabled
remote_timeline_fetchall = disabled
0 Karma

bnorthway_splun
Splunk Employee
Splunk Employee

FYI this is no longer considered a "best practice" by Splunk Professional Services.

0 Karma

aladda_splunk
Splunk Employee
Splunk Employee

@bnorthway - Are you saying setting these settings to false is no longer a PS best practice or the other way around?

fetch_remote_search_log = false
remote_timeline_fetchall = false

0 Karma

bnorthway_splun
Splunk Employee
Splunk Employee

According to Sanford, it is not recommended to change this setting for versions after 6.1, 6.2, etc.

0 Karma

pcsegal
Explorer

What would be the recommended fix for this problem?

0 Karma

Lucas_K
Motivator

I had the same thought as you so I looked into it. Turned both of them off and the change was AMAZING.

The "fetch_remote_search_log" option is the search.log files generated by each search peer. These are accessable via the job inspector at the very bottom. You will normally see them in the "Additional info" section. You can use these for troubleshooting but are generally not used very often at all. The more search peers you have the more of these search logs are created and sent back to the search head. Seems quite safe to turn off. Honestly this should probably be off by default and turned on for debugging purposes only.

The "remote_timeline_fetchall" im not 100% sure as the description is a little too vague. I'm guessing its the little bar chart in the flashtimeline/search bar. shrugs

abhijitmishra
Explorer

These were turned on by default in Splunk 6 for some reason. In the limits.conf on local folder, add these two configurations.

fetch_remote_search_log = false

remote_timeline_fetchall = false

View solution in original post

alacercogitatus
SplunkTrust
SplunkTrust

I know this is an old post, but this changes should take place on the SH, not needed on IDX. Just FYI for future me.

anwarmian
Communicator

I have not tried it but isn't "remote_timeline_fetchall = false" good for reporting searches instead of search that you run on search window?

0 Karma

luhadia_aditya
Path Finder

What is the impact of turning this off?

0 Karma

anupkpal
New Member

Where do I find limits.conf file? Can some one send me the full path name of limits.conf file?

Thanks,

Anup Pal

0 Karma

a212830
Champion

Same deal here - all Splunk 6. Search runs fine, but the finalizing takes a while.

0 Karma

dshpritz
SplunkTrust
SplunkTrust

Nope. Splunk 6 all the way around.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Are you using a Splunk 6 searchhead on a Splunk 5 indexer? If so, take a look at http://docs.splunk.com/Documentation/Splunk/6.0.1/DistSearch/Versioncompatibility#Remote_timeline_fe...

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!