How to connect heavy forwarder running Splunk DB Connect to Splunk Cloud?

[dbcase]

Hi,

I have a heavy forwarder running Splunk DB Connect (Splunk DB Connect is configured and working properly). What I need to do is get the data from Splunk DB Connect searches to Splunk Cloud. I've looked at several different documentation pages and answers but for the life of me I can't figure out where this went sideways.

on the Splunk Cloud instance if I run this search

index=_internal 10.30.28.220 

I do see some data getting from the heavy forwarder (10.30.28.220) to Splunk Cloud

2/10/17
1:26:31.143 PM  
02-10-2017 19:26:31.143 +0000 INFO  StreamedSearch - Streamed search connection terminated: search_id=remote_sh1.icontrol.splunkcloud.com_1486754790.435, server=sh1.icontrol.splunkcloud.com, active_searches=3, elapsedTime=0.481, search='litsearch ( index=_internal 10.30.28.220 ) | fields  keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"  | remotetl  nb=300 et=2147483647.000000 lt=0.000000 remove=true max_count=1000 max_prefetch=100', savedsearch_name=""
date_hour = 19 date_mday =  10 date_minute =    26 date_month = february date_second =  31 date_wday =  friday date_year =  2017 date_zone =    0 eventtype =   external-referer    eventtype = nix-all-logs    eventtype = visitor-type-referred host =    idx5.icontrol.splunkcloud.com index =   _internal linecount =   1 punct =   --_::._+____-____:_=....,_=...,_=,_=.,_='_(_=_..._ search = 'litsearch ( index=_internal 10.30.28.220 ) | fields keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server" | remotetl nb=300 et=2147483647.000000 lt=0.000000 remove=true max_count=1000 max_prefetch=100' server =    sh1.icontrol.splunkcloud.com source =   /opt/splunk/var/log/splunk/remote_searches.log sourcetype = splunkd_remote_searches splunk_server = idx5.icontrol.splunkcloud.com timeendpos =  29 timestartpos =   0 unix_category =   all_hosts unix_group =  default
2/10/17
1:26:30.674 PM  
02-10-2017 19:26:30.674 +0000 INFO  StreamedSearch - Streamed search search starting: search_id=remote_sh1.icontrol.splunkcloud.com_1486754790.435, server=sh1.icontrol.splunkcloud.com, active_searches=4, search='litsearch ( index=_internal 10.30.28.220 ) | fields  keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"  | remotetl  nb=300 et=2147483647.000000 lt=0.000000 remove=true max_count=1000 max_prefetch=100', remote_ttl=600, apiStartTime='ZERO_TIME', apiEndTime='ZERO_TIME', savedsearch_name=""
date_hour = 19 date_mday =  10 date_minute =    26 date_month = february date_second =  30 date_wday =  friday date_year =  2017 date_zone =    0 eventtype =   external-referer    eventtype = nix-all-logs    eventtype = visitor-type-referred host =    idx1.icontrol.splunkcloud.com index =   _internal linecount =   1 punct =   --_::._+____-____:_=....,_=...,_=,_='_(_=_..._)_|_ search = 'litsearch ( index=_internal 10.30.28.220 ) | fields keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server" | remotetl nb=300 et=2147483647.000000 lt=0.000000 remove=true max_count=1000 max_prefetch=100' server =    sh1.icontrol.splunkcloud.com source =   /opt/splunk/var/log/splunk/remote_searches.log sourcetype = splunkd_remote_searches splunk_server = idx1.icontrol.splunkcloud.com timeendpos =  29 timestartpos =   0 unix_category =   all_hosts unix_group =  default
2/10/17
1:26:30.672 PM  
02-10-2017 19:26:30.672 +0000 INFO  StreamedSearch - Streamed search search starting: search_id=remote_sh1.icontrol.splunkcloud.com_1486754790.435, server=sh1.icontrol.splunkcloud.com, active_searches=4, search='litsearch ( index=_internal 10.30.28.220 ) | fields  keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"  | remotetl  nb=300 et=2147483647.000000 lt=0.000000 remove=true max_count=1000 max_prefetch=100', remote_ttl=600, apiStartTime='ZERO_TIME', apiEndTime='ZERO_TIME', savedsearch_name=""
date_hour = 19 date_mday =  10 date_minute =    26 date_month = february date_second =  30 date_wday =  friday date_year =  2017 date_zone =    0 eventtype =   external-referer    eventtype = nix-all-logs    eventtype = visitor-type-referred host =    idx3.icontrol.splunkcloud.com index =   _internal linecount =   1 punct =   --_::._+____-____:_=....,_=...,_=,_='_(_=_..._)_|_ search = 'litsearch ( index=_internal 10.30.28.220 ) | fields keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server" | remotetl nb=300 et=2147483647.000000 lt=0.000000 remove=true max_count=1000 max_prefetch=100' server =    sh1.icontrol.splunkcloud.com source =   /opt/splunk/var/log/splunk/remote_searches.log sourcetype = splunkd_remote_searches splunk_server = idx3.icontrol.splunkcloud.com timeendpos =  29 timestartpos =   0 unix_category =   all_hosts unix_group =  default
2/10/17
1:26:30.671 PM  
02-10-2017 19:26:30.671 +0000 INFO  StreamedSearch - Streamed search search starting: search_id=remote_sh1.icontrol.splunkcloud.com_1486754790.435, server=sh1.icontrol.splunkcloud.com, active_searches=4, search='litsearch ( index=_internal 10.30.28.220 ) | fields  keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"  | remotetl  nb=300 et=2147483647.000000 lt=0.000000 remove=true max_count=1000 max_prefetch=100', remote_ttl=600, apiStartTime='ZERO_TIME', apiEndTime='ZERO_TIME', savedsearch_name=""
date_hour = 19 date_mday =  10 date_minute =    26 date_month = february date_second =  30 date_wday =  friday date_year =  2017 date_zone =    0 eventtype =   external-referer    eventtype = nix-all-logs    eventtype = visitor-type-referred host =    idx6.icontrol.splunkcloud.com index =   _internal linecount =   1 punct =   --_::._+____-____:_=....,_=...,_=,_='_(_=_..._)_|_ search = 'litsearch ( index=_internal 10.30.28.220 ) | fields keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server" | remotetl nb=300 et=2147483647.000000 lt=0.000000 remove=true max_count=1000 max_prefetch=100' server =    sh1.icontrol.splunkcloud.com source =   /opt/splunk/var/log/splunk/remote_searches.log sourcetype = splunkd_remote_searches splunk_server = idx6.icontrol.splunkcloud.com timeendpos =  29 timestartpos =   0 unix_category =   all_hosts unix_group =  default

but if I run this search

index="dcdbtest" 

which is the index I need the data in, there are zero results. What do I need to look at to get this connection working? THANK YOU!!!!

[dbcase · Accepted solution]

Its now working! It turns out Oracle (who authored the DB Connect App) disables by default any user created queries. The gotcha is that it is an IMPLIED default disable. Meaning disabled=1 is NOT reflected in inputs.conf but it is "there". Once I put in disabled =0 everything started working

[esix_splunk]

Have you installed the Splunk Cloud Forwarder App on your Heavy Forwarder running DBConnect? You should be able to find this app on your Splunk Cloud Search Head. Its a spl file that you install and it contains the neccessary outputs.conf and SSL keys to send data to your Splunk Cloud instance.

[grittonc]

I just posted this question: https://answers.splunk.com/answers/665941/what-is-being-forwarded-when-db-connect-3-is-insta.html . You seem to know a lot about the topic. Would you mind taking a look?

[ehudb]

Please post the following:
outputs.conf of the heavy forwarder
inputs.conf of the DB connect of heavy forwarder

Also please run in the cloud search head
index=_internal |top host
index=_internal host=[heavy forwarder hostname]

[dbcase]

Oh also, the data that does show up on the internal index, is NOT the DB query data. I can't find the DB query data anywhere.

[pradeepkumarg]

I believe what you are seeing there is not the logs from your 10.30.28.220. Instead the logs of your activity of the search (index=_internal 10.30.28.220 )

Cross check outputs on your heavyforwarder, where the data is being sent to? Does it have one?
If yes, are those indexers in your outputs.conf are configured as search peers on the search head where you are searching from?