Getting Data In

How can we read local data on HF and send them to indexers?

kaboom1
Explorer

Hello,

Here is the deal, I am following this link to ingest cisco umbrella logs into splunk: https://support.umbrella.com/hc/en-us/articles/360001388406-Configuring-Splunk-with-a-Cisco-managed-... (I know, you will say: why not use one of the existing apps in SplunkBase? I will say: I really don't know which one to use, they seem doing the same thing, plus I just want the raw logs, so getting the logs directly from the bucket is good enough 🙂 )
I downloaded the logs and stored them in the HF under /opt/ciscologs/  and i configured a simple inputs.conf file to read those files with the idea that these logs will be sent to the indexers (because I already have the output file with the config to send data from HF to indexers) 

But, I dont see any logs being indexed, I dont see any events on the search head 😞 

here is the inputs.conf in the HF:

[monitor:///opt/ciscologs/dnslogs/*]
index = index_name
sourcetype = csv
#whitelist = 2023-*/*
disabled = 0
crcSalt = <SOURCE>
#_TCP_ROUTING = default-autolb-group

(the commented lines are settings that i have tried but still no luck.)

Any suggestions here? I am out of ideas 

Thanks.

 

Labels (4)
Tags (3)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The HF still must restart to force it to re-read the config files.  The DS has an option that tells the client to restart when it gets an update to the app.  Be sure that option is selected.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

Did you restart the HF after creating the inputs.conf file?  The changes don't take effect until restart.

---
If this reply helps you, Karma would be appreciated.
0 Karma

kaboom1
Explorer

I didnt restart the HF because I am managing the conf files from a deployment server.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The HF still must restart to force it to re-read the config files.  The DS has an option that tells the client to restart when it gets an update to the app.  Be sure that option is selected.

---
If this reply helps you, Karma would be appreciated.

kaboom1
Explorer

Thank you @richgalloway , indeed by simply selecting the option Restart Splunkd from the deployment server GUI (in Settings > Forwarder management > select your app then click Edit abd select: After installation: Restart Splunkd)  logs start coming in 🙂 

isoutamo
SplunkTrust
SplunkTrust

One way to check when UF has restarted

index=_internal host=<your UF name known by splunk> sourcetype=splunkd source=*splunkd.log component IN (ModularInputs ExecProcessor TailingProcessor)
| rex "New scheduled exec process: (?<mod_input_script>.*)" 
| rex "Parsing configuration stanza: (?<input_file>.*)\." 
| eval input = coalesce(input_file, mod_input_script) 
| where isnotnull(input) 
| eval input = if (input == mod_input_script, "modinput:" . input, input) 
| bin span=1d _time as time 
| bin span=1h _time as bTime
| stats min(_time) as firstBoot max(_time) as lastBoot values(bTime) as BootTimes values(input) as input by time host 
| table firstBoot lastBoot BootTimes host input
| convert timeformat="%F %H:%M" ctime(BootTimes) 
| sort + firstBoot -lastBoot
| convert timeformat="%F %T" ctime(firstBoot) ctime(lastBoot)

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

can you give a sample of those filenames and sampled content of those files?

Have Splunk HF's splunk server process access right to those files? This instructions shows that sync is done as root and if your splunk is running as splunk-user (never run it as root), then maybe splunk-user haven't access to those files.

r. Ismo

0 Karma

kaboom1
Explorer

hello @isoutamo ,
so the downloaded files have this structure: /opt/ciscologs/dnslogs/date/date-hour-text-csv.gz

for example: /opt/ciscologs/dnslogs/2023-06-01/2023-06-01-12-20-732d.csv.gz

I didnt run the script as root, I only use splunk user, so the files are owned by splunk user.

 

 

Tags (3)
0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...