search does not return anything

[dvuichor]

I have tried to add to monitor several log files but so far search returns nothing
I am using trial version with max 500M so not sure i have exceeded 500M. How do i verify it?
thanks

[dvuichor]

hi,
it seems to be fine when I executed wget
wget 172.16.128.155:9997
--2018-05-07 14:22:56-- http://172.16.128.155:9997/
Connecting to 172.16.128.155:9997... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

[dvuichor]

Hi Guiseppe,
so far i only execute at CLI
i have 2 linux servers: one hosts Splunk instance and the other one hosts forwarder

on splunk forwarder server:
i execute:
./splunk add forward-server 172.16.128.155:9997
Splunk username: admin
Password:
Added forwarding to: 172.16.128.155:9997.

then add path and log files to be monitored:
[root@dagapps bin]# ./splunk list monitor
Monitored Directories:
[No directories monitored.]
Monitored Files:
/u01/app/agile/agile935/agileDomain/bin
/u01/app/agile/agile935/agileDomain/bin/nohup.out
/var/log
/var/log/lastlog

on splunk server i execute:
$ ./splunk enable listen 9997
Splunk username: admin
Password:
Listening for Splunk data on TCP port 9997.

did i miss any steps?

thanks for your help

[amitm05]

Can you try to telnet on 172.16.128.155:9997 from your forwarder server and see if the connection is successful.

Successful connection - Check the splunkd.log file on Splunk Indexer for any errors.

UnSuccessful connection - It might be a firewall block. Crosscheck it. Or check splunkd.log file on your forwarder to see error details.

[dvuichor]

Hi,
I tried
index=* |head 100
still returns nothing
Is there any logs which splunk generates to see any error?

[gcusello]

Hi dvuichor,
you can check if you're in violation opening [Settings -- Licenses], anyway how many times you exceeded license limit? if you exceeded less than 3 times it's also OK.

In addition, when there's a violation Splunk gives an error message.
To verify if there'se a problem run a simple search

index=* | head 100

using "always" as time period and see if there are results: maybe the problem is a different one: time error or ingestion error, etc...

Bye.
Giuseppe

[gcusello]

Hi dvuichor,,
if you haven't any violation message the problem is probably on ingestion.
Try to ingest a local log:

• [Settings -- Inputs -- Windows Event Logs] if you're using a Windows server,
  
• [Settings -- inputs -- Files or Directories -- /var/log/messages] if you're using a Linux server

If in this way tou find logs, you have to troubleshoot your log ingestion (see at https://docs.splunk.com/Documentation/Splunk/7.1.0/Forwarding/Receiverconnection ).

Are you using an Universal Forwarder or not?
Can you share your inputs.conf from UF or (if you haven't) from system local?
If you're using a Universal Forwarder, please share also outputs.conf.

Bye.
Giuseppe

[gcusello]

please, share inputs.conf of your Forwarder.
it should be

[monitor///u01/app/agile/agile935/agileDomain/bin/nohup.out]
disabled=0
index=my_index
[monitor///var/log/lastlog]
disabled=0
index=my_index

running the search

index=my_index

you should have results.

As additional check, verify that the time of both the servers are aligned and remember to restart Universal Forwarder after inputs.conf updates.

Bye.
Giuseppe