Data Onboarding issue- unable to see the data in splunk

[blbr123]

Hello All,

I have configured the inputs and props but unable to see the data in splunk.

I have around 20 monitor stanza and all of them have same source type, below is my monitor stanza

File to be monitored is below

archive.log.DYYYYMMDD.Tnnnnnn

[monitor:///opt/sw/ss/splunklogs/archive.log.*.*]

index=abc

disabled = 0

sourcetype=es:test:sd:logs

Sample log file is below:

where YYYYMMDD-Date ex-20220412

nnnnnn-6 digit timestamp ex- 171300

Below is props conf

[es:test:sd:logs]

SHOULD_LINEMERGE=true

BREAK_ONLY_BEFORE= ^[\d+\-\d+\-\d+\s+\d+\d:+\d+:\d+.\d+\d+]

MAX_TIMESTAMP_LOOKAHEAD=28

TIME_FORMAT=%d-%m-%y %H:%M:%S.%N

TIME_PREFIX=^\w

Below is the data on which REGEX was done.

[2022-04-04 23:10:30.643]

Please let me know if there anything wrong in my configurations

in internal logs for log level error it shows below error.

StreamId:123456 had parsing error:unexpected character while expecting ' : ' :  ' , '

[sperkins]

I am not sure it is your whole issue, but your time format doesn't match the example: [2022-04-04 23:10:30.643] 

It should be %Y-%m-%d %H:%M:%S.%N

I would also add in a 

LINE_BREAKER = <REGEX>

[gcusello]

Hi @blbr123,

some little question to better understand:

• are you trying to read logs on the same Splunk server or in a different target server (using Universal Forwarder)?
• the running Splunk user has the rights on that folder?
• what does it happen if you run by cli the following command: "ls -la /opt/sw/ss/splunklogs/archive.log.*.*"?

Ciao.

Giuseppe

[blbr123]

@gcusello 

I am trying to read logs from another host using universal forwarder.

Yes the splunk user has the read access to the log paths and files.

Cannot check third one as user is not available.

[gcusello]

Hi @blbr123 ,

the third check is a linux command to check if the path you're using is correct that you have to run using an SSH terminal.

ls -la /opt/sw/ss/splunklogs/archive.log.*.*

Ciao.

Giuseppe

[blbr123]

i get this output when i run the command:

-rw-r----- 1 abc xyz 716 Apr 22 01:16

[gcusello]

Hi @blbr123,

if "xyz" is the file that you want to monitor the command in the stanza is correct.

did you see in Splunk Enterprise server the internal Splunk logs from that server?

index=_internal host=<your_host>

if not there's a problem in connection.

Ciao.

Giuseppe

[blbr123]

There are 2 hots sending the logs:

and can see the internal logs for both the hosts.

For one of the hosts it gives error in internal logs: for log_level WARN

TcpOutEloop] - Connect to x:x:x:x:9997 failed. Connection refused

for log_level ERROR getting below error:

StreamId:1234567 had parsing error:Unexpected character while expecting ':': ',' - data_source="/opt/splunkforwarder/var/spool/splunk/tracker.log"

Does this indicate something wrong for monitoring?

[gcusello]

Hi @blbr123,

the error message says that there's a connection problem, but I don't see any configuration error (except the one indicated by @sperkins).

if you're receiving the Splunk internal logs from that Universal Forwarder the connection is correctly established, are you sure about internal logs?

What does it happen if you use a larger time period (e.g. always)?

ciao.

Giuseppe 

[blbr123]

when i select always i get below error:

FilesystemChangeWatcher [xxxxxx MainTailingThread] - error getting attributes of path "/opt/sw/ss/splunklogs/system.log.xxxxxxxxx.xxxxxx": Permission denied

Insufficient permissions to read file='/opt/sw/ss/si/install/logs/noapp.log.xxxx.xxxx' (hint: Permission denied , UID: xxxxxx, GID: xxxxxxx)

and the user has changed the permission to read the files for splunk user, and i have added restartSplunkd=true in server class to restart the splunk service for changes to be applied.

But still same issue