Splunk Dev

Splunk is not indexing my text file

Abilan1
Path Finder

Hi Team,

It looks like splunk is not indexing my log file, If I create a test file then it is indexing it but not my log file.

input.conf

[monitor://D:\logs]
disabled = false
index = prd
sourcetype = sched

Log File sample:

2017-03-24 00:00:00,848 [Thread-2] WARN com.JobInitiator- executed CALL
2017-03-24 00:00:20,927 [Thread-2] WARN com.JobInitiator- executed CALL
2017-03-24 00:00:41,161 [Thread-2] WARN com.JobInitiator- executed CALL
2017-03-24 00:01:01,208 [Thread-2] WARN com.JobInitiator- executed CALL

Please help me on this issue. Thank You!

Tags (2)
0 Karma

danielsofoulis
Path Finder

When searching select ALL TIME, this has tripped me up before especially if the test logs are old.

I'm not sure if you want to ingest the whole directory or a file named logs.

Try using creating a folder logs and putting your test file inside and then use the following stanza.

[monitor://D:\logs\*]

You could also confirm connectivity by using telnet
telnet SPLUNK_IP 9997

Another thing you could do is a one shot command. This will ingest the file regardless of whether the file has been ingested before.

SPLUNK_HOME\bin\splunk add oneshot D:\logs -index pad -sourcetype shed

http://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorfilesanddirectoriesusingtheCLI

0 Karma

jpass
Contributor

Check that the user splunk runs under has permission to access the folders & log files you want to index.

0 Karma

Abilan1
Path Finder

Hi ,

Permission looks good, that is not the issue.

0 Karma

mattymo
Splunk Employee
Splunk Employee

Hi Abilan1

You can try checking index=_internal tailreader with your log path in the GUI to see perhaps the tail reader is ignoring the file due to CRC checks or permissions otherwise, or check

./splunk list inputstatus to check for the files and the tailprocessor's status.

- MattyMo
0 Karma

mattymo
Splunk Employee
Splunk Employee

your user is set to UTC in the Splunk GUI, correct? click the username dropdown (top right of your screen) > Account Settings > Set timezone

With my user set in UTC, I see these events at 3AM GMT in my Time field....is that what you are trying to achieve? Note the time in the event has not changed, just the Time field now has a local conversion in the gui.

can you please collect the output of ./splunk btool props list sched --debug on your forwarder and your indexer? ( not sure of your setup, I think you said forwarder)

you have the sched sourcetype updated in both a forwarder and indexer, correct?

`[splunker@n00bserver bin]$ ./splunk btool props list sched --debug
/home/splunker/splunk/etc/apps/search/local/props.conf [sched]

.........

/home/splunker/splunk/etc/apps/search/local/props.conf TIME_FORMAT = %Y-%m-%d %H:%M:%S,%f
/home/splunker/splunk/etc/apps/search/local/props.conf TIME_PREFIX = ^
/home/splunker/splunk/etc/system/default/props.conf TRANSFORMS =
/home/splunker/splunk/etc/system/default/props.conf TRUNCATE = 10000
/home/splunker/splunk/etc/apps/search/local/props.conf TZ = Brazil/East

........

`

alt text
alt text

Brasil event time, UTC Splunk web user, splunker sitting in Canada 😉

- MattyMo
0 Karma

Abilan1
Path Finder

Hi ,

Thanks for your reply, I have verified the permission it looks fine. Also If I create any other test file in that folder that is getting indexed properly. so it doesn't seem to be a permission issue.

Checked the tail-processor's status, it says finished reading. No errors related to CRC.

Regards,
Abilan

0 Karma

mattymo
Splunk Employee
Splunk Employee

if the files are read, then maybe we are looking in the wrong spot?

does searching index=prd sourcetype=sched source=*\logs* searched over alltime find them?

- MattyMo
0 Karma

Abilan1
Path Finder

Hi,

Hi,

Yes I see that logs are indexing, but If I want to search for current log then I have to search for last 4 hours. I mean indexing time stamp is 4 hour behind.

If I search for last 4 hours in the search, I can see the latest logs there.

We have the same forwarder on linux machines and that is giving proper time stamps. This is happening on windows server's universal forwarder. All our universal forwarder machines are in same time zone but not indexer.

Thanks ,
Abilan

0 Karma

mattymo
Splunk Employee
Splunk Employee

Sounds like you need to configure your sourcetype with timestamp settings. There are a few best practices when it comes to defining a sourcetype...but for now lets focus on the props.conf for timestamp recogonition.

What timezone should these logs be in?

can you share the props.conf you are using for sourcetype sched ?

./splunk btool props list sched --debug

Try adding TZ= <logTimezone> to the props to help Splunk determine the correct stamp.

- MattyMo
0 Karma

Abilan1
Path Finder

Hi ,

My Indexer is in GMT. but all my forwarders are in Brazil time zone.

I didn't change anything in props.conf file after installation.

Also please let me know which timezone I need to add and any standard format for the same?

Thanks for your help!

Regards,
Abilan

0 Karma

mattymo
Splunk Employee
Splunk Employee

You need to set the timezone to that of the machines spitting the logs. IT best practice puts all machines in UTC, but alas, no one is perfect ;).

If i run date on the machine generating the logs, I want to match that timezone as that is the TZ it will stamp the logs with.

As long as Splunk knows the proper TZ at indextime, the GUI settings will allow the user to set their local timezone and the GUI will adjust for them. But you gotta get the initial TZ correct.

TZ =
* The algorithm for determining the time zone for a particular event is as
follows:
* If the event has a timezone in its raw text (for example, UTC, -08:00),
use that.
* If TZ is set to a valid timezone string, use that.
* If the event was forwarded, and the forwarder-indexer connection is using
the 6.0+ forwarding protocol, use the timezone provided by the forwarder.
* Otherwise, use the timezone of the system that is running splunkd.
* Defaults to empty.

http://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/Propsconf

- MattyMo
0 Karma

Abilan1
Path Finder

Hi ,

I have updated the below in my props.conf on forwarder machine.

[ sched ]
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%f
TIME_PREFIX=^
TZ=UTC-3
MAX_TIMESTAMP_LOOKAHEAD=25

And restarted the splunk forwarder service, but still I don't see any change in time from Splunk web.

Regards,
Abilan

0 Karma

mattymo
Splunk Employee
Splunk Employee

try

TZ=America/Sao_Paulo

not sure that utc-3 value is legit...

- MattyMo
0 Karma

Abilan1
Path Finder

Hi ,

I have updated this for UTC in my props.conf on forwarder machine.

[sched]
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%f
TIME_PREFIX=^
TZ=Africa/Abidjan
MAX_TIMESTAMP_LOOKAHEAD=25

Still it is not working as expected. Logs are still coming in Brazil Time Zone.

Regards,
Abilan

0 Karma

Abilan1
Path Finder

Hi ,

Yes, I have tried with Brazil Time zone still it is not working!

0 Karma

mattymo
Splunk Employee
Splunk Employee

UTC is wrong. the logs are Brazil time. Set it to brazil, that way Splunk can translate it to your chosen timezone in the GUI

- MattyMo
0 Karma

mattymo
Splunk Employee
Splunk Employee

Try this and set your timezone according to the machines that are spitting the sched logs to your windows box. Setting SHOULD_LINEMERGE, TIME_FORMAT, TIME_PREFIX, MAX_TIMESTAMP_LOOKAHEAD, TZ, and LINE_BREAKER in props.conf, should become part of all your sourcetyping. Splunk is really good at auto recognizing things..but you can gain indexing performance by not making it work so hard. Try using the add data wizard! it makes creating these configs real easy and lets you validate your choices.

[ sched ]
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%f
TIME_PREFIX=^
TZ=UTC
MAX_TIMESTAMP_LOOKAHEAD=25
- MattyMo
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...