When Splunk is configured to read a file, does it need to be done on the indexer side and how to handle a file edited weekly?

[mmohiuddin]

Hi:

I know it is possible for Splunk to read data from a file, but I just had some questions that I need to be addressed. Does this need to be done on the indexer side? Also, the user will be editing that file every week, so does that file need to be uploaded every week? Please let me know.

[dflodstrom]

You can read the file in using a Splunk Forwarder or a Splunk Indexer. You can configure an input in Splunk to monitor the file for changes.

Use this as a guide for inputs.conf http://docs.splunk.com/Documentation/Splunk/6.2.1/admin/inputsconf

[dflodstrom]

Feel free to comment back with a more specific question when you get to that point.

[mmohiuddin]

I am unable to upload the file in Splunk. It is getting a timestamp automatically assigned, 1/7/02. I am trying to create a plain text file and upload in Splunk. But unable to do so as timestamp is coming as 1/7/02.

How do I address this issue.

Thanks,

[mmohiuddin]

The following is the content of this file :

105.158.114.214
108.110.44.89
108.113.192.90
108.117.33.43
109.184.100.232

[mmohiuddin]

Timestamp   Event

1 1/7/02 6:53:58.000 AM

105.158.114.214
2 1/7/02 6:53:58.000 AM

108.110.44.89
3 1/7/02 6:53:58.000 AM

108.113.192.90
4 1/7/02 6:53:58.000 AM

108.117.33.43

[dflodstrom]

If you're creating this file manually you should format it in a way that makes splunk understand it by default. Splunk is looking for key value pairs, it should pick a timestamp from each line if you use a standard format for time. You've got a list of IP addresses, if these correspond to a specific date and time you will want to include that in your file:

1603 2/24/2015 ip = 192.168.1.1

Even a simple file like this will get processed more accurately by Splunk.

[mmohiuddin]

Actually it is just a list of IP addresses. I am not assigning any timestamp to it. But still getting default timestamp 1/7/02

Please advise what needs to be done to fix timestamp issue

[dflodstrom]

Do you want to create a lookup or do you want to index this file? Splunk assigns a timestamp to every event. If you don't provide one it will take its best guess.

[mmohiuddin]

Isn't it possible for Splunk to make it determine the IP addresses based on current time, instead of taking a wild guess

[dflodstrom]

Yes there is an option to use current time. The instructions may vary slightly depending on your Splunk version. Splunk 6.2.x: When you're using the web interface to add data you first select the source, click next and set sourcetype, on the left you'll see a dropdown for "Timestamp" select current time instead of Auto which is default.

[mmohiuddin]

I tried doing that but Splunk is still assigning a default time of 01/07/12

i have read in splunk documents that if the event does not have a timestamp, If the timestamp is not located at the very start of each event, you must also specify a prefix in the Timestamp is always prefixed by a pattern field in the "Location" section. Could someone tell me how do I need to specify the prefix pattern.

This is my props.conf

NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TIME_FORMAT=%m/%d/%Y
TIME_PREFIX=\d+.\d+.\d+.\d+

Please find below my sample data:

Timestamp                              Event

1 1/8/02 10:55:12.000 PM 10.10.91.108
2 1/8/02 10:55:12.000 PM 10.15.54.333
3 1/8/02 10:55:12.000 PM 11.16.14.133

These IP addresses do not have any timestamp. These are just a list of IP's which I am trying to upload as a file on Splunk Indexer.

But splunk is randomly assigning timestamp, and defaulting to 2002 year.

[vincenteous]

If you want to default splunk timestamp to current time when indexing data, how about you use DATETIME_CONFIG property instead? So, your props.conf should be something like this:

[<stanza name: be it your sourcetype, source, etc>]
DATETIME_CONFIG = CURRENT
...
...

Reference: http://docs.splunk.com/Documentation/Splunk/6.2.1/Data/Configuretimestamprecognition

[mmohiuddin]

I tried using it:

your settings

DATETIME_CONFIG=CURRENT
NO_BINARY_CHECK=1
TZ=America/Detroit

and the following is my data preview:

              Timestamp                    Event

1 1/9/02 12:41:12.000 AM 10.10.91.108
2 1/9/02 12:41:12.000 AM 10.15.54.333
3 1/9/02 12:41:12.000 AM 11.16.14.133

Still unable to extract timestamp properly to (EST)

[dflodstrom]

For sanity sake can you confirm the date and time on your server are correct? Do you really want to index this 'data' or do you want to use it as a lookup?

[mmohiuddin]

The data preview is from our prod indexer server. This server has current date and time as per EST. We want to index this data and not use it as a lookup.