topic Re: Can't get all rows from CSV in Getting Data In

Can't get all rows from CSV

twtyj — Tue, 29 Sep 2020 20:16:06 GMT

I monitor folder on one server with SplunkUniversalForwarder installed. the configuration of input.conf as below:
[monitor://E:\A530\archive\Powershell_User]

  sourcetype = Powershell_User

  crcSalt = <SOURCE>

  disabled = false

  followTail = 0

then I do search in search head with Splunk query:
source="E:\A530\archive\Powershell_User\20180614_get_aduser_all_for_Splunk.csv"

the total rows of this CSV file are 7168, but only return 5139 rows.

the CSV is well formatted

How can I get all rows in Splunk query?

Thanks

Re: Can't get all rows from CSV

renjith_nair — Tue, 03 Jul 2018 05:02:53 GMT

Do you have a timestamp column in csv and what's the timerange you are using in search?

Re: Can't get all rows from CSV

twtyj — Tue, 03 Jul 2018 05:20:49 GMT

Hi nair,

I am fresh to use SPLUNK, I use all time range in search.

How can i find timestamp information?

thanks.

Re: Can't get all rows from CSV

renjith_nair — Tue, 03 Jul 2018 09:04:41 GMT

If you are selecting "All Time" timestamp does not really matter. Are you uploading it to the same index or do you have some props to split it? also try index=* source="E:\\A530\\archive\\Powershell_User\\20180614_get_aduser_all_for_Splunk.csv" Is it possible to identify one of the rows which is missing and find the difference of that from other rows?

Re: Can't get all rows from CSV

twtyj — Thu, 05 Jul 2018 03:17:41 GMT

Hi Nair,
I know why only part of rows are returned. Splunk index the CSV file with timestamp but there is no dedicated filed for that.
Splunk extract the timestamp of CSV from modified time. the problem is some rows can added the timestamp with modified time.

do you know how to extract timestamp based on file name? we have 20180614 within file name.

Re: Can't get all rows from CSV

woodcock — Thu, 05 Jul 2018 03:43:50 GMT

Are you ABSOLUTELY sure that it is well-formatted? Splunk will abort the import (possibly without indicating an error) if it hits something that is wonky. The best way to check is using the Lookup Editor App (https://splunkbase.splunk.com/app/1724/). You simply upload the lookup file and it checks for you and you will see the error straight away because it will complain. It will even show you what kind of error and on what line!

Re: Can't get all rows from CSV

renjith_nair — Thu, 05 Jul 2018 03:52:26 GMT

Refer : https://answers.splunk.com/answers/320978/how-to-extract-the-timestamp-from-a-filename-at-in.html

http://docs.splunk.com/Documentation/Splunk/latest/Data/HowSplunkextractstimestamps

Re: Can't get all rows from CSV

woodcock — Thu, 05 Jul 2018 04:19:18 GMT

You are pretty much out of luck. You can get the date from your filename but the events must have a timestamp in them and yours do not. See here:

https://www.splunk.com/blog/2009/12/02/configure-splunk-to-pull-a-date-out-of-a-non-standard-filename.html

You can use DATETIME_CONFIG = CURRENT to timestamp with the index time.