topic Re: CSV file not getting indexed in correct format through UF but parses correctly through WEB UI? in Getting Data In

CSV file not getting indexed in correct format through UF but parses correctly through WEB UI?

vrmandadi — Tue, 29 Sep 2020 22:17:53 GMT

Has any one installed Splunk UF on Kali linux and faced any issues?.We have Splunk UF(7.1.1) installed on Kali linux and monitoring a path as below.The csv file is not coming in right format from the forwarder but when tried uploading in test environment through WEB UI(Settings-->Add Data--->Upload file ) shows the correct format

Below is the path of the csv file

/home/reports/8e20594b-282a-493e-ad9a-dc69e0ac676c.csv and I am using the monitor stanza as below

[monitor:///home/reports/*.csv]
recursive = true
index = main
sourcetype = rf
initCrcLength = 1024
crcSalt =

props.conf

SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
CHARSET=UTF-8
INDEXED_EXTRACTIONS=csv
KV_MODE=none
category=Structured
disabled=false
pulldown_type=true
TIMESTAMP_FIELDS=Timestamp
HEADER_FIELD_LINE_NUMBER=1

Re: CSV file not getting indexed in correct format through UF but parses correctly through WEB UI?

dflodstrom — Wed, 05 Dec 2018 01:09:09 GMT

Can you give an example of what is different when you ingest this file with a UF? Are the fields not being parsed properly?

Re: CSV file not getting indexed in correct format through UF but parses correctly through WEB UI?

vrmandadi — Wed, 05 Dec 2018 01:24:13 GMT

I have attached screen shot in the question of images which worked and did not work

Re: CSV file not getting indexed in correct format through UF but parses correctly through WEB UI?

dflodstrom — Wed, 05 Dec 2018 02:40:44 GMT

It looks like your events are many lines. Is that true in the source file? You may need a custom LINE_BREAKER if these are multi-line events.

Re: CSV file not getting indexed in correct format through UF but parses correctly through WEB UI?

bjoernjensen — Wed, 05 Dec 2018 08:11:21 GMT

Hey,

is your test environment also Kali Linux? If yes, I would start and try to monitor there too. The input assitant (WEB UI) as some rare effects (shouldlinemerge).

Looking at wrong-format.png, splunk might take the wrong delimiter/quote char. You could try to define these explicitly:

HEADER_FIELD_DELIMITER
FIELD_DELIMITER
FIELD_QUOTE

https://docs.splunk.com/Documentation/Splunk/7.1.1/Admin/Propsconf

Could you post what the file looks like? Maybe check on Kali Linux something like cat -T yourfile.csv
https://www.if-not-true-then-false.com/2011/linux-display-show-file-contents-tabs-line-breaks-non-printing-characters/

Cheerz,
Björn

Re: CSV file not getting indexed in correct format through UF but parses correctly through WEB UI?

FrankVl — Wed, 05 Dec 2018 08:20:30 GMT

To double check: you have that props.conf deployed on your universal forwarder as well as your indexer(s)? Normally UFs don't do much with props.conf of course, but INDEXED_EXTRACTIONS are one of the exceptions to that.

Re: CSV file not getting indexed in correct format through UF but parses correctly through WEB UI?

vrmandadi — Wed, 05 Dec 2018 14:59:06 GMT

The WEB UI is not a kali linux.The timestamp field shows none and I think it is not picking the time as mentioned in the props. Below is the format of the file after running the command

cat -T yourfile.csv

IP,Hostname,Port,Port Protocol,CVSS,Severity,Solution Type,NVT Name,Summary,Specific Result,NVT OID,CVEs,Task ID,Task Name,Timestamp,Result ID,Impact,Solution,Affected Software/OS,Vulnerability Insight,Vulnerability Detection Method,Product Detection Result,BIDs,CERTs,Other References 10.22.19.1,,,,0.0,Log,"","CPE Inventory","This routine uses information collected by other routines about
CPE identities (http://cpe.mitre.org/) of operating systems, services and
applications detected during the scan.","10.22.19.1|cpe:/o:cisco",1.3.6.1.4.1.25623.1.0.810002,"NOCVE",ed32074a-1188-45f4-9c59-50ec456a43f2,"Harbor East 10.22.19.0/24",2018-11-22T16:23:20-05:00,0100f392-2d3e-4c39-b7db-45b2b1674018,"","","","","
Details:
CPE Inventory
(OID: 1.3.6.1.4.1.25623.1.0.810002)
Version used: $Revision: 8140 $
","","","",""
10.22.19.2,,,,0.0,Log,"","CPE Inventory","This routine uses information collected by other routines about
CPE identities (http://cpe.mitre.org/) of operating systems, services and
applications detected during the scan.","10.22.19.2|cpe:/o:cisco",1.3.6.1.4.1.25623.1.0.810002,"NOCVE",ed32074a-1188-45f4-9c59-50ec456a43f2,"Harbor East 10.22.19.0/24",2018-11-22T16:23:20-05:00,993222f0-69af-4454-b20f-d7ae8fc041f5,"","","","","
Details:
CPE Inventory
(OID: 1.3.6.1.4.1.25623.1.0.810002)
Version used: $Revision: 8140 $
","","","",""
10.22.19.3,,,,0.0,Log,"","CPE Inventory","This routine uses information collected by other routines about
CPE identities (http://cpe.mitre.org/) of operating systems, services and
applications detected during the scan.","10.22.19.3|cpe:/o:cisco",1.3.6.1.4.1.25623.1.0.810002,"NOCVE",ed32074a-1188-45f4-9c59-50ec456a43f2,"Harbor East 10.22.19.0/24",2018-11-22T16:23:20-05:00,9452ff23-4c0a-4962-a81c-25d43064f956,"","","","","
Details:
CPE Inventory
(OID: 1.3.6.1.4.1.25623.1.0.810002)
Version used: $Revision: 8140 $
","","","",""
10.22.19.1,,,,0.0,Log,"","ICMP Timestamp Detection","The remote host responded to an ICMP timestamp request.
The Timestamp Reply is an ICMP message which replies to a Timestamp message. It consists
of the originating timestamp sent by the sender of the Timestamp as well as a receive
timestamp and a transmit timestamp. This information could theoretically be used to
exploit weak time-based random number generators in other services.","Vulnerability was detected according to the Vulnerability Detection Method.",1.3.6.1.4.1.25623.1.0.103190,"CVE-1999-0524",ed32074a-1188-45f4-9c59-50ec456a43f2,"Harbor East 10.22.19.0/24",2018-11-22T16:23:20-05:00,75d32440-245d-4c9f-83ed-eca8980aff16,"","","","","
Details:
ICMP Timestamp Detection
(OID: 1.3.6.1.4.1.25623.1.0.103190)
Version used: $Revision: 10411 $
","","","CB-K15/1514, CB-K14/0632, DFN-CERT-2014-0658","http://www.ietf.org/rfc/rfc0792.txt"
10.22.19.2,,,,0.0,Log,"","ICMP Timestamp Detection","The remote host responded to an ICMP timestamp request.
The Timestamp Reply is an ICMP message which replies to a Timestamp message. It consists
of the originating timestamp sent by the sender of the Timestamp as well as a receive
timestamp and a transmit timestamp. This information could theoretically be used to
exploit weak time-based random number generators in other services.","Vulnerability was detected according to the Vulnerability Detection Method.",1.3.6.1.4.1.25623.1.0.103190,"CVE-1999-0524",ed32074a-1188-45f4-9c59-50ec456a43f2,"Harbor East 10.22.19.0/24",2018-11-22T16:23:20-05:00,d6f5ad45-4d5e-4deb-8619-a2f85b329097,"","","","","
Details:
ICMP Timestamp Detection
(OID: 1.3.6.1.4.1.25623.1.0.103190)
Version used: $Revision: 10411 $
","","","CB-K15/1514, CB-K14/0632, DFN-CERT-2014-0658","http://www.ietf.org/rfc/rfc0792.txt"
10.22.19.3,,,,0.0,Log,"","ICMP Timestamp Detection","The remote host responded to an ICMP timestamp request.
The Timestamp Reply is an ICMP message which replies to a Timestamp message. It consists
of the originating timestamp sent by the sender of the Timestamp as well as a receive
timestamp and a transmit timestamp. This information could theoretically be used to
exploit weak time-based random number generators in other services.","Vulnerability was detected according to the Vulnerability Detection Method.",1.3.6.1.4.1.25623.1.0.103190,"CVE-1999-0524",ed32074a-1188-45f4-9c59-50ec456a43f2,"Harbor East 10.22.19.0/24",2018-11-22T16:23:20-05:00,cfbbd926-6eef-4e27-a068-6c803cf9e76b,"","","","","
Details:
ICMP Timestamp Detection
(OID: 1.3.6.1.4.1.25623.1.0.103190)
Version used: $Revision: 10411 $
","","","CB-K15/1514, CB-K14/0632, DFN-CERT-2014-0658","http://www.ietf.org/rfc/rfc0792.txt"
10.22.19.1,,,,0.0,Log,"","OS Detection Consolidation and Reporting","This script consolidates the OS information detected by several NVTs and tries to find the best matching OS.

Furthermore it reports all previously collected information leading to this best matching OS. It also reports possible additional information
which might help to improve the OS detection.

If any of this information is wrong or could be improved please consider to report these to the references community portal.","Best matching OS:...........................

Re: CSV file not getting indexed in correct format through UF but parses correctly through WEB UI?

vrmandadi — Wed, 05 Dec 2018 15:10:51 GMT

Yes there are many lines.The props.conf is not picking for this file