Splunk Search
Highlighted

Indexing a log file comes up with empty results.

New Member

Hello-

I am trying to index some files in a directory local to the splunk server (/tmp/risqiq/). I can see in the S.O.S that it it searching the log files that I want with since it has an "Action Status" of "finished reading" and a "Read_status" of "read". I put them in their own index but when I go look in the index there are 0 events in the index and its 1kb in size. I believe that its becase of my event type and the fact that its parsing the file and determining that there is nothing in there. I have tried both the continuous import mode as well as the direct file import.

I have tried to have Splunk auto detect the data type but that does not work. The log contents do show in the preview so I know it can access the files and folder. It has the date highlighted (which is not what I want). The problem is the log file is one giant line with events buried in to it. I believe each of the events begin with "id". I have also tried to search with a regex search of "\b\w[id]\b".

I have tried the following settings in the advanced mode of data preview for props.conf (I dont have shell access to the server itself):

# your settings
BREAK_ONLY_BEFORE=\b\w[id]\b
MAX_TIMESTAMP_LOOKAHEAD=150
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false

# set by detected source type
TRUNCATE=0
index=riskiq
pulldown_type=1

Here is a snippet from the log:

{"startDateInclusive":"2015-04-19T19:04:39.000-0700","endDateExclusive":"2015-04-19T20:04:39.000-0700","totalResults":2669,"resources":[{"id":249138,"url":"http://www.oneclicktools.com/oct/RingtoneConverterIcon.gif","hostname":"www.oneclicktools.com","rank":2147483647,"phishing":false,"malware":true,"spam":false,"matchType":"HOST","score":49,"description":"Binary file MD5 found in AV(21%)","contentType":"image/gif","httpResponseCode":200,"detectedAt":"2010-01-01T00:00:00.000-0800","lastSeenAt":"2015-04-19T05:37:40.000-0700","entries":[{"type":"GSBMalware","matchType":"DOMAIN","url":"oneclicktools.com/"},{"type":"CymruMalware","matchType":"HOST","id":34761,"url":"http://www.oneclicktools.com/cdwriter15.exe","description":"Binary file MD5 found in AV(21%)","detectedAt":"2012-12-25T10:45:49.000-0800"}]},{"id":1130541,"url":"http://www.nutsvolts.com/","hostname":"www.nutsvolts.com","ip":"76.12.26.68","asn":"20021","rank":747367,"phishing":false,"malware":true,"spam":false,"matchType":"HOST","score":49,"description":"Binary file detected by AV(21%)","contentType":"text/html","httpResponseCode":200,"detectedAt":"2010-01-01T00:00:00.000-0800","lastSeenAt":"2015-04-19T20:00:55.000-0700","entries":[{"type":"GSBMalware","matchType":"DOMAIN","url":"nutsvolts.com/"},{"type":"VirusTotal","matchType":"HOST","id":7770421,"url":"http://www.nutsvolts.com/uploads/magazine_downloads/workshop55.zip","description":"Binary file detected by AV(21%). {Avast=Win32:Malware-gen, Avira=TR/Graftor.101096.2, Comodo=UnclassifiedMalware, GData=Archive.Trojan.Agent.MV9FTA, Ikarus=Backdoor.Poison, Jiangmin=Backdoor/Poison.adbb, K7AntiVirus=Riskware ( 0040eff71 ), K7GW=Riskware ( 0040eff71 ), McAfee=RDN/Generic.dx!d2b, McAfee-GW-Edition=RDN/Generic.dx!d2b, Symantec=Trojan.Gen.2, TrendMicro-HouseCall=Suspicious_GEN.F47V0106}. md5=05dea7390de10298c277683f4d75646f","detectedAt":"2013-07-11T13:47:00.000-0700"}]},{"id":2692729,"url":"http://files.informer.com/siinst.exe","hostname":"files.informer.com","ip":"208.88.224.211","asn":"40824","rank":849,"phishing":false,"malware":true,"spam":false,"matchType":"URL","score":54,"contentType":"application/octet-stream","httpResponseCode":200,"detectedAt":"2011-02-20T06:51:54.000-0800","lastSeenAt":"2015-04-19T16:33:50.000-0700","entries":[{"type":"Malc0de","matchType":"URL","id":1860233,"url":"http://files.informer.com/siinst.exe","description":"MD5: 3ad2f4b0b0ce0875da4dd58bced17db9, IP: 208.88.224.211, Country: us, ASN: 40824","detectedAt":"2011-11-09T06:20:00.000-0800"},{"type":"VirusTotal","matchType":"URL","id":20347336,"url":"http://files.informer.com/siinst.exe","description":"Binary file detected by AV(4%). {Ikarus=Win32.SuspectCrc, TrendMicro-HouseCall=Suspicious_GEN.F47V1021}. md5=e49ef284df6d6516c0bf8851c76d081e","detectedAt":"2014-09-24T09:51:34.000-0700"}]},{"id":3795387,"url":"http://files.brothersoft.com/internet/miscellaneous/brothersoftextreme_ct2776682.exe","hostname":"files.brothersoft.com","ip":"65.49.92.216","asn":"6939","rank":7440,"phishing":false,"malware":true,"spam":false,"matchType":"URL","score":77,"description":"Binary file detected by AV(7%)","contentType":"application/octet-stream","httpResponseCode":200,"detectedAt":"2011-04-28T17:07:14.000-0700","lastSeenAt":"2015-04-19T08:31:19.000-0700","entries":[{"type":"GSBMalware","matchType":"HOST","url":"files.brothersoft.com/"},{"type":"RiskIQ","matchType":"URL","id":330552,"url":"http://files.brothersoft.com/internet/miscellaneous/BrotherSoftExtreme_CT2776682.exe","description":"Confidence: 75. FakeSoftwareUpdate: alert:Your Flash Player may be out of date","detectedAt":"2014-08-23T12:01:31.000-0700"},{"type":"CymruMalware","matchType":"HOST","id":1589,"url":"http://files.brothersoft.com/security/monitoring_software/spyagent7.zip","description":"Binary file MD5 found in AV(20%)","detectedAt":"2011-10-04T07:24:54.000-0700"},{"type":"VirusTotal","matchType":"URL","id":448557,"url":"http://files.brothersoft.com/internet/miscellaneous/BrotherSoftExtreme_CT2776682.exe","description":"Binary file detected by AV(7%). {Cyren=W32/A-68608136!Eldorado, DrWeb=Program.BrotherSoft.4, F-Prot=W32/A-68608136!Eldorado, NANO-Antivirus=Riskware.Win32.BrotherSoft.diumlo}. md5=b02a24d94306e494994ef41b55be7d07","detectedAt":"2011-09-21T09:25:34.000-0700"}]},{"id":8119183,"url":"http://www.ascentive.com/","hostname":"www.ascentive.com","ip":"64.62.158.147","asn":"6939","rank":81185,"phishing":false,"malware":true,"spam":false,"matchType":"HOST","score":45,"description":"Binary file MD5 found in AV(16%)","contentType":"text/html","httpResponseCode":200,"detectedAt":"2011-08-07T22:46:15.000-0700","lastSeenAt":"2015-04-18T22:55:39.000-0700","entries":[{"type":"CymruMalware","matchType":"HOST","id":713,"url":"http://www.ascentive.com/support/new/libraryfiles.exe","description":"Binary file MD5 found in AV(16%)","detectedAt":"2011-09-19T13:20:42.000-0700"},{"type":"VirusTotal","matchType":"HOST","id":22960038,"url":"http://www.ascentive.com/run/download?service=SpeedScan&debug=0&loadlink=http%3A%2F%2Fwww.ascentive.com%2Frun%2Fclick%2F%403040667546237%2Fproducts%2FSP_finallyfastpc%2Fselect_finallyfastpc_speedscan_download_op1.html%3Ftheme%3Dselect_speedscan_finallyfastpc_download%26plan1id%3D%26orderpackage1id%3DSSCN1YEAR29%26plan1c%3D%26upsell_code%3D%26popuppage%3D%26display%3D%26referredby%3D%403040667546237%26loadlink%3D","description":"Binary file detected by AV(11%). {ESET-NOD32=a variant of Win32/Ascentive, Kingsoft=VIRUS_UNKNOWN, McAfee=Artemis!B18049048917, McAfee-GW-Edition=Artemis, TrendMicro-HouseCall=Suspicious_GEN.F47V1029, Zillya=Trojan.Patched.Win32.80083}. md5=b18049048917e904f3c1a544d869abe1","detectedAt":"2014-11-09T22:16:08.000-0800"}]},{"id":8119273,"url":"http://www.ascentive.com/partners/affiliate-program/","hostname":"www.ascentive.com","ip":"64.62.158.147","asn":"6939","rank":81185,"phishing":false,"malware":true,"spam":false,"matchType":"HOST","score":45,"description":"Binary file MD5 found in AV(16%)","contentType":"text/html","httpResponseCode":200,"detectedAt":"2011-08-07T22:46:34.000-0700","lastSeenAt":"2015-04-19T19:14:23.000-0700","entries":[{"type":"CymruMalware","matchType":"HOST","id":713,"url":"http://www.ascentive.com/support/new/libraryfiles.exe","description":"Binary file MD5 found in AV(16%)","detectedAt":"2011-09-19T13:20:42.000-0700"},{"type":"VirusTotal","matchType":"HOST","id":13017376,"url":"http://www.ascentive.com/run/download?service=SpeedScan&debug=0&loadlink=http://www.ascentive.com/run/click/@2022810294200/products/SP/select_finallyfast_download_op1.html?theme%3Dselect_finallyfast_partner_download%26plan1id%3D%26orderpackage1id%3DFFST1YR29%26plan1c%3D%26upsell_code%3Dff2%26popuppage%3D%26display%3D%26referredby%3D@2022810294200%26c1%3D03_59531325_093feaa1-e60c-4ae0-a3c9-514147985d45%26loadlink%3D","description":"Binary file detected by AV(16%). {Comodo=Application.Win32.Conduit.~A, DrWeb=Adware.Conduit.6, ESET-NOD32=Win32/Toolbar.Conduit.R, Malwarebytes=PUP.Optional.Conduit.A, McAfee=Artemis!43583D1CCFCE, McAfee-GW-Edition=Artemis!43583D1CCFCE, Symantec=Trojan.ADH.2, TrendMicro-HouseCall=TROJ_GEN.F47V0305}. md5=43583d1ccfce25d7c422f4ecdafa662b","detectedAt":"2014-03-05T15:14:26.000-0800"}]},{"id":8145893,"url":"http://knorrnetworks.net/","hostname":"knorrnetworks.net","ip":"75.145.143.169","asn":"33491","rank":2147483647,"phishing":false,"malware":true,"spam":false,"matchType":"HOST","score":30,"description":"Binary file detected by AV(22%)","contentType":"text/html","httpResponseCode":200,"detectedAt":"2011-08-08T13:45:33.000-0700","lastSeenAt":"2015-04-18T20:36:28.000-0700","entries":[{"type":"RiskIQ","matchType":"HOST","id":87799,"url":"http://knorrnetworks.net/SUPPORT/VNC/vnc-4.0-x86_win32.exe","description":"Confidence: 100. Binary file detected by AV(14%)","detectedAt":"2011-08-08T14:02:00.000-0700"},{"type":"VirusTotal","matchType":"HOST","id":47853,"url":"http://knorrnetworks.net/kn.exe","description":"Binary file detected by AV(22%). {Antiy-AVL=Trojan[RemoteAdmin:not-a-virus]/Win32.WinVNC-based, Baidu-International=HackTool.Win32.WinVNC.AsK, CMC=RemoteAdmin.Win32.WinVNC-based!O, Comodo=UnclassifiedMalware, Kaspersky=not-a-virus:RemoteAdmin.Win32.WinVNC-based.c, Malwarebytes=PUP.Radmin, McAfee=Artemis!193A45B47AD8, McAfee-GW-Edition=Heuristic.BehavesLike.Win32.Suspicious-BAY.G, NANO-Antivirus=Riskware.Win32.WinVNC-based.cqjdov, Symantec=WS.Reputation.1, TheHacker=Trojan/Buzus.bpyo}. md5=193a45b47ad85b15c39cfc32d0d88164","detectedAt":"2011-08-07T10:47:00.000-0700"}]},{"id":8278779,"url":"http://www.ascentive.com/about_us/ascentive/press_releases.php","hostname":"www.ascentive.com","ip":"64.62.158.147","asn":"6939","rank":81185,"phishing":false,"malware":true,"spam":false,"matchType":"HOST","score":34,"description":"Binary file MD5 found in AV(16%)","contentType":"text/html","httpResponseCode":200,"detectedAt":"2011-08-11T21:26:19.000-0700","lastSeenAt":"2015-04-19T10:38:00.000-0700","entries":[{"type":"RiskIQ","matchType":"HOST","id":87457,"url":"http://www.ascentive.com/support/new/libraryfiles.exe","description":"Confidence: 100. Binary file MD5 found in AV(16%)","detectedAt":"2011-08-07T14:52:10.000-0700"},{"type":"CymruMalware","matchType":"HOST","id":713,"url":"http://www.ascentive.com/support/new/libraryfiles.exe","description":"Binary file MD5 found in AV(16%)","detectedAt":"2011-09-19T13:20:42.000-0700"},{"type":"VirusTotal","matchType":"HOST",

Does anyone have any ideas on what the best way to go about getting these log files into Splunk?

0 Karma
Highlighted

Re: Indexing a log file comes up with empty results.

New Member

I got a little closer with:

INDEXED_EXTRACTIONS=json

0 Karma
Highlighted

Re: Indexing a log file comes up with empty results.

Splunk Employee
Splunk Employee

If you use the "upload data" wizard, and select Structured>Json you get the first example.
The second example uses LINE
BREAKER and will break at the point designated...
Splunk understands JSON and will extract the field for you. First one indexes the fields second one doesn't.

 [indexed_json]
    INDEXED_EXTRACTIONS = json
    KV_MODE = none
    NO_BINARY_CHECK = true
    category = Structured
    description = JavaScript Object Notation format. For more information, visit http://json.org/
    disabled = false
    pulldown_type = true

    [JSON_LB]
    LINE_BREAKER = ([\r\n]+){"startDateInclusive":
    NO_BINARY_CHECK = true
    SHOULD_LINEMERGE = false
    category = Custom
    pulldown_type = true
With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma