Indexing a log file comes up with empty results.

[agregory23]

Hello-

I am trying to index some files in a directory local to the splunk server (/tmp/risqiq/). I can see in the S.O.S that it it searching the log files that I want with since it has an "Action Status" of "finished reading" and a "Read_status" of "read". I put them in their own index but when I go look in the index there are 0 events in the index and its 1kb in size. I believe that its becase of my event type and the fact that its parsing the file and determining that there is nothing in there. I have tried both the continuous import mode as well as the direct file import.

I have tried to have Splunk auto detect the data type but that does not work. The log contents do show in the preview so I know it can access the files and folder. It has the date highlighted (which is not what I want). The problem is the log file is one giant line with events buried in to it. I believe each of the events begin with "id". I have also tried to search with a regex search of "\b\w[id]\b".

I have tried the following settings in the advanced mode of data preview for props.conf (I dont have shell access to the server itself):

# your settings
BREAK_ONLY_BEFORE=\b\w[id]\b
MAX_TIMESTAMP_LOOKAHEAD=150
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false

# set by detected source type
TRUNCATE=0
index=riskiq
pulldown_type=1

Here is a snippet from the log:

{"startDateInclusive":"2015-04-19T19:04:39.000-0700","endDateExclusive":"2015-04-19T20:04:39.000-0700","totalResults":2669,"resources":[{"id":249138,"url":"http://www.oneclicktools.com/oct/RingtoneConverterIcon.gif","hostname":"www.oneclicktools.com","rank":2147483647,"phishing":false,"malware":true,"spam":false,"matchType":"HOST","score":49,"description":"Binary file MD5 found in AV(21%)","contentType":"image/gif","httpResponseCode":200,"detectedAt":"2010-01-01T00:00:00.000-0800","lastSeenAt":"2015-04-19T05:37:40.000-0700","entries":[{"type":"GSBMalware","matchType":"DOMAIN","url":"oneclicktools.com/"},{"type":"CymruMalware","matchType":"HOST","id":34761,"url":"http://www.oneclicktools.com/cdwriter15.exe","description":"Binary file MD5 found in AV(21%)","detectedAt":"2012-12-25T10:45:49.000-0800"}]},{"id":1130541,"url":"http://www.nutsvolts.com/","hostname":"www.nutsvolts.com","ip":"76.12.26.68","asn":"20021","rank":747367,"phishing":false,"malware":true,"spam":false,"matchType":"HOST","score":49,"description":"Binary file detected by AV(21%)","contentType":"text/html","httpResponseCode":200,"detectedAt":"2010-01-01T00:00:00.000-0800","lastSeenAt":"2015-04-19T20:00:55.000-0700","entries":[{"type":"GSBMalware","matchType":"DOMAIN","url":"nutsvolts.com/"},{"type":"VirusTotal","matchType":"HOST","id":7770421,"url":"http://www.nutsvolts.com/uploads/magazine_downloads/workshop55.zip","description":"Binary file detected by AV(21%). {Avast=Win32:Malware-gen, Avira=TR/Graftor.101096.2, Comodo=UnclassifiedMalware, GData=Archive.Trojan.Agent.MV9FTA, Ikarus=Backdoor.Poison, Jiangmin=Backdoor/Poison.adbb, K7AntiVirus=Riskware ( 0040eff71 ), K7GW=Riskware ( 0040eff71 ), McAfee=RDN/Generic.dx!d2b, McAfee-GW-Edition=RDN/Generic.dx!d2b, Symantec=Trojan.Gen.2, TrendMicro-HouseCall=Suspicious_GEN.F47V0106}. md5=05dea7390de10298c277683f4d75646f","detectedAt":"2013-07-11T13:47:00.000-0700"}]},{"id":2692729,"url":"http://files.informer.com/siinst.exe","hostname":"files.informer.com","ip":"208.88.224.211","asn":"40824","rank":849,"phishing":false,"malware":true,"spam":false,"matchType":"URL","score":54,"contentType":"application/octet-stream","httpResponseCode":200,"detectedAt":"2011-02-20T06:51:54.000-0800","lastSeenAt":"2015-04-19T16:33:50.000-0700","entries":[{"type":"Malc0de","matchType":"URL","id":1860233,"url":"http://files.informer.com/siinst.exe","description":"MD5: 3ad2f4b0b0ce0875da4dd58bced17db9, IP: 208.88.224.211, Country: us, ASN: 40824","detectedAt":"2011-11-09T06:20:00.000-0800"},{"type":"VirusTotal","matchType":"URL","id":20347336,"url":"http://files.informer.com/siinst.exe","description":"Binary file detected by AV(4%). {Ikarus=Win32.SuspectCrc, TrendMicro-HouseCall=Suspicious_GEN.F47V1021}. md5=e49ef284df6d6516c0bf8851c76d081e","detectedAt":"2014-09-24T09:51:34.000-0700"}]},{"id":3795387,"url":"http://files.brothersoft.com/internet/miscellaneous/brothersoftextreme_ct2776682.exe","hostname":"files.brothersoft.com","ip":"65.49.92.216","asn":"6939","rank":7440,"phishing":false,"malware":true,"spam":false,"matchType":"URL","score":77,"description":"Binary file detected by AV(7%)","contentType":"application/octet-stream","httpResponseCode":200,"detectedAt":"2011-04-28T17:07:14.000-0700","lastSeenAt":"2015-04-19T08:31:19.000-0700","entries":[{"type":"GSBMalware","matchType":"HOST","url":"files.brothersoft.com/"},{"type":"RiskIQ","matchType":"URL","id":330552,"url":"http://files.brothersoft.com/internet/miscellaneous/BrotherSoftExtreme_CT2776682.exe","description":"Confidence: 75. FakeSoftwareUpdate: alert:Your Flash Player may be out of date","detectedAt":"2014-08-23T12:01:31.000-0700"},{"type":"CymruMalware","matchType":"HOST","id":1589,"url":"http://files.brothersoft.com/security/monitoring_software/spyagent7.zip","description":"Binary file MD5 found in AV(20%)","detectedAt":"2011-10-04T07:24:54.000-0700"},{"type":"VirusTotal","matchType":"URL","id":448557,"url":"http://files.brothersoft.com/internet/miscellaneous/BrotherSoftExtreme_CT2776682.exe","description":"Binary file detected by AV(7%). {Cyren=W32/A-68608136!Eldorado, DrWeb=Program.BrotherSoft.4, F-Prot=W32/A-68608136!Eldorado, NANO-Antivirus=Riskware.Win32.BrotherSoft.diumlo}. md5=b02a24d94306e494994ef41b55be7d07","detectedAt":"2011-09-21T09:25:34.000-0700"}]},{"id":8119183,"url":"http://www.ascentive.com/","hostname":"www.ascentive.com","ip":"64.62.158.147","asn":"6939","rank":81185,"phishing":false,"malware":true,"spam":false,"matchType":"HOST","score":45,"description":"Binary file MD5 found in AV(16%)","contentType":"text/html","httpResponseCode":200,"detectedAt":"2011-08-07T22:46:15.000-0700","lastSeenAt":"2015-04-18T22:55:39.000-0700","entries":[{"type":"CymruMalware","matchType":"HOST","id":713,"url":"http://www.ascentive.com/support/new/libraryfiles.exe","description":"Binary file MD5 found in AV(16%)","detectedAt":"2011-09-19T13:20:42.000-0700"},{"type":"VirusTotal","matchType":"HOST","id":22960038,"url":"http://www.ascentive.com/run/download?service=SpeedScan&debug=0&loadlink=http%3A%2F%2Fwww.ascentive.com%2Frun%2Fclick%2F%403040667546237%2Fproducts%2FSP_finallyfastpc%2Fselect_finallyfastpc_speedscan_download_op1.html%3Ftheme%3Dselect_speedscan_finallyfastpc_download%26plan1id%3D%26orderpackage1id%3DSSCN1YEAR29%26plan1c%3D%26upsell_code%3D%26popuppage%3D%26display%3D%26referredby%3D%403040667546237%26loadlink%3D","description":"Binary file detected by AV(11%). {ESET-NOD32=a variant of Win32/Ascentive, Kingsoft=VIRUS_UNKNOWN, McAfee=Artemis!B18049048917, McAfee-GW-Edition=Artemis, TrendMicro-HouseCall=Suspicious_GEN.F47V1029, Zillya=Trojan.Patched.Win32.80083}. md5=b18049048917e904f3c1a544d869abe1","detectedAt":"2014-11-09T22:16:08.000-0800"}]},{"id":8119273,"url":"http://www.ascentive.com/partners/affiliate-program/","hostname":"www.ascentive.com","ip":"64.62.158.147","asn":"6939","rank":81185,"phishing":false,"malware":true,"spam":false,"matchType":"HOST","score":45,"description":"Binary file MD5 found in AV(16%)","contentType":"text/html","httpResponseCode":200,"detectedAt":"2011-08-07T22:46:34.000-0700","lastSeenAt":"2015-04-19T19:14:23.000-0700","entries":[{"type":"CymruMalware","matchType":"HOST","id":713,"url":"http://www.ascentive.com/support/new/libraryfiles.exe","description":"Binary file MD5 found in AV(16%)","detectedAt":"2011-09-19T13:20:42.000-0700"},{"type":"VirusTotal","matchType":"HOST","id":13017376,"url":"http://www.ascentive.com/run/download?service=SpeedScan&debug=0&loadlink=http://www.ascentive.com/run/click/@2022810294200/products/SP/select_finallyfast_download_op1.html?theme%3Dselect_finallyfast_partner_download%26plan1id%3D%26orderpackage1id%3DFFST1YR29%26plan1c%3D%26upsell_code%3Dff2%26popuppage%3D%26display%3D%26referredby%3D@2022810294200%26c1%3D03_59531325_093feaa1-e60c-4ae0-a3c9-514147985d45%26loadlink%3D","description":"Binary file detected by AV(16%). {Comodo=Application.Win32.Conduit.~A, DrWeb=Adware.Conduit.6, ESET-NOD32=Win32/Toolbar.Conduit.R, Malwarebytes=PUP.Optional.Conduit.A, McAfee=Artemis!43583D1CCFCE, McAfee-GW-Edition=Artemis!43583D1CCFCE, Symantec=Trojan.ADH.2, TrendMicro-HouseCall=TROJ_GEN.F47V0305}. md5=43583d1ccfce25d7c422f4ecdafa662b","detectedAt":"2014-03-05T15:14:26.000-0800"}]},{"id":8145893,"url":"http://knorrnetworks.net/","hostname":"knorrnetworks.net","ip":"75.145.143.169","asn":"33491","rank":2147483647,"phishing":false,"malware":true,"spam":false,"matchType":"HOST","score":30,"description":"Binary file detected by AV(22%)","contentType":"text/html","httpResponseCode":200,"detectedAt":"2011-08-08T13:45:33.000-0700","lastSeenAt":"2015-04-18T20:36:28.000-0700","entries":[{"type":"RiskIQ","matchType":"HOST","id":87799,"url":"http://knorrnetworks.net/SUPPORT/VNC/vnc-4.0-x86_win32.exe","description":"Confidence: 100. Binary file detected by AV(14%)","detectedAt":"2011-08-08T14:02:00.000-0700"},{"type":"VirusTotal","matchType":"HOST","id":47853,"url":"http://knorrnetworks.net/kn.exe","description":"Binary file detected by AV(22%). {Antiy-AVL=Trojan[RemoteAdmin:not-a-virus]/Win32.WinVNC-based, Baidu-International=HackTool.Win32.WinVNC.AsK, CMC=RemoteAdmin.Win32.WinVNC-based!O, Comodo=UnclassifiedMalware, Kaspersky=not-a-virus:RemoteAdmin.Win32.WinVNC-based.c, Malwarebytes=PUP.Radmin, McAfee=Artemis!193A45B47AD8, McAfee-GW-Edition=Heuristic.BehavesLike.Win32.Suspicious-BAY.G, NANO-Antivirus=Riskware.Win32.WinVNC-based.cqjdov, Symantec=WS.Reputation.1, TheHacker=Trojan/Buzus.bpyo}. md5=193a45b47ad85b15c39cfc32d0d88164","detectedAt":"2011-08-07T10:47:00.000-0700"}]},{"id":8278779,"url":"http://www.ascentive.com/about_us/ascentive/press_releases.php","hostname":"www.ascentive.com","ip":"64.62.158.147","asn":"6939","rank":81185,"phishing":false,"malware":true,"spam":false,"matchType":"HOST","score":34,"description":"Binary file MD5 found in AV(16%)","contentType":"text/html","httpResponseCode":200,"detectedAt":"2011-08-11T21:26:19.000-0700","lastSeenAt":"2015-04-19T10:38:00.000-0700","entries":[{"type":"RiskIQ","matchType":"HOST","id":87457,"url":"http://www.ascentive.com/support/new/libraryfiles.exe","description":"Confidence: 100. Binary file MD5 found in AV(16%)","detectedAt":"2011-08-07T14:52:10.000-0700"},{"type":"CymruMalware","matchType":"HOST","id":713,"url":"http://www.ascentive.com/support/new/libraryfiles.exe","description":"Binary file MD5 found in AV(16%)","detectedAt":"2011-09-19T13:20:42.000-0700"},{"type":"VirusTotal","matchType":"HOST",

Does anyone have any ideas on what the best way to go about getting these log files into Splunk?

[rsennett_splunk]

If you use the "upload data" wizard, and select Structured>_Json you get the first example.
The second example uses LINE_BREAKER and will break at the point designated...
Splunk understands JSON and will extract the field for you. First one indexes the fields second one doesn't.

 [indexed_json]
    INDEXED_EXTRACTIONS = json
    KV_MODE = none
    NO_BINARY_CHECK = true
    category = Structured
    description = JavaScript Object Notation format. For more information, visit http://json.org/
    disabled = false
    pulldown_type = true

    [JSON_LB]
    LINE_BREAKER = ([\r\n]+){"startDateInclusive":
    NO_BINARY_CHECK = true
    SHOULD_LINEMERGE = false
    category = Custom
    pulldown_type = true

[agregory23]

I got a little closer with:

INDEXED_EXTRACTIONS=json