Hi all,   So here is the deal, I have to prepare some( a lot) db_outputs(using db_connect), however the corresponding tables are not yet existing. Colleagues responsible for that are on different tasks. I would like to configure the exports in advance, so that once the tables are ready, the output would just flow. (and I might not be able to work on that later)  I did not manage to find a way using the GUI for that , as it always requires every step of the the way to be fulfilled, so even if I had data waiting for me, I would not be able to prep the field matching. So my idea is to configure them in db_outputs.conf, then a restart of the HF, should (or at least i think) be the solution. However, there is this.... customized_mappings = <string> # required # Specifies the output data name (fieldx) and database column number (1...n) mappings. # The expected format is: # field1:column1:type1,field2:column2:type2,…,fieldN:columnN:typeN And I do not know where to get the values for the types( I already know what field will be varchar, timestamp etc.... what I do not know is numeric representation of the field types). So it is a two fold question, 1) does anybody know this numeric to field type mapping (for example varchar=12, unassigned integer=4.... these I got from previous tables) ? 2) has anyone configured outputs in advance, before the corresponding table is even created, and does it start later automatically?  have fun! rd ... View more

Hi all, I need some support regarding ingestion of .pcap files to SPLUNK.  We have another application that already creates .pcap files in a directory, and they rotate in a good enough manner.... on all of the needed machines(and I already have UF there as I am ingesting log files from these machines). I do not want /or need to push splunk to create new pcaps for me. I just want to transfer the files to SPLUNK, and hopefully be able to then analyze them there. I have read what I found regarding Stream App, and I have to say it is a bit inconsistent and unclear. I have architecture with UFs->HFs->IDXs->SHs. We have installed Stream app on the SH, the TA_stream on the SH, Indexers and HFs, I have Stream installed on UF as well. However I still do not see option for PCAPs in Data Input menu of settings.  It is probably lack of configuration on my side, but I fail to find the documentation how to configure the app on each of the nodes. SPLUNK is 9.0.2/on Linux If someone has already gone through that challenge, I'd appreciate a piece of advice.     ... View more

Hi all,  I am at an impasse, and I do need some ideas how to overcome that. So here is the challenge.  1)I have CSV data coming into SPLUNK via UF(50+ sources); 2)Data is slightly restructured (not even always) , and later sent as DB_Output via SPLUNK DBConnect. 3)Data is output either using upsert(for several of the DBs, that need it), or without upsert(for the rest) 4)The automated search(for the DBoutput) is performed periodically (about once/hour, for X-amount of hours back), and it covers our needs regarding newly added data into previously added data sources. HOWEVER, and here is the main challenge, 5)When a new source is added, the CSV source files are already full of old data( older than X amount of hours, some times months) , thus though the data is indexed, it does not get DB-outputted(great word). However I have to send the old data as well. 6) Automated search for "all time" all the time is not applicable( 50+ sources, with often 10+ source files each) , the amount of "select"s sent to the target DB is too much and causes too much time for splunk to search plus the target DB is overwhelmed and stops responding. At this moment, I have a workaround solution , that involves modifying the automated output to search "All time" for that particular source, for 1 iteration, and then I change it back to normal, but, the sources keep increasing their numbers, and this solution is more and more inconvenient. I have a feeling that there is a better solution(easy solution) but it eludes me. If anyone could share ideas, that would be helpful. Have a great day!     ... View more

Hi all,  I am facing strange behavior,  for which I can't find anything in the docs. I have a source that generates CSV files(comma sep.). They are indexed in a dedicated index, and sourcetype. A look-alike example : id,service_id,product_id,shop_id,user_id,blah_blah,whatever,name,date,client_id 1,34456789,12234,23,4,f,45678,ivan,2022-01-13 07:04:49,1 2,34452789,12134,25,4,f,45678,ivan,2022-01-13 07:14:49,1 3,34451789,12134,27,4,f,45678,ivan,2022-01-13 07:14:49,1 4,34451789,12134,27,4,f,45678,ivan,2022-01-13 07:15:49,1 5,34451789,12133,23,4,f,45678,ivan,2022-01-13 07:15:49,1 6,34456789,12234,23,4,f,45678,ivan,2022-01-13 07:04:49,1 7,34452789,12134,25,4,f,45678,ivan,2022-01-13 07:14:49,1 8,34451789,12134,27,4,f,45678,ivan,2022-01-13 07:14:49,1 9,34451789,12134,27,4,f,45678,ivan,2022-01-13 07:15:49,1 Now, the challenge no1 is that the script that generates the csv, can edit on already existing lines. challenge no2 is that this does not result in one and the same behaviors all the time. If a simple value is changed ( from 1 to 2, or from ivan to ivag - important is same number of characters) , the change is no-were to be found in the indexed data. However, if the change includes a change in the number of characters(say ivan becomes johnathan) then, the whole file is re-indexed with the new value, causing lots of duplications. I am sure that this must be documented somewhere....but I can not find it, thus can not really understand it.  Does anyone know what is going on( I managed to find something in the community about splunk checking the first 256 char. of a file to decide, but I have tested changing both before 256 threshold and after it).....?   Kind regards! rd   ... View more