topic Re: Import a text data with a index in Splunk Search

Import a text data with a index

gsfiorese_au — Tue, 29 Sep 2020 06:57:53 GMT

Hello everyone, I'm starting with the development in splunk...

Each time a new database or datatable and created on the server, I have a service that generates a text file with the respective Info thereof, for example:

[hive_APOLLO_APOLLO_DATA]
vix.description = (hive orc) edb exit nbn database
vix.input.1.path = /data/tdc/prd/corp/base/edb/APOLLO/...
vix.provider = bud_hive_orc
vix.input.1.splitter.hive.dbname = APOLLO
vix.input.1.splitter.hive.tablename = APOLLO_DATA
vix.input.1.splitter.hive.fileformat = orc

[hive_APOLLO_EPI_IV_CLI]
vix.description = (hive orc) edb exit nbn database
vix.input.1.path = /data/tdc/prd/corp/base/edb/APOLLO/...
vix.provider = bud_hive_orc
vix.input.1.splitter.hive.dbname = APOLLO
vix.input.1.splitter.hive.tablename = EPI_IV_CLI
vix.input.1.splitter.hive.fileformat = orc

[hive_APOLLO_EPI_IV_NBN_CLI]
vix.description = (hive orc) edb exit nbn database
vix.input.1.path = /data/tdc/prd/corp/base/edb/APOLLO/...
vix.provider = bud_hive_orc
vix.input.1.splitter.hive.dbname = APOLLO
vix.input.1.splitter.hive.tablename = EPI_IV_NBN_CLI
vix.input.1.splitter.hive.fileformat = orc

...

Once the data is imported into splunk the need to show the dashboard the entire information for the file.

In this case when I click one of the results the system does the search and returns the information from block concerning the same, however, the index generated and generally for the file (main):

What do I need and the index is generated based on information contained in square brackets ([hive_APOLLO_TEST_PARTITION]) so when I run the search system bring something like this:

Main View
   DB1
      DB1.TBL1 (virtual.index.1) -> links to Child View (virtual.index.1)

 Child View (virtual.index.1)
 <Search results for splunk search: index=virtual.index.1>

 DB1=APOLLO
 TBL1=APOLLO_DATA
 virtual.index.1=hive_APOLLO_APOLLO_DATA

Sorry for the long text, someone help me?

Thank you all!

Re: Import a text data with a index

woodcock — Thu, 13 Aug 2015 18:06:05 GMT

I am having a great deal of trouble understanding very much of your request and it would help if you added more detail to your question (you can re-edit it). I think that I understand this statement:

Once the data is imported into splunk the need to show the dashboard the entire information for the file.

To do this, you can use a search like this:

... | reverse | stats list(_raw) by source

This will reconstitute every file almost exactly the way the file was when it was indexed. Hopefully I am understanding that part of your question and it helps you.

Re: Import a text data with a index

gsfiorese_au — Tue, 29 Sep 2020 06:58:31 GMT

Hi Woodcock,

Thanks for the reply and I'm sorry if I was not clear, if it is not uncomfortable try to explain better now.

Every day the txt file (myfile.txt) is imported into splunk and the data are shown through the dashboard below:

As I have many informations (1 for n) the visualization is compromised and too long, and many pages are generated and this complicates the search.

Apollo >> Table_BlaBla_1
Apollo >> Table_BlaBla_2
Apollo >> Table_BlaBla_3

Gendesk >> Table_Genx_1
Gendesk >> Table_Genx_2
Gendesk >> Table_Genx_3
Gendesk >> Table_Genx_4

I need the header once appears, and the dependencies come down (something like a tree view)

Apollo

Table_BlaBla_1
Table_BlaBla_2
Table_BlaBla_3

Gendesk

Table_Genx_1
Table_Genx_2
Table_Genx_3
Table_Genx_4

Below is the code I used to create the dashboard, if his reach to help me or anyone else will be grateful.

Re: Import a text data with a index

gsfiorese_au — Fri, 14 Aug 2015 05:12:08 GMT

Sorry I forget my code for dashboard:

<form>
  <label>Databases and Datatables - Search</label>
  <description>List of Databases and Datatables from HIVE</description>
  <fieldset autoRun="true" submitButton="true">    
    <input type="text" token="database">
      <label>Enter an Database Name</label>
      <default></default>
      <prefix>vix_input_1_splitter_hive_dbname=</prefix>
      <sufix></sufix>
    </input>
    <input type="text" token="datatable">
      <label>Enter an Database Name</label>
      <default></default>
      <prefix>vix_input_1_splitter_hive_tablename=</prefix>
      <sufix></sufix>
    </input>
  </fieldset>
  <row>
    <table>
       <title>Databases and Datatables list found in the file:</title>
       <search>
       <query>
         search $database$ $datatable$ sourcetype="my_data"
         | table vix_input_1_splitter_hive_dbname vix_input_1_splitter_hive_tablename 
         | rename vix_input_1_splitter_hive_dbname AS "Database Name" 
         | rename vix_input_1_splitter_hive_tablename AS "Datatable Name"           
       </query>
       </search>   
       <!--earliestTime>-60m@m</earliestTime-->
       <latestTime>now</latestTime>
       <option name="drilldown">row</option>
       <option name="count">10</option>
     </table>
  </row>
</form>

Tks,

Gabriel

Re: Import a text data with a index

woodcock — Fri, 14 Aug 2015 14:31:17 GMT

Your clarification was excellent; try this search:

search $database$ $datatable$ sourcetype="my_data" | stats values(vix_input_1_splitter_hive_tablename) BY vix_input_1_splitter_hive_dbname | rename vix_input_1_splitter_hive_dbname AS "Database Name" | rename vix_input_1_splitter_hive_tablename AS "Datatable Name"

Re: Import a text data with a index

gsfiorese_au — Mon, 17 Aug 2015 00:59:35 GMT

Thank you!! Woodcock!