customize Indexing

[vaibhavbeohar]

I need your help to configure Splunk for unstructured text file, and how do I upload a text file in Splunk which is having field separated by comma?
Can we customize Index and if yes, how do I create my own index with different fields?

[jbsplunk]

You can use DELIMS and FIELDS in transforms.conf to define search time field extractions for a specific type of data(source/sourcetype/host) as called in props.conf:

From Transforms.conf.spec(http://docs.splunk.com/Documentation/Splunk/latest/admin/transformsconf):

DELIMS = <quoted string list>
* NOTE: This attribute is only valid for search-time field extractions.
* IMPORTANT: If a value may contain an embedded unescaped double quote character, 
  such as "foo"bar", use REGEX, not DELIMS. An escaped double quote (\") is ok.
* Optional. Used in place of REGEX when dealing with delimiter-based field extractions, 
  where field values (or field/value pairs) are separated by delimiters such as colons, 
  spaces, line breaks, and so on.
* Sets delimiter characters, first to separate data into field/value pairs, and then to 
  separate field from value.
* Each individual character in the delimiter string is used as a delimiter to split the event.
* Delimiters must be quoted with " " (use \ to escape).
* When the event contains full delimiter-separated field/value pairs, you enter two sets of 
  quoted characters for DELIMS: 
    * The first set of quoted delimiters extracts the field/value pairs.
    * The second set of quoted delimiters separates the field name from its corresponding
      value.
* When the event only contains delimiter-separated values (no field names) you use just one set
  of quoted delimiters to separate the field values. Then you use the FIELDS attribute to
  apply field names to the extracted values (see FIELDS, below).
    * Alternately, Splunk reads even tokens as field names and odd tokens as field values.
* Splunk consumes consecutive delimiter characters unless you specify a list of field names.
* The following example of DELIMS usage applies to an event where field/value pairs are 
  seperated by '|' symbols and the field names are separated from their corresponding values 
  by '=' symbols:
    [pipe_eq]
    DELIMS = "|", "="
* Defaults to "".   

FIELDS = <quoted string list>
* NOTE: This attribute is only valid for search-time field extractions.
* Used in conjunction with DELIMS when you are performing delimiter-based field extraction 
  and only have field values to extract. 
* FIELDS enables you to provide field names for the extracted field values, in list format 
  according to the order in which the values are extracted.
* NOTE: If field names contain spaces or commas they must be quoted with " " (to escape, 
  use \).
* The following example is a delimiter-based field extraction where three field values appear
  in an event. They are separated by a comma and then a space.
    [commalist]
    DELIMS = ", "
    FIELDS = field1, field2, field3
* Defaults to "".

An example from the same file:

# Extract comma-delimited values into fields:

[extract_csv]
DELIMS = ","
FIELDS = "field1", "field2", "field3"

# This example assigns the extracted values from _raw to field1, field2 and field3 (in order of 
# extraction). If more than three values are extracted the values without a matching field name 
# are ignored.

[lguinn2]

An index is not like an SQL table - a single index can hold many different types of data. All of these different data types can have different fields. So you can have a separate index if you want -- but you don't need to have a different index for each type of data. Fields are not defined by the index.

[Ayn]

Have you understood the concept of search-time extraction of fields in Splunk? http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Aboutfields

[RubenOlsen]

Could you provide us with some sample data?