Extract New Field - First x characters

[robMZ]

Hi, I want to create a new field which will simply pull out the first x number of characters from a line on an event log. I am not sure of the regex to use as I assume that's the option to go for?

As per the image this log brings back an initial datetime stamp followed by certain text (which is what I am searching on). If I can get this in a specific field it will help for amending my query, it's output and the linked alert which uses this. 

[ITWhisperer]

There is no image that you refer to, however, if you want to say skip the first 5 characters and extract the next 10 you would do something like this

| rex "^.{5}(?<field>.{10})"

[jmartens]

You don't need a regular expression for that, substring will do perfectly well.

[robMZ]

Apologies, here is the image. The text displayed i.e. the lines with the datetime stamp and then text isn't a field so how can I create this as a field, the first x characters.

[ITWhisperer]

Is the timestamp always the same length? That is, can you assume that the data you want always starts at the same position or do you need to have an expression to locate the start of the desired string?

[jmartens]

I think you also need to revisit your sourcetype configuration as it seems to split you log entry in (at least) three lines judging from your image. I am assuming that all three lines should be in one event and you want to extract fields from there.
This way it is hardly impossible to correlate data to one event concerning your image.

As a start you might want to take a look at the Getting data in primer, especially the section on Configure event line breaking 

[robMZ]

This is the original log file, each line is a new event. I am using an OR statement to pick up on particular lines.

There's no pattern hence I think the best solution to have each line captured in a new field is to use the first x amount of characters, maybe 50.

Let me know if that makes sense. 

[jmartens]

Did you even try bother reading the documentation section I referred to?

The second example in the examples section more or less handles your case of one event running over multiple lines where each line is prepended with a timestamp.

[jmartens]

Can you post some example data and an example of desired results? I think your current question is kind of cryptic.

From what I could make of it you might be better of using one of Splunks text functions for instance substr:

As a simple example:

| makeresults
| eval test=substr("string", 1, 3)

Which should create a dummy test event with a field test with the value 'str' (first three characters of the text 'string')

[robMZ]

Apologies if not explaining well (newbie). Do the attached images help in regards to the Splunk query and the log in it's original format.

The lines which start with a datetime stamp aren't a field I can reference by name - or am I wrong?

I want to create this as a field, there's no pattern hence my suggestion of first x characters.

Splunk search

original log (txt file)

Extract New Field

[ITWhisperer · Accepted solution]

Each event is in a field called _raw - this is also the default field used by some commands (e.g. rex) and searches

[jmartens]

This is an oversimplification. In searches you can search on any field you like, but you will have to define them if you are not using a default sourcetype.

Relying on _raw and extracting all your fields from there at search time is cumbersome and not future proof.

[robMZ]

AHA!! 💡

I had no idea as assumed it wasn't captured and I had to create my own field. Thank you for this

[jmartens]

I really suggest you read up on how to get data in. If you continue on this path you will have to redo a lot off work and will not benefit very much from your data. Even though you accepted the answer pointing you to the _raw event using SPL and text filtering is not the wat to get data in robustly for the future.

It all starts with getting your data in in a ordered and structured way, hence my link to the primer on Getting data in.

You first need to setup your sourcetype correctly as I mentioned earlier, get all your data in to one event if it is concerning one entity.

From there define fields to be extracted at index time. These fields can than be (re-)used when searching and reporting. Creating your fields each time in  the query window (e.g. in your SPL query) is cumbersome and not future proof.

When defining your sourcetype you can then also set other stuff like custom delimiters so key value pair extraction might be done for up you (automatically/automagically), for instance extracting the values after the colon in your screenshots.

Once agian, do yourself a favour and invest in a proper start, it will gain you time and joy in the long run. 😄

[robMZ]

Sure, appreciate the feedback. This is at a client site and something I have inherited, just after a short term solution to utilise what is being captured in Splunk. For that I can use what you suggested.

The points made are very valid in regards to a "better solution" and can be a different conversation.

Thanks again.