Splunk Search

Lookup csv with Ñ/ü/ä/í... characters (German/Spanish/French)

Contributor

Hello all,

I have some csv files that I'm updating to splunk as lookup files, but there are some german/spanish/french characters that are not being recognized. Recently I found out the text iso necessary for this and I've changed the sourcetype for the index.

How can I do it for the lookups? If not possible, maybe it's easier to change them to index data?

Thank you in advance,

0 Karma
1 Solution

Legend

Hi marina_rovira,
If your lookups are built from indexed data, you could rebuild them using a search with outputlookup command.
otherwise the easiest way is to export them (or to take the original csv files), modify them in Excel or Notepad++ and then upload the updated files.

Bye.
Giuseppe

View solution in original post

Ultra Champion

The following seems to be a good approach -

-- I used a specialized converter to change it to UTF-8 and now it works fine.

It's at Lookup files with foreign characters

alt text

0 Karma

Explorer

Hi Marina,

What it has worked for me is to edit the file with Notepad++
In the Coding menu, select "Convert to UTF-8 and save as a new file.CSV.
Open the new file.CSV, select "Code as ANSI" and save again.
With this new file.CSV, inputlookup show all characters OK.

0 Karma

Explorer

Update: I've tried again with Notepad++ "Convert to UTF-8" and save.
It worked also perfecty.

Contributor

One thing here, if in one index, I have Informàtica and in the lookup I have Infromatica because of the conversion, Will I have problems in the search and mix them?

0 Karma

Ultra Champion

Not sure Marina about the Informàtica's data - we need to find out in which character encoding your data is...

0 Karma

Legend

Hi marina_rovira,
If your lookups are built from indexed data, you could rebuild them using a search with outputlookup command.
otherwise the easiest way is to export them (or to take the original csv files), modify them in Excel or Notepad++ and then upload the updated files.

Bye.
Giuseppe

View solution in original post

Contributor

Hey, they are not build from indexed data. I just export and move them directly to the lookup directory in splunk app search. This is because they should be static, they are updated with minimal changed once a month, and for looking info with them easier, but there is no previous data indexed.

What I need is something as the ISO field in the sourcetype data for the lookup tables.

0 Karma

Legend

Hi marina_rovira,
instead to move lookup files, try to newly import them in your SHs , eventually using Lookup Editor App, so you can immediately see result.
Bye.
Giuseppe

0 Karma

Contributor

I've started that, but I have a little question. The good thing about moving it directly is that I have a script doing it, which implies no manual work from my part.

Adding it as new by the app, then I have to update it manually every month? Or it will recognize if the script overwrites the file?

Thank you for your help.

0 Karma

Legend

Hi marina_rovira,
what's the output of your script?
if it's something that Splunk can recognize, you can ingest every month the script's results in Splunk (with the correct management of characters) and then update your lookup using outputlookup command.
Bye.
Giuseppe

0 Karma

Contributor

Do you have near you some understanding information about how to sue the OUTPUT thing?

The output of my script is a csv file directly.

0 Karma

Legend

You could ingest in Splunk the csv files in an index and then create a search with outputlookup command that one time a month rebuild your lookup

index=this_new_index earliest=-24h latest=now
| fields field1 field2 ... fieldn
| outputlookup yourlookup.csv

scheduled after your script execution.

Bye.
Giuseppe

Contributor

Sorry for bother you one more time.

I've done everything and it seems it works, just one more question. Now the csv file will be injected as an index, where or how can I schedule this rebuilt search?

Thank you in advance.

0 Karma

Contributor

I think I found it 🙂 Tahnk you anyway for all the help.

0 Karma

Legend

Hi Marina,
If this answer satisfies your question, please accept it.
Thank you.
Giuseppe

0 Karma

Contributor

Hello! Sorry for bringing this live again, but I have one question.

I seemed to work, but now, I have again the bad characters for the languages special ones. The problem is I don't remember how to modify it. Also when I try to open a lookup, it says me he file is too big to open it.

Can you help me please?

Thank you

0 Karma

Contributor

I think I'm starting to understand what you meant.

I will try this. Thank you 🙂

0 Karma

Contributor

Just, one more question.

This file, for example, I can not pick the time range for exporting, as is a view itself. So, every month, if there is any change, is because there's something new, something to add.
If I use an index, I will have the information repeated every time right?
Do you know some way in bash or in splunk itself, to not have all the lines repeated? (besides the dedup command in the search)

Thank you so much, you're helping me a lot.

0 Karma

Legend

Hi marina_rovira,
If you ingest csv in an index one time a month and use for it the automatic Splunk Timestamp, you'll have all the csv raw with the same timestamp in index.
So choosing last month as time period you're sure to take only the last one raw for each one and you can replace the full lookup.
If instead you want update only the different from lookup raws, you could run a search like this:

index=your_index earliest=-mon latest now NOT [ | inputlookup your_lookup.csv ] 
| table field1 field2 ... fieldn
| outputlookup your_lookup.csv append=true

Bye.
Giuseppe

0 Karma