I wonder if anyone can help. We currently process feeds using perl which is of course the perfect language for this sort of thing. However we have a real techie problem.
Our database is UTF-8, the feeds we collect are largely a windows type encoding, something like iso-8859-1 or similar.
If we import the products the data inside the database still effectively is windows binary for those characters (excuse my terminology, im the designer not the programmers) which means if i was to display that data in UTF-8 then it would for the most part look like junk, especially for characters like ä or ö for example. Resetting a browser (if its for web use) to windows encoding would show the characters correctly.
Now there is nothing wrong with that and to be honest i dont care but we are dealing with datas from many different countries and using them for many different applications and sometimes that data might not be returning to environments that render the same encoding. For example we import data from russia in KOI8-R but would be displaying it to people that might not have cyrillic installed in their operating system to view it.
There are allsorts of other reasons but the only way to move forward with it properly is to convert all datas, no matter what they are into utf-8 characters before they go into our database and if we want to return them to their native state then we can do that when we export them from the database. We need a base to work from, a standard so everyone knows what they are doing all the time, not messing things up because of this or that encoding.
What i really need to know is if there is any kind of perl libraries that can handle this, there is a huge open source project for nearly everything alive but i havent a clue what i
am looking for or even which of them is the best for what we want to do. Is there such a thing. Basically we want the library to be a complete engine of its own, converting the characters into UTF-8 as we import them so the whole of our datas are standard across the world. Anyone have a clue or do we have to resort to signing up to those god awful techie forums where they want to know how much ram we have in our coffee cups before they even answer the question?
Thanks for any help, or even if you just took the time to read and laugh
Linear Mode
