Talk:Unicode/Archive 1

Archive 1

Archive 2

Archive 3

→

Archive 5

Intro

This paragraph was added to the start of the article:

Unicode is a standard used in computer software for encoding human readable characters in digital form. The most common encoding is the ASCII code, which can encode a maximum of 127 characters, which is enough for the English language. As computer use spread to other languages, the shortcomings of ASCII became more and more apparent. There are many other languages with many other characters; Asian languages in particular contain many, many characters.

I removed it because it is inaccurate (It overplays Unicode-as-a-standard rather than Unicode as a consortium that produces lots of standards), confusing (its mention of ASCII is not clearly historical), and adds no information that isn't already in the article. I assume, however, that it was added because someone thought the existing first paragraph was unclear, so I'm open to suggestions about how to improve it. -- Lee Daniel Crocker 12:20, 9 November 2001 (UTC)

Section 0 improvements

I think section 0 which currently reads could be improved by changing: In computing, Unicode is the international standard whose goal is to specify a code matching every character needed by every written human language, including many dead languages in small scholarly use, to a single unique integer number, called a code point.

To: In computing, Unicode is the international standard whose goal is to specify a code matching every character needed by every written human language, including many dead languages in small scholarly use such as foo and bar as well as [some other good example, perhaps a made-up language?], to a single unique integer number, called a code point.

I think the intro would be better by adding two examples there, furthermore i think is the international standard should be is an international standard, or has it been approved by a major authoraty as the standard? -- Ævar Arnfjörð Bjarmason 17:44, 2004 Oct 5 (UTC)

The parts of Unicode which are also in ISO 10646 most likely define it as the standard, aslo given the fact that the maintaince of the ISO 8859 has been put into hibernation. Pjacobi 20:27, 5 Oct 2004 (UTC)

And, argueing about section 0, what about: in internationalization of software. A Thai programmer writing a program with Thai user interface for Thai customers doesn't fit at all the definition of internationalization. -- Pjacobi 20:30, 5 Oct 2004 (UTC)

The interesting thing for most people is that it provides a way to store text in any language in a computer. Starting off by mentioning "unique integer numbers" doesn't make Unicode easier to understand. Even as a computer programmer, I have a bit of trouble reading that sentence and understanding what it means. And it's not really true as given; characters in Unicode is a polite fiction. Many characters (Maltese "ie", Lakota p with bar above, many Khmer characters) are more then characters in Unicode-ese. Going to rewrite boldly. --Prosfilaes 21:47, 11 Oct 2004 (UTC)

Unicode Lacks ASL glphys

I can't believe that Unicode 5.0 doesn't have American Sign Language characters. Linguistic research during the past thirty years has demonstrated that American Sign Language (and indeed any of the world's indigenous sign languages) meets all of the requirements for human languages - it is a rule-governed, grammatical symbol system that changes over time and that members of a community share —Preceding unsigned comment added by 64.40.46.183 (talk) 22:34, 19 September 2007 (UTC)

New sections should go at the end — I generally assume that stuff near the top has already been resolved. ASL is not a written language, so it doesn't warrant being in Unicode. I don't know of any reasonable amount of text in Unicode, and if there is any, I suspect it's largely to help people learn Unicode. ⇌Elektron 11:22, 22 September 2007 (UTC)

Downloads

I am not a techie! Nevertheless I can see the usefulness of much of the material available in Unicode. Neither am I the sort of anti-techie that complains that anything in other than plain-Jane unaccented English alphabetical characters must be thrown out of Wikipedia, or that articles should not be displaying meaningless question marks. I was visiting the chess page, and someone there has made a valiant effort to produce diagrams of how the pieces move by using only ordinary keyboard characters. I'm sure that he would not take it as a sign of disrespect when I say that it looks like shit.

I see no such chart there. Evertype 15:43, 2004 Jun 20 (UTC)

That's because they've been removed. They were there when the comment was posted in March 2002. --Zundark 16:11, 20 Jun 2004 (UTC)

I'm sure that most of us would like to see the special symbols, letters, or chinese characters at the appropriate time and place. At the same time I understand that for many Wikipedians there are technical reasons which prevent their hardware from dealing with this material (eg. limited memory). Then there are others for whom only the appropriate software is missing. Even some of the people with hardware restrictions may be able to handle Greek or Russian, though probably not Chinese. In cases where I've tried to find the code, I've ended up wading through reams of technical discussions. These discussions may be very interesting, but they don't provide a solution to my immediate problem.

The practical suggestion may be a notice at the head of any article containing symbols not in ISO 8859-1 saying in effect. "This article contains non-standard characters. You may download these characters by activating this LINK". Eclecticology

Just because an HTML document contains characters that are not in the ISO 8859-1 range doesn't mean that the characters are nonstandard. HTML 4.0 allows nearly all of Unicode to be used in a document, and all web browsers make an attempt to handle any character they encounter. The problem is merely that the underlying operating systems upon which the browsers rely to provide character rendering tends to be either not Unicode aware or just does not have a good selection of fonts (character-to-glyph mappings) installed.

There's no reliable way to guess the user's character rendering capabilities, so we really don't know when to tell people when it would be a good idea to download font files, and fonts tend to be OS-specific anyway. I prefer just to acknowledge in the prose that any non-ASCII characters may or may not render as they are *supposed* to. I don't think we should dumb down the HTML and avoid those characters though. - mjb 18:21 Feb 20, 2003 (UTC)

---

In cases where I've tried to find the code, ...

What exactly were you looking for ? Do you have the Unicode value, and you're looking for a typical glyph (like a ASCII chart) ? Are you looking for the Unicode value ?

These discussions may be very interesting, but they don't provide a solution to my immediate problem.

What exactly is your "immediate problem" ?

---

Is there a reason to use '<code>foo</code> <code>bar</code> <code>baz</code> ...' instead of '<code>foo bar baz ...</code>'? -- Miciah

UTF-7

Isn't there a UTF-7? Or is an invetion of Microsoft (it's in .NET)? CGS 21:54, 16 Sep 2003 (UTC).

Yes[1], but it's virtually never used. --Brion 23:33, 16 Sep 2003 (UTC)

---

The oldest of Unicode's encodings is UTF-16, a variable-length encoding that uses either one or two 16-bit words, manifesting on most platforms as 2 or 4 8-bit bytes, for each character. {NB: This can't be true; UCS-2 has to predate UTF-16!}

66.44.102.169 wrote "{NB: This can't be true; UCS-2 has to predate UTF-16!}" in the article. UTF-16 was previously UCS-2 but I'm not sure that makes the statement untrue as such but I reworded it anyway. Angela.

In the way the terminology today is used, UCS-2 doesn't have surrogate support, and certainly a 16-bit encoding without surrogate support existed before one with it. I don't think either of these were called UCS-2 and UTF-16 at the time though. Morwen 11:52, 6 Dec 2003 (UTC)

I wrote the comment that UTF-16 can't be the oldest encoding. Currently text may be encoded in UCS-2 or it may be encoded in UTF-16. Many Windows application designers pay no heed to the difference, but by their assumptions clearly support UCS-2 and not UTF-16. I speak of the MS-Windows world, wherein UCS-2LE holds dominant sway. In fact, Microsoft documents very commonly use the term "Unicode" as a synonym for UCS-2LE. Anyway, I meant that in the current time, we have both UCS-2 encodings and UTF-16 encodings, and I suspect we will all agree that the UCS-2 encodings (by whatever name) predate the UTF-16 encodings. :)

Windows supports UTF-16 surrogates though the support isn't enabled by default. For an application that doesn't support stuff outside the BMP there is no difference between UCS-2 and UTF-16 for one that does UCS-2 would be pointless to support since you can read UCS-2 data as UTF-16. So they can be considered to be essentially the same encoding with different feature restrictions. Plugwash 02:19, 18 September 2005 (UTC)

A Brain Dropping follows:

Wasn't Unicode created to encode all languages - not just 'human' languages? In the future then, why couldn't Unicode conceivably be used to encode extraterrestrial languages as well?(well, why not? hehehehe) Therefore, shouldn't the 'human' be removed from this page? One possible alternative: Unicode is the international standard created, whose goal is to specify a code matching every character needed by every known written language to a single unique integer number, called a code point.

The Universal Character Set, whether in its Unicode Standard or its ISO/IEC 10646 manifestation, was made to encode the writing systems of the world, not the languages of the world. Evertype 15:43, 2004 Jun 20 (UTC)

I'm not an expert on this, but I don't believe the Unicode Consortium seeks to encode writing systems whose existence we don't yet know of (and if we ever meet aliens who use more than 2³² characters, Unicode will have a problem). I believe that Tolkein's Elvish scripts are in there, but other fictional scripts like Klingon are not. So they're not being wholly anthropocentric. adamrice 00:02, 11 Jul 2004 (UTC)

Tengwar and Cirth are not yet encoded, but are roadmapped for encoding. To answer the brain-dropping: Were we to meet aliens who had an encodable writing system, it is likely that their characters would fit. Evertype 11:59, 2004 Jul 11 (UTC)

I find that statement a little overreaching. The aliens could easily have a writing system with a million symbols, or several Chinese-size writing systems or a history of writing that dates back millions of years instead five or six thousand. Or simply be a group of ten different species of aliens with writing histories as complex as ours. The best we can say is that humans, all told, will use about 3 planes, and there's 17 planes of characters. --Prosfilaes 03:49, 12 Jul 2004 (UTC)

More likely, I think, is the possibility that another intelligent species of life will perceive the universe in entirely different ways than humans do, and will ascribe little to no value at all to written language as we know it. -Factorial, 15:52, 28 Nov 2005 UTC

There is currently no Klingon alphabet/writing system that is suitable for encoding. The glyphs shown in Star Trek are merely a nearly-1:1-mapping of the Latin alphabet. The Star Trek folks seem to have a real Klingon alphabet, but have not yet published it. JensMueller 10:06, 5 Sep 2004 (UTC)

Any alphabet is going to have a nearly-1:1 mapping to the phonemes of the language, and hence to the Latin transcription. What must a "real" Klingon alphabet be ill-fitted to the Klingon language? --Prosfilaes 04:11, 9 Sep 2004 (UTC)

There has been no sufficient klingon alphabet published, and nearly all klingon users online are using a "roman transcription". The reason the klingon encoding proposal was turned down was because it was hardly used, (and also because the sounds were mapped to english sounds, rather than klingon). If a canonical klingon alphabet would appear, I guess a unicode encoding is likely. Klingon seems to have 26 sounds, according to Wiki article and www.kli.org, which shouldn't be too difficult to find a mapping area for.

The Klingon piqaD alphabet has a mapping in the Private Use Area of Unicode, and has recently come into occasional use on the Internet. See, for example, Chatting in piqaD and qurgh's blog The requirement for getting Klingon piqaD an assignment of regular Unicode code points is some level of use in data interchange. We can expect that it will qualify at some time in the future. --Cherlin

Perhaps for "Issues?"

One of classicists' issues with Unicode has been the omission of the LATIN Y WITH MACRON characters. While the omission has been corrected in Unicode 3, most user agents don't know to render anything for that codepoint. Somewhere in that story is an issue that perhaps might make sense in the article -- either the omission of the letter, or the outdated support available by user agents (I don't see Microsoft rushing to update its fonts and packaging them as an update to Windows or Internet Explorer just to comply with recent standards).

Definitely not. This is not an "issue" with Unicode, but with implementation. We add support for classicists constantly, and it wasn't Y WITH MACRON alone either. Evertype 21:49, 2004 Jul 10 (UTC)

UTF-8 as the basis for multilingual text?

"UNIX-like operating systems such as GNU/Linux, BSD and Mac OS X have adopted Unicode, more specifically UTF-8, as the basis of representation of multilingual text."

Mac OS X stores a lot of text in UTF-8, but the other UTF's are also supported throughout the system and widely used. I agree that UTf-8 is currently the most widely used Unicode encoding (because it is the most legacy-compatible encoding), and that is important enough to mention in the leading section, but perhaps it should be rephrased so that it doesn't mislead the reader into believing those OSes don't support other kinds of Unicode? — David Remahl 04:16, 9 Sep 2004 (UTC)

afaict UCS-2/UTF-16 (which can essentialy be regarded as different implementation levels of the same thing) is the norm in the windows and java worlds whilst UTF-8 is the norm in the unix and web worlds. Yes other encodings are supported by the conversion libs but UTF-8 and UTF-16 are the ones the systems are built arround.

Microsoft Word for Windows gives me the option to save plain text in 4 flavors of Unicode: Big-Endian (UTF-16BE?), UTF-7, UTF-8, or "Unicode". If I choose "Unicode" what exactly do I get? From the above, UTF-16, is that correct?--84.188.146.200 02:35, 9 February 2006 (UTC)

Hmm which version of word are you using and can you say exactly where you found those options i can't find them in word 2K here. I'd guess its plain unicode option is little endian utf-16 though. Plugwash 22:46, 18 February 2006 (UTC)

Largest and most complete

This phrase has appeared recently without much discussion. "Unicode is the most complete character set, and one of the largest." Could anyone give justification? -- Taku 06:14, Oct 12, 2004 (UTC)

ISO/IEC 10646? Unicode reserves 1,114,112 (2^20 + 2^16) code points, and currently assigns characters to more than 96,000 of those code points. No other encoding even comes close. {Ανάριον} 09:09, 12 Oct 2004 (UTC)

As for the completeness, take a look at mapping of Unicode characters for all the scripts encoded. {Ανάριον} 09:10, 12 Oct 2004 (UTC)

GB18030 is by defintion as large as Unicode, but except for the pre-existing mappings, all GB18030 codepoints and Unicode codepoints, including yet unassigned ones, are algorithmically mapped. So, it is more like a strange encoding form of Unicode.

For certain scripts, there are character sets with more precomposed glyphs, e.g. VISCII for Vietnamese, TSCII for Tamil, or some scholarly encoding for pointed Hebrew. But they don't count as larger, as they don't support more than one or two scripts, and they don't count as more complete, as the encoded characters are uniquely representable in Unicode as sequences including combining characters.

So yes, according to all my knowledge and research, Unicode is the most complete and one of the largest character set, for which information is freely available in languages I can read.

If you have knowledge of implemented characters sets (not counting proposals, shich are cheap to made) which are more complete than Unicode, please elaborate.

Otherwise, I'll revert your reversal.

Pjacobi 09:12, 12 Oct 2004 (UTC)

Unicode is meant to be a superset of all known character sets, so it is hardly possible that there are character sets not covered by it. (And surely all VISCII and TSCII characters are included in Unicode as they are). — Monedula 11:11, 12 Oct 2004 (UTC)

Unicode is only a superset of the character sets in use when they started out. In practice, they were a superset of most new character sets up to 2000, at which they stopped encoding new precomposed characters. (So they're still a superset of them, in a practical sense.) One of the Chinese standards that encoded every minor variation on the ideograph seen wasn't added to Unicode, which decided to adopt a most unifing encoding policy, but all the Chinese and Japanese standards in widespread use are subsets. --Prosfilaes 21:53, 12 Oct 2004 (UTC)

I agree with your first half-sentence, but Unicode has decided to not encode any more precomposed characters. Only for pre-existing national and international standards there was a consensus to include the precomposed characters. Now, new suggestions for precomposed characters are routinely declined, and for good reasons. In fact it is hoped that in some future version (6.0?) all exsiting precomposed characters will become deprecated. A like case exist for glyph variant. What got in, is in, but new additions will be declined. See also: http://www.unicode.org/standard/where/

So there is no chance (neither is there a necessity) that TSCII codepoint 0xE0 "tU" will be assigned a single codepoint in Unicode. Instead it transcodes as [U+0ba4 U+0bc2] --Pjacobi 12:37, 12 Oct 2004 (UTC)

Ignoring the whole Han ideographs, and ignoring the sets that are basically new encodings of Unicode, what is there? TRON? --Prosfilaes 21:53, 12 Oct 2004 (UTC)

The idea of switching to several character encodings isn't unique to TRON, it was already included in ISO-2022. And both have in common, that it makes implementation difficults scatters the design process of new script encodings instead of unifiing it. I don't think much of it is still in use. Heck, you can't even use both (TRON and full ISO-2022 with escape switching) on the Web or in e-Mail. Most Unicode criticism on the TRON advocate's pages are just outdated or a result of misunderstandings. --Pjacobi 22:32, 12 Oct 2004 (UTC)

I am convinced that probably Unicode is the largest and most complete character set but can we still ignore criticism on unicode? What I am often heard about unicode, it is not inadequate in handling old text or text containing outdated characters. Maybe most of criticism are pointless or a result of misunderstandings but I still hear them and I don't think we should make a general statement which not everyone agrees with. Unicode is meant to be the largest and the most complete but if it is really so is disputed, if such dispute is nonsense in actuallity. -- Taku 22:41, Oct 12, 2004 (UTC)

Yes, of course the criticisms must be included, but we must try hard to find the right criticisms and the right way to present them. Don't forget the long expertise of Unicode in this field and the large number of field experts contributing to the evolving Unicode effort. We would achieve nothing but spoil the creditability of the Wikipedia, if we hastily add criticisms of mediocre quality.

A generic problem with Unicode is the long process it takes, to get additions done. This is the downside of centralism. And you need somebody with "weight" to get major additions and changes done. Either a national standards body are field experts of value.

A brainstorming list of criticisms:

• Unicode got the Hebrew points for biblical texts all wrong (or something like that, I'm no expert)
• Unicode has unified scripts (and requires different fonts and markup to differentiate), which should not have been unified.
• Unicode has not unified scripts, which should have been unified, as they are only font differces.
• Unicode has too few presentation forms for complex shaping scripts
• Unicode has not enough presentation forms for complex shaping scripts
• Unicode has too few precomposed glyphs
• Unicode has not enough precomposed glyphs

As you can seen, some criticisms arise out of the fact, that decisions must be made in a standard on questions which are viewed differently by different people.

Pjacobi 00:32, 13 Oct 2004 (UTC)

If you want to say that Unicode is not the largest and most complete, then there must be something that's larger or more complete. If you tell us what it is, we can discuss it.

Most of the complaints about Unicode don't stem from size or completeness. Most of the scripts and characters that are left are very obscure and almost invariably not used for writing new material. The complaints come from how Unicode treats the existing scripts; often the question is whether two entities should be treated as distinct. Since in all these cases, they are distinct in some ways, and not in others, there's no "right" answer that will satisfy everyone. The Chinese and Japanese encodings that are supposedly more "complete" are in reality more fine-grained, in that they seperate characters that Unicode unifies. --Prosfilaes 20:01, 13 Oct 2004 (UTC)

Reorganization

I made some reorganization of the sections and the continuing work on the leading section. I think the new 4 big sections make good sense: origin and development, mapping and encoding, process and issues and in use. In addition to this, we probably need:

• difference in character and glyph; we should give some example
• difference in mapping and encoding; particularly, what is code point, what is plane?
• short summary of utf; what is utf? and why we want it
• size comparison, particularly what unicode not to include; perhaps Pjacobi is right that some criticism are wrong but it is still true that many people advertise their sets as being larger and more complete. We need some response to them.

If I have some time, I will try to address them but you can also help me. Finally, I'm sorry for late reply to unicode as largest and the most complete question. I slighly reworded the mention. Please make further edit if you think necessary. -- Taku 20:42, Oct 17, 2004 (UTC)

You must give some concrete examples who advertises which character set to be larger in what specific sense. --Pjacobi 22:18, 17 Oct 2004 (UTC)

ok, the press release of chokanji 3 [2] (in Japanese) says it supports 170,000 kanji while Unicode handles 20,000 chinese characters, 12,000 of which are kanji. -- Taku 02:15, Oct 18, 2004 (UTC)

a) If I'm not mistaken, the press release is dated 2001-01-06. So, nearly four years later, are there any implementations? Can you give the URL of a single webpage in this charset? Is the IANA registration in progress? Does somebody work on GNU iconv support? Does somebody worl on IBM ICU support? You can't compare vaporware to a widely implemented standard.

b) As of version 4.0, Unicode supports 71,000 Han characters, it is a horribly outdated or mis-informed to state the number 20,000. And PRC is busily adding more. It is a political decision of JIS, not to propose adding more kanji. Either because JIS doesn't see the necessity or for other reasons.

Pjacobi 06:12, 18 Oct 2004 (UTC)

We don't compare. You wanted "some concrete examples who advertises which character set to be larger in what specific sense.". So this is the answer. Again and again, I didn't mean what they are saying so fair. I won't use their product because there is just so little compatibility and besides, I don't have any pratical problem with unicode. I mean I agree with you so I am not sure whom you try to convince. -- Taku 13:14, Oct 18, 2004 (UTC)

Thank you for giving the concrete example. Yes, I specifically asked for it. I apologize for replying in flame-war style. --Pjacobi 14:41, 18 Oct 2004 (UTC)

Many documents in non-western languages, for instance, are still represented in other character sets. Which languages? Which character sets In this generality it doesn't help. Please state languages and character sets used. And remember, GB18030 is now fully harmonized with Unicode and cannot be considered a different character set, but Unicode encoding form standardized by somebody other than ISO or Unicode Org, namely the Guobiao. --Pjacobi 22:23, 17 Oct 2004 (UTC)

Maybe Shift-JIS? I don't think it is only character set used beside Unicode. If you know more, that would help. -- Taku 02:15, Oct 18, 2004 (UTC)

The largest use of a non-Unicode charset is still EBCDIC, ASCII and ISO-8859-1, as seen on this Wiki. So this doesn't look like a west vs east problem to me. The difference is, that almost universally all other charsets are considered to be subsets of Unicode nowadays. And especially the HTML and XML character model explicetely states, that while the physical charset may vary, the logical charset is always Unicode. Also in programming, it is nearly always assumed, that everything can be converted (and most things reversibly) to Unicode.

So if I can judge this correctly, the Unicode character encoding models is only challenged by some users of Japanese and not much is known outside of Japan of this. As said above, I'm very skeptical about the practical relevance of the Unicode challengers. But the interesting point, why this happens in Japan, seems to be good stuff to write a separate article about Japanese character encoding.

Pjacobi 06:23, 18 Oct 2004 (UTC)

I had absolutely no intention to make a case like west vs east problem. If you think some sentences are problematic, then go ahead to edit. I just wanted to illustrate the adoptation of unicode and the sentence absolutely never mean to imply the use of unicode is problematic or anything. Besides, I am not sure what you are saying. I don't think you believe any non-unicode character sets have died out completely. We want to show when when unicode is used and when it is not. I mean what you want after all? -- Taku 13:14, Oct 18, 2004 (UTC)

Sorry for being unclear. And apologies for not contributing to the article itself in the moment. I am of the opinion some non-trivial additions are dearly needed (on the character model, on character vs glyphs vs graphemes), but I feel unable to do it myself. Perhaps I'll try it next week.

No, I surely don't want to say non-unicode character sets have died out completely. What I tried to say, is that the character encoding model of Unicode is nearly universal success and nowadays other character sets are mostly seen as subsets of Unicode. This wasn't the case ten years ago.

Pjacobi 14:41, 18 Oct 2004 (UTC)

It's fine. I was just puzzled about what upset you so much. As a matter of fact, I am neither the backer of unicode nor the detractor. I am only interested in making the article informative for those who have questions about unicode. It's very surprising that many people don't know well about unicode, even computer programmers. The article could be a help for them. -- Taku 15:54, Oct 23, 2004 (UTC)

Fully agree. When supporting charset issues (as I do sometimes for Firebird SQL) it's quite amazing that some programmers at first don't even see a problem in the different mappings between characters and bytes. --Pjacobi 17:55, 23 Oct 2004 (UTC)

Phishing

In the section that talks about pre-composed characters vs. composing with several codepoints, how about mentioning that this capability opens up lots of opportunities for phishing once URLs are more universally excepted in UTF-8? For example, once accented characters are common in website addresses, links with a pre-composed "è" and separate "e" plus an accent will point to different sites, but look identical to the user (in fact the intent is for them to look the same). I don't know if this info belongs here, but it's an interesting tidbit. Rlobkovsky 00:06, 6 Dec 2004 (UTC) Insert non-formatted text here

If and when URIs start supporting characters beyond ASCII in a standard way, some decomposing must take place, as according to the principles behind unicode the precomposed character à is exactly equivalent to ` + a. Any future internet domain funkèynáme.ext will have to point to the same IP(v6?) address for all its possible decomposings.

-- Jordi·✆ 12:26, 28 Dec 2004 (UTC)

According to the article on IDN, names are supposed to go through Nameprep which normalises to NFKC. I imagine it's left to browsers to always normalise names (so a decomposed e-plus-accent will be recomposed), and for registrars to reject names which aren't normalised. There are more serious attack (characters which look the same), also mentioned in the article, and for that reason, they fixed Safari so that a name like xn--pypal-4ve.com looks exactly like that, and not like "pаypal.com" (the first "a" is actually a cyrillic small a). Elektron 15:00, 24 August 2007 (UTC)