Jim Woolsey, a hippie and early-ish computer hacker from New Hope, Pennsylvania, was an important and early force in the digitization of the Tibetan language. This interview[0] with him from 1993 is a fascinating time capsule, and interesting in its own right. He was a family friend and I always admired his singular commitment to this important and underappreciated work.
"... relatively short paragraphs (possibly up to a few pages)"
I love things like this that just shows me how much I view the world from a certain perspective. I don't think I've ever had a paragraph even on one page! The closest I know is that some writer, that I forgot the name of, had several pages of stream of consciousness that I think was without paragraphs and punctuations.
In some countries, legal documents are required to not have any paragraph breaks, so you can have a document with one paragraph spanning 100s of pages. OpenOffice has a hard limit of 65534 per paragraph, and it took LibreOffice quite some work to left it: https://bugs.documentfoundation.org/show_bug.cgi?id=30668
I never thought about this element of cross language structure before. Text direction, diacritics, punctuation, sure - but I always assumed that chunking was universal. Turns out no:
“the typographical notion of the paragraph does not really exist in a Tibetan text the way it does in European languages. As a result, Tibetan texts often need to be processed as a long stream of uninterrupted text with no forced line breaks, sometimes over hundreds or thousands of pages. “
The same applies to old Chinese, and in fact most ancient languages. Latin and Greek were originally written in scriptio continua, meaning no punctuation or spacing:
That kind of organization takes time, editing, multiple revisions to get right, etc.
And a mindset that it is useful. In many cases (like if you’re a religious caste), having a giant wall of text that requires skill to identify elements from, is a plus.
Do you want your ciphertexts formatted into paragraphs too?
Are you really saying that Tibetan doesn't use spaces and paragraphs because the creators or perhaps all Tibetans don't want to spend time editing or can't organize their writing well enough?
Not saying can’t. Arguably, there isn’t anything anyone can’t do. At least until they go broke.
I’m saying that different societies have different priorities/expectations/motivations, and there is clearly a reason they don’t do it, or it wouldn’t be so consistent eh?
It’s not like white space isn’t the default on a writing surface.
Do you have any alternative theories?
I could also imagine scrolls being expensive, so ‘fluff’ like white space is discouraged, and not easily re-used or overwritten based on the inks, so re-editing or the like doesn’t actually work.
But I’m just speculating here.
Edit: the scripta continua link above had a good reference to Greek/roman examples where they were transcriptions by slaves of spoken monologues. They didn’t have paragraphs or the like because people don’t speak in paragraphs.
They also don’t edit their words when they speak, and rarely do ‘chapters’.
Footnotes, bibliographic references, etc. also don’t really make sense in the way we might think if it’s ’a record of spoken words’ vs ‘words representing ideas on their own’.
So writing used more like transcriptions of famous speeches or lectures, less as standalone and independent works.
And I assume every culture has the equivalent of 2 hour long speeches that could have been a one page email.
“Before and after the advent of the codex, Latin and Greek script was written on scrolls by slave scribes. The role of the scribes was to simply record everything they heard to create documentation. Because speech is continuous, there was no need to add spaces.”
Languages, are very interesting beasts. As in, they are easy to learn and marshal communication across large swaths of the world- or they are hard to master and allow to construct very complex constructs and ideas- which are then transported from one speaker to another. In which part of the field does the Tibetan language fall?
This has been in the works for a while. There is an old HyperCard stack to teach Tibetan pronunciation (with 16bit sound) you can try: https://hcsimulator.com/Learn-Tibetan
The site seems incomplete. Tibetan does have 5 vowels, and it looks like the non intrinsic vowels are written at the bottom section of the view, but I can't get them to work. I assume the intention would be that you click one of the other vowels to toggle it, but it no worky.
Pretty short change for reducing O(n^2) impact with a cache.
This change includes the following scalability improvements for
documents containing extremely large paragraphs:
- Reduces the size of layout contexts to account for LF control chars.
- Due to typical access patterns while laying out paragraphs, VCL was
making O(n^2) calls to vcl::ScriptRun::next(). VCL now uses an
existing global LRU cache for script runs, avoiding much of this
overhead.
Probably shows a bit how little that software is used with Tibetan text if this bug was able to stay open for almost 10 years for what ultimately was a 5 line fix.
The fix looks like a 5 line fix because it is a last step in a very long process of optimizing LibreOffice text layout that started years ago. This 5 line fix could not have been possible 10 years ago simply because the code it is fixing didn't exist back then.
i don't know how it is phrased in the book itself but in Tibetan Buddhism there is no god. And their innovation is far beyond this book (at least the plot summary on wikipedia).
Hey everyone, I'm Eyal, a LibreOffice project volunteer who does a lot of QA regarding Right-to-Left and Complex-Text-Layout scripts (= written languages). I want to thank thunderbong3 for posting a link to that post - and heartily thank Jonathan Clark, the new RTL-CTL-CJK-focused developer at The Document Foundation, who implemented the performance improvement for Tibetan.
Most bugs we encounter and report in LibreOffice are more general, and aren't script specific (e.g. code which forgets that the content may be right-to-left resulting in wrong behavior in those cases); and a lot of the script-specific bugs are about the most popular script, which is Arabic (that is also used for Farsi, Urdu, Javanese etc.)
But we do have some issues regarding less-commonly-used scripts,
like Tibetan or Mongolian. Here:
is the meta-bug which tracks issues with: Mongolian, Tibetan, Uyghur, Zhuang,Kazak, Xibo, Dai, Yi, Miao, Jingpo, Lisu, Lahu, Wa, etc.
We don't know if there are really very few issues specific to those languages (which is quite possible), or whether it's just that they're not used so much and the users aren't motivated enough to file bugs.
Still, as Jonathan's recent fix demonstrates, there is certainly the interest to address them when developer-time-resources become available.
I would like to encourage everyone who cares about these scripts, and "document editing fairness" across countries and cultures, to consider:
1. Try using LibreOffice with such languages which you know at least a little bit of - and if you find any bugs, file them at our BugZilla: https://bugs.documentfoundation.org/
2. Consider supproting The Document Foundation, which manages the LibreOffice project, financially:
We are one of the larger FOSS projects in the world, with tens of Millions of regular users (if not > 100 Million) and a board of trustees with members from dozens of countries; but - we don't have large corporations investing money nor time in the project. While a few commercial companies do contribute to LibreOffice (like Collabora and Allotropia) - many fundamental issues are not close enough to their customers' needs - which is why it was decided to hire Jonathan directly to give RTL-CTL-CJK support a boost. Individual user donations are what enables this work.
Well, the "minimal" support is there, as buovjaga noted - but... we need you to tell us what aspects of that support is missing - by filing bugs, or at the very least talking to us about this (for example - there are "LibreOffice RTL" and a "LibreOffice CJK" groups on Telegram).
Why are we anthropomorphizing languages now? Are there really no appropriately-charged words to describe what we wanna say without resorting to political speech?
This is a well-established phrase in computer science and programming languages, and it’s likely that its use here is meant to be evocative of those principles rather than of an anthropomorphic sense.
Can you give an example or two of the type of terminology you have in mind? Something like first tier, or high level, or primary, or what? Almost any term that I could think of still carries with it a sense of politics. Perhaps someone from the first class citizen language, like English or French doesn’t really notice this dynamic, but someone who uses these other languages sure notice the lack of importance, or urgency, when they can’t live digitally with their own language. Whether or not they use the term,” political”, it’s still there. But I’m curious to see what your suggestions might include.
Jim Woolsey, a hippie and early-ish computer hacker from New Hope, Pennsylvania, was an important and early force in the digitization of the Tibetan language. This interview[0] with him from 1993 is a fascinating time capsule, and interesting in its own right. He was a family friend and I always admired his singular commitment to this important and underappreciated work.
[0] https://www.mcall.com/1993/10/08/new-hope-man-computer-guru-...
"... relatively short paragraphs (possibly up to a few pages)"
I love things like this that just shows me how much I view the world from a certain perspective. I don't think I've ever had a paragraph even on one page! The closest I know is that some writer, that I forgot the name of, had several pages of stream of consciousness that I think was without paragraphs and punctuations.
Apparently "documents have reasonably short paragraphs" should be added to "falsehoods programmers believe about text."
In some countries, legal documents are required to not have any paragraph breaks, so you can have a document with one paragraph spanning 100s of pages. OpenOffice has a hard limit of 65534 per paragraph, and it took LibreOffice quite some work to left it: https://bugs.documentfoundation.org/show_bug.cgi?id=30668
I never thought about this element of cross language structure before. Text direction, diacritics, punctuation, sure - but I always assumed that chunking was universal. Turns out no:
“the typographical notion of the paragraph does not really exist in a Tibetan text the way it does in European languages. As a result, Tibetan texts often need to be processed as a long stream of uninterrupted text with no forced line breaks, sometimes over hundreds or thousands of pages. “
The same applies to old Chinese, and in fact most ancient languages. Latin and Greek were originally written in scriptio continua, meaning no punctuation or spacing:
https://en.wikipedia.org/wiki/Scriptio_continua
Tens of pages, sure.
But hundreds? Thousands?
Do they not have the concepts of headers? Sections? Chapters?
Both in non-fiction and fiction, there are a lot more means of content separation than just paragraphs.
Stream processing before it was cool.
‘I don’t have time to make a short letter, so I made a long one instead’ -Mark Twain [https://www.goodreads.com/quotes/21422-i-didn-t-have-time-to...]
That kind of organization takes time, editing, multiple revisions to get right, etc.
And a mindset that it is useful. In many cases (like if you’re a religious caste), having a giant wall of text that requires skill to identify elements from, is a plus.
Do you want your ciphertexts formatted into paragraphs too?
Are you really saying that Tibetan doesn't use spaces and paragraphs because the creators or perhaps all Tibetans don't want to spend time editing or can't organize their writing well enough?
Not saying can’t. Arguably, there isn’t anything anyone can’t do. At least until they go broke.
I’m saying that different societies have different priorities/expectations/motivations, and there is clearly a reason they don’t do it, or it wouldn’t be so consistent eh?
It’s not like white space isn’t the default on a writing surface.
Do you have any alternative theories?
I could also imagine scrolls being expensive, so ‘fluff’ like white space is discouraged, and not easily re-used or overwritten based on the inks, so re-editing or the like doesn’t actually work.
But I’m just speculating here.
Edit: the scripta continua link above had a good reference to Greek/roman examples where they were transcriptions by slaves of spoken monologues. They didn’t have paragraphs or the like because people don’t speak in paragraphs.
They also don’t edit their words when they speak, and rarely do ‘chapters’.
Footnotes, bibliographic references, etc. also don’t really make sense in the way we might think if it’s ’a record of spoken words’ vs ‘words representing ideas on their own’.
So writing used more like transcriptions of famous speeches or lectures, less as standalone and independent works.
And I assume every culture has the equivalent of 2 hour long speeches that could have been a one page email.
“Before and after the advent of the codex, Latin and Greek script was written on scrolls by slave scribes. The role of the scribes was to simply record everything they heard to create documentation. Because speech is continuous, there was no need to add spaces.”
"Continued on next scroll"
Its all about the scrolls, man
Somewhere, a programmer created a 4096 character buffer and sought the next '\n' only to be defeated by Tibetan.
Languages, are very interesting beasts. As in, they are easy to learn and marshal communication across large swaths of the world- or they are hard to master and allow to construct very complex constructs and ideas- which are then transported from one speaker to another. In which part of the field does the Tibetan language fall?
This has been in the works for a while. There is an old HyperCard stack to teach Tibetan pronunciation (with 16bit sound) you can try: https://hcsimulator.com/Learn-Tibetan
the only vowel is AH ?
In text, the singular vowels are built on the ah syllable with modifying marks.
The site seems incomplete. Tibetan does have 5 vowels, and it looks like the non intrinsic vowels are written at the bottom section of the view, but I can't get them to work. I assume the intention would be that you click one of the other vowels to toggle it, but it no worky.
I don't know who created it, or if it was part of a larger proto-Duolingo language product.
I want to know the details how they achieved it (the support for super-long paragraphs, or rather, the absence thereof).
Does anyone know?
https://gerrit.libreoffice.org/c/core/+/172801
Pretty short change for reducing O(n^2) impact with a cache.
This change includes the following scalability improvements for documents containing extremely large paragraphs:
- Reduces the size of layout contexts to account for LF control chars.
- Due to typical access patterns while laying out paragraphs, VCL was making O(n^2) calls to vcl::ScriptRun::next(). VCL now uses an existing global LRU cache for script runs, avoiding much of this overhead.
First commit from July: https://git.libreoffice.org/core/commit/4c8f88bef948b18f3d81...
Thank you. Also https://bugs.documentfoundation.org/show_bug.cgi?id=92064
I lack the context - are they still layong out the widths of characters when wrapping?
Probably shows a bit how little that software is used with Tibetan text if this bug was able to stay open for almost 10 years for what ultimately was a 5 line fix.
The fix looks like a 5 line fix because it is a last step in a very long process of optimizing LibreOffice text layout that started years ago. This 5 line fix could not have been possible 10 years ago simply because the code it is fixing didn't exist back then.
> Probably shows a bit how little that software is used with Tibetan text
... by the LibreOffice devs in Indo-European-speaking countries.
Apparently by anyone if the bug description is accurate. Seemingly one cannot open sufficiently long documents let alone write into them.
Perhaps: from the article:
So long as LibreOffice could not handle long paragraphs there was essentially no free tool to publish Tibetan.
With all due respect, the innovation side of Tibetans is also appreciated in "The Nine Billion Names of God" [1].
[1] <https://en.wikipedia.org/wiki/The_Nine_Billion_Names_of_God>
Unsong takes inspiration from this as well -
https://unsongbook.com/
i don't know how it is phrased in the book itself but in Tibetan Buddhism there is no god. And their innovation is far beyond this book (at least the plot summary on wikipedia).
If I were a Tibetan Buddhist, I might say we were just having some fun with Arthur C. Clarke's imagination.
Classic!
Hey everyone, I'm Eyal, a LibreOffice project volunteer who does a lot of QA regarding Right-to-Left and Complex-Text-Layout scripts (= written languages). I want to thank thunderbong3 for posting a link to that post - and heartily thank Jonathan Clark, the new RTL-CTL-CJK-focused developer at The Document Foundation, who implemented the performance improvement for Tibetan.
Most bugs we encounter and report in LibreOffice are more general, and aren't script specific (e.g. code which forgets that the content may be right-to-left resulting in wrong behavior in those cases); and a lot of the script-specific bugs are about the most popular script, which is Arabic (that is also used for Farsi, Urdu, Javanese etc.)
But we do have some issues regarding less-commonly-used scripts, like Tibetan or Mongolian. Here:
https://bugs.documentfoundation.org/show_bug.cgi?id=115607
is the meta-bug which tracks issues with: Mongolian, Tibetan, Uyghur, Zhuang,Kazak, Xibo, Dai, Yi, Miao, Jingpo, Lisu, Lahu, Wa, etc.
We don't know if there are really very few issues specific to those languages (which is quite possible), or whether it's just that they're not used so much and the users aren't motivated enough to file bugs.
Still, as Jonathan's recent fix demonstrates, there is certainly the interest to address them when developer-time-resources become available.
I would like to encourage everyone who cares about these scripts, and "document editing fairness" across countries and cultures, to consider:
1. Try using LibreOffice with such languages which you know at least a little bit of - and if you find any bugs, file them at our BugZilla: https://bugs.documentfoundation.org/
2. Consider supproting The Document Foundation, which manages the LibreOffice project, financially:
https://www.libreoffice.org/donate/
We are one of the larger FOSS projects in the world, with tens of Millions of regular users (if not > 100 Million) and a board of trustees with members from dozens of countries; but - we don't have large corporations investing money nor time in the project. While a few commercial companies do contribute to LibreOffice (like Collabora and Allotropia) - many fundamental issues are not close enough to their customers' needs - which is why it was decided to hire Jonathan directly to give RTL-CTL-CJK support a boost. Individual user donations are what enables this work.
Hi Eyal - Your hard work as a volunteer does so much for so many - look at that blog post, for example. I really admire it.
The bug report about long paragraphs and the blog post are by Élie Roux, the CTO of BDRC :)
I did not author that blog post... I just noticed it as a HackerNews reads :-)
I did give a talk on the state of Right-to-Left language support at the annual LibreOffice conference, a few days ago:
https://events.documentfoundation.org/libreoffice-conference...
I would appreciate you if you supported Dzongkha as well!
Support is included: https://git.libreoffice.org/core/+/8a11d2e0b1ff09b3637a369eb...
Well, the "minimal" support is there, as buovjaga noted - but... we need you to tell us what aspects of that support is missing - by filing bugs, or at the very least talking to us about this (for example - there are "LibreOffice RTL" and a "LibreOffice CJK" groups on Telegram).
There are over 50 tibetic languages, which one do you choose?
There is only one (modern) written Tibetan script.
How about Dzongkha?
It also uses Tibetan script.
> first-class citizen
Why are we anthropomorphizing languages now? Are there really no appropriately-charged words to describe what we wanna say without resorting to political speech?
This is a well-established phrase in computer science and programming languages, and it’s likely that its use here is meant to be evocative of those principles rather than of an anthropomorphic sense.
https://en.wikipedia.org/wiki/First-class_citizen
Well let's not use it anymore if I agree on principle, computer science itself is a new field, so not sure how deeply entrenched it really is.
Can you give an example or two of the type of terminology you have in mind? Something like first tier, or high level, or primary, or what? Almost any term that I could think of still carries with it a sense of politics. Perhaps someone from the first class citizen language, like English or French doesn’t really notice this dynamic, but someone who uses these other languages sure notice the lack of importance, or urgency, when they can’t live digitally with their own language. Whether or not they use the term,” political”, it’s still there. But I’m curious to see what your suggestions might include.