• Print

BeyondPrint Offers Helpful Review of StartWithXML

George Alexander, who attended the StartWithXML forum in New York on Tuesday and made quick work of reading the research paper (thank you!), offers a helpful review of both.

In his review, George also offers a view he shared with the StartWithXML team the day after the forum: the current tools are not yet ready for widespread use, and the forum and the research paper were largely silent on his concerns.

I think that George makes an important point about the tools for authoring and editing. I responded yesterday to say that what may have felt like a “middling” position at the forum reflects a range of opinion within the project team.

At the forum, O’Reilly’s Andrew Savikas, for example, advocated use of XML authoring tools in
his afternoon remarks, showing some examples of what worked. In contrast, Laura Dawson, who co-wrote the research paper, is more critical of the tools, something she made clear in her comments. I’m somewhat in the middle, feeling that the tools are not necessarily ready for widespread deployment, but that balanced changes in processes, technology/tools and organizational structures can provide a path to moving the tagging work upstream.

One thing less evident at the forum or in the paper is the healthy discussion that took place within the team about this issue. At one point in the e-mail exchanges, I wrote (paraphrasing) that “waiting until the tools are “ready” isn’t the right answer; people developing the tools will improve them when publishers in adequate numbers use the tools and advocate for better and more features.

When I presented the “solutions” grid in the afternoon, I pointed out that the bulk of the most developed software and systems are in the production editorial and operational areas, but that upstream options were becoming more available. I stopped short of saying “not ready,” in part because I don’t want publishers to hear me and walk out saying “we’ll wait until the tools come on line” and let production worry about tagging until then. Changing workflows is painful, and people are prone to avoiding pain. That’s smart in the short term and potentially disastrous in the mid-term, so I stuck with the recommendation to push upstream as much and as fast as you can.

We view the research paper as a living document, and we expect to revise it based on feedback from the forum as well as an evolving understanding of the number of case studies that the paper and forum started to capture. Look for a subsequent draft to articulate a position on XML tools that may not match what George sees but more clearly captures the project team’s thinking.

tags: , , , , ,
  • bowerbird

    brian said:
    > waiting until the tools are “ready” isn’t the right answer;
    > people developing the tools will improve them
    > when publishers in adequate numbers use the tools
    > and advocate for better and more features.

    seems like x.m.l. tools should have been built _years_ ago…

    what’s the delay? who are the “people developing the tools”?

    -bowerbird

  • http://www.magellanmediapartners.com Brian O'Leary

    Nice to see you again, bowerbird. Sorry I was away for a while – writing and editing the research paper took a lot of my bandwidth.

    To qualify, I’m not really the person making the tools argument. I feel that the tools used to create, compose and design content formats are adequate, if we make necessary changes in process and structure/roles. Others have made the case that writers, editors or designers would not want to work with XML tools as currently configured, because doing so requires changes in how people work.

    That said, I do wish that the commonly used content creation and design tools (Word, for example, or InDesign and Quark) had embraced XML earlier. Hindsight is easy, especially when I don’t have to rewrite a software package. Few companies are as bold as Apple was in rewriting its operating system, and as a result we get legacy systems that look creakier over time.

    Part of the delay is that development/retooling lag. A second part reflects what I posted about initially: not so many publishers are working in XML right now.

    A third part is reality: How big is the publishing market? How fast would you expect a firm like Quark to move to serve a need that is emerging in a narrow part of its market? Industrial content uses (pharmaceuticals, for example), with established taxonomies and a regulatory incentive to follow a structured approach, are faster to adopt and easier to serve. Publishers represent an idiosyncratic workflow market.

  • bowerbird

    brian said:
    > I feel that the tools used to create, compose and
    > design content formats are adequate, if we make
    > necessary changes in process and structure/roles.

    the common tools (by which i mean ms-word and indesign
    and quark) _are_ adequate — if you know how to do x.m.l.

    if you don’t — and most writers don’t — they ain’t even close.

    but if you don’t expect writers to do much of the markup,
    that’s fine. it just means you pay somebody else to do it…

    > Others have made the case that writers, editors or designers
    > would not want to work with XML tools as currently configured,
    > because doing so requires changes in how people work.

    i think it’s more that xml-specialized tools are crappy, and
    most people simply do not want to work with crappy tools…

    and people aren’t going to change much, not in that regard.

    > I do wish that the commonly used content creation and
    > design tools (Word, for example, or InDesign and Quark)
    > had embraced XML earlier. Hindsight is easy, especially
    > when I don’t have to rewrite a software package.

    it’s _difficult_ to write software that can manipulate x.m.l.
    it’s not a format that a programmer would have developed.

    > A third part is reality: How big is the publishing market?
    > How fast would you expect a firm like Quark to move to
    > serve a need that is emerging in a narrow part of its market?
    > Industrial content uses (pharmaceuticals, for example),
    > with established taxonomies and a regulatory incentive
    > to follow a structured approach, are faster to adopt and
    > easier to serve. Publishers represent an idiosyncratic workflow

    still, x.m.l. has been “the trend” for a very long time already.
    and before it, there were literally _decades_ of s.g.m.l. hype.

    so in spite of the difficulty of programming around x.m.l.,
    i’d expect that _someone_ would’ve taken up the challenge.
    but pretty much zilch. i can’t explain it any better than you.

    but i _do_ think it would be dangerous to proceed “as if”
    new tools were right around the corner. as time has shown,
    for quite some time now, they’re not…

    -bowerbird

  • http://www.magellanmediapartners.com Brian O'Leary

    Your points are well-reasoned; I can’t add anything.

    At the forum, I did say “don’t wait” to create more flexible, reusable content files, even if the work done to create those files occurs downstream. On the tools front, much as you are saying “don’t just hope they arrive soon”, I was trying to say “and don’t avoid doing what will create more agile content just because you don’t like the tools.” Maybe we can get George Alexander to weigh in on this.

    Not even a “welcome back” ?

  • bowerbird

    brian said:
    > Not even a “welcome back” ?

    wherever are my manners? of course, “welcome back!” :+)

    (honestly, i didn’t know you went away. i figured that
    while you were writing and editing the research paper,
    your thoughts never strayed very far from this place!)

    > Your points are well-reasoned; I can’t add anything.

    you know, i write my comments as a rorschach blot –
    how people interpret it provides interesting information
    about them. and your interpretation speaks well of you.

    > At the forum, I did say “don’t wait” to
    > create more flexible, reusable content files

    sounds good to me…

    > even if the work done
    > to create those files
    > occurs downstream.

    …still sounds good…

    > On the tools front, much as you are saying
    > “don’t just hope they arrive soon”, I was trying to say
    > “and don’t avoid doing what will create more agile content
    > just because you don’t like the tools.”

    …and agile content still sounds good to me…

    > Maybe we can get George Alexander to weigh in on this.

    …the more the merrier…

    -bowerbird

    p.s. it’s always good to have a well-reasoned guy back…

  • http://www.beyond-print.net George Alexander

    OK, I’ll take the bait. My general thinking is outlined below. First, let me say that I don’t know much about the inner workings of XML, so some of my ideas may come across as naïve, and I may be wrong on some points. But I do have a sense of what an author or an editor needs, and I do have an idea of how I would like to see those needs addressed in an XML environment.

    The user interface for an author or copy editor should be very simple: perhaps a window showing a diagram of the document’s structure, an editing window, and a handful of tools across the top. It should be much closer to Windows Notepad than to Microsoft Word. Training time for new users should be on the order of 15 minutes.

    The cost should be very low — no more than $50 — so that every author and every freelance copy editor could own a copy.

    The software should give the authors and editors only what they need to do their job.

    In the simplest case, if the author is working on a novel, the software should provide access to one or two head levels (chapter and subchapter), and a few front-matter elements (author’s name, title, dedication, preface). It should provide bold and italic emphasis.

    Elements are the only “coding” that the author should see. I would argue that there is no need for attributes at all in the majority of book manuscripts, so that aspect of the UI should be absent entirely. Where needed, elements can be used instead. Entities, too, should be hidden from the user. Special characters that are not on the keyboard should be picked from a list or selected using keyboard shortcuts, as in MS Word and other word processors, and displayed as WYSIWYG.

    If the book is a memoir, then support for inserting photos (and their captions) and references should be added to what the author can do.

    If it is a business book, several more things should be added: bulleted and numbered lists, tables, URLs.

    The mechanism for providing the right set of tools for a given book could be a small library of schemas (one for novels, one for business books, etc.) The publisher would tell the author which schema to select at the start of the project.

    Editors would have similar tools to those used by the authors, but with more elements available (for example, copyright-page information and the ability to embed index references).

    Formatting and printing would be done based on stylesheets provided by the publisher.

    I think that this simple type of editing software would be sufficient to write and edit 80% of books, and it would result in an XML file that would provide the basis for an all-XML workflow. I think authors and editors would like it: they would see the value of the structured approach and would not be intimidated by lots of unfamiliar tools and jargon. It would be a practical way to truly “start with XML.”

    I’m sure I have over-simplified here, and I look forward to hearing from others who can help me understand the respects in which this bare-bones approach is insufficient.

    – George

  • http://www.beyond-print.net George Alexander

    To come back to the topic of “waiting for the right tools”:

    It seems to me that any vendor who already produces a full-featured XML editor could produce an editor like the above in very short order. It is mostly a matter of removing features and simplifying the UI. If the publishers clamor for this, I imagine it will become available very soon.

    It is clear that the vendor community, on its own, is unlikely to do this (since they have had the opportunity for the last decade or more and haven’t done it). The StartWithXML initiative is the perfect vehicle to make the vendors aware of this need. Since authors and editors are where the publishing process starts, it seems to me that satisfactory tools for them should be the highest-priority item on the StartWithXML agenda.

    – George

  • http://www.magellanmediapartners.com Brian O'Leary

    George, thank you for making the time to weigh in. Your article and your comments here are really helpful.

    I’ll admit to being somewhat catholic in advocating for a balanced consideration of tools in concert with process and possibly structural changes, but I don’t feel that what you are proposing here argues against that. I particularly like the idea of a low price point. O’Reilly’s Andrew Savikas is a big fan of open-source here, in part because it allows widespread access and adoption, as you recognize.

    The only thing that I might shift in your comments is the sense that tagging beyond structure isn’t useful for most books. Maybe we could agree that it is not needed “yet”, and if so, then structure at least gets us started. Our research suggested that there may be a lot of new uses for structured content with a range of contextual attributes included. The old saw, of course, is “Don’t let the perfect be the enemy of the good”, so let’s see the good in your idea as an effective way to get going.

  • bowerbird

    george said:
    > I’m sure I have over-simplified here, and
    > I look forward to hearing from others who can
    > help me understand the respects in which
    > this bare-bones approach is insufficient.

    you’ve focused on the user-interface, george.
    that’s not the hard part of the equation here…

    one hard part is ingestion of “arbitrary” x.m.l.

    even within “uniform” aspects of a “standard”
    like .epub, there is a huge amount of “wiggle”
    — that’s the scientific name for it — and thus
    the complexity grows immense quite quickly.

    another hard part is the hiding of the markup
    while the text itself is being written or edited,
    but still presenting the text in a wysiwyg way.

    in general, this isn’t hard — realistically, all
    modern word-processors since wordperfect
    have worked in essentially this way — _but_
    the particular design of most x.m.l.-formats
    presents some especially difficult challenges.

    but again, given all the hype of the last decade
    how x.m.l. was taking over the world, i’m a bit
    perplexed why nobody made an authoring-tool
    simple enough and cheap enough for a writer…

    at least you guys are honest enough to admit it,
    rather than just repeating the normal claptrap on
    how there are “lots” of authoring-tools out there.

    -bowerbird

  • http://www.beyond-print.net George Alexander

    I’m relieved that both of you agree with me, at least partially. Perhaps I’m not so far off the mark as I feared I might be.

    Brian said:
    >Maybe we could agree that it is not needed “yet”
    I think we are in agreement, if “yet” means at this stage in the publishing workflow. I would argue that tagging that goes beyond elements (and here I include in-line elements, not just structural tags) is not necessary for authors and copy editors to do their work. If that is true, they should not deal with it. Let it be added by someone, or some automatic conversion step, later in the process.

    bowerbird said:
    > one hard part is ingestion of “arbitrary” x.m.l.
    I’m not suggesting that the author’s tool should ingest any XML at all; and it should impose enough structure on the file so that its output should be easily “ingestible” by other XML tools. Am I missing something here?

    >another hard part is the hiding of the markup
    >while the text itself is being written or edited,
    >but still presenting the text in a wysiwyg way.
    >in general, this isn’t hard — realistically, all
    >modern word-processors since wordperfect
    >have worked in essentially this way — _but_
    >the particular design of most x.m.l.-formats
    >presents some especially difficult challenges.
    You are obviously familiar with the details of dealing with XML from a programmer’s point of view, and I am not. But I’d like to learn a bit more. Can you provide an example of an XML structure that is troublesome in the sense you mean?

  • bowerbird

    george said:
    > I’m not suggesting that the author’s tool
    > should ingest any XML at all

    including work on the manuscript the author saved yesterday?

    it’s convenient to say “everybody has to use the same tool”, but
    that means that tool needs to be good enough for everybody…

    on the other hand, if multiple tools are used in the workflow,
    then each tool needs to be able to ingest the other’s output…

    > Can you provide an example of an XML structure
    > that is troublesome in the sense you mean?

    no, because i haven’t seriously tried to program an x.m.l. tool.
    but if you ask the people who have, they can probably tell you.
    if it was easy, it woulda been done already. it hasn’t. it’s hard.

    however, i’d like to send you in another direction entirely.

    are you aware of “light markup” and its authoring tools?

    the idea is that you store your text in a mostly-text format
    — one with very simple rules which don’t impede writing –
    and then do conversions _from_ that to your desired output,
    where one of those “desired output” formats can be (x)html.

    the best-known of the light-markup systems is “markdown”.

    you can see markdown in action at this website, on this page:
    > http://daringfireball.net/projects/markdown/dingus

    but a real-time implementation i like better is “showdown”:
    > http://attacklab.net/showdown/

    later on, i’ll show you how easy the use of light-markup can be.

    -bowerbird

  • http://www.beyond-print.net George Alexander

    I’m sorry, bowerbird, but after half an hour playing with Markdown and Showdown, I have to say this is NOT what I mean when I say “simple tools” for authors and editors. Yes, it is true you can produce remarkable results with the simplest of text editors; yes, it is easy to read; and, yes, the price is right; but none of that means an author will take to it. And it certainly doesn’t meet my “15 minutes of training” criterion.

    I can’t imagine advising an author “try to avoid starting a line with a number — you might accidentally trigger a numbered list.” Or explaining the difference between ending a line with one space before the return vs. two spaces. Or admitting to the author that the only way to include a table is by using some pretty advanced HTML coding. These examples show that, despite appearances, there really is a lot of coding implicit in how you type the light-markup file, so you have to be very careful and follow the rules precisely. What they also show (to me at least) is that it is just extremely difficult to represent the structure of a book in plain, relatively code-free monospaced text; and teaching an author or editor how to do it will therefore also be extremely difficult. I believe that actually seeing the tags is generally going to be easier for the author or editor to deal with than having the structure be implicit in the way the file is typed.

    It’s obvious that the light-markup approach is useful in some contexts — it is probably a great tool to give non-HTML types for updating a web site — but it doesn’t serve what I see as the needs of authors and editors in book publishing. They need a tool that will allow them to easily work with both the structure and content of their manuscript, with the confidence that they can write and edit with no surprises and no dead ends. Markdown does not seem to me to be that tool, and I suspect the light-markup approach in general will have the same kinds of shortcomings.

    I continue to think that a bare-bones version of one of the XML editing packages (or something similar) would be a much better tool for the task.

    – George

  • bowerbird

    george-

    sorry i didn’t get back to you sooner with some example text;
    it woulda gone a long way toward showing you the big picture.

    as it was, you appear to have gotten snared by some minutia…

    but no matter, since you came to the correct conclusion anyway,
    in regard to _markdown_, anyway — it’s not quite “light” enough.

    to get what you really need, for books, you still must do “coding”;
    you just do “markdown” coding instead of doing “x.m.l.” coding…
    but there isn’t much sense in trading one boss for another boss.

    > These examples show that, despite appearances,
    > there really is a lot of coding implicit in how you type
    > the light-markup file, so you have to be very careful
    > and follow the rules precisely.

    i think you’re stretching it a bit with the “very careful” part, since
    — with the right user-interface — an author can see immediately
    via the wysiwyg display when they’ve made some kind of mistake.

    > What they also show (to me at least) is that it is just
    > extremely difficult to represent the structure of a book
    > in plain, relatively code-free monospaced text

    well, you’ve also nailed another shortcoming of markdown there, but
    you’ve attributed it to light-markup, which is a mistake, in my view…
    (and, just for the record, i almost never ever use a monospaced font.)

    i still intend to share some examples of how markdown _can_ handle
    a book, but i will then move on to some other examples of yet another
    form of light-markup — this one my own — that does so much better.

    i assure you that it is _not_ at all “extremely difficult to represent the
    structure of a book” using such light-markup. it is straightforward…

    > I believe that actually seeing the tags is generally
    > going to be easier for the author or editor to deal with than
    > having the structure be implicit in the way the file is typed.

    when a writer is writing, no way they want to “actually see the tags”…

    might wanna see the _effects_ of the tags — italics rendered as such,
    blockquotes nicely indented, and so on –but not the tags themselves.
    this battle was decided on the desktop; wysiwyg scored a massive win.

    -bowerbird

  • http://www.beyond-print.net George Alexander

    Sorry about the minutia, bowerbird. But I’m glad my point came through.

    > i assure you that it is _not_ at all “extremely difficult to represent the
    > structure of a book” using such light-markup. it is straightforward…

    It is hard for me to visualize this. I’m eager to see how it works.

    > might wanna see the _effects_ of the tags — italics rendered as such,
    > blockquotes nicely indented, and so on –but not the tags themselves.
    > this battle was decided on the desktop; wysiwyg scored a massive win.

    Here, we part company. WYSIWYG is good, and it is the ideal approach for office communications, but it is not sufficient for a large percentage of books.

    A pure WYSIWYG approach depends on typography to convey structure implicitly, and that’s hard to do once a book reaches a certain level of complexity. Consider the case where a book has photo captions and end-of-chapter references, and both the captions and the references use the same font, style, and point size. That’s a problem, because later in the workflow the distinction between captions and references will be important, and the author and editor are the people who should be in charge of making this distinction.

    A strategy of depending on slight differences in typography (references are 10pt but captions are 9.5pt) makes it hard for the author or editor, who must constantly check a list to see which size to use in a given situation or what 9.5pt “means.” This is a symptom that the limits of pure WYSIWYG have been reached. It is better to make the meaning explicit via tagging or by applying named styles. The WYSIWYG display continues to be useful, because it highlights the document structure and gives the user visual feedback to make it more likely that the correct tagging has been done.

    You can argue about whether all the tags should be visible constantly, or whether they should be hidden but the tagging of the current cursor context should be displayed in a window at the side. Either way works. I think it would be wise to provide the user with the option of turning the display of tags on and off. Many XML editors have this capability.

    – George

  • bowerbird

    george said:

    > It is hard for me to visualize this. I’m eager to see how it works.

    ok, good.

    if you’ve got the content of a book, clean content, well-named,

    i’d be happy to work it up for you, as a demo. i do this for fun.

    > Here, we part company. WYSIWYG is good, and it is

    > the ideal approach for office communications, but

    > it is not sufficient for a large percentage of books.

    it sounds like you think it needs to be wysiwyg _or_ structural.

    i’m of the opinion that it can (and should) be _both_together_.

    we want structural tagging, of course, but the way people work

    – as you yourself just implied above — is via _visualization_.

    if the tags they have applied — either directly or “implicitly” –

    exert an effect on the _appearance_ of the text, it helps people

    see when the tags are correct… i extend this in my tools, and

    _colorize_ each of the different structures with a different color.

    they’ll all be turned black in the end, but this colorization helps

    keep everyone straight during the writing and editing process…

    > A pure WYSIWYG approach depends on typography

    > to convey structure implicitly

    this is very similar to something i say all the time, which is that

    _a_big_purpose_of_typography_is_to_convey_the_structure_.

    to a large degree, you can ascertain the structure of a book by

    examining its typography. even if you can’t read its language!

    (as just a few obvious examples, it doesn’t take much to tell if

    something is a chapter-header, or a footnote, or a pull-quote.)

    this realization mocks the mindset of those who pedantically

    preach that we must _separate_ “content” and “presentation”.

    if we’re looking at something that has already been published,

    structure and presentation are inextricably _mixed,_ and that

    experience is how we’re raised; we understand it inexplicably.

    > A pure WYSIWYG approach depends on typography

    > to convey structure implicitly, and that’s hard to do

    > once a book reaches a certain level of complexity.

    i’ve been able to handle some books of extreme complexity…

    so if you want to have me do a demo for you, make it hard…

    > Consider the case where a book has photo captions and

    > end-of-chapter references, and both the captions and

    > the references use the same font, style, and point size.

    > That’s a problem, because

    um, that’s not really a “problem”… indeed, for the most part,

    my e-book authoring-tools — like my format itself — assumes

    that _all_ the book uses the same font, style, and point-size…

    that’s because the specification of font, style, point-size, and

    a myriad of other variables, is given over to the end-reader.

    authors no longer exert any control of those things, except in

    their “canonical” version of their book. (more on that later.)

    and while the author and the editor are in the authoring-tool,

    the _colors_ of these two different entities make ‘em distinct.

    > later in the workflow the distinction between captions and

    > references will be important, and the author and editor are

    > the people who should be in charge of making this distinction.

    that’s a good example, so i’ll run with it.

    in my authoring-tools, anything entered as a photo-caption will

    _look_and_act_ like a photo-caption. something entered as an

    end-of-chapter reference will _look_and_act_ like that entity…

    if something doesn’t _look_and_act_ “appropriately”, the author

    will know the way that they entered it was wrong, and until they

    correct the entry, the entity continues to look-and-act wrong.

    > A strategy of depending on slight differences in typography

    > (references are 10pt but captions are 9.5pt) makes it hard

    > for the author or editor, who must constantly check a list to

    > see which size to use in a given situation or what 9.5pt “means.”

    again, i use _colors_ to signal various structural components,

    for the most part. (for the color-blind, i can use other means,

    like fonts, sizes, or whatever, but color works well in general.)

    > This is a symptom that the limits of pure WYSIWYG have been reached.

    i just don’t define “wysiwyg” in the same way that you specify it.

    > It is better to make the meaning explicit via tagging

    > or by applying named styles.

    well, you’re free to create tools based on that approach, but

    i don’t think you’re gonna get far with it, especially with tags,

    not if they’re explicit in the text, because they are obfuscatory,

    and interfere with the writing process. even with named styles,

    if the display doesn’t _show_the_effect_, it won’t help authors…

    > The WYSIWYG display continues to be useful, because

    > it highlights the document structure and gives the user

    > visual feedback to make it more likely that

    > the correct tagging has been done.

    couldn’t have said it better myself. :+)

    but like they say, a picture is worth a thousand words.

    so how about some screenshots of my user-interface?

    > http://z-m-l.com/go/zml-sandbox01.jpg

    > http://z-m-l.com/go/zml-sandbox02.jpg

    > http://z-m-l.com/go/zml-sandbox03.jpg

    > http://z-m-l.com/go/zml-sandbox04.jpg

    the right side is where the author works on the text;

    the left side is a wysiwyg display as their feedback…

    (and yes, the user can flip the sides if they want to.)

    > You can argue about whether all the tags should be visible constantly,

    > or whether they should be hidden but the tagging of the current

    > cursor context should be displayed in a window at the side.

    > Either way works. I think it would be wise to provide the user

    > with the option of turning the display of tags on and off.

    > Many XML editors have this capability.

    we’ve talked about “tagging” a lot here, but another problem

    is the complexity of the learning-curve for x.m.l. in general,

    and especially as it is applied to books. authors simply don’t

    see a need for those added difficulties to their part of the job.

    writing, by itself, is hard enough, thankyouverymuch… :+)

    -bowerbird

  • http://www.beyond-print.net George Alexander

    Bowerbird:

    Thanks for the extensive explanation, and especially for the sample screen captures, which really helped me to begin to understand your approach.

    Some of your points I agree with and some I don’t. Let’s start with a key philosophical point that I don’t share, which comes at the very end of your post.

    > we’ve talked about “tagging” a lot here, but another problem
    > is the complexity of the learning-curve for x.m.l. in general,
    > and especially as it is applied to books. authors simply don’t
    > see a need for those added difficulties to their part of the job.

    Authors need the ability to express their intent—no more and no less. If the book is simple, they need very little. If the book is more complex, they need to be able to spell out exactly the structure they intend. There do not need to be any “added difficulties” here, only tools that support precisely what the author needs. I see no reason why an XML-based tool should have the slightest degree of extra complexity beyond what the author actually requires to do the job. The fact that there is XML under the hood should not add to the learning curve in the slightest. (If it does, the tool has been poorly designed.)

    On the other hand:

    > it sounds like you think it needs to be wysiwyg _or_ structural.
    > i’m of the opinion that it can (and should) be _both_together_.

    No, we’re in agreement on this. The author’s tool certainly needs to be structural, and it is very helpful if it is to some degree “WYSIWYG”. Let’s agree, though, that the way you and I are using the term “WYSIWYG” has little to do with the way it is used in word-processing and page-layout programs. Those programs try to display what you will get when the document is printed (“what you see is what you get”), and that degree of “literalness” is not required in an author’s tool.

    Since the tool need not reflect the actual fonts, point sizes, etc. of the final book, perhaps it could better be called “WYSLLAB” (What You See Looks Like A Book). That is to say, it gives the user the same kinds of visual clues to structure that a book’s typography gives.

    I see that your approach uses different colors of text to disambiguate types of text that might otherwise appear similar. That’s certainly a useful technique and one I wouldn’t have thought of. One thing I don’t quite understand about the use of color is this: does the color of the text change automatically, based on something in the way the author has typed the text; or does the author explicitly choose the color of a given block of text, as a way of indicating what type of text it is intended to be?

    Your work on your light-markup approach is impressive, and I can see that some authors might adapt to it, but my experience in book publishing (primarily medical, reference, technical, and history books) tells me that most authors and copy editors in these disciplines will actually prefer to select explicit structural tags from a menu (or some equivalent interface), rather than having to remember how to type the text in order to elicit the structure they have in mind. Perhaps your system was developed to serve a different group of authors.

    It is becoming clear that we will probably not find common ground on how an authoring and editing tool should be designed, but I would like to say how helpful this discussion has been in helping me clarify my own position and opening my eyes to an alternative approach.

    – George

  • bowerbird

    george said:
    > Thanks for the extensive explanation,
    > and especially for the sample screen captures, which
    > really helped me to begin to understand your approach.

    you’re welcome. glad we’re finding some understanding.
    even if there’s not agreement, it still takes two to tango…
    so i thank you in return. (and hey brian, i thank you too!)

    > Authors need the ability to express their intent—
    > no more and no less. If the book is simple,
    > they need very little. If the book is more complex,
    > they need to be able to spell out exactly the structure
    > they intend. There do not need to be any “added difficulties”
    > here, only tools that support precisely what the author needs.

    it would help me understand if you gave an example of
    something that will fall under the “more complex” class.

    > The fact that there is XML under the hood should not
    > add to the learning curve in the slightest. (If it does,
    > the tool has been poorly designed.)

    i haven’t seen an x.m.l. editor yet that kept it “under the hood”.
    i think it’s a great thing to strive for. don’t know if it’s possible.

    > Let’s agree, though, that the way you and I are using the
    > term “WYSIWYG” has little to do with the way it is used in
    > word-processing and page-layout programs. Those programs
    > try to display what you will get when the document is printed
    > (“what you see is what you get”), and that degree of
    > “literalness” is not required in an author’s tool.

    well, i can agree that it is “not required”, but that “literalness”
    is something to which i do aspire with my format and tools…

    a lot of my work is based on o.c.r. derived from scan-sets, and
    my intent is to be able to replicate a book as it was published…

    i don’t take it as far as matching the font of the p-book, but i
    do keep pagebreaks and linebreaks (even end-line hyphenates).

    and i’d imagine a person writing a book would want to output
    work-in-progress as a .pdf that looks like it’s already “typeset”
    (e.g., with bottom-balanced pages, widow/orphan control, and
    even the hotlinks that readers now expect within an e-book),
    or as an .html-book that, again, has the “regular” features…

    this is particularly so if _manuscript_length_ is a consideration.
    why should a publisher _estimate_ the length, or pay to have it
    _manipulated_ via typography, when the author could control it?

    > I see that your approach uses different colors of text to
    > disambiguate types of text that might otherwise appear similar.

    based on long experience — i’ve used that tactic back to the
    days of ventura publisher v1.0 — i can say it’s _very_ useful.

    > One thing I don’t quite understand about the use of color is this:
    > does the color of the text change automatically, based on
    > something in the way the author has typed the text

    yes.

    > or does the author explicitly choose the color of a given block of text

    no. (the colorful display-side is view-only; it cannot be edited.)

    > or does the author explicitly choose the color of a given block of text
    > as a way of indicating what type of text it is intended to be?

    the author “indicates” what the text is — i.e., its _structure_ –
    by how s/he enters it, according to the (simple) rules of “z.m.l.”
    (that’s “zen markup language”, which is my format.)

    for instance, to indicate something is a header, you precede it
    with 4 (or more) blank lines, and follow it with 2 blank lines…

    a 2-part header (e.g., with chapter-number and chapter-title)
    has a blank line between the two parts to indicate the split…

    if you have different _levels_ of headers, you use 4-5-6-7-8-9
    blank lines; additional blank lines indicate higher-level headers.

    a typical reaction when i _explain_ it to people is that “it’s silly
    to have to count blank lines”, but what happens in reality is that
    an outline is made, and the author checks the outline is correct.
    and the headers that are incorrect stick out like sore thumbs…
    i’ve successfully done books with 5 different levels of headers.

    once you indicate the headers, my tools automatically construct
    links between the contents-page (which they can also generate,
    automatically, if you haven’t) and each of the chapter-headers.

    there are also links _back_ from headers to the contents-page,
    plus links that let a reader skim the chapters back and forward.

    all the links are formed automatically, with no markup required.

    ***

    for another example…

    foot/endnotes are indicated in the body by bracketed text.[1]

    in the foot/endnote section, there is a similar bracketed note.

    [1] this is the footnote matched to the bracketed text above.

    it’s _shown_ as a footnote if the reader wants to see it that way.

    or it can be displayed as a two-way _link_ between the two…

    again, all links are formed automatically, without any markup…

    ***

    i examined lots of books, and compiled the structural elements
    commonly required, so as to devise my z.m.l. format for them…

    ***

    > my experience in book publishing (primarily medical, reference,
    > technical, and history books) tells me that most authors and
    > copy editors in these disciplines will actually prefer to select
    > explicit structural tags from a menu (or some equivalent interface),
    > rather than having to remember how to type the text in order to
    > elicit the structure they have in mind.

    perhaps they haven’t used a system as simple as mine… :+)

    > Perhaps your system was developed
    > to serve a different group of authors.

    that too… :+)

    my aim is authors who want to self-publish, quickly and easily.
    without buying expensive software, or hiring any consultants…

    at the aggregate level, my aim is a system that is so simple that
    one person (e.g., me) can maintain the 20,000+ books found in
    project gutenberg, or even a library that is 10 times that size…

    and, to be complete, my authoring-tool _does_ let you select
    some text and then choose a menu — or click a button — that
    applies a style to it. so you could type a quote as body-text,
    then select it and click “blockquote”, and it’d get reformatted.

    the main idea of the z.m.l. is that — once you’ve learned the
    rules — you can use _any_ text-editor to create a z.m.l. file…

    > It is becoming clear that we will probably not find common ground
    > on how an authoring and editing tool should be designed, but
    > I would like to say how helpful this discussion has been in helping me
    > clarify my own position and opening my eyes to an alternative approach.

    discussion is always good. for both sides. :+)

    collaboration — it’s the obama way… ;+)

    anyway, it sounds like you’re opting out of the conversation…
    which is fine. but if you would like to see a bit more, i wasn’t
    quite done yet. what you need to see are some examples of
    how a z.m.l. file gets converted into .pdf and .html output.

    so here’s an simple book — the jungle, by sinclair — in .zml:
    > http://z-m-l.com/go/tjbus/tjbus.zml

    here’s page-by-page .html that recreates the p-book pages:
    > http://z-m-l.com/go/tjbus/tjbusf001.html
    this series puts each scan next to its text, for easy verification.

    here’s a reflowed single .html file, from the z.m.l. “master file”:
    > http://z-m-l.com/go/tjbus/tjbus.html
    you’ll see this has the navigational features mentioned above.

    and here’s the created .pdf:
    > http://z-m-l.com/go/tjbus/tjbus.pdf
    again we have the navigational chapter-links mentioned above.

    the page-by-page .html series is the public “canonical” version.

    what’s cool about the .pdf is each page mimics the “canonical”
    version, even _linking_ up to the online .html canonical page…

    so you can use the canonical page as a public “gathering point”
    for each page in the book, where public annotations are made,
    error-reports can be submitted and commented on, and so on.
    very nifty.

    it’s a big download — at 64 megs — but here’s another .pdf,
    which displays my digital text with the image from each page:
    > http://z-m-l.com/go/tjbus/tjbus-hybrid.pdf
    the size, of course, results from the scans. but this .pdf shows
    how accurate z.m.l. can replicate the look of the paper-book…

    if you have any comments, or questions, i’d love to hear them…

    if not, thanks for the tango. enjoy the new president… :+)

    -bowerbird

  • http://www.beyond-print.net George Alexander

    Thanks for additional links, bowerbird. I think I now have a good sense of your approach. I am also getting a sense of the types of books that interest you, and how they differ from those that interest me.

    > i haven’t seen an x.m.l. editor yet that kept it “under the hood”.
    > i think it’s a great thing to strive for. don’t know if it’s possible.

    I’m glad you see it that way, and I think it IS possible. Personally, I think such an editor is the only hope for “starting with XML” for many books.

    > well, i can agree that it is “not required”, but that “literalness”
    > is something to which i do aspire with my format and tools…

    You are misinterpreting my point of view. To me, “literal WYSIWYG” means that what you interact with on the screen is a close facsimile of the printed page. That is clearly NOT what your editing approach tries to do (although you do propose to support the production of a PDF preview that is “literally WYSIWYG” in this sense).

    > my aim is authors who want to self-publish, quickly and easily.
    > without buying expensive software, or hiring any consultants…
    > at the aggregate level, my aim is a system that is so simple that
    > one person (e.g., me) can maintain the 20,000+ books found in
    > project gutenberg, or even a library that is 10 times that size…

    I can see that your approach would work well for many of these authors and for the vast majority of the Gutenberg texts. My impression of the Gutenberg texts is that almost all are fiction, and almost none have photographs, diagrams, tables, lists, or references. There are very few texts in the disciplines that I have experience in (medicine, technology, reference, and history), and the few texts in those disciplines that do appear seem to be entirely composed of running text (or very nearly so). Given the authors and texts that you mention, it makes sense that the type of editing tool that would interest me would be different from one that would interest you.

    For the record, I think a target price of $50 for the software (and no consultants) would do the job. I am not, however, thinking at all about authors who self-publish. For most of them, XML would add little or no value. This is probably true for most fiction, whether self-published or not.

    Your work in putting “The Jungle” into your format and deriving different spin-offs from it is impressive, and you’ve convinced me that you have an opportunity for an interesting re-publishing business there.

    But for authors and editors in the disciplines I mentioned a few paragraphs back, I am still convinced that the right tool is one that allows explicit labeling the structure of the document (with the option to suppress those tags when desired). Books in those disciplines, and ones with similar needs, are the books that can benefit most from XML.

    – George

  • bowerbird

    george said:
    > I think I now have a good sense of your approach. I am
    > also getting a sense of the types of books that interest you,
    > and how they differ from those that interest me.

    my approach isn’t limited to any specific “type” of book.

    > You are misinterpreting my point of view. To me,
    > “literal WYSIWYG” means that what you interact with
    > on the screen is a close facsimile of the printed page.

    that’s what it means to me too… :+)

    > That is clearly NOT what your editing approach tries to do
    > (although you do propose to support the production of
    > a PDF preview that is “literally WYSIWYG” in this sense).

    if you take a look at those “sandbox” links i gave above,
    you’ll see i actually draw the “pages” on the display-side.

    i encourage authors to consider the _display_ of their work.

    while i can say it’s true that it “doesn’t matter” to me whether
    that display is on the computer-screen or on the printed-page,
    i will then hasten to add that i want authors to think of _both_.

    i believe in electronic-books, but i also want to give the reader
    a good solid option to _print_ the book in high-quality fashion.

    so i _do_ indeed think of the screen as “a close facsimile” of
    the printed page. early in my electronic-book research, i was
    of the opinion that we shouldn’t concern ourselves with paper.
    over the years, i’ve come to believe that that was shortsighted,
    for a number of reasons (which we can discuss, if you care to).

    so — in my book — if an e-book authoring tool doesn’t _remind_
    the writer to consider how their work will look when it’s printed,
    it is doing that writer a disservice… that’s my firm opinion…

    > I can see that your approach would work well for many of
    > these authors and for the vast majority of the Gutenberg texts.
    > My impression of the Gutenberg texts is that almost all are fiction, and
    > almost none have photographs, diagrams, tables, lists, or references.

    i figured you would think my approach was limited in that regard.

    but it’s not true. not at all. not in the slightest. not even a tiny bit.

    i handle photographs, diagrams — all types of images — just fine.

    tables and lists aren’t too difficult either, although sometimes you
    do have to put some thinking into the best way to present them…

    as far as references, they’re just another kind of “footnote” to me.

    i’d have to refresh my memory to be sure, but with adjustments,
    i can manage a.p.a., m.l.a., turabian, and chicago citation-styles.

    i can even do “complex” things, like footnotes _within_ footnotes.
    and — as per usual — these references can be displayed however
    the reader wants — as footnotes, endnotes, or in-line pop-ups…

    you didn’t mention audio or video, but my viewer-app can handle
    those files natively, thanks to quicktime. try that with your .pdf,
    or e-pub, and let me know how easy it is. (i really don’t know.)

    z.m.l. even has a type of “plug-in” architecture, in the sense that
    it will “launch” an unsupported file on the end-readers’ machine,
    using whatever app they have set up to handle that type of file…

    so, if you want to have a latex, .svg, or lilypond file in your e-book,
    you include that file, and it will be launched on the user’s machine,
    using whatever program that they’ve associated with that filetype.

    this gives my format tremendous versatility, as you might imagine.

    in addition, it can interface with the web, including sites like flickr
    and photobucket and picasa — since you mentioned photographs.

    you also saw, in my .pdf example, how z.m.l. can call up a web-page
    in the browser, which is just a subset of the “launch” capability above.

    finally, i’ve considered the aggregate — or “library” — level as well.
    each book in my library can reference any other book in the library.

    with all this taken together, i think my format is _quite_ capable of
    representing some very sophisticated e-books, including functionality
    that has heretofore been largely unconsidered by the .epub format…

    indeed, my format is even more valuable when a book is complicated.
    a simple book is simple to code, whether you’re using z.m.l. or x.m.l.
    since a complex book means more coding, z.m.l. saves you from that.

    > There are very few texts in the disciplines that I have experience in
    > (medicine, technology, reference, and history), and the few texts
    > in those disciplines that do appear seem to be entirely composed of
    > running text (or very nearly so).

    i’m not sure that that sentence says exactly what you wanted it to say.

    but that’s beside the point. my format can handle sophisticated books.
    it can handle multimedia. even apps i wrote more than a decade ago
    have been fully capable of doing multimedia. it’s easy with quicktime.
    and now that quicktime is ubiquitous — even on windows machines –
    (thanks to the popularity of the itunes store), multimedia is simple…

    (and, just for the record, i’m a mac person. and the basic text-editor
    on the mac for the last 15 years has been able to incorporate graphics,
    and has supported styled text too. so i take these things for granted.)

    > For the record, I think a target price of $50 for the software
    > (and no consultants) would do the job. I am not, however,
    > thinking at all about authors who self-publish. For most of them,
    > XML would add little or no value.

    most of the books that random house puts out — or harlequin,
    or penguin, or any number of other major publishing houses –
    are text-streams first and foremost. they still seem to want x.m.l.

    so they must think that it adds value.

    (by the way, i absolutely agree that x.m.l. _does_ “add value”.
    the only thing i question is whether it adds _sufficient_ value to
    offset the fairly high cost incurred by coding the book with it.)

    > This is probably true for most fiction, whether self-published or not.

    i can tell you the self-publishing world will be wider than “fiction”.

    _much_ wider.

    do you know how many young academics write text-books? lots.

    because if they get it picked up by a publisher, they can be golden.

    do you know how many of those text-books get picked up? very few.

    but down the line, how many of those young academics will allow
    their rejected manuscripts gather dust up on the shelf? very few.

    they’re going to make those manuscripts freely available as e-books.

    and do you know what that’s going to do to the textbook publishers?

    i do.

    furthermore, books will be displaced by other forms of media too.
    there was a post on this o’reilly blog not too long ago noting that
    many “how-to” topics are being tackled with video instead of text.
    instead of buying a book on “tiling the bathroom”, people are now
    turning to youtube, where they find plenty of informative content,
    stuff that — frankly — is likely even better than a text-based book.
    perhaps you don’t consider a youtube video to be “self-publishing”,
    but that’s no consolation if that video means lost sales of your book.

    > Your work in putting “The Jungle” into your format and deriving
    > different spin-offs from it is impressive, and you’ve convinced me that
    > you have an opportunity for an interesting re-publishing business there.

    ok. good. now, would you like to see some more complex examples?

    and, better, would you like to furnish me with the content for a demo?
    make it as hard as you want — i’d like the challenge of a difficult job…

    > But for authors and editors in the disciplines I mentioned
    > a few paragraphs back, I am still convinced that
    > the right tool is one that allows explicit labeling
    > the structure of the document
    > (with the option to suppress those tags when desired).
    > Books in those disciplines, and ones with similar needs,
    > are the books that can benefit most from XML.

    different people can believe different things. indeed, as the saying goes,
    “that’s what makes a horse-race”… :+)

    -bowerbird

  • bowerbird

    it looks like george has left the dancefloor.

    thanks for the tango, george! :+)

    i’m still willing to demonstrate z.m.l. on a more-difficult book,
    if anyone would like to present me with their content to do that.

    or perhaps the o’reilly gang would like to see a z.m.l. version of
    “the missing manual”, by david pogue. it has some 395 graphics,
    spread amongst ~500k of text, with a 4-level header-structure,
    so it would be a moderately difficult example. how about it, guys?

    nothing like head-to-head comparisons to determine whether
    .epub is superior enough to be adopted as “the sole standard”…

    (or even good enough to remain in the race…)

    -bowerbird

  • http://www.beyond-print.net George Alexander

    Hi again. It’s true I had to drop this conversation for a while because of pressing deadlines–and I don’t have much too add at this point.

    Bowerbird, I don’t know about “The Missing Manual”, but let me suggest “The Whole Internet User’s Guide & Catalog” from 1992. (You could fit it all into 400 pages back then!) O’Reilly now offers this as a an open-source book, and it has a medium level of complexity: a good range of heads, bulleted lists, tables, footnotes, etc. Why don’t you sink your teeth into that one? I’d be very interested to see what you can do with it.

    – George

  • bowerbird

    george pops back up on groundhog day! :+)

    welcome back…

    i’d already started on “iphone — the missing manual”,
    so i’ll release it when i’ve finished processing it, which
    will include sufficiently disguising the text and graphics,
    of course, so as not to worry o’reilly folks unnecessarily.
    i’m sure it’ll continue to be a bestseller for ‘em for a while.

    it looks like “the whole internet user’s guide and catalog”
    is only available at archive.org. i’d need clean digital text
    — as opposed to the unclean o.c.r. from the scan-set –
    and all of the graphics burst out individually to proceed…
    if that’s available somewhere, or you can provide it to me
    — if you re-do the o.c.r., save the output as styled text –
    let me know and i will be happy to do it. otherwise, it will
    be too much trouble for the value of the resulting product.
    sure is a nice-looking book though…

    -bowerbird

    p.s. but i don’t see anything in it that would be hard for me,
    except maybe that i’ve resisted a decision to support columns.
    otherwise, it’s straightforward.

  • http://www.beyond-print.net George Alexander

    The digital text is here, in case you are still interested:

    http://www.archive.org/stream/wholeinternet00krolmiss/wholeinternet00krolmiss.txt

    – George

  • bowerbird

    yeah, that’s the unclean o.c.r. to which i made reference.

    to make a worthwhile book, it’d have to be cleaned first…

    to show yourself the difference between o.c.r. versus clean,
    compare the archive.org text from “the jungle” with mine…

    > http://www.archive.org/stream/thejungle00sincuoft/thejungle00sincuoft_djvu.txt

    > http://z-m-l.com/go/tjbus/tjbus.zml

    i know more about cleaning up archive.org o.c.r. text than
    probably anyone else in the world now, but i just won’t do it
    unless i have a _very_ compelling reason… and even then,
    i’m quite reluctant to do it unless i also have a comparatively
    clean text (usually from project gutenberg) against which to compare the archive.org text. and for this book, i have none.

    -bowerbird

    p.s. even with the project gutenberg e-text for “the jungle”,
    i had to resolve well over a thousand contradictions between
    that and the archive.org o.c.r., which was a very tedious task,
    with the only offset being that this was an important classic.
    without a comparison text, you need to proofread every word.
    i won’t generally do that even for an important classic, and as
    interesting as this book might be, it’s certainly not a “classic”…
    but if you were to sponsor the book at distributed proofreaders,
    so as to have them clean up the text, i’d certainly do it in z.m.l.
    or, as i said earlier, you might be able to get better o.c.r. from it.

  • bowerbird

    george-

    my work on “iphone: the missing manual” is coming along fine.

    it looks like it will be good not just as a demonstration of z.m.l.,
    but as material for interesting dialog on various e-book aspects.

    the secret has been spilled on how to extract the .epub file
    from the iphone-app version of this book, if you have that.
    > http://toc.oreilly.com/2009/02/popping-the-hood-on-the-iphone-mm-app.html

    if not, you can get a feel for the book by looking at its “preview”:
    > http://oreilly.com/catalog/9780596521677/preview.html

    that preview shows the p-book pages. the pagesize is 6*9,
    so the book-block is probably 5*8. this means, of course,
    that the content flows significantly differently on the iphone.
    so that’s one of the issues where a dialog could be interesting.

    -bowerbird

  • bowerbird

    ok, here’s an example of the way i will disguise the text
    in the version i release of “the iphone missing manual”.
    i’ve done random replacement of the lowercase vowels.
    i trust o’reilly will agree that it is sufficiently distorted…

    will it also be necessary for me to disguise the pictures?
    if so, i can reverse ‘em, or draw a red slash through ‘em,
    or whatever else you might suggest will meet your needs.

    ***

    the top version is an example of my vowel-substitution.
    the bottom version is the original.

    > In tha ferst yier af thi uPhena’s axustincu,
    > Applu seld 6 mollaen if tham; bruaght thi thung
    > tu 70 caentruus; ind unsperud un andestry af
    > masbagettun ePhinu laikolekos frum uthir campunees.
    > By thu ind uf Yier Ona, yai ciald typa uPhina
    > untu Gaegla und git 229 mullaan huts.

    > In the first year of the iPhone’s existence,
    > Apple sold 6 million of them; brought the thing
    > to 70 countries; and inspired an industry of
    > misbegotten iPhone lookalikes from other companies.
    > By the end of Year One, you could type iPhone
    > into Google and get 229 million hits.

    ***

    > Niw thura’s i niw ePhuna, thu aPhani 3G.
    > Muru empirtintly, tharu’s u naw virseun ef
    > thu aPhanu’s suftwura, cillid ePhana 2.0.
    > And than thiru’s thu uPhani App Stura,
    > whuch iffirs thaesunds af idd-en prigrims
    > wretton by andevodails, suftwura campinuas,
    > und uvarythung an batwuun.

    > Now there’s a new iPhone, the iPhone 3G.
    > More importantly, there’s a new version of
    > the iPhone’s software, called iPhone 2.0.
    > And then there’s the iPhone App Store,
    > which offers thousands of add-on programs
    > written by individuals, software companies,
    > and everything in between.

    ***

    > Thas es higi. Rumambar hiw mystufaud
    > aviryunu wus whin Appla cullid uts
    > mesac pliyar thi ePud — unstied af, sey,
    > aMasac er eSungs er sumuthung? Thi rausan
    > wis thut Applu hid mach baggur plins fur
    > tha uPid — phates, vudeus, dacamunts, ind se un.
    > Miyba thuy shueld hiva suvud thit numi fir tha aPhanu.

    > This is huge. Remember how mystified
    > everyone was when Apple called its
    > music player the iPod — instead of, say,
    > iMusic or iSongs or something? The reason
    > was that Apple had much bigger plans for
    > the iPod — photos, videos, documents, and so on.
    > Maybe they should have saved that name for the iPhone.

    -bowerbird