ENTRIES TAGGED "idpf"
The publishing industry has a problem, and EPUB is not the solution
Ebooks are deliberately being made defective through digital restrictions
This article contains my personal views, not those of my employer Lonely Planet.
I’ll be blunt. Ebooks and EPUB are to the publishing industry what Blu-Ray is to the movie industry: a solution to yesterday’s problem made irrelevant by broader change in the industry. Both have a couple of years left in them, and there’s good money to be made while the kinks get worked out from the alternatives, but the way the wind is blowing is clear.
Whenever someone proposes EPUB as a solution, ask yourself a question: what’s the problem they’re trying to solve? As a standard drafted by the IDPF, a self-proclaimed “organization for the Digital Publishing Industry”, EPUB is built squarely to address the industry’s biggest headache: ensuring that, in the digital age, they retain the ability to charge money for distributing content. The best interests of authors or readers simply do not figure in the equation. Read more…
O’Reilly’s journey to EPUB 3
Upgrading to EPUB 3 is not a trivial undertaking
We at O’Reilly are very pleased to announce that we have officially upgraded to EPUB 3, and ebook bundles purchased from oreilly.com will now include EPUB 3 files, in addition to Mobi and PDF files. All O’Reilly ebooks released in 2013 are now available in EPUB 3 format, and in the coming weeks, we will be updating and rereleasing our backlist ebooks in EPUB 3 as well.
But while we’re excited to share this news, this article is not merely a press release. The decision of when and how to upgrade to EPUB 3 has been challenging for many in the publishing community, and it has been a long journey for O’Reilly as well. I’d like to talk more about why we chose to take this step now, what additional value we believe EPUB 3 provides to our customers, and the challenges and tradeoffs we’ve tackled in making our EPUBs backward compatible with EPUB 2 platforms.
Are we over-thinking EPUB?
The future of the book is inherently linked to the browser
One common misnomer I have come across is that EPUB3 is ‘a technology’ – something in and of itself. I believe this category mistake is largely a result of the the IDPF’s (the organisation that maintains EPUB3) success in promoting EPUB as a ‘standalone’ technology to the publishing world.
While all content is trending towards CSS and JavaScript, the core technologies of the browser, it seems a little weird to position EPUB as being a collection of things that do something different from what browsers do. The nuance might not be clear so here goes…
TOC to Host W3C eBook Workshop
CFP deadline is December 10
Among the many exciting events taking place during TOC this year, we’ll be hosting the W3C’s first eBook workshop, “eBooks: Great Expectations for Web Standards – A W3C Workshop on Electronic Books and the Open Web Platform.”
The workshop – organized in partnership with IDPF and BISG – is free, but is limited to 80 registrants (with no more than two attendees per any one organization). To be considered for participation, attendees are required to submit a position paper explaining their perspectives on a workshop topic of their choice (see http://www.w3.org/2012/08/electronic-books/topics.html for workshop topics) by December 10. Participants should have an active interest in the area chosen, ensuring other workshop attendees will benefit from the their presence in the discussion. Click here for more information on the CFP.
Ebook problem areas that need standardisation
A user experience plea for more consistency across platforms
Ebook publishing is full of problem areas, most of which cannot be addressed through standardisation but can only come about via a sea-change in the behaviour and nature of the various participants in the ebook industry.
There are, however, several issues that could be addressed, at least partially, via standardisation, that would make everybody’s life easier if implemented.
Overrides
One of the major issues facing publishers today is the spiralling complexity of dealing with vendor rendering overrides.
Each vendor applies different CSS overrides with differing behaviours, sometimes even only enabling features through server-side manipulation, which means that proper testing of an ebook is not only difficult, but impossible.
If vendors cannot be talked out of requiring these overrides then they need to be standardised and normalised. Any reading system that implements a CSS override is in violation of how the CSS standard defines the cascade and so is in violation of the EPUB 3 standard.
CSS overrides come in four broad types:
- Vendor styles only — The publisher’s styles are completely ignored in favour of the vendor’s.
- Aggressive vendor styles, but publisher styles enabled — Very little is seen of the publisher styles in this scenario. They mainly surface in edge cases that weren’t accounted for in the vendor’s stylesheet.
- Minimal overrides — The vendor only really enforces control over margins, backgrounds, and possibly font styles.
- Publisher styles — The mode that the reading app goes into when the reader deliberately selects ‘publisher styles’. Under ordinary circumstances this would simply disable the overrides but in most reading apps this mode has a unique behaviour.
Ebooks as native apps vs. web apps
Our two experts wrap up their debate on the best path towards richer content
Over the past week I’ve posted excerpts from a very insightful email exchange between Bill McCoy of the IDPF and Sanders Kleinfeld of O’Reilly (first piece here and second piece here). This is the final installment of that thread and we pick it up where Bill suggests a tighter connection between the simple ebooks of today and the richer ones we’re likely to see in the future:
Distinguishing ebooks from apps
Bill: There’s a continuum between non-enhanced text-centric publications and enhanced interactive publications, not a dichotomy.
Sanders: Well, I think you always need to draw a line somewhere between “ebook” and “app”. That’s true right now, in that “Finding Nemo: My Puzzle Book” is an app and “Does the Noise in My Head Bother You (Enhanced Edition)” is an ebook. My current prediction is that in the future, the line is going to be drawn such that more and more content slides over to the “app” side, and more specifically that more of these will be web apps, as opposed to the native iOS/Android book apps that predominate now. It sounds like you believe the reverse will be true, and you may very well be right.
Honestly, I just want publishers to put out more and more awesome, innovative digital content, preferably crafted using open Web standards. If it’s easier for them to do it using EPUB 3, then I’m all for it. But if circumstances militate against this route, as they do now, I’d rather see them explore web apps than throw in the towel.
Bill: Your argument would seem to imply that if you need to add a video or an equation visualization widget to an eBook you should move to the “other side” and build a web app instead. That’s not realistic for publishers. A lot of content will bring limited enhancements to largely text-centric linear documents.
Your own “HTML5 for Publishers” may even be a good example. It obviously has a lot of enhanced capabilities. It can live as a web app but as far as I know it doesn’t, or if it does it isn’t the normal way it’s consumed. And if you did make it into a standalone web app you’d have to code up a whole skeleton of supporting infrastructure around it, using non-standard tools. Do you really expect every publisher to do this for every title?
Sanders: I disagree with this point, in that it implies that in circumstances where you want to add a video or another widget, it’s necessarily a lot of additional work to do a web app than an EPUB 3. There’s no reason why you can’t take the very same content that you zip up in an EPUB 3 and instead post it wholesale on the Internet as a website. And if you did want to add additional supporting infrastructure for a title, it could likely be reused for other titles; I don’t think it’s true that publishers would have to start over from scratch for every book.
Bill: And a whole lot of content is going to be less enhanced than “HTML5 for Publishers” (since after all it’s subject is interactivity). I think any title being created today with iBooks Author and Habitat will be doable with off-the-shelf EPUB 3 based tools and deliverable to multiple off-the-shelf EPUB 3 reading systems within the year.
Sanders: That’s great. I really do hope that proves to be true.
Closing the gap between HTML5 and EPUB 3 support
Bill: We are still in the initial rollout stage of EPUB 3 support. As EPUB 3 proliferates, there’s no reason for EPUB 3 support to continue substantially lag browser support.
If I thought reading system implementations would remain well behind browser stack level capabilities forever, that it wouldn’t become normal to expect to deploy the same wide variety of JavaScript libraries in an interactive ebook as in a web app, I’d be less positive. But I see things converging to a model where everyone will be using WebKit, in many cases the same one that the platform’s web browser is using, so no reason that it should be any less feature-rich. iBooks is already there, and things that work in browser-based web apps that are disabled there are only for business or security considerations, and I view the former as temporary. We’re catching up more than 10 years in one transition – it’s painful that it’s taking over a year to accomplish but that’s no reason to presume it won’t be successful. In fact I’d much rather have your help getting us there than having you arguing that we shouldn’t bother. But maybe I’m misunderstanding your POV and would welcome your thoughts and feedback.
Sanders: It’s great to hear you say that. I do acknowledge that my perspective on this debate may be overly biased by my experiences as an ebook developer for the past year, who has been greatly frustrated by the current state of the ereader landscape’s support for HTML5 features and how much it’s limiting innovation in the EPUB (and Mobi) space. But I know you’re traveling all over, fiercely advocating for EPUB 3, HTML5, and open Web standards, and have a much better perspective than I do where things are headed, and how fast we’ll get there. I’m not sure what you’re advocating publishers do in the meantime, though. Should they just wait? Should they predicate their digital publishing initiatives on whether NOOK will support Canvas in the next year?
I have a great deal of admiration for the IDPF and their mission to achieve broad-based support for EPUB3, and I most certainly regret that anything I said may have diminished the value of this goal or argued against it. At the same time, I hope it’s understandable why publishers might view putting all their eggs in the EPUB 3 basket to be undesirable at this time.
I believe this statement by Bill was one of the most important ones in the whole discussion:
As EPUB 3 proliferates, there’s no reason for EPUB 3 support to continue substantially lag browser support.
That lag is one of my biggest concerns as well and why I feel publishers should focus on HTML5 itself and not something else that’s built upon HTML5 (and therefore requires more time for development and adoption).
What’s your opinion? Will the gap between HTML functionality and EPUB reader support for those features shrink or will there always be a lengthy delay in the latter’s ability to keep up with the former?
EPUB 3 facts and forecasts
Why ebook publishing will look more like software development than print production
In an article posted a few days ago I shared the first part of an email exchange between Bill McCoy of the IDPF and Sanders Kleinfeld of O’Reilly. They were debating the merits of HTML5 and EPUB 3. In the second of this three-part series they dig deeper into the capabilities of EPUB 3 and what the future of this format might look like:
Why isn’t EPUB 3 leading to more innovative products?
Bill: It seems to me that your argument amounts to (to paraphrase): “EPUB 2 is good enough for plain text, everything else can and should be a web app, so let’s not bother with EPUB 3”. That’s not an entirely new argument, but I’m a bit surprised to hear it from you.
Sanders: I definitely did imply this, and I really wish I hadn’t, because at its heart, EPUB 3 *is a web app*; it’s just packaged according to a specific standard. I have absolutely no qualms about the feature set EPUB 3 was designed to support, and I think those features are exactly what is needed to do Category 2 digital content.
My complaint is that there’s really no sane way to do full-fledged Category 2 [more sophisticated, feature-rich] emedia *right now* in EPUB 3 given today’s landscape of ereaders and the features they support. But that’s really a knock against the tablet vendors, not EPUB 3, as I feel they’re the ones stifling innovation.
Anyway, if you are a publisher who wants to press forward today, you really have to pursue an app, in the sense of something that can either be served via a Web browser or compiled and distributed in an app store. I don’t really feel I can advocate that publishers do an EPUB 3 at this time, unless they’re content with limiting its marketability largely to the iPad, and even then, Apple does not make it especially easy to do a standard EPUB 3 for iBooks (because presumably they’d much rather you use iBooks Author).
But I do regret sounding so dismissive of EPUB 3 in my remarks. I also wish I had specifically mentioned Readium as an example of an app that holds a lot of promise in terms of furthering the goals of an open HTML5-based pathway for modern “ebook” content. I hope it helps crack the stranglehold that tablet vendors have over feature support.
Bill: [My first issue with what you noted earlier is that] publishers can’t afford to create custom web apps for each title at scale. A declarative content model and associated toolset will be required. Unless we want that model and toolset to be closed and proprietary, EPUB 3 or something very much like it will be needed.
As you know, creating compelling engaging web apps isn’t easy. Most publishers don’t have O’Reilly’s technical talent and resources and I doubt even O’Reilly has enough JavaScript software developers to make web apps for every title. There’s a reason there’s only one Financial Times and a reason there’s only one Principles of Biology, and that’s in large part that they are extremely expensive endeavors. At Google I/O the Chrome team spent a lot of time talking about how offline in the browser didn’t deliver a great user experience for things like Google Reader, and that’s why they are rolling out new packaged apps.
Publishers will demand tooling and that tooling will require underlying content representation that’s not just JavaScript, and will largely be assembling and configuring widgets, not writing them and the apps around them from scratch. HTML5-based tools like Habitat and iBooks Author have underlying representations but they are entirely closed and proprietary. And, I don’t think you are implicitly arguing that either of these toolsets is the path forward rather than a free and open content representation that enables multiple tools from different vendors as well as multiple distribution options.
Sanders: Agree with all your points above, and heck no, I am definitely not arguing for iBooks Author or Habitat over EPUB 3 🙂
Open source development will lead to additional EPUB 3 growth
Bill: If you get to the more micro level of features, EPUB 3’s “extra layer” (as you called it) has features like Media Overlays and declarative triggers. Do you really think that publishers who need to synch audio with text or wire up buttons to media controls should have to hand-code home-grown solutions every time they need to do it? How about other accessibility features?
Sanders: Excellent points. I agree, and I do not think I gave credit where it was due here to the packaging layer of EPUB 3. However, much of the hard work of creating an interactive web app exists whether or not it’s packaged in EPUB 3 format or not. Composing interactive games in <canvas> is basically the same challenge either way. I think that in the future, ebook publishing is going to look a lot more like software development than print production, so if content producers want to stay competitive, I think they will need to hire more software engineers.
I do think, though, that just as there’s been a huge wave of open source innovation that’s sprouted in the past several years around doing web apps for mobile and tablet devices (I’m thinking of software like jQuery Mobile and bootstrap.js), there will be a similar wave of development around tools and widgets facilitating the creation of interactive ebook content. It seems likely that this will facilitate and complement the growth of EPUB 3, which is great. The bummer right now is that EPUB 3 feels like more of a hindrance than a help, because I can’t get it to render the way I want in Nook Tablet’s ereader, but I can just unzip that same EPUB file and put it up as a web site, and come much closer to the results I want.
Bill: [My second issue with your earlier points is that] EPUB 2 isn’t good enough even for text-centric non-enhanced content.
For one thing we have fixed layout. For another, vertical writing and other global language features: even for a novel, EPUB 2 isn’t enough in Japan. Thirdly, accessibility. These are all defined in terms of modern HTML5/CSS features that aren’t part of EPUB 2. Fourthly, we have a much tighter spec for content and reading system conformance in EPUB 3 than in EPUB 2. So even if we accepted the proposition that publishers should create web apps to deliver highly interactive content, there’s enough reason to move non-interactive content to EPUB 3 to get these other benefits.
Sanders: Yes, I agree with this point. In retrospect I feel I was wrong to suggest that EPUB 2.1 was good enough as is for text-centric content. I do see the need for broader accessibility and foreign-language support.
Sanders makes a great point about EPUB 3 not being fully supported in most ereader apps. I’d like to think once that happens we’ll see much more innovation and less of a need for publishers to create platform-specific apps. I’m skeptical though. Ours tends to be an industry of followers where we wait for others to take the risk, prove the investment worthy and then jump in. I figure it will be the small, lean startups that will do more exciting things with EPUB 3 and the rest of the industry will follow. What’s your opinion?
Here’s the link to the third and final portion of this series.
HTML5, EPUB 3, and ebooks vs. web apps
Two experts help us navigate the ebook format jungle
One of the benefits of working on TOC is that I get to see some of the behind-the-scenes industry debates that take place via email. Since it’s “formats” month here in TOC-land I thought it would be fun to share a thread about HTML5 vs. EPUB 3 featuring O’Reilly’s Sanders Kleinfeld and the IDPF’s Bill McCoy. They’ve both agreed to share this thread with the TOC community since it helps clarify the state of both EPUB 3 and HTML5.
It all started with an HTML5 interview I did with Sanders earlier this month. Bill reached out to Sanders as follows:
Your mileage may vary, especially on the Nook
Bill: Your interview with Joe for TOC was interesting. As you know from prior discussion I totally agree with you that the current situation is sub-optimal. It’s disappointing that nearly a year after EPUB 3 is out, B&N’s flagship tablet device doesn’t support EPUB 3 yet, even though it ships with a reasonably modern browser that does support HTML5. This is due to delays by their current vendor for eBook rendering software (Adobe) rather than a lack of desire on B&N’s part, but it’s still a problem. I also agree with you that where EPUB 3 is supported like Apple iBooks and Kobo the consistency in terms of being able to safely use leading-edge JavaScript libraries and other browser-stack features in disparate reading systems is not where it needs to be. While it’s true that EPUB 3 needs to be able to be supported on limited capability devices I think we didn’t help by making so many things optional and being silent on reading system support for various APIs that are de facto part of the browser stack but not officially part of the HTML5 spec normatively referenced by EPUB 3 spec (e.g. geolocation, local storage, XMLHttpRequest).
I also agree with you that Web-technologies-based apps are the future for experience delivery, both in browser and increasingly for native-class apps that are liberated from the browser (whether wrapped in PhoneGap or CEF, W8 Metro apps or the new Chrome Packaged App model Google rolled out this summer at I/O).
Sanders: Yes, I think what I find so frustrating is that we’ve got tablets on the market like the Nook, which use two different engines to render Web content–one for the Web browser and one for the ereader–and the ereader is lagging so far behind the browser in HTML5 support. The ereader is being treated like a second-class citizen, even though it’s ostensibly the primary feature of the tablet (or at least the primary feature by which the tablet is being marketed; I concede that there are many people buying the Nook who just want a low-cost Android tablet, and have no intention of reading ebooks on it).
One of the things I appreciate most about iBooks is that it uses the same Webkit engine as Mobile Safari, and if you test the same HTML5 content in Safari and in iBooks, you usually get the same results.
Distinguishing apps from ebooks
Bill: But… despite all this violent agreement… I’m having a hard time with your contention in the interview that “a lot less is going to fall in the eBook side than it does now for enhanced text and graphics… anything that’s more enhanced is going to drift over to the app side… we’ll have our standard EPUBs for fiction/nonfiction and then biology textbooks will be on the other side”. That’s what I want to probe on in this email.
Sanders: I’ve thought about this quite a bit after my initial conversation with Joe, and I think the debate of “ebook” vs. “app” can be pretty facile if you’re not careful about how you define those somewhat loaded terms. And I think I failed to do a good job of that in my interview. So let me step back and try to do better now.
If you define “ebook” as “something that can be represented in EPUB 2/3 or Mobi format and rendered in an ereader app”, and you define “web app” as “something that can be rendered in a Web browser and/or sold in an app store”, then you’re really just arguing about packaging, not the underlying content, because an EPUB 3 file is composed of HTML5, CSS, and JavaScript, just like a web app. In the quote you cite above, I wasn’t really trying to make an argument about packaging, as much as about the kinds of electronic media that publishers are creating right now, and will be creating in the future.
I think Phase 1 of ebook creation for publishers was basically, “Let’s take all our print books and digitize them so they can be read on a Kindle or iPad”, without much in the way of innovation in terms of interactivity, customized rendering for a reflowable context, or even hyperlinking.
I think we’ve now graduated to Phase 2, where publishers are thinking, “How can we make customized digital content for tablet devices, instead of plain-old text-and-graphics ebooks? Do I make an ‘app’ or do I make an ‘enhanced ebook?'” I think publishers are generally taking two approaches to this:
Approach #1: Hire on software developers to make full-fledged native apps they can sell in the app stores for iPad/iPhone/iPad and Android (lots of these are children’s titles, e.g. “Finding Nemo: My Puzzle Book“)
Approach #2: Make an “enhanced ebook”, which takes the standard Phase 1 text-and-graphics context, and then grafts on some multimedia features, like audio and video clips (here I’m thinking of the Steven Tyler memoir “Does the Noise in My Head Bother You (Enhanced Edition)“, which was featured prominently in Ana Maria Allessi’s talk at Editech)
I’m rather against Approach #2, because I don’t think it’s especially innovative, and I don’t think it’s what customers really want out of next-generation e-content. I feel like the whole notion of “enhanced ebooks” is somewhat of a transitional concept, as publishers start making baby steps in rethinking how they produce content for a Digital First world. In the long term (next 5 years or so), I think a large part of the “enhanced-ebook” middle ground is going to go away, and ebook content is going to fall more neatly into one of two categories:
Category 1: The standard text-and-graphic content that you can read on even the lowest-end eInk ereader (because I don’t think all-text fiction/nonfiction is ever going to go out of favor)
Category 2: Everything else, which will be more and more app-like in the sense that it will be highly interactive, to-some-degree social (commenting, linked to Facebook/Twitter, etc.), and conceived from the start for a Web context (densely hyperlinked, with sophisticated mechanisms to search content and navigate it in a nonlinear fashion)
So, that’s what I was getting at above; I just wish I had articulated it better in my interview.
That’s just part one, folks. Stay tuned later this week for excerpts from the rest of the thread where Bill and Sanders talk further about the subtleties of EPUB 3, HTML5 and web apps.
What’s new with EPUB?
EPUB3 is a big step forward but we haven't seen all the benefits it has to offer
This month’s TOC theme is “formats.” Even though our customers still tend to favor PDF it’s clear that mobi and EPUB are the formats with all the momentum. In order to get the scoop on EPUB I decided to go right to the source. The IDPF is the organization that develops and maintains the EPUB standard and Bill McCoy is the IDPF’s Executive Director. Bill was kind enough to sit down and talk with me about the current state of EPUB and where it’s heading.
Key points from the full audio interview (below) include:
- HTML5’s impact — EPUB3 is based on HTML5 and brings with it all the rich functionality that HTML5 has to offer. Global language support and accessibility are two very important improvements in EPUB3.
- Why do we need both EPUB3 and HTML5? — Bill points out that EPUB is a “reliable container” within which HTML5 content can be distributed. EPUB also allows you to step back from HTML5 coding and focus more on the structure of the document itself.
- Fixed layout — PDF is designed with fixed layout in mind whereas EPUB not only supports fixed layout but it also offers the reflowable option as well as a hybrid solution for those situations where both fixed and reflowable are needed in the same product.
- 12 years in the making — I’m puzzled over why it’s taking so long for EPUB3 support to show up on devices and in apps. Bill reminds me it was a pretty significant jump from EPUB2 to EPUB3 but that we’re already seeing signs of significant progress.
- Beyond today’s EPUB3 — Bill prefers to focus on EPUB3 but I did manage to pry a few details from him about what we can expect in the future: Advanced fixed layout (e.g., Manga), adaptive layout (page templates), dictionaries, indexes and annotations.
For more information be sure to download a copy of our free ebook, co-published with the IDPF, What Is EPUB3? If you’re attending the Frankfurt Book Fair you’ll also want to register for TOC Frankfurt to hear Bill and I debate the pros and cons of DRM. (Are there really any DRM pros?!)
You can listen the entire interview in the audio player below.
Portable Documents for the Open Web (Part 3)
EPUB 3: The future of digital publications
The first two parts of this three-part series covered the enduring need for portable documents and why PDF’s fundamental architecture is too dated and too limited to fill this need. In this final part, we’ll take a look at EPUB, the format that has rapidly emerged as the open standard for eBooks. It’s my contention that EPUB, not PDF, represents the future of portable documents in our increasingly Web-based world. Why? In short, EPUB addresses all the key limitations of PDF. EPUB is reflowable, accessible, modular (with packaging and content cleanly separated), and based on HTML5 and related Web Standards. It’s a truly open format, developed in a collaborative process to meet global requirements rather than by a single vendor to support its proprietary products. Let’s take a harder look at these points.