ADC Punch List

No, it’s not something that one makes before sitting down around the Thanksgiving table. From Wikipedia, a punch list:

…Takes its name from the historical process of punching a hole in the margin of the document, next to one of the items on the list. This indicated that the work was completed for that particular construction task. Two copies of the list were punched at the same time to provide an identical record for the architect and contractor. [citation needed]

Although, I suppose, a punch list could be used for nefarious purposes, Mandated Memoranda Publishing formulated and used one to develop and market our newest book: A Digital Carol – A Tale for Our Generation. We refer to it as ADC.

We’ve become fairly practiced at generating Kindle books. We’ve reported on our eBook development for previous books: Tragic Wonders – Stories, Poems, and Essays to Ponder and Tiānmìng – Mandate of Heaven.

Earlier descriptions give more insight into the detailed development. As we’ve said before, your mileage may vary. Please consider this list as what it is: steps in a conditional and evolving process meant for adaptation and improvisation.

As an amusing aside, we helped surface a software bug in the Kindle Fire HD device’s Kindle Reader application (version 9.5_1190027510 (before Oct. 28, 2014) progressing to 9.5_1190027710 (as of Dec. 4, 2014)). According to some Amazon Customer Service representatives, it’s been widely reported from multiple users. I managed to send Amazon a device log to examine (using the Fire HD device’s help service). Additionally, one worker was able to duplicate the issue on her Fire HD device with her own book selection. It seems to occur generally with all Kindle eBooks.

We assured ourselves it was a device application issue by downloading our eBook (the AZW file) to our PC. Then we ran Kindle Previewer, version 2.923 in Fire HD emulation mode to check the book cover under midnight, sepia, and normal backgrounds. It rendered correctly and repeatedly in emulation mode. We also tested four randomly selected books we had purchased before the update (system update 4.5.1) was installed.

This testing followed running the original KDP generated MOBI file of our own book through a full Quality Assurance schedule using Kindle Previewer in E-Ink, Fire, and iPad (w/ side-loading) modes. We also tested the original KDP generated MOBI with the Kindle PC application and an AZK file side-loaded onto an iPad. Our book rendered correctly on all platforms but the Fire HD device. We almost halted publication because of the issue but went through with it anyway based on our intuition and experience.

What happens when the issue occurs is that any eBook’s cover (the JPG or GIF image at the book front) renders correctly when the selected background is midnight. It renders incorrectly when the background is changed to sepia or normal (white). The cover is surrounded by a background toned frame but the cover is either blacked (or whited) out or obscured by a dark semitransparent overlay.

The phenomenon is repeatable so long as the book is open. Once the book is closed and reopened, the issue changes. In this second instance, the cover renders correctly with sepia and normal background, but this time the midnight background obscures the cover.

One Amazon Customer Service rep said the issue was viewed as content related (i.e., the eBooks were at fault). She recognized the issue was a device application issue.

The best analogy I can give is this. Pretend you’ve taken a photo with your phone. You view the photo with the camera app and it looks good. You decide to improve how the photo looks so you open a photo-enhancement app on the phone. There, the photo is blacked out. Puzzled, you reopen the camera app. The photo is there and seems unchanged from when you took it. You close the camera app and reopen the enhancement app. This time the photo is whited out. Concerned, you email the photo to your PC where it looks just as colorful as it did in the camera app. Clearly, the enhancement app on the phone is at fault, not the photograph itself.

The photo represents all Kindle eBooks. The camera app and the PC app represent all Kindle Reader apps except the one on the Fire HD device. The enhancement app on the phone represents the Kindle Reader app on the Fire HD device. It is the Kindle Reader app on the Fire HD device that is at fault.

I hope Amazon Software personnel find and fix the bug (perhaps a test software artifact left active after product release?). Not resolving the issue could adversely affect all Amazon eBook sales this Christmas buying season. I wouldn’t want to be on the customer service staff if it is not fixed.

Obviously, these are our personal observations and opinions. EBook development seems never to be without excitement. Maybe it will be different next year? We resolved the previous bug we reported on our own (a rectifiable Windows ‘run with graphics processor’ selection for Kindle Previewer’s phantomjs_mobi82html executable file). Not so with this one, I’m afraid. As of yesterday, I think they still think it’s a data (or user!) issue. Such is the way of medium to large-sized software organizations.

POST UPDATE (12182014): Amazon has upgraded the Kindle Fire HD (3rd Gen) System software version to 4.5.2 and the HD device’s Kindle Reader to 9.6_1190216910. The reader was updated before the system software and appears to have resolved the cover rendering problem. This puts Amazon software in the ranks of Microsoft who actually fix their issues in a timely manner rather than letting them linger for weeks, months, and years.

Here’s the structured procedure we followed in the development and marketing of A Digital Carol – A Tale for Our Generation. We’ve also added some details on our HTML generation. This is our punch list:

Fact find from recent KDP newsletters

Kindlefy ADC version 1 – 3

  1. Add ISBN
  2. Spell check and search for errant spaces
  3. Create HTML
  4. Prepare Two Structure Files (Check To Press archive)
  5. Create File Folders (Check To Press archive)
  6. Update Kindle Previewer (KP)
  7. Generate MOBI and AZK files using KP
  8. Examine with KP (use spreadsheet for QA)
  9. Examine on Fire, PW, and iPad
  10. Revise original and go to 3 or Finish

Revise Blurb (do word count based page estimate ~130 pages; ASIN assignment)

Author interview, ADC Status, and MM posts: ADC status, Author interview (hit themes: Economic divide, Jobs automated, AI demons, War, and Population), Character Interview, Excerpts (?)

Dry run finalization of manuscript and generation of PDF

Recheck KF8 on HW

Start KDP entry for ADC [DO NOT SUBMIT]

Receive Copyedited Manuscript back from Kirkus (due November 17, received November 12)

Finalize manuscript, generate PDF

Submit PDF for review

Kirkus review ($575 for rush – suppressible if bad)

Books and Culture (?) – delayed

Publishers Weekly Indie aka Booklife (free if accepted)

Red City Review ($40) – delayed

Kindlefy ADC version 4

  1. Compare new manuscript with existing HTML (Use KDP generated version)
  2. Transcribe deltas into existing HTML
  3. Spell check updated HTML for transcription errors
  4. Buff em> versus i> and em> punctuation issues leave well enough alone (also French sp. not perfect; issue with images and font size, too)
  5. Repeat general HTML check over (used IE)
  6. Compare PDF source with HTML in Word
  7. Update Kindle Previewer (KP)
  8. Generate MOBI and AZK files using KP
  9. Do Kindle Fire only QA check on KP version
  10. Run KP MOBI (KF8) through KDP to get testable MOBI file, folders with HTML [check book data; DO NOT SUBMIT]
  11. Generate AZK from KDP file
  12. Examine KDP MOBI with KP (use spreadsheet for QA)
  13. Examine on Fire, PW, and iPad
  14. Revise and go to 3 or submit to KDP

Investigate Bowker listing?

Submit ADC for sale via KDP [Submitted to KDP Wednesday Nov 19, 2014, Published on Wednsday, Nov. 19, 2014] [ADC Pre-order unrealistic (Up to 90 days early; By Nov. 18 for 28 release?)]

Submit application for Copyright to LOC (need publication date)

Buy book and push to all devices, check out background color on cover issue

Follow up with KDP and Kindle development on this Fire HD device issue (kindle app version 9.5_1190027510).

Submit PDF for review:

Red City Review ($40)

Books and Culture (Free)

Update Author Central as necessary

Solicit reviews from Vine and other Amazon reviewers (see Dickens’s works)

Use Goodreads Authors program posts and adverts

Solicit Amazon Singles

Update Blurb/Press Release with ASIN and Amazon page estimate

Update MM Blog posts with ASIN and Amazon page estimate

Create PR Newswire Press Release from Blurb and Red City or Kirkus Reviews (Times Square and don’t forget Twitter Leisure, $400~)

Solicit other Reviews (if Kirkus Review good)

Economist

WSJ

Here’s the HTML process:

On Word file (97–2003 versions seem cleaner upon HTML conversion)

Remove cover

Reinforce styles (especially in author bio)

Substitute en dashs for hyphens (to account for Kindle rendering)

Replace book signing image with jpg

Save as ‘filtered web file’

Name: Adolphus Writer

Title: A Digital Carol – A Tale for Our Generation

On HTML

Simplify page breaks as per KDP guidelines

Move ‘<a> </a>’ out of ‘<h1> </h1>’ for chapter titles

Change ‘a name’ to ‘a id’ in all occurrences

Fix centered stars custom style

Remove excess formatting styles (i.e., those not used in manuscript body)

Clean Styles

Add ‘../image/’ to JPGs

Add text size: 200%, 150%, and 120% to centered title, centered subtitle, and h1 respectively

Add back font variant small caps to centered title

Make sure all styles have text indent as appropriate (0pt)

Remove color, text size (except as specified above), spacing

Adjust margin left to 0in instead of 0.3in as appropriate

Replace i> with em> except for foreign words

Assure punctuation italicized appropriately (see copyedited version)

Remove font color, text size, text spacing, lang[uage] references, and all span references in text

There you have it. I wish you well in your publishing adventures.

ADC Cover quarter scale, Copyrighted, All Rights Reserved

A Digital Carol – A Tale for Our Generation Cover – quarter scale (copyrighted, all rights reserved)

Self–Publishing — Love It or Leave It

What a long, strange trip… Sing it with me. We here at Mandated Memoranda Publishing LLC have just given birth to: Tiānmìng – Mandate of Heaven. I had to check with Ninja to make sure I wasn’t exaggerating. We can’t provide a clear, error-proof method to do self-publishing, but we’ll tell you all we can remember (and we won’t charge a cent). Please forgive the length of the post. As I said, it was a long trip.

The following process assumes development on a Windows PC. It may work on a Mac but your mileage may vary. We read a lot of references. Most were $2.99, or less. Given their long-term control over the publishing formats, I cannot fathom folks reluctance to stick to Amazon’s preferred rules stated in their Kindle Publishing Guidelines, methods overview, one layer deeper guide, and examples (see right hand list). The publishing method we followed gives you almost total control over your book. The publishing method is “simple” assuming you are writing from a fairly well thought out story line (I use detailed graphics to develop the plot).

Develop your Cover (an important marketing tool) according to Amazon’s Publishing Guidelines (1.6 high to 1 wide ratio and greater than 1000 pixels wide). I use royalty free (one time charge for perpetual use) images from Getty Images. Getty requires attribution on your book’s copyrights page.

Type out your book in Microsoft Word. Early in the typing (first thing is best), modify Microsoft’s Normal style. Change the font to +body (Calibri, 11 point). Actually, you’ll be removing this later in the filtered HTML but the PDF for copyright submittal to the Library of Congress looks modern in this the format. As you modify the Normal style, select the paragraph attribute. In the paragraph dialog box, under ‘indentation’, set “special” to ‘first line’ and set the “by” value to 0.01” (the smallest inches value Word recognizes). Set the “spacing after” value to 6 pt. and “line spacing” to ‘single’. Find and set the “tabs” value to 0.01”, too. The 0.01″ settings influence something important called ‘text indent’ (see below). Leave the others at their default values (if you still had them at their original defaults in the first place).

Use user defined format styles based on your slightly modified Microsoft Normal style (undo what formatting you have if you have to and rebuild it based on Normal, it’s worth it). Define other styles based on the Normal style (for instance, those for Title and Subtitle, the *** symbol, page breaks, etc.). What you want is a well-controlled set of styles that you can modify globally solely by modifying the Normal style. A few styles must diverge from this approach. Use Microsoft’s Heading 1 style for chapter titles and Table of Contents (TOC) style for the table of contents.

Use the insert bookmarks dialog box in Microsoft Word to add two special bookmarks: toc and start. These are case-sensitive and define where the table of contents and beginning of your book are, respectively. I recommend that the toc bookmark go before the heading for the table of contents but after the previous page’s page break (you can adjust its placement in the filtered HTML if necessary). I strongly urge placing the start bookmark at the top of the title page.

Proof the manuscript thoroughly. Make sure to delete blank lines and spaces (you should use page breaks between chapters and the *** symbol between scenes in a chapter). Submit what you have to an editor. Pay one to tell you what your friends or family will not. A good developmental editor is worth their salt. Fold editor comments back into the primary copy. Proof again and, if possible, resubmit to the editor for a final look-see.

Then, and here’s the part we spun over for weeks, save the Microsoft Word file as filtered HTML. Make sure to apply the book’s title in the “title” box (where it says: add a title) on the “save as” dialog box before saving the file (it gets placed in the HTML and is used by Kindle devices). You’ll also use the original file to create a PDF for the Library of Congress Copyright process (you’ll imbed the Cover image in the original Word document for the PDF but not embed it for the filtered HTML). Your baseline has just diverged so make sure that you’ll make almost no changes to the text at this point. You’ll have to fold back any format changes to the PDF and master Word copies (what a pain that was).

We went back to square one several times. One of the go-rounds centered on updating the Microsoft Word TOC without first removing the old one (go to the reference tab and the TOC pull down menu for the remove TOC command). Not doing this added extraneous bookmarks (‘< a id=“_Tocnumbers” > < /a >’) in the HTML in front of headings (i.e., two sets of bookmarks, only one of which matched the TOC ‘href=#_Tocnumbers’ entries). When you open the filtered HTML, if you’ve made that mistake, you’ll see what I mean.

Open the filtered HTML file with Microsoft Notepad (or your favorite text editor that won’t add anything behind the scenes like Word does). Now remove all ‘font size=’ statements (you might leave some for Heading and Title styles but use relative sizes like 120%). Also remove all ‘font color=’ statements since the Kindle devices use their own defaults (and black font disappears on a black background selectable for Fire and PC readers). Now, your text will scale (more) properly on Paperwhite and Fire devices. The font will change on Paperwhite. And Fire and PC readers with black or sepia backgrounds will show all your text.

Change all ‘text indent=value’ to 0 from whatever Microsoft Word set them. This prevents the Kindle devices that naturally insert tabs at the beginning of paragraphs from doing so. Obviously, don’t do it if you like that formatting. I left most of the absolute margin specifications. I also left extraneous styles Microsoft added. If I were braver or surer of what I was doing, I’d have removed them (be careful, it’s easy to remove stuff you actually need).

I recommend formatting the page breaks separately with a slight bottom margin (0.01”). This prevents certain devices from ‘dangling a sentence’ when a reader lands on a chapter heading and pages back one page (I do that a lot). It does trade the aforementioned error for another. The page back command has to be performed twice to take. Not sure why that is but it only happens when you first land on a chapter heading. Such ‘path dependencies’ are likely database related, so go figure.

I also, foolishly, added language references all over the text. It helps the Microsoft Word spellchecker but is unnecessary (ever get an error that a foreign language dictionary wasn’t available, that’s why). I won’t even tell you how I did this so you won’t be tempted to try it. Hundreds of remnants are still in the filtered HTML to this day (‘< span > … < /span >’ items). However, I did neuter their effect by removing the ‘lang=en’ field.

Review your filtered HTML file in a web browser at this point. I found some extraneous HTML symbols that I failed to remove by doing this. It is much easier to check now than later in the process. I also spell checked a version (that I discarded) in Word for errors I might have introduced in the format editing process.

At this point you have a choice. You can upload your filtered HTML file to Amazon KDP for them to convert into a book. After they process your HTML, you download the files they generate and iteratively refine the HTML (I did not do this, exactly). However, do not submit your book for publication until it is finished; it will go live (more below). Or, you can construct two more files and use Amazon KDP’s Kindle generation program to build your book. You will make the same number of iterations to your book if you use the Kindlegen program, but, you may have more insight into the changes you make.

You will need two things from Amazon to continue. The ‘compiler’ or book generator: Kindlegen (I used PC version 2.8) and the test tool: Kindle Previewer (I used PC version 2.85). You can also get the latest versions here.

The two other files you’ll need are the Open Package File (or OPF) and the Navigation Control file for XML applications (or NCX) files. The OPF file defines to the Kindlegen program your books cover file, the metadata about your book, the text files, and how to find the ‘toc’ and ‘start’ bookmarks. The NCX file tells Kindlegen how to construct the pop-up table of contents folks see on their Kindle devices. It’s called a logical TOC. Amazon provides examples.

The example that stood out far above all others was the annotated Kindle Users Guide OPF (Guide.opf in the downloadable Guide example). Whoever did the annotation is our friend for life. The OPF file tells Kindlegen how to build your book (where your files are and how to process them). Populate the file (in Notepad or equivalent) and save as ‘all files’ (not the default ‘txt’ file). You will overwrite the existing OPF file, so archive it first. I had 28 different versions of the three files archived by the end of the process (no, I’m not proud of that, just really miffed).

There is a good example of an NCX file in the Guide example folder, too. With your HTML (or HTM), OPF, and NCX files you are ready to have Kindlegen build the book. Explicitly follow the instructions in the manual.html ‘read-me’ entry that comes in the folder with the Kindlegen.exe file. All this may sound like gobbledygook, but you can do it.

Just in case it hasn’t become obvious, you are, in essence, programming your book in a formatting computer language. You generate MOBI files (Mobipocket eBook file) by running Kindlegen on the OPF file. The MOBI that results is then opened in the Kindle Previewer to test it. Here’s a list of my test procedures that I ran in all 8 Kindle emulations and the PC viewer I had downloaded separately.

  • Open the book (tests where the beginning is sensed no matter what the beginning tab showed).
  • Use the pull down menus to select Cover, End, TOC, etc.
  • Use the pull down menus to confirm that the metadata (language, author, etc.) was recognized from the OPF file.
  • Page through from the Cover to the TOC.
  • Step through the TOC and return (using the Kindle Previewer return arrow) on each entry (tests landing and for any weird artifacts).
  • Step through the logical TOC the same way.
  • Finally, on the slowest setting, use auto-flip tool to review each page for formatting issues (I went backward and forward many times). I checked the images at this point after I had settled some issues early on (I forget what they were).

This does not include the many more times I reviewed the straight, filtered HTML in Internet Explorer and did spell checks on it (the HTML) in Microsoft Word as a kind of error checking (in the software sense). At no time did I save the filtered HTML from Word (so no new artifacts would arise).

Early on, I also verified all font sizes and types (using the Paperwhite emulation) and colors (using the Fire emulations and PC reader) were removed correctly. Needless to say, I must have read the text twenty times including once on my hardware Paperwhite.

Along the way I ran into issues. My Paperwhite gave me an error when I got to 1% of the end. It ‘erased’ the file from itself (it actually was wrapped in something called a .fused file). This occurred because of the non-breaking spaces (essentially, extra carriage-return line-feeds) at the end of the book. When I removed them and recompiled the MOBI file, my Paperwhite stopped choking on the book.

On closer examination, this effect showed up in the Paperwhite emulation when the auto-flip tool stopped at 99%. I did not buy a Fire and Kindle DX to test the book on hardware (we’re really not fixed to do so, at this time). However, I did ask a friend about his iPhone app. He had the older version and he mentioned the ‘dangling sentence’ issue described above. I fixed many other issues and generated at least 13 versions of MOBI files (more I think, but I lost count).

I urge you to use the keywords: toc and start, in the OPF and HTML files. I foolishly used the _Tocnumber reference for chapter one in my OPF file instead of the start keyword. Kindlegen gave me a pass but Amazon KDP processing did not. As they say, follow the directions. I made many other changes that I can no longer recount.

Finally (so I thought), I tested my ‘last’ MOBI file on all 9 emulated devices and my Paperwhite and it worked flawlessly. Each version had its beginning on the first chapter. I uploaded my Kindlegen’ed MOBI file and Amazon KDP processed it. The MOBI file I downloaded was bigger and opened to the page before the first chapter at the end of the internal table of content on Kindle Fires. It looked unprofessional. Turns out that is one way the Fires differ from the other Kindles, at least according to the KDP forums. Unfortunately, I published this version in frustration.

The best way to modify things is one change at a time; but the turnaround is so intensive that I did more. Bad mistake. I made changes, rebuilt the book, and uploaded it to Amazon KDP. The download from Amazon KDP had the beginning page issue resolved, sort of. I had changed the ‘beginning’ from the first chapter to the title page.

Now the Fire and PC versions landed on the Cover (which was okay), and the Paperwhite, Apple (at least in emulation), and older e-ink Kindles opened to the title page (same issue, just finessed). I spent time on the forums and gleaned some true facts (and lots of opinions) about this and other issues. Whether you’ll use the forums depends on your frustration tolerance level, I guess.

However, paging through the TOC brought up an artifact in the e-ink Kindle emulation (the $69 model) that should have remained hidden. All the other versions did not exhibit this bug; but I wasn’t going to completely test the new MOBI only to find further bugs. It turns out that I had encapsulated the entire HTML text after the TOC in a ‘< div > … < /div >’ pair (more than one change at a time). I removed it (Heaven know why, I don’t) and the artifact disappeared.

I uploaded a new MOBI generated with the corrected HTML, let Amazon KDP process it, and tested the downloaded MOBI version completely (see above). I remember doing this cycle 5 times. When I determined that it worked (the beginning issue was resolved), I re-published it (no new upload was necessary at this point). ‘Belt and suspenders’ that I am, I downloaded the sample when it came online hours later. I ordered a full copy from the sample and made sure it went to the end and opened where it was supposed to on my Paperwhite.

Turns out that if you delete the .sdr folder associated with a Kindle file it’s as if the file were brand new on the device (all reader entered bookmarks are lost, of course). You can delete the .sdr folder by attaching the device to your PC and deleting the folder in Windows. I did this with every MOBI version I tested. Don’t do this directly with a purchased book (you can reset the book from the ‘manage my Kindle’ page at Amazon).

I’m now a publisher. But I may never read Tiānmìng – Mandate of Heaven again. On to book two: Tragic Wonders. Come to think of it, this ordeal might actually qualify for the new book.