Thursday, July 29, 2010

Turning DRM'ed PDB eBooks into iBooks

Suppose you, like me, have e-books that you've purchased and downloaded in the PDB format, also known as the eReader format, and now you'd like to read them in iBooks on your iPhone. If they're copy-protected using DRM, no dice ...

... unless you decrypt them first.


(Warning: decrypting a DRM-protected e-book, thereby removing copy protection and other forms of access control, is a violation of the law ... even if the decrypted copy is never shared with others. Sorry about that.)


eReader2html is a Python script that does exactly that. You can view and download the eReader2html Python script here. Then you can use it on your Mac OS X platform — as well as on Windows and other platforms — to turn your PDBs into HTML files. (My platform is Mac OS X 10.5.8 "Leopard.")

You can then read the HTML version in any web browser ... or you can use calibre to turn it into an EPUB for use in iBooks. More on calibre later.

Don't know how to read Python code? Never mind ... just click on the Download button. Your browser (mine is Safari) will put the downloaded script file into whatever folder it is set to put downloads into, using the filename f1fc790cb.txt. If, for you, the folder does not land on the Desktop, you should move the downloaded file manually to the Desktop. Then, for comprehensibility, you should click on the f1fc790cb filename and change it to eReader2html. (The original filename extension, .txt, remains.)

Now open Terminal (it's in the Utilities folder) and type the command python, followed by a space. Drag the icon of the eReader2html.txt file into the Terminal window and hit Return. You should see something like:


iMac:~ eric$ python ~/Desktop/eReader2html.txt 
eReader2Html v0.03. Copyright (c) 2008 The Dark Reverser
Converts eReader books to HTML
Usage:
  ereader2html infile.pdb outdir "your name" credit_card_number 
Note:
  It's enough to enter the last 8 digits of the credit card number
iMac:~ eric$ 


If you get an error message indicating that the python command is unknown, you probably need to install the Python interpreter. Make sure the optionally installed Apple Developer package is present on your Mac.

If you already, at the outermost root level of your Mac's folder hierarchy, have a folder called Developer, you're fine — and something else must be wrong. Post a comment to this blog entry, and I'll try to help you.

If Developer is not there, create it by inserting your OS X 10.5 Install DVD and navigating to and then double-clicking Mac OS X Install DVD:Optional Installs:Xcode Tools:Xcode Tools.mpkg. You will be led, by the installer program which opens, through the necessary steps to install the Developer Tools package, thereby creating the Developer folder.

Once you have a Developer folder, try invoking python again in Terminal. If that works, try python ~/Desktop/eReader2html.txt again.

Now do something like:


iMac:~ eric$ cd ~/Desktop
iMac:Desktop eric$ python eReader2html.txt "The Lost Symbol.pdb" "The Lost Symbol HTML" "Eric Palmer Stewart" 4264********0937
eReader2Html v0.03. Copyright (c) 2008 The Dark Reverser
Processing... done
iMac:Desktop eric$ 


Notes:

  1. Before I did this, I'd made a copy on my Desktop of my original The Lost Symbol.pdb file as downloaded from eReader.com. To do this, I'd dragged the icon of the original file to the Desktop with the Option key held down.
  2. I'd also created an empty folder, The Lost Symbol HTML, on my Desktop.
  3. eReader2html.txt, the Python script file, was also sitting on my Desktop.
  4. To enter commands in the Terminal command line, I simply typed them in. If I made a mistake, I used the Delete key to back up over (and erase) some or all what I'd previously typed, and then I retyped it correctly.
  5. First, I typed cd , followed by a space. (A space needed to be typed in after each part, other than the last part, of each command line.)
  6. Then I typed ~/Desktop and hit Return. That caused the first command to be executed.
  7. The first command, cd ~/Desktop, made the Desktop the "active" directory in Terminal. Later on, when I executed my main command, any non-"qualified' references to files and folders would implicitly point to the Desktop.
  8. In that first command, ~ represented the current user's (my own) Home folder, so ~/Desktop represented the Desktop of the current Mac OS X user: me.
  9. After cd ~/Desktop was executed, the prompt on the next line in the Terminal window changed to iMac:Desktop eric$. , since eric represents the short name of the current user at the time, me.)
  10. The main command line began when I typed in python, followed by a space.
  11. I then typed in eReader2html.txt, followed by a space. That was to tell Terminal to find the eReader2html.txt Python script file sitting in the currently active directory, i.e., the Desktop.
  12. I next typed "The Lost Symbol.pdb", in double quotes, followed by a space. This was the name of the input PDB file, sitting on my Desktop.
  13. The filename was enclosed in double quotes so the embedded spaces wouldn't cause confusion. If I didn't like the typing-filenames-in-double-quotes method, then instead I could simply have dragged the file's icon from the Desktop to the Terminal window at the point when I would ordinarily have typed in the argument — following the manually typed-in space character, mind you — resulting in /Users/eric/Desktop/The\ Lost\ Symbol.pdb. This is the so-called "fully specified path to the file." It has \ characters strewn within it to tame the included embedded spaces. No enclosing double quotes are necessary.
  14. I next typed "The Lost Symbol HTML", in double quotes. It pointed to the empty The Lost Symbol HTML folder, sitting on the Desktop, where I wanted the output of eReader2html.txt to go.
  15. Again, I could simply have dragged the folder's icon from the Desktop into the Terminal window, following my manually typed-in space character, to enter the fully specified path to the folder, /Users/eric/Desktop/The\ Lost\ Symbol\ HTML, into the command line.
  16. I next typed "Eric Palmer Stewart",  in double quotes. It was the name under which I had purchased the original PDB version of the original The Lost Symbol.pdb e-book from eReader.com. My "your name" specification had to be capitalized in the exact same way as on my eReader.com account.
  17. Finally, I typed in 4264********0937. No double quotes were needed, since there were no internal spaces, dashes, etc. It was the 16-digit number on the credit card that I used to buy the The Lost Symbol.pdb e-book from eReader.com. I have obfuscated the middle eight digits here, but what I actually typed in in Terminal was the correct credit card number in its entirety. I could alternatively have entered just the final eight digits****0937 — without obfuscation, of course — instead of the whole 16-digit number.

It took a few minutes for Processing... done to appear in the Terminal window. Once the processing was done, my The Lost Symbol HTML folder contained:

The Lost Symbol HTML folder's contents
(click to enlarge)


The book.html file is the main item. The .png files are graphic elements, most of which are illustrations from the book. cover.png is an image of the book's cover:

The Lost Symbol Cover Image
from cover.png
as seen in Preview


I double-clicked book.html, and the whole book, graphics and all, opened for inspection in my Safari browser:

The Lost Symbol HTML folder's book.html file
open in a Safari window
(click to enlarge)


Instead of turning pages, I was able to use the scroll bar to slide vertically through the book.

+++++++

Now it's time to use calibre to convert the HTML into an EPUB:

Open calibre (website here; download page here; online manual here).


(Click on this and the other screenshot images to enlarge them.)

Click on the Add Books icon.

Navigate to and open the book.html file.

Wait a minute for the Adding... window to disappear.

Make sure the e-book you just added is highlighted.

Now, since calibre won't necessarily use the cover.png file, you can give the e-book you just added a cover:

Click on the Edit Metadata icon.

Click on the Download cover button in the Edit Meta Information window, and after the cover downloads, click OK.

Verify the downloaded cover in the Edit Meta Information window, then click OK.

Verify the downloaded cover in the main window.

Now you can convert the e-book to an EPUB:

Click on the Convert Books icon.

When the Convert (your e-book name) window appears ...

... choose Output format: EPUB, then click OK to dismiss the Convert (your e-book name) window.  

In the main window, wait a minute for the spinning progress icon next to Jobs: 1 to stop spinning.

Now, with Jobs: 0 again, the main window looks like this.

Click on Click to open to reveal the EPUB file in the Finder ...

...and (in this example) the new EPUB file is shown as The Lost Symbol - Dan Brown.epub.


Next, import the EPUB file into iTunes by dragging it into the Library, as per this movie:







(You can play a larger version of the movie in QuickTime by clicking on the image above. You can also just position your mouse pointer on the image above and use the scroll wheel to move through the movie.)

The EPUB is now visible in the Books section of the iTunes library:



To copy it into iBooks on your iPhone or other iDevice, connect the iDevice to your Mac, and it will automatically sync. If it's already connected at the time you import the EPUB into iTunes, just highlight the iDevice on the left side of the iTunes window and click on the Sync button at lower right:



Now the EPUB shows up in your Library in iBooks on the iDevice:


Very neat!

12 comments:

Anonymous said...

Would it be possible to get a copy of your original pdb version of this? I cant find it anywhere.

defianse@live.com

Thank you so much

eric said...

Anonymous said...

Would it be possible to get a copy of your original pdb version of this? I cant find it anywhere.

Sorry, I can't give you a copy of my original PDB version of Dan Brown's The Lost Symbol.

For one thing, it would be illegal.

For another, it violates my own rules against sharing copyrighted e-books. I don't mind decrypting them for my own use, but sharing them with others is out of the question. Sorry 'bout that.

Here is a link to where I bought mine, at eReader.com.

Unfortunately, the price at the moment is a steep $22.46.! I bought my copy in Sept. 2009 for just $9.99! I consider the current price highway robbery ...

A way to locate and price-shop e-books is to visit http://ebooks.addall.com/. You can enter "Dan Brown" as author and "The Lost Symbol" as title, then click on Show more options. To search for just PDB versions, unselect all formats in Search all formats except eReader, which is synonymous with PDB. Scroll down and click Search.

After about 5 seconds, you'll see a list of hits. I got 3 hits for this search. One of them showed the e-book available for $22.46 at Fictionwise, which is the same company as eReader.com. BooksOnBoard supposedly has it for $19.98 ... but when I followed the link provided to that outfit, it turned out the real price was $25.40!

I know I'm going way beyond your original question here ... but I need to vent! The prices of certain e-books, in certain formats, have skyrocketed! I think this may be in part attributable to the move away from the eReader/PDB format in favor of EPUBs with Adobe encryption.

But it is pretty clear also that publishers are leaning on e-book distributors to adopt a different model for computing prices. This model does not favor us e-book consumers!

Anyway, keep in mind that the list of hits you see at http://ebooks.addall.com/ shows vendors that offer e-books in various formats. It is up to you to be sure you are getting the format you want from the vendor you select ... Let the buyer beware!

Anonymous said...

I cant get the .pdb file to process I don't get an error here is what I get:
Shane-Wilsons-MacBook-Pro:Trial scya$ python ereader2html.txt lifeontheedge.pdb [lifeontheedge HTML] "Shane Wilson" 1******0
eReader2Html v0.09. Copyright (c) 2008 The Dark Reverser
Converts DRMed eReader books to PML Source and HTML
Usage:
ereader2html infile.pdb [outdir] "your name" credit_card_number
Note:
if ommitted, outdir defaults based on 'infile.pdb'
It's enough to enter the last 8 digits of the credit card number
Shane-Wilsons-MacBook-Pro:Trial scya$

I have tried every variation and am at a loss

Thanks

eric said...

Anonymous said...

I cant get the .pdb file to process I don't get an error here is what I get:
Shane-Wilsons-MacBook-Pro:Trial scya$ python ereader2html.txt lifeontheedge.pdb [lifeontheedge HTML] "Shane Wilson" 1******0
eReader2Html v0.09. Copyright (c) 2008 The Dark Reverser
Converts DRMed eReader books to PML Source and HTML
Usage:
ereader2html infile.pdb [outdir] "your name" credit_card_number
Note:
if ommitted, outdir defaults based on 'infile.pdb'
It's enough to enter the last 8 digits of the credit card number
Shane-Wilsons-MacBook-Pro:Trial scya$

I have tried every variation and am at a loss

Thanks


Anon,

It looks like you have been using a different version of eReader2Html, v0.09. I am using v0.03, available here.

It looks as if you enclosed the output directory in square brackets — [lifeontheedge HTML] — which indeed v0.09 seems to call for. But I am fairly certain the author doesn't really want you to type in those square brackets literally, when you use eReader2Html.

Instead, try enclosing the output directory in double quotes: "lifeontheedge HTML" ...

If there is still a problem, try downloading v.0.03 of eReader2Html and using that, again with double quotes and not square brackets around the output directory name.

For those who want to try even later versions of eReader2Html than v0.09, I have found a v0.11 here.

Anonymous said...

It still is not working what python application and version are you using?
I tried with"" with out and every way possible.

Thanks

eric said...

Anonymous said...

It still is not working what python application and version are you using?
I tried with"" with out and every way possible.

Thanks


Anon,

I am using Python 2.5.1, which is the Python that is installed from the Mac OS X 10.5.n DVD.

Using your OS X 10.5.n Install DVD, navigate to and then double-click Mac OS X Install DVD:Optional Installs:Xcode Tools:Xcode Tools.mpkg on that DVD. You will, by the installer program which opens, be led through the necessary steps to install the Developer Tools package.

If you alreaady have a /Developers/ folder at the root directory level on your Mac, this would replace it with a folder structure that contains Python 2.5.

Are you on Snow Leopard, i.e., Mac OS X 10.6? That might explain the problems you have. If you're on 10.6, then I think the default Python version may be other than Python 2.5. In the Terminal command line that you've been using, try substituting python2.5 for python. That may give you a version of Python that works for you ...

eric said...

More about the problem that the anonymous earlier poster was having:

I have looked at the Python code for eReader2Html. The code spits out the brief guide to its usage seen by the anonymous poster if and only if the number of arguments on the command line is incorrect.

The only acceptable number of arguments for eReader2Html v0.03, the version I am using, is 5. So this is correct:

python a b c d e

where

a = eReader2Html.txt
b = the input PDB file name
c = the output HTML folder name, in quotes
d = your name, in quotes
e = your credit card no., or the last 8 digits of it

If there are more or fewer arguments, the script spits out the usage guide and quits.

Later versions of the script, such as v0.09, allow either 4 or 5 arguments. If only 4 are present, then it derives the output folder name from the input file name.

There are certain things that might cause the number of arguments to be wrong:

• If you type Return before entering any remaining argument(s), the Terminal "shell" will think you are done typing in arguments and will try to execute the command line as is.

• Any time any one argument contains embedded spaces, then it needs to be put in quotes. The quotes can actually be either single quotes (' ... ') or double quotes (" ... ") as long as both are the same kind. Don't try to copy paste the command line from any text editor that is using "smart quotes," though, since those represent special kinds of quote characters that don't work right in the Terminal command line. Also, if there happens to be a quote character embedded in the argument itself, that can cause a problem!

• If you don't want to type in the argument within enclosing quotes, then (only in the case of a file or folder) you can drag the file or folder's icon from Finder into the Terminal window and drop the icon there. As long as there is at least one space character following the previous argument when you do that, you're fine. A space character will automatically be generated following the new argument. The spaces and other troublesome characters in the new argument will automatically be preceded by escape chracters (\ characters).

•While extra spaces between arguments (more than one space per argument, that is) do not themselves cause problems, there can conceivably be a problem when any of the so-called spaces is actually some other invisible or "whitespace" character, such as a tab character. Again, be careful when typing in the command line, and especially careful if you are entering it into a text editor and then copying-and-pasting it into Terminal.

However, you will not get the result I have been discussing if you enter the wrong name or credit card number. You will see "Processing... Error: Incorrect Name and/or Credit Card" instead. So if (for example) you get mixed up and type in a credit card number with embedded * characters (because you wish to conceal part of the number), you'll get "Processing... Error: Incorrect Name and/or Credit Card".

A "wrong" name or credit card number is any version of your name or any CC number other than the ones used on the account at the vendor's site at which you purchased the PDB e-book. Specifically, they are the name and CC number on the account at the time you downloaded the book. If for any reason you later changed either the name on the account or the CC number, the name and number that were in force at the time you downloaded the book still apply to that particular download. If, however, you change the name or CC number on the account, then download the book again, the new download may (though I am not sure about this) bear the updated name/number combination.

Anonymous said...

Thanks a Million .... I Got it working!!!

Anonymous said...

Hello eric

Thanks so much for posting these instructions. I can tell I"m very close to making this work.

I followed all the steps, and got a message saying "Processing . . . done", but the folder I created is empty! Strange.

Here's what I did:

Last login: Thu Aug 26 17:23:42 on ttys000
joe-smith-macbook-pro:~ joesmith$ python /Users/joesmith/Desktop/eReader2html.txt
/Users/joesmith/Desktop/eReader2html.txt:10: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
import struct, binascii, zlib, os, sha, sys, os.path
eReader2Html v0.03. Copyright (c) 2008 The Dark Reverser
Converts eReader books to HTML
Usage:
ereader2html infile.pdb outdir "your name" credit_card_number
Note:
It's enough to enter the last 8 digits of the credit card number
joe-smiths-macbook-pro:~ fredericknaff$ cd ~/Desktop
frederick-naffs-macbook-pro:Desktop joesmith$ python eReader2html.txt "ChangebyDesign_95152.pdb" "ChangeHTML" "Joe Smith" 5311********7005
eReader2html.txt:10: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
import struct, binascii, zlib, os, sha, sys, os.path
eReader2Html v0.03. Copyright (c) 2008 The Dark Reverser
Processing... done
joe-smiths-macbook-pro:Desktop joesmith$

I"m not sure why my "sha module is deprecated", but it's not as painful as it sounds!

Any suggestions?

Thanks in advance!

eric said...

Anonymous said...

I followed all the steps, and got a message saying "Processing . . . done", but the folder I created is empty! Strange.

Here's what I did:

Last login: Thu Aug 26 17:23:42 on ttys000
joe-smith-macbook-pro:~ joesmith$ python /Users/joesmith/Desktop/eReader2html.txt
/Users/joesmith/Desktop/eReader2html.txt:10: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
import struct, binascii, zlib, os, sha, sys, os.path
eReader2Html v0.03. Copyright (c) 2008 The Dark Reverser
Converts eReader books to HTML
Usage:
ereader2html infile.pdb outdir "your name" credit_card_number
Note:
It's enough to enter the last 8 digits of the credit card number
joe-smiths-macbook-pro:~ fredericknaff$ cd ~/Desktop
frederick-naffs-macbook-pro:Desktop joesmith$ python eReader2html.txt "ChangebyDesign_95152.pdb" "ChangeHTML" "Joe Smith" 5311********7005
eReader2html.txt:10: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
import struct, binascii, zlib, os, sha, sys, os.path
eReader2Html v0.03. Copyright (c) 2008 The Dark Reverser
Processing... done
joe-smiths-macbook-pro:Desktop joesmith$

I"m not sure why my "sha module is deprecated", but it's not as painful as it sounds!


Hi, Anon,

I'd like to see what you get in Terminal if you enter just

python

after the prompt, with no further arguments or parameters, and hit the return key. I get

iMac:~ eric$ python
Python 2.5.1 (r251:54863, Jun 17 2009, 20:37:34)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.


where iMac:~ eric$ is my prompt. (To exit from python, type exit() and hit the return key.)

As you see, I'm using Python 2.5.1, as installed from my Leopard 10.5.8 install DVD. I wonder if you are on Mac OS X 10.5.8 and have that (and only that) version of Python on your machine. That malarkey about "sha module is deprecated" suggests you may have another Python (or more than one) installed on your platform. This is likely if, for instance, you are on Snow Leopard, in which case I believe Python 2.6 gets installed with the Developer Tools.

I've been told that Python 2.5 is also there, for Snow Leopard users. If so, you may be able to type the line

python2.5

after the prompt in Terminal and use that version instead of the "default" version on your platform.

If that works, then you can substitute python2.5 for python in the procedure in my main blog entry above. It may well allow you to get the procedure working ...

Best of luck,
Eric

Anonymous said...

Hello Eric

Thank you so much for getting back to me so quickly. I did what you said and used python2.5 instead of just python and it worked! Success! This is great. Now I can read all those books that I bought, legally, paid for, didn't borrow or steal, on the new device I have. Thank you for posting this information.

Yes, I'm using a MacBook Pro running Snow Leopard, and to answer your question, when I type just python in Terminal, I get the following:

Last login: Fri Aug 27 08:42:37 on ttys000
clarence-darrows-macbook-pro:~ clarencedarrow$ python
Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

Now that you've solved this mystery, can someone tell me why Barnes and Noble doesn't sell e-books to customers outside the US, but Amazon does, and the same books? How about why books that were purchased from Fictionwise.com can't be read on the B & N Nook, even though B & N owns Fictionwise???

Sorry if I'm getting into issues unrelated to the matter at hand, but I would truly like to understand these things. All they do is deter people from using e-books.

Thanks again for this blog and for your help.

CD

eric said...

CD said...

I did what you said and used python2.5 instead of just python and it worked! Success! This is great. Now I can read all those books that I bought, legally, paid for, didn't borrow or steal, on the new device I have. Thank you for posting this information.

You're very welcome.

Yes, I'm using a MacBook Pro running Snow Leopard, and to answer your question, when I type just python in Terminal, I get the following:

Last login: Fri Aug 27 08:42:37 on ttys000
clarence-darrows-macbook-pro:~ clarencedarrow$ python
Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.


It looks like Snow Leopard users should always substitute python2.5 for python in the procedure I gave. Thanks for helping me learn this!

Now that you've solved this mystery, can someone tell me why Barnes and Noble doesn't sell e-books to customers outside the US, but Amazon does, and the same books? How about why books that were purchased from Fictionwise.com can't be read on the B & N Nook, even though B & N owns Fictionwise???

Sorry if I'm getting into issues unrelated to the matter at hand, but I would truly like to understand these things. All they do is deter people from using e-books.


Agreed!

I don't know the answers, but I do know that e-book vendors have contractual agreements with publishers, and those contracts specify what books can be sold, in what manner, and in what parts of the world those books can be made available.

A new model for e-book selling came to the fore when Steve Jobs and Apple put iBooks on the iPad and other similar devices. This so-called "agency model" for the first time lets the publishers, not the sellers, set the selling price to the customer. I assume it gives publishers other means of distribution control as well. If you check various Amazon Kindle e-books, you will now see several that sell for a price that is higher than $9.99, the price that formerly prevailed for new bestsellers.

That's the "agency model" at work. It means we consumers can expect to pay higher prices. Amazon's Jeff Bezos, wh was not using the "agency model," had been selling e-books at at loss, at $9.99, to get the Kindle off the ground.

Publishers are afraid of the e-book market ... afraid to ignore it, but afraid to exploit it for fear that it will undermine the bricks-and-mortar stores that they have long depended on for sales. They are afraid lest e-books get widely pirated, as happened with Napster for songs. Worst case scenario for book publishers: e-books kill off bricks-and-mortar stores, and then, after DRM has been removed, get distributed illegally via bit torrents or whatever ...

You have to sympathize to a certain extent ...

Best,
Eric