Thursday, July 29, 2010

Turning DRM'ed PDB eBooks into iBooks

Suppose you, like me, have e-books that you've purchased and downloaded in the PDB format, also known as the eReader format, and now you'd like to read them in iBooks on your iPhone. If they're copy-protected using DRM, no dice ...

... unless you decrypt them first.


(Warning: decrypting a DRM-protected e-book, thereby removing copy protection and other forms of access control, is a violation of the law ... even if the decrypted copy is never shared with others. Sorry about that.)


eReader2html is a Python script that does exactly that. You can view and download the eReader2html Python script here. Then you can use it on your Mac OS X platform — as well as on Windows and other platforms — to turn your PDBs into HTML files. (My platform is Mac OS X 10.5.8 "Leopard.")

You can then read the HTML version in any web browser ... or you can use calibre to turn it into an EPUB for use in iBooks. More on calibre later.

Don't know how to read Python code? Never mind ... just click on the Download button. Your browser (mine is Safari) will put the downloaded script file into whatever folder it is set to put downloads into, using the filename f1fc790cb.txt. If, for you, the folder does not land on the Desktop, you should move the downloaded file manually to the Desktop. Then, for comprehensibility, you should click on the f1fc790cb filename and change it to eReader2html. (The original filename extension, .txt, remains.)

Now open Terminal (it's in the Utilities folder) and type the command python, followed by a space. Drag the icon of the eReader2html.txt file into the Terminal window and hit Return. You should see something like:


iMac:~ eric$ python ~/Desktop/eReader2html.txt 
eReader2Html v0.03. Copyright (c) 2008 The Dark Reverser
Converts eReader books to HTML
Usage:
  ereader2html infile.pdb outdir "your name" credit_card_number 
Note:
  It's enough to enter the last 8 digits of the credit card number
iMac:~ eric$ 


If you get an error message indicating that the python command is unknown, you probably need to install the Python interpreter. Make sure the optionally installed Apple Developer package is present on your Mac.

If you already, at the outermost root level of your Mac's folder hierarchy, have a folder called Developer, you're fine — and something else must be wrong. Post a comment to this blog entry, and I'll try to help you.

If Developer is not there, create it by inserting your OS X 10.5 Install DVD and navigating to and then double-clicking Mac OS X Install DVD:Optional Installs:Xcode Tools:Xcode Tools.mpkg. You will be led, by the installer program which opens, through the necessary steps to install the Developer Tools package, thereby creating the Developer folder.

Once you have a Developer folder, try invoking python again in Terminal. If that works, try python ~/Desktop/eReader2html.txt again.

Now do something like:


iMac:~ eric$ cd ~/Desktop
iMac:Desktop eric$ python eReader2html.txt "The Lost Symbol.pdb" "The Lost Symbol HTML" "Eric Palmer Stewart" 4264********0937
eReader2Html v0.03. Copyright (c) 2008 The Dark Reverser
Processing... done
iMac:Desktop eric$ 


Notes:

  1. Before I did this, I'd made a copy on my Desktop of my original The Lost Symbol.pdb file as downloaded from eReader.com. To do this, I'd dragged the icon of the original file to the Desktop with the Option key held down.
  2. I'd also created an empty folder, The Lost Symbol HTML, on my Desktop.
  3. eReader2html.txt, the Python script file, was also sitting on my Desktop.
  4. To enter commands in the Terminal command line, I simply typed them in. If I made a mistake, I used the Delete key to back up over (and erase) some or all what I'd previously typed, and then I retyped it correctly.
  5. First, I typed cd , followed by a space. (A space needed to be typed in after each part, other than the last part, of each command line.)
  6. Then I typed ~/Desktop and hit Return. That caused the first command to be executed.
  7. The first command, cd ~/Desktop, made the Desktop the "active" directory in Terminal. Later on, when I executed my main command, any non-"qualified' references to files and folders would implicitly point to the Desktop.
  8. In that first command, ~ represented the current user's (my own) Home folder, so ~/Desktop represented the Desktop of the current Mac OS X user: me.
  9. After cd ~/Desktop was executed, the prompt on the next line in the Terminal window changed to iMac:Desktop eric$. , since eric represents the short name of the current user at the time, me.)
  10. The main command line began when I typed in python, followed by a space.
  11. I then typed in eReader2html.txt, followed by a space. That was to tell Terminal to find the eReader2html.txt Python script file sitting in the currently active directory, i.e., the Desktop.
  12. I next typed "The Lost Symbol.pdb", in double quotes, followed by a space. This was the name of the input PDB file, sitting on my Desktop.
  13. The filename was enclosed in double quotes so the embedded spaces wouldn't cause confusion. If I didn't like the typing-filenames-in-double-quotes method, then instead I could simply have dragged the file's icon from the Desktop to the Terminal window at the point when I would ordinarily have typed in the argument — following the manually typed-in space character, mind you — resulting in /Users/eric/Desktop/The\ Lost\ Symbol.pdb. This is the so-called "fully specified path to the file." It has \ characters strewn within it to tame the included embedded spaces. No enclosing double quotes are necessary.
  14. I next typed "The Lost Symbol HTML", in double quotes. It pointed to the empty The Lost Symbol HTML folder, sitting on the Desktop, where I wanted the output of eReader2html.txt to go.
  15. Again, I could simply have dragged the folder's icon from the Desktop into the Terminal window, following my manually typed-in space character, to enter the fully specified path to the folder, /Users/eric/Desktop/The\ Lost\ Symbol\ HTML, into the command line.
  16. I next typed "Eric Palmer Stewart",  in double quotes. It was the name under which I had purchased the original PDB version of the original The Lost Symbol.pdb e-book from eReader.com. My "your name" specification had to be capitalized in the exact same way as on my eReader.com account.
  17. Finally, I typed in 4264********0937. No double quotes were needed, since there were no internal spaces, dashes, etc. It was the 16-digit number on the credit card that I used to buy the The Lost Symbol.pdb e-book from eReader.com. I have obfuscated the middle eight digits here, but what I actually typed in in Terminal was the correct credit card number in its entirety. I could alternatively have entered just the final eight digits****0937 — without obfuscation, of course — instead of the whole 16-digit number.

It took a few minutes for Processing... done to appear in the Terminal window. Once the processing was done, my The Lost Symbol HTML folder contained:

The Lost Symbol HTML folder's contents
(click to enlarge)


The book.html file is the main item. The .png files are graphic elements, most of which are illustrations from the book. cover.png is an image of the book's cover:

The Lost Symbol Cover Image
from cover.png
as seen in Preview


I double-clicked book.html, and the whole book, graphics and all, opened for inspection in my Safari browser:

The Lost Symbol HTML folder's book.html file
open in a Safari window
(click to enlarge)


Instead of turning pages, I was able to use the scroll bar to slide vertically through the book.

+++++++

Now it's time to use calibre to convert the HTML into an EPUB:

Open calibre (website here; download page here; online manual here).


(Click on this and the other screenshot images to enlarge them.)

Click on the Add Books icon.

Navigate to and open the book.html file.

Wait a minute for the Adding... window to disappear.

Make sure the e-book you just added is highlighted.

Now, since calibre won't necessarily use the cover.png file, you can give the e-book you just added a cover:

Click on the Edit Metadata icon.

Click on the Download cover button in the Edit Meta Information window, and after the cover downloads, click OK.

Verify the downloaded cover in the Edit Meta Information window, then click OK.

Verify the downloaded cover in the main window.

Now you can convert the e-book to an EPUB:

Click on the Convert Books icon.

When the Convert (your e-book name) window appears ...

... choose Output format: EPUB, then click OK to dismiss the Convert (your e-book name) window.  

In the main window, wait a minute for the spinning progress icon next to Jobs: 1 to stop spinning.

Now, with Jobs: 0 again, the main window looks like this.

Click on Click to open to reveal the EPUB file in the Finder ...

...and (in this example) the new EPUB file is shown as The Lost Symbol - Dan Brown.epub.


Next, import the EPUB file into iTunes by dragging it into the Library, as per this movie:







(You can play a larger version of the movie in QuickTime by clicking on the image above. You can also just position your mouse pointer on the image above and use the scroll wheel to move through the movie.)

The EPUB is now visible in the Books section of the iTunes library:



To copy it into iBooks on your iPhone or other iDevice, connect the iDevice to your Mac, and it will automatically sync. If it's already connected at the time you import the EPUB into iTunes, just highlight the iDevice on the left side of the iTunes window and click on the Sync button at lower right:



Now the EPUB shows up in your Library in iBooks on the iDevice:


Very neat!

Tuesday, July 20, 2010

iBooks♥B&N: Let's Get Ignoble!

ignoble is the moniker that I♥CABBAGES has chosen for his tools to decrypt e-books from Barnes & Noble that are in the EPUB format. This post by him tells generally how to use these tools. Herein, I show how to do that on a Mac running OS X 10.5 "Leopard."

First, some background information. B&N sell e-books here. For purposes of making my post more concrete, later on in this post I'm going to buy and decrypt the e-book version of Stieg Larsson's The Girl with the Dragon Tattoo, which is the first of (currently) three books in the author's Millennium Trilogy Series. The books in Larsson's series are among the most popular e-books around, and presently you cannot buy them in Apple's iBookstore. So one of the things I'll be doing in this post is showing how to use ignoble to decrypt that B&N e-book so that it can be imported into an iBooks library.

What is iBooks? It's the main e-book reader app for Apple's new iPad, and it also works on iPhones and iPods that run the new iOS 4.0 operating software. Using iBooks, you can buy e-books at the iBookstore, part of Apple's familiar iTunes Store. However, Apple has failed to line up iBookstore support from the largest publishing concern, Random House. Hence a huge number of e-book titles are missing from the iBookstore!

That's presumably why Larsson's books are absent. Their absence is not intolerable for those who are OK with, using their iDevice, switching between iBooks and the B&N eReader app, available for free at the iTunes Store. But then they have to remember which books are in which apps — not fun! In this post I'll show how to move your B&N downloads into iTunes and hence into iBooks.

Here's what you have to do to get and use ignoble:

• Get a Python interpreter

For the Mac platform, you do not need to download Python at all, as Python 2.5 (which works fine for purposes of ignoble) can be installed from the "Leopard" install DVD.

Even though Python 2.5 is part of the Mac OS X software package, in order to use it you do need to make sure the optionally installed Apple Developer package is present on your Mac. If you already, at the outermost root level of your Mac's folder hierarchy, have a folder called Developer, you're fine. If not, create it by inserting your OS X 10.5 Install DVD and navigating to and then double-clicking Mac OS X Install DVD:Optional Installs:Xcode Tools:Xcode Tools.mpkg. You will be led by the installer program which opens through the necessary steps to install the Developer Tools package.

• Get PyCrypto, the Python cryptography toolkit

Go here and click on the "Download" button for PyCrypto 2.0.1. The pycrypto-2.0.1.tar.gz file you'll receive will most likely land right on your Desktop (depending on how you have your browser configured for downloads; if it doesn't download right to the Desktop, you can just move it there). Upon being double-clicked, the pycrypto-2.0.1.tar.gz file expands into a pycrypto-2.0.1 folder, also on the Desktop. You can move this folder wherever you like, or leave it right on the Desktop.

Next, click here to download PyCrypto_2.0.1_Installer.app.zip, which, when unzipped — which will probably happen for you automatically — expands into an AppleScript app called PyCrypto_2.0.1_Installer. Double-click PyCrypto_2.0.1_Installer to run the installer itself. It will ask you to navigate to and "Choose" your pycrypto-2.0.1 folder, the one you created in the preceding paragraph of these instructions. Once you have clicked "Choose," PyCrypto_2.0.1_Installer will open a window in the Mac's Terminal application and execute some Unix commands in it.

The first time you use the PyCrypto_2.0.1_Installer, the Unix commands will produce a large amount of output in the Terminal window, due to the fact that the PyCrypto software is, for the first time on your computer, undergoing a full "build." Any subsequent times you use PyCrypto_2.0.1_Installer, an "install" without a preliminary "build" will be done, and you will see less output in the Terminal window. In either case, the final line of the Terminal-window output should say Writing /Library/Python/2.5/site-packages/pycrypto-2.0.1-py2.5.egg-info.

At this point you can quit Terminal. If you look in Finder — starting at the root-level directory, not at the Home folder for your user account — and navigate to /Library/Python/2.5/, you should find a site-packages folder. In it there should now be a Crypto folder and a pycrypto-2.0.1-py2.5.egg-info file.

• Get the any-platform key-generation script, ignoblekeygen

ignoblekeygen is a Python script that will generate, and save to a file, a Barnes & Noble EPUB user key derived from your name and credit card number.

Before you download this script, create a folder on your Desktop called ignoble. In Finder, click on the Desktop and type Command-Shift-N. A new folder appears with the name untitled folder highlighted for editing. Type ignoble to replace that generic name, and hit Return.

Open the Safari web browser and type Command-, (Command-comma). In the Preferences window that opens, click on General at top left. In the Save downloaded files: popup, choose Other... . In the File Open dialog which appears, navigate to and select the ignoble folder you just created. Click on the red close button at the upper left of the Preferences window.

Now, go here to download the ignoblekeygen script. This is a page at Pastie.org, and in the upper right corner is a Download link. Click on it. A downloaded file called pastie-753985.py will appear in your ignoble folder. (If Safari's Downloads window remains open at this point, click on the red close button at the upper left of the window to close it.)

Rename the pastie-753985.py file ignoblekeygen.py. To do that, double-click on the ignoble folder to open it in Finder, then click twice (but do not do the second click too fast, or it will cause a double-click) on the filename pastie-753985.py. The main part of the filename, pastie-753985, will be highlighted. Type ignoblekeygen in its place and hit Return. This will cause the name of the file to become ignoblekeygen.py.

Now right-click on the ignoblekeygen.py file. In the resulting popup menu, select Open With: Build Applet (default).

If you do not see that option in the popup menu, chances are you do not in fact have Python installed on your Mac. Go back to the first step of this post to see how to install it.

If you see multiple Build Applet choices, choose the one whose version number begins with 2.5.

When you select Build Applet, you may see a warning dialog, "ignoblekeygen.py" is a script application which was downloaded from the Internet. Are you sure you want to open it? Click Open.

After you have run Build Applet — which takes only a second, after which Build Applet disappears — an application file called ignoblekeygen appears in your ignoble folder.

• Run ignoblekeygen

Double-click the ignoblekeygen application file (not ignoblekeygen.py itself). You will see:



Click on the ... button to the right of the Output file field. Type bnepubkey.b64 (or use any other filename you prefer, but make sure you use the .b64 extension) into the Save As: field of the window that appears, then navigate to your ignoble folder and click Save. The Output file field will now contain a fully qualified path name to the key file that is about to be created in your ignoble folder.

Now type your name and credit card number into the appropriate fields. The name should be the one on your account at Barnes & Noble.com. (If you don't yet have a B&N account, click here to set one up.) Likewise, the credit card number you enter should be the one you are using for your B&N account.

Forgotten those details? If you sign in at B&N and then click on the My Account link, you'll see, under PERSONAL INFORMATION just above your e-mail address, the name you specified for your B&N account. You need to make sure that you use that name exactly in Ignoble EPUB Keyfile Generator. For example, if it contains your middle initial on the B&N website, then be sure to use the middle initial for Ignoble EPUB Keyfile Generator.

Under PAYMENT SETTINGS on the website you'll see Default Payment and, beneath that, ****nnnn, with nnnn representing the last four digits of your credit card number. Type the name on your account into the Name field of Ignoble EPUB Keyfile Generator, and the full 16-digit number of your credit card (without spaces or dashes) into the CC# field. Then click on the Generate button to generate the key file (by whatever name you have chosen to give it).

After you have done that, click on the Quit button. You only have to run the Ignoble EPUB Keyfile Generator once ... unless you have multiple B&N accounts, that is, in which case you should generate one key file for each, and the filename you enter in the the Output file field should be different for each — for example, bob's B&N key.b64 and mary's B&N key.b64.

The Ignoble EPUB Keyfile Generator saves your name and credit card number in its output key file in encrypted form, so no one who inspects the contents of the key file will be able to steal the information.

• Get the any-platform book-decryption script, ignobleepub

ignobleepub is a Python script that will use the key file just generated to decrypt any encrypted B&N EPUB e-book.

Go here to download the ignobleepub script. This is a page at Pastie.org, and in the upper right corner is a Download link. Click on it. A downloaded file called pastie-751366.py will appear in your ignoble folder. (If Safari's Downloads window remains open at this point, click on the red close button at the upper left of the window to close it.)

Rename the pastie-751366.py file ignobleepub.py. To do that, double-click on the ignoble folder to open it in Finder, then click twice (but do not do the second click too fast, or it will cause a double-click) on the filename pastie-751366.py. The main part of the filename, pastie-751366, will be highlighted. Type ignobleepub in its place and hit Return. This will cause the name of the file to become ignobleepub.py.

Now right-click on the ignobleepub.py file. In the resulting popup menu, select Open With: Build Applet (default).

If you do not see that option in the popup menu, chances are you do not in fact have Python installed on your Mac. Go back to the first step of this post to see how to install it.

If you see multiple Build Applet choices, choose the one whose version number begins with 2.5.

When you select Build Applet, you may see a warning dialog, "ignobleepub.py" is a script application which was downloaded from the Internet. Are you sure you want to open it? Click Open.

After you have run Build Applet — which takes only a second, after which Build Applet disappears — an application file called ignobleepub appears in your ignoble folder.

• Buy and download a Barnes & Noble EPUB e-book

I'm going to buy Stieg Larsson's The Girl with the Dragon Tattoo, which at the moment of this writing costs a reasonable $7.16 as a B&N e-book. To do that, I go to the webpage for that book and (having already signed in) click on the BUY NOW button. A dialog box appears in which I need to confirm my order. I click on the Confirm button. A "Thank You for Your Order" page appears, on which I click the Go to Your eBooks Library link.

On my My eBooks Library page there now appears:



I click on the  Download Now  .   button. That causes Safari to download the file 9780307272119_Stie_TheGirl.epub into my ignoble folder. (Close the Safari Downloads window manually at this point if you desire.)

The filename extension, .epub, indicates that this file is indeed an EPUB. B&N used to sell e-books with a .pdb filename extension. That outdated so-called "eReader format" does not work with ignoble. It cannot easily be converted for use with iBooks.

Tip: If you have e-books that you downloaded some time ago from B&N, go to your e-books library and re-download the books. Click here, then sign in, then click on the  Download Now   button next to each e-book you want to re-download in turn. The newly downloaded copy of each book will very likely have an .epub extension and will therefore work with ignoble. If you like, you can trash the old .pdb versions. Likewise, if you are using a Nook or any mobile device (such as an iPhone) with the B&N eReader app, you can and probably should re-download your legacy .pdb e-books that are already on that device, so that you will have up-to-date .epub versions on the device instead.

• Decrypt your e-book

To convert 9780307272119_Stie_TheGirl.epub, I just double-click on ignobleepub (not the original script, ignobleepub.py) in my ignoble folder, and I see:




Using the ... button next to the Key file field, I navigate to and select the bnepubkey.b64 key file I generated earlier, using ignoblekeygen.

Using the ... button next to the Input file field, I navigate to and open 9780307272119_Stie_TheGirl.epub from my ignoble folder.

Using the ... button next to the Output file field, I navigate to my ignoble folder and enter Girl Decrypted.epub as the output filename. It does not really matter what output filename I use ... but I must be sure to specify the .epub filename extension.

To see what the full contents of the three fields are, I drag the resize corner at lower right of the window rightward, to see:


Now I click the Decrypt button and quickly see:


Unless I want to buy and decrypt another e-book, I click Quit to exit the ignobleepub application. At any future time I can buy more B&N e-books, download them, and use ignobleepub all over again to decrypt them. I do not have to run ignoblekeygen again, unless of course I have misplaced my bnepubkey.b64 file.

At this point, I have a file called Girl Decrypted.epub that is no longer encrypted, so iBooks can use it. I now have to ...

• Import the Decrypted E-Book into iTunes

I simply drag the Girl Decrypted.epub file onto the LIBRARY area at the top left of my main iTunes window (click image to enlarge it):



This causes The Girl with the Dragon Tattoo to be imported into the Books area of iTunes (click image to enlarge it):



Behind the scenes, iTunes has copied the e-book into a /Books/StiegLarsson/ folder it creates in the folder structure of my iTunes library (click image to enlarge it):



The filename is The Girl with the Dragon Tattoo.epub. iTunes has figured out what the actual title of the book is, along with the author's name!

This all happens in part because I have used the following iTunes Advanced Preferences (click image to enlarge it):


You may need to type Command-, (Command-comma) while in iTunes, click on the Advanced icon at top right, and click on the check marks for Keep iTunes Media folder organized and Copy files to iTunes Media folder when adding to library, if you haven't already done so.

Notice that after I imported my e-book into iTunes, its cover image showed up automatically in Cover Flow view. That doesn't always happen. If you end up with a missing cover image, you can ...

• Manually Give the E-Book a Cover Image

First, go back to the B&N page from which you purchased your e-book. In my case, it was The Girl with the Dragon Tattoo.

Now, click on the cover image you see there, bringing up a new window with a larger version of the cover image.

Right click on the larger cover image and choose Copy Image from the pop-up menu that appears.

Now, switch to iTunes and, with the new e-book selected in the Books portion of the Library (here, my e-book already shows its cover image, but never mind )...


... type Command-I to bring up its Get Info window. Then click on the Artwork tab, to see:



Now, click in the empty artwork box — it's empty even though my cover image was already visible in my main iTunes window — and type Command-V to paste into it the cover artwork that presently is residing on the Clipboard, in order to see:



Click OK to dismiss the iTunes Get Info window. A few seconds later, the new cover artwork appears in the main Tunes window (replacing the cover image, if any, that was already visible there).

Now you're ready to ...

• Sync your iDevice to iTunes

Your iPad, iPhone, or iPod Touch will generally be synced to iTunes automatically, each time you connect it to your Mac. Syncing books requires iTunes version 9.2 or later. But syncing doesn't necessarily include e-books, unless you (click image to enlarge it) ...



... have Sync Books checkmarked in the iDevice's Books tab, as I do here. I've likewise checked Selected books rather than All books, so only the e-books that I've put individual checkmarks by are going to get synced to my iDevice. Notice The Girl with the Dragon Tattoo is shown at lower left of the Books pane, and it does have a checkmark by it. So when I click on the Sync button, at lower right of the window, I expect it (and all other checkmarked books) to be synced to my iDevice.

And that is exactly what happens in iBooks on my iPhone:



Here's my e-book actually being read in iBooks:




You can read more about iBooks, the free app from Apple that runs on iOS 4.0 iDevices — iPad, iPhone 3GS, iPod Touch — in my iBooks Cook! post.

The iBooks app itself can be downloaded here.

Apple offers this iBooks: Frequently Asked Questions (FAQ) to fill you in on the basic capabilities of iBooks.

If you read that FAQ, notice that there is no mention of any ability to read iBooks e-books right on your Mac, in iTunes or in any other app, even though the e-books reside within the iTunes library. For B&N books that you have decrypted according to the procedure herein given, you can just open them in the B&N eReader app for your Mac, available here.

Unfortunately, I know of no way to read, on your Mac, e-books (other than those that are not DRM-encrypted) from the iBookstore.

• What about Windows?

Windows users can also use ignoble, but installing Python and PyCrypto requires a different strategy.

Python:


If you use Windows you can currently get Python version 2.6, which you can download and install in Windows by going to the
 Python download page and then clicking on (as of this writing) the hotlink to Python 2.6.4 Windows installer. (There is also a version 3.1, but I can't guarantee it will work with this procedure.) You'll download a file called python-2.6.4.msi which you'll proceed to double-click, and then you'll follow the installer's on-screen prompts that allow you to install the Python 2.6 package in a directory the installer creates,C:\Python26. The actual app within that folder is called python.

PyCrypto:


Windows users can obtain PyCrypto
 here (on that page, click on the link to pycrypto-2.0.1.win32-py2.6.exe). After downloading the .exe file, double-click on it to run the Setup Wizard. As long as you have already downloaded and installed Python as detailed above, the Wizard (after you click "Next >" on the introductory panel) will indicate Python 2.6 has been duly found in the Windows Registry, in C:\Python26\. The Wizard will accordingly use the subfolder C:\Python26\Lib\site-packages\ as its installation directory. After you click on "Next >" once more, you will see a "Ready to install" panel. Click "Next >" yet again to actually perform the installation. Then click "Finish" to exit the Wizard. You should now be able to locate a Crypto folder and also a pycrypto-2.0.1-py2.6.egg-info file in C:\Python26\Lib\site-packages\.

Notice that Windows users are going to be using Python version 2.6, not 2.5 as on a Mac. As far as I know, this is not a problem.

There will also be different OS-dependent mechanics involved with downloading, installing, and using ignoblekeygen and ignobleepub. I'm not going to go into detail on all of those mechanics, but I will give some hints.

After downloading the two Python scripts, you need to arrange for them to have the filenames ignoblekeygen.pyw and ignobleepub.pyw, respectively. Notice that the filename extensions in each case are .pyw, with a "w" at the end. The "w" stands for "Windows." It allows you to do the following ...

When you want to run, say, ignoblekeygen, you right-click on ignoblekeygen.pyw and select Open With: python to designate the python.exe app (which in turn is "hosted" by the cmd.exe interface, whose window, though empty, appears behind that of the Python script's user interface). Or, if you instead select Open With: Choose Program ... , you can check "Always use the selected program to open this kind of file," and then select python and click "OK." From then on, you will be able to double-click any Python script file whose filename extension is .pyw, and it will automatically open in python.exe.

• What about Mac OS X 10.6 "Snow Leopard"?

I myself have yet to upgrade to Mac OS X 10.6, so I don't have hands-on experience with "Snow Leopard." You will have to take what I say with a grain of salt ...

Based on comments that users have made to my iPhone: Reading Adobe/EPUB eBooks post, which involves using similar Python scripts from I♥CABBAGES to decrypt e-books that have Adobe DRM, it is my understanding that the instructions I gave above basically work on Snow Leopard, except ...

... some Snow Leopard users have had trouble with installing PyCrypto, the Python Cryptography Toolkit. The basic problem has to do with Snow Leopard's use of two Python versions. Version 2.5 is the one that you need to use to run the two Python scripts, ignoblekeygen and ignobleepub, that my instructions call for.

Some users of Snow Leopard have been finding that the PyCrypto installer is putting PyCrypto in

/Library/Frameworks/Python.frameworks/versions/2.6/lib/python2.6/site-packages/

That is the wrong place! The right place for it is

/Library/Python/2.5/site-packages/

Solution: after installing PyCrypto as per the procedure above, simply copy all of the contents of the

/Library/Frameworks/Python.frameworks/versions/2.6/lib/python2.6/site-packages/

folder to

/Library/Python/2.5/site-packages/

In Finder, you can do this by highlighting all of the items contained in the Python 2.6 /site-packages/ folder and dragging them with the option key held down to the Python 2.5 /site-packages/ folder. (Be aware that you may have to create the latter folder manually in the /Library/Python/2.5/ folder.) This option-drag procedure makes a copy in the destination /site-packages/ folder of all of the highlighted items.

Among the items that need to be highlighted and copied in this way are a folder called Crypto, along with all of its contents, and a file called pycrypto-2.0.1-py2.5.egg-info. You should check to make sure that these items and (in the case of the Crypto folder) their contained items get properly copied. If the latter file is named pycrypto-2.0.1-py2.6.egg-info, you should change the name to pycrypto-2.0.1-py2.5.egg-info manually.

Once you have manually copied the PyCrypto installation folder/files to where they need to be, I believe the remaining steps in the procedure above should work.


Well, that's about it for now. Happy iBooking, everyone!

Saturday, July 17, 2010

Breaking Adobe DRM — Tips, Tricks, Workarounds

In iPhone: Reading Adobe/EPUB eBooks I gave instructions on how to move Adobe e-books, which are normally DRM-encrypted, to an iPhone for reading in an "open" e-book format. Digital rights management: it's "copy protection" that locks an e-book into a particular handheld reading device or a particular piece of software on a Windows PC or a Mac OS X computer. Many people who have paid for an e-book prefer to read it outside its prescribed environment. Hence, posts like this one.

For example, those who have paid for an e-book encrypted with Adobe's DRM scheme may want to read it on a iPhone, and iPad, or an iPod Touch. That can't happen unless the DRM encryption is removed.

Caution: Removing DRM encryption is illegal! You might think that, as long as you do not share the decrypted e-book with unauthorized users, it should be legally permissible to de-DRM a legally owned copy of an e-book.

Alas, it is not. 
In the United States, according to Wikipedia, per the Digital Millenium Copyright Act of 1998 " ... circumventing an access control, whether or not there is actual infringement of copyright itself" is illegal. DRM ("digital rights management") amounts to "access control": who may gain access to what copyrighted stuff, and when and how. DRM is also copy protection. DRM does both.

So when we strip DRM in order to move an e-book to an iPhone, we are "circumventing an access control."

Actually, this and other blog posts of mine are likewise in violation of DMCA, in that they tell people how to strip DRM and provide links to software that does that. I guess I'll be seeing you all in the old proverbial cellblock someday ...


Anyway ... taken from the long series of comments people have left below that original post, here are some tips, tricks, and workarounds that may help those who run into trouble in removing DRM encryption from Adobe e-books.


The Unzip/Rezip Trick

Some users of the ineptepub Python script see ...

Error: problem decrypting session key

... when they shouldn't. This error message is a general one that basically means ineptepub couldn't decrypt the e-book, for any of a variety of reasons. One possible reason is that the internal format of the e-book is faulty.

Some ineptepub users get a more-specific message like:

Error: File name in directory "OEBPS/BookTitle_chap-nn ... "

This error is typically caused by the same problem with the e-book and has the same solution, and it likewise means that ineptepub has been unable to decrypt the e-book.

What is the underlying cause? The internal format of an Adobe e-book is the same as that of any standard EPUB: it's a ZIP file. (See this Wikipedia article for more on the "open" EPUB format.) The various folders and files that make up any EPUB e-book, including any EPUB e-book that uses Adobe DRM, will have been compressed into a ZIP format archive. (The Wikipedia article says that the "OEBPS Container Format (OCF) 1.0," defines the standard for how the e-book's contents, which are a set of folders that contain a bunch of files, are put in a single "container" file. The OCF container file "collects all files as a ZIP archive.")

Whenever the e-book is read, it first has to be expanded from its ZIP form. Normally, this is done by the reader app. ineptepub in fact tries to do the same thing, and when the internal format is not as expected, an error results.

This happens a lot on Adobe e-books downloaded by Adobe Reader on a Mac. It doesn't seem to affect Adobe e-books downloaded by Adobe Reader on a Windows PC.

A workaround that sometimes helps is to unzip the original Adobe e-book, rezip it again while using the correct internal format, and then use ineptepub with the rezipped copy as its input file.

That can be done in the Mac OS X environment in the following way. For purposes of this example, I'll assume that the name of the downloaded .epub file is The_Poisonwood_Bible.epub. Like most downloaded Adobe e-books, it is typically located in ~/Documents/Digital Editions/. (The first node, ~, refers to your username's Home directory. To open that directory in Finder, simply type Shift-Command-H.)

Here are the steps you will need to take after opening ~/Documents/Digital Editions/ in Finder.

  1. Make a copy on the Desktop of The_Poisonwood_Bible.epub. (Working with a copy of the original file is a good idea because you don't want to take a chance on ruining the original.) The easiest way to copy the file to the Desktop is to drag it there from its original folder while holding down the Option key.
  2. Change the filename extension on the copied version of The_Poisonwood_Bible.epub from .epub to .zip. To do that in Finder, click on the file itself as it sits on the Desktop, click on the name of the file to highlight it, double-click on the .epub extension, and use the keyboard to change that extension to .zip. Or, if the extension is not visible, select the file and type Command-I — or choose Get Info from Finder's File menu — and edit the extension as it appears in the Name & Extension text box in the Get Info window; you simply replace .epub with .zip in the text box. Then close the Get Info window.
  3. If asked whether you really want to change .epub to .zip, select "Use .zip".
  4. You should now have a file on your Desktop called The_Poisonwood_Bible.zip. It is exactly the same file as The_Poisonwood_Bible.epub, but with a different filename extension.
  5. Double click The_Poisonwood_Bible.zip on the Desktop, so that Archive Utility on your Mac unzips it into a new folder, also on your Desktop. (If double clicking the .zip file doesn't start Archive Utility, you may have to right-click on the .zip file and, from the pop-up menu, choose Open With: Archive Utility. Or, if that option doesn't seem to be available in the pop-up menu, you can manually open Archive Utility from /System/Library/CoreServices/Archive Utility.app and choose Expand Library from the Archive Utility's File menu, then navigate to and select The_Poisonwood_Bible.zip on the Desktop in the standard file open dialog.)
  6. You should now have a new folder on your Desktop called The_Poisonwood_Bible. (Check that it contains a file called mimetype and two folders, META-INF and OEBPS. However, the contents of this folder are not really of interest to you.)
  7. Trash the The_Poisonwood_Bible.zip file on the Desktop, as you do not need the copy you made of your original e-book file any more.
  8. Download and unzip ePub Zip 1.0.2.app.zip, if you don't already have it. Double click it to expand it to ePub Zip 1.0.2 on the Desktop. (See above for what to do if double-clicking does not automatically expand a ZIP file in Archive Utility.)
  9. Drag the icon for the new The_Poisonwood_Bible folder to ePub Zip 1.0.2 and drop it onto the ePub Zip 1.0.2 icon. (Note: You must drag and drop your The_Poisonwood_Bible folder and not your The_Poisonwood_Bible.zip file, which should now be in the Trash anyway.) This drag-and-drop operation causes an AppleScript to run in the background. That AppleScript's function is to rezip your e-book the right way!
  10. You should at this point see a dialog box, "The_Poisonwood_Bible.epub created successfully," and a The_Poisonwood_Bible.epub file should magically show up on your Desktop. This The_Poisonwood_Bible.epub file is your correctly rezipped e-book.
  11.  Use the new The_Poisonwood_Bible.epub file as the input file to the ineptepub script.

If ineptepub still doesn't work, the problem lies elsewhere than with the internal ZIP format of the e-book.

If you use Windows, I imagine something like WinZip might offer the ability to do the same unzip/rezip trick. Sorry I can't give you detailed instructions about that. However, as I said earlier, this problem doesn't seem to affect Windows users very often, if at all.


Installing PyCrypto on Snow Leopard

The instructions for Mac OS X users that I gave in iPhone: Reading Adobe/EPUB eBooks are basically for users of the 10.5.x (" Leopard") version of that operating system. Many users are now on the 10.6.x ("Snow Leopard") version. Though my procedure basically works with Snow Leopard, some Snow Leopard users have had trouble with installing PyCrypto, the Python Cryptography Toolkit, in Step 3 of my original procedure.

The basic problem has to do with Snow Leopard's use of two Python versions. Version 2.5 is the one that you need to use to run the two Python scripts, ineptkey and ineptepub, that my instructions call for. ineptkey is the script that generates the adeptkey.der ADEPT key file, and as such does not require PyCrypto, but ineptepub, which is the script that actually decrypts the e-book, does need PyCrypto. PyCrypto is basically a set of library files Python needs to access when it runs ineptepub. This set of library files needs to be installed in the right place in your Mac's folder hierarchy. (When I refer generically to just plain ineptepub herein, I mean specifically the ineptepub_v2 applet that is created in Step 6 of my original instructions.)

Some users of Snow Leopard are finding that the PyCrypto installer is putting PyCrypto in

/Library/Frameworks/Python.frameworks/versions/2.6/lib/python2.6/site-packages/

That is the wrong place! The right place for it is

/Library/Python/2.5/site-packages/

Solution: after installing PyCrypto in Step 3 of the original procedure, simply copy all of the contents of the

/Library/Frameworks/Python.frameworks/versions/2.6/lib/python2.6/site-packages/

folder to

/Library/Python/2.5/site-packages/

In Finder, you can do this by highlighting all of the items contained in the Python 2.6 /site-packages/ folder and dragging them with the option key held down to the Python 2.5 /site-packages/ folder. (Be aware that you may have to create the latter folder manually in the /Library/Python/2.5/ folder.) This option-drag procedure makes a copy in the destination /site-packages/ folder of all of the highlighted items.

Among the items that need to be copied in this way are a folder called Crypto, along with all its contents, and a file called pycrypto-2.0.1-py2.5.egg-info. You should check to make sure that these items and (in the case of the Crypto folder) their contained items get properly copied. If the latter file is named pycrypto-2.0.1-py2.6.egg-info, change the name to pycrypto-2.0.1-py2.5.egg-info manually.

At that point, you will hopefully be able to proceed with Steps 4-7 of the original procedure.

This applies to a Mac, by the way, and does not apply to Windows computers.


Which Build Applet Version To Use?

Some Mac users have gotten to Step 6 and had trouble where I tell them to right-click the ineptepub_v2.py script file and choose Open With: Build Applet. This is intended to cause an application, Build Applet, that is included with Python on a Mac, to change the text file ineptepub_v2.py into an "applet" — a double-clickable application — called ineptepub_v2.

Accordingly, right-clicking on ineptepub_v2.py brings up a menu that has an Open With: item, and one of the sub-items of that item should be just plain Build Applet.

But some users double-click on the ineptepub_v2.py script file and see, for example, Open With: Build Applet 2.6.0 and Open With: Build Applet: 2.5.4!

How you handle this problem depends on whether you are on Snow Leopard or Leopard.

(1) If you are on Snow Leopard, just choose Open With: Build Applet: 2.5.n — i.e., choose the version of Build Applet that has 2.5 in it.

(2) If you are on Leopard, which natively supports only Python 2.5, you probably have at some point tried to install some version of Python 2.6. The two versions of Python, 2.5 and 2.6, are different enough that ineptkey and ineptepub won't work properly with Python 2.6 on a Mac, and yet it may be the case that the default version of Python that your Mac is trying to use is in fact 2.6.

Since Mac OS X 10.5.n normally doesn't come with any version of Python other than 2.5, your having at some point installed (or tried to install) 2.6 may gum up the works.

I recommend that you do a clean install of Mac OS X 10.5.n, using the original install DVD, and also re-installed the Xcode Developer tools, which is a package that contains Python 2.5, at the same time. This will eliminate any vestiges of whatever Python 2.6 installation currently inhabits your Mac. (See the next topic for how to do a clean install.)

On the other hand, you may feel you have some reason not to do a clean Mac OS X 10.5 install. If so, your best bet may just be to choose Open With: Build Applet: 2.5.n, just as if you were on Snow Leopard. The result may be a usable ineptepub_v2 applet. If so, fine. If not, a clean install may be your only recourse.


Doing a Clean Install of Mac OS X 10.5.n

I recommend doing a clean install of Mac OS X 10.5.n, if you are on Leopard and not Snow Leopard and you are having trouble with getting the procedure to work that I laid out in my original post, iPhone: Reading Adobe/EPUB eBooks. This of course applies to Mac users only, not users of Windows PCs.

Mac OS X 10.5.n is the version that I currently use on my Mac, and the instructions in my original post apply mainly to it. As far as I know, they also work for Snow Leopard users, except for the problem mentioned earlier in which PyCrypto gets installed into the wrong folder. Snow Leopard users who run into other problems with my procedure may want to consider doing their own clean install of Mac OS X 10.6.n, but I cannot advise them about that.

To do a clean install of Mac OS X 10.5.n:

Click on this link to an excerpt from Mac OS X: Missing Manual  — Leopard, by David Pogue. Scroll about halfway down the page to the The Clean Install ("Archive and Install") section. Follow the instructions there.

To install the Xcode Developer tools:

Using your OS X 10.5.n Install DVD, navigate to and then double-click Mac OS X Install DVD:Optional Installs:Xcode Tools:Xcode Tools.mpkg on that DVD. You will be led, by the installer program which opens, through the necessary steps to install the Developer Tools package.

(See "Complete Steps to Perform a Clean OS X Reinstall on Your MacBook Air" for a how-to on doing a clean install on a Mac that lacks a DVD drive.)

Once you have done the clean install, including the Xcode Developer tools, go back and right-click the ineptepub_v2.py script file and choose Open With: Build Applet, which now should be the only Build Applet choice available. The result should be a usable ineptepub_v2 applet.

(2) If you are on Snow Leopard, then there probably should be two versions of Build Applet on your Mac, just as there are two versions of Python! In this case, just choose Open With: Build Applet: 2.5.n. Again, the result should be a usable ineptepub_v2 applet.


Reading Your Adobe E-Books on an iPad in iBooks

My original post was about reading your Adobe e-books on an iPhone in a reader app like Stanza, but you can now read them on an iPad (or an iPhone or iPod Touch) using Apple's free iBooks app.

If the free iBooks app from Apple works the same on an iPad as it does on my iPhone — and I believe it does, except I'm not sure to what extent you'll be able to take advantage of the larger screen or the standard assortment of fonts with the "open" EPUB books you derive from Adobe DRM'ed e-books — you're definitely in luck.

If you follow the instructions in my original iPhone: Reading Adobe/EPUB eBooks post (and, if need be, the problem-solving workarounds detailed herein) then for each Adobe e-book you decrypt, you will wind up with an open EPUB e-book file that is no longer locked into the Adobe Reader world.

Add that file (it will have an .epub filename extension, whether or not that extension is visible in Finder) to your iTunes library. On a Mac, you can just drag the file's Finder icon to the iTunes icon in the Dock or to the Books icon on the left side of the main iTunes window. On Windows, you can use, I believe, similar shortcuts.

Once the file shows up in iTunes' Books section, you can, in iTunes, just sync your iDevice — your iPad, iPhone, or iPod touch, as long as that iDevice is running iOS 4.0, Apple's latest operating system for mobiles — and then you can read the book in the iBooks app on the iDevice!

I find that the Adobe-derived open EPUBs I have tried so far in iBooks work exactly the same as Apple-sanctioned e-books from the iBookstore.


Fixing Missing Cover Artwork on open EPUBs in iBooks

But I have found that some but not all of the e-books I have treated in the way just described do not show up as having a pretty cover, either in iTunes or in the iBooks app on the iDevice. Some have no cover at all. Here's how to fix that:

In your web browser, bring up Google and click on Google's Images tab at the upper left of the browser window. In the Google search box enter the name of your e-book book and, to make sure Google knows exactly what you're looking for, the author's name, both in quotation marks. For example:

"an echo in the bone" "diana gabaldon"

Then, click on Search Images. Most of the results you see will be images of the actual book's cover!

Click on any thumbnail image you like — the larger the dimensions in pixels of the actual image, the better. At the top of the page that comes up, click on See full size image. Then left-click on the image that comes up and select Copy Image. (This works on a Mac using Safari. If you are on Windows, or if you are using another browser, you may have to make adjustments.)

You now have a copy of the image on your Clipboard. In iTunes, click on Books on the left side of the main window, then select the book the cover goes with and choose the Get Info menu item. (Mac shortcut: Command-I.) Click on the Artwork tab, then click in the (typically empty) artwork box. Paste the Clipboard's contents there, and click OK.

In a few seconds you will see the new cover artwork show up in the main iTunes window. Now sync your iDevice in iTunes. Once you do that, the new cover artwork will show up on your iBooks library bookshelf!