New Blog
Hi all,
I’ve finally made a blog to describe my reversing endeavours. I’m not used to blogging, so I might not write so much.
Anyway, here’s the latest versions of my scripts:
eReader2html.py 0.03 : http://pastebin.com/m28dad336
mobihuff.py 0.03 : http://pastebin.com/mbdcaf3b
mobidedrm.py 0.02 : http://pastebin.com/m12cec25b
Darkreverser's Weblog 
February 13, 2008 at 5:40 pm
MobiHuff 0.02 does not work with the Huff dic compressed mobi file at http://www.mobileread.com/forums/attachment.php?attachmentid=10249&d=1202841000
It was generated using mobigen.
mobihuff produced an empty output file
February 14, 2008 at 7:36 am
Try with version 0.03
February 14, 2008 at 4:09 pm
version 0.03 extracts the html successfully, but it doesn’t handle the and tags.
February 14, 2008 at 4:27 pm
Version 0.2 of mobidedrm works fine with huffdic compressed files, but it produces an error with some files that worked fine in v.0.1. I sent you an e-mail with the details.
February 14, 2008 at 4:56 pm
It’s not designed to. It’s just made as a proof of concept to show how the compression algorithm works.
February 15, 2008 at 5:42 pm
I’ve processed well over 200 eReader books with v0.3 and received the following errors:
“Incorrect ereader version (error 2)” This was on 5 files.
“Index Error: String index out of range” This was on 1 file.
Many, many thanks for my huge amount of successful conversions.
February 15, 2008 at 7:27 pm
Thanks much for these scripts! Although I can use DRMed Mobi files on my iLiad, I hate the thought of not being able to read the books I’ve purchased in the future if my reader dies. (although to date the number of books purchased with DRM =1) Baen.com and other places that sell DRM-free books are great.
February 16, 2008 at 9:35 pm
Thanks for the eReader2Html script. It even works with dictionaries, simply they have different id: PDctPPrs.
What is not working: footnotes. Not only they are not tagged in the output file, but even their bodies are missing completely…
February 17, 2008 at 5:16 pm
Any chance of a mobi2html script? It would be nice to be able to get the text out of already DRM-free mobi files.
February 17, 2008 at 6:42 pm
nevermind, found one at: http://www.ida.liu.se/~tompe/mobiperl/
February 22, 2008 at 9:17 pm
Is there a way, short of brute force, to determine if a Mobi or Ereader file is DRMed?
February 25, 2008 at 7:31 pm
I am trying to get mobidedrm to work , unfortunately i am a noob on programming an d terminal. Can anyone geve me some feedback on the error message I got seen below, thnx in advance, Andre awi100_hotmail.com
mobidedrm.py:49: FutureWarning: hex/oct constants > sys.maxint will return positive values in Python 2.4 and up
crc = (~binascii.crc32(s,-1))&0xFFFFFFFF
MobiDeDrm v0.02. Copyright (c) 2008 The Dark Reverser
Traceback (most recent call last):
File “mobidedrm.py”, line 176, in ?
file(outfile, ‘wb’).write(DrmStripper(data_file, pid).getResult())
File “mobidedrm.py”, line 144, in __init__
found_key = self.parseDRM(sect[drm_ptr:drm_ptr+drm_size], drm_count, pid)
File “mobidedrm.py”, line 101, in parseDRM
pid = pid.ljust(16,”)
TypeError: ljust() takes exactly 1 argument (2 given)
February 28, 2008 at 9:28 pm
When running mobodedrm v0.02 I recieve Error: no key found. maybe the PID is incorrect
when I put in a bogus pid I get the error: invalid PID checksum
I would expect an error with a bogus PID, but not with the real PID
August 13, 2010 at 2:47 pm
did you get a response to this? i’m having the same problem
September 15, 2010 at 3:41 pm
I am getting the same error. I just started to get the error with new books from Kindle. Did anybody get a fix??
Thanks
September 15, 2010 at 4:39 pm
Hi Brian,
That version of MobiDeDRM (v0.02) is quite old and is no longer used. The correct version is version 16. You can grab it from Apprentice-Alf site or from the links to tools-v1.9.zip provided many times in this forum (simply search within this web page).
That said if your books come from a Kindle that is running firmware 2.5 or later, there is NO way to remove the DRM anymore as Amazon has changed things inside your Kindle that has to do with DRM.
The only solution now that is guaranteed to work is to use “Kindle For PC” (Kindle For Mac will NOT work) or use Kindle for iPhone, iPad, or iPodTouch.
I highly recommend the Apprentice-Alf blog (a Google search term), where in the comments, there are detailed instructions for removing the DRM from both of those two approaches.
February 28, 2008 at 9:30 pm
this is a mobi book (.prc) purchased from mobipocket.com
February 28, 2008 at 10:00 pm
Can these decoder tools work for muliple PIDs? Mobipocket now allows up to 3 (or 4?). Or is it case of running it through the decoder twice?
February 28, 2008 at 10:48 pm
^ Answered my own question ~ just using one PID seems to strip them all or at least opens the file. A quick check shows 3 out of 4 worked fine. The 4th I got the same error as Johnny. Brilliant little utility though ~ I’ve been worried about being stuck with unusable DRM’d books if mobi went belly up!
March 3, 2008 at 10:06 am
Andre: Upgrade your Python installation. From your log, you seem to be running a version pre-2.4, while 2.5 is the current version.
Johnny/Jason: I think that only the first PID in a file works for decryption. (I get “no key found” errors for PIDs not in the file, and “invalid checksum” when it’s in there but not the first one.)
March 3, 2008 at 3:18 pm
How do I get the first PID? I installed mobireader on one PC and when I opened the file I went to the question mark – about and copied the PID from there.
March 4, 2008 at 8:04 am
@Atkinson ~ I thought of that at the time and tried it with both PIDs, neither worked. The vast majority of files do convert fine though, it just seems to be the odd one.
March 4, 2008 at 10:41 am
I;m about 50% way through my collection ~ 212 have converted fine, 73 haven’t. All the ones that fail show basically this series of error messages. The books come from a variety of mobi retailers. Quite often some in the books series, bought at the same time will convert okay. Anyway here’s the error message ~ any help to shed light on what could be the issue would be appreciated 🙂
C:\>mobidedrm.py Patterson_1sttoDie.prc C:\zzznodrm\Patterson_1sttoDie.prc 44E6J
JC84G
MobiDeDrm v0.02. Copyright (c) 2008 The Dark Reverser
Decrypting. Please wait…
Traceback (most recent call last):
File “C:\mobidedrm.py”, line 176, in
file(outfile, ‘wb’).write(DrmStripper(data_file, pid).getResult())
File “C:\mobidedrm.py”, line 159, in __init__
extra_size = getSizeOfTrailingDataEntries(data, len(data), extra_data_flags)
File “C:\mobidedrm.py”, line 74, in getSizeOfTrailingDataEntries
num += getSizeOfTrailingDataEntry(ptr, size – num)
File “C:\mobidedrm.py”, line 64, in getSizeOfTrailingDataEntry
v = ord(ptr[size-1])
IndexError: string index out of range
March 4, 2008 at 1:50 pm
my problem was because I was trying to de-code a copy of the original. I had to use the one located in My Documents\My Ebooks
Hope this helps some people
March 4, 2008 at 3:50 pm
Well my problem @10:41 post was cured by using v1 for those which failed v2. I’ve tested all 409 files and every one worked which is pretty brilliant! Now, I’m much happier to buy DRM’d books in the knowledge that I could still access even if mobipocket had a melt down. Thanks Dark Reverser ~ 10/10!
March 9, 2008 at 10:31 pm
Jason (or anyone else) where do I get V1? I am having the same issue.
Thanks Dark Reverser
March 10, 2008 at 7:57 am
Try searching Google for MobiDeDrm v0.01, tell Google to include all results previously excluded, then look at cached copy of page
March 10, 2008 at 1:32 pm
@ atimson
thnx, After upgrading to OSX 10.5 (python is included) the script worked fine, thnx!
March 19, 2008 at 8:24 am
Unfortunately the links to the files on pastebin are reported to be expired or removed!
Is there any mirror out there?
THX in advance
March 20, 2008 at 10:34 am
ereader2html 0.03 has been uploaded at http://pastebin.com/f1fc790cb
March 24, 2008 at 8:46 pm
MobiDeDrm v0.02
March 24, 2008 at 8:48 pm
Trying one more time..
MobiDeDrm v0.02 uploaded at:
http://pastebin.com/m398d265a
March 26, 2008 at 8:58 pm
I’m new to this game so I would appreciate some help.
Can you give me an example of the syntax to use when converting mobipocket files.
The instructions given when I load the script state mobidedrm infile.mobi outfile.mobi PID. How do I point it in the right direction when all my mobipocket files are .prc files.
Thamks in advance.
March 30, 2008 at 5:56 pm
Thanks for the great tools!
Could you repost mobihuff v03? The pastebin entry for it has expired.
Thanks again!
March 31, 2008 at 12:06 am
MobiDeDrm v0.03
March 31, 2008 at 12:07 am
opps v0.02 is the latest, thanks 🙂
April 1, 2008 at 3:49 am
Like tdprofffitt said, please repost mobihuff v03 for those of us who missed it.
Thanks!
April 2, 2008 at 6:50 pm
I have a PRC file of a book I bought from BooksOnBoard. The book is no longer available for download. I very much want to be able to read it on my Amazon Kindle. I’ve tried using MOBIDEDRM. (Versions 0.01 and 0.02) I get an error message: “no key found. maybe the PID is incorrect. I’ve verified the PID at BooksOnBoard, and on my Palm T/X. Am I missing a step?
Thanks.
April 5, 2008 at 12:34 am
Both books I used mobihuff on truncated the first 15 or so pages of the book. The cover, TOC, and about half of the first chapter. Both were run through mobidedrm first but that shouldn’t affect the content should it?
Both deDRMed files crashed MobiReader but the DRMed files read fine. Guess I’ll have to stick with LIT books.
April 18, 2008 at 12:00 am
The de drmer is a fantastic tool, I don’t mind buying books but
I hate the idea of not being able to read MY books when my
ereader goes away!
I have a problem with FBReader, a de-DRMed book will frequently
fail to open with an “Unknown Compression Method” error. About
half of my fixed books have this problem…
Any Ideas?
Jerry
April 18, 2008 at 12:02 am
The de drmer is a fantastic tool, I don’t mind buying books but
I hate the idea of not being able to read MY books when my
ereader goes away!
I have a problem with FBReader, a de-DRMed book will frequently
fail to open with an “Unknown Compression Method” error. About
half of my fixed books have this problem…
Any Ideas?
Jerry
April 19, 2008 at 1:28 am
Hi DarkReverser,
I want to thank you for your great work and wish you the very best for the future.
But one last question is on my mind:
MobiDeDRM v0.2 works fine, but if I read the decrypted files with my MobipocketReader, the text (the characters) are replaced with weird signs a/o letters or symbols.
Am I doing something wrong? Or is it a “Python-Error”?
Using:
Python v2.5 (Eng.)
MobiDeDRM v0.1 & v0.2
non-english *.prc-files (like French, Spanish or Sweden)
Any ideas?
So long, and thanks for the scripts … 😉
April 23, 2008 at 10:17 pm
Any working links to mobihuff 0.03 and mobidedrm 0.02?
Please?
April 30, 2008 at 6:55 pm
[…] search (search for MobiDeDrm and Rapid share, for now at least). These nice scripts were developed here. It is easy to use (assuming you can use the command line), you need your PID (as described above) […]
May 1, 2008 at 9:41 pm
Any working links to mobihuff 0.03 and mobiledrm 0.02 would be greatly appreciated.
Thanks
May 2, 2008 at 1:51 am
You also want MobiDeDRM.py version 0.01 as well. It works in cases where 0.02 does not.
May 12, 2008 at 2:14 pm
Hi DarkReverser,
Would you please repost all three versions of MobiDeDRM as the links say that they have expired.
Thanks
May 13, 2008 at 1:11 pm
[…] do this a script is needed from here. The script is no longer there, but I’ll put the source code up […]
May 13, 2008 at 9:23 pm
Thanks Mr Reverser (posting this to the right page now)
I get: “Error: no key found. maybe the PID is incorrect”
even on books that “don’t have DRM,” but other tools like mobi2oeb or mobihuff say that the file is DRM encoded.
I think maybe the files are DRM’d to some generic PID that all mobi-browsers test, but I don’t know what this PID might be,
Try the file at:
http://www.lds.org/handheld/pdafiles/pocketpc/MobipocketNewTestamentStudyGuide.zip
May 16, 2008 at 11:04 pm
These scripts as well as the Kindle DRM scripts have been compiled to EXE files and are here:
http://www.demonoid.com/files/details/1479930/775222/
May 22, 2008 at 7:53 pm
Found them: http://rapidshare.com/files/102854939/4DeDRMfiles.zip.html
June 1, 2008 at 2:39 am
[…] New Blog « Darkreverser’s Weblog (tags: drm ebooks convert ereader scripts mobile html) […]
June 4, 2008 at 5:53 pm
Have tried pastebin for the mobidrm but seems to have expired can it be reposted please
June 6, 2008 at 12:02 pm
liverpool;
it’s here:
http://www.demonoid.com/files/details/1479930/775222/
June 21, 2008 at 5:01 pm
Anyone know where the ereader2html script can be found. Tried searching but cannot find it. Thanks.
June 21, 2008 at 9:11 pm
I’ve had success with Mobidedrm before, now all I’m getting is …
MobiDeDrm v0.02. Copyright (c) 2008 The Dark Reverser
Removes protection from Mobipocket books
Usage mobidedrm infile.mobi outfile.mobi PID”
June 27, 2008 at 8:22 am
Version 0.03
# This is a python script. You need a Python interpreter to run it.
# For example, ActiveState Python, which exists for windows.
#
# Changelog
# 0.01 – Initial version
# 0.02 – Huffdic compressed books were not properly decrypted
# 0.03 – http://www.mobileread.com/forums/showpost.php?p=202684&postcount=76
import sys,struct,binascii
class DrmException(Exception):
pass
#implementation of Pukall Cipher 1
def PC1(key, src, decryption=True):
sum1 = 0;
sum2 = 0;
keyXorVal = 0;
if len(key)!=16:
print “Bad key length!”
return None
wkey = []
for i in xrange(8):
wkey.append(ord(key[i*2])<> 8)) ^ byteXorVal) & 0xFF
if decryption:
keyXorVal = curByte * 257;
for j in xrange(8):
wkey[j] ^= keyXorVal;
dst+=chr(curByte)
return dst
def checksumPid(s):
letters = “ABCDEFGHIJKLMNPQRSTUVWXYZ123456789”
crc = (~binascii.crc32(s,-1))&0xFFFFFFFF
crc = crc ^ (crc >> 16)
res = s
l = len(letters)
for i in (0,1):
b = crc & 0xff
pos = (b // l) ^ (b % l)
res += letters[pos%l]
crc >>= 8
return res
def getSizeOfTrailingDataEntries(ptr, size, flags):
def getSizeOfTrailingDataEntry(ptr, size):
bitpos, result = 0, 0
if size <= 0:
return result
while True:
v = ord(ptr[size-1])
result |= (v & 0x7F) <= 28) or (size == 0):
return result
num = 0
flags >>= 1
# while flags:
if flags & 1:
num += getSizeOfTrailingDataEntry(ptr, size – num)
flags >>= 1
return num
class DrmStripper:
def loadSection(self, section):
if (section + 1 == self.num_sections):
endoff = len(self.data_file)
else:
endoff = self.sections[section + 1][0]
off = self.sections[section][0]
return self.data_file[off:endoff]
def patch(self, off, new):
self.data_file = self.data_file[:off] + new + self.data_file[off+len(new):]
def patchSection(self, section, new, in_off = 0):
if (section + 1 == self.num_sections):
endoff = len(self.data_file)
else:
endoff = self.sections[section + 1][0]
off = self.sections[section][0]
assert off + in_off + len(new) LLLBxxx32s’, data[i*0x30:i*0x30+0x30])
cookie = PC1(temp_key, cookie)
ver,flags,finalkey,expiry,expiry2 = struct.unpack(‘>LL16sLL’, cookie)
if verification == ver and cksum == temp_key_sum and (flags & 0x1F) == 1:
found_key = finalkey
break
return found_key
def __init__(self, data_file, pid):
if checksumPid(pid[0:-2]) != pid:
raise DrmException(“invalid PID checksum”)
pid = pid[0:-2]
self.data_file = data_file
header = data_file[0:72]
if header[0x3C:0x3C+8] != ‘BOOKMOBI’:
raise DrmException(“invalid file format”)
self.num_sections, = struct.unpack(‘>H’, data_file[76:78])
self.sections = []
for i in xrange(self.num_sections):
offset, a1,a2,a3,a4 = struct.unpack(‘>LBBBB’, data_file[78+i*8:78+i*8+8])
flags, val = a1, a2<<16|a3<H’, sect[0x8:0x8+2])
extra_data_flags, = struct.unpack(‘>L’, sect[0xF0:0xF4])
crypto_type, = struct.unpack(‘>H’, sect[0xC:0xC+2])
if crypto_type != 2:
raise DrmException(“invalid encryption type: %d” % crypto_type)
# calculate the keys
drm_ptr, drm_count, drm_size, drm_flags = struct.unpack(‘>LLLL’, sect[0xA8:0xA8+16])
found_key = self.parseDRM(sect[drm_ptr:drm_ptr+drm_size], drm_count, pid)
if not found_key:
raise DrmException(“no key found. maybe the PID is incorrect”)
# kill the drm keys
self.patchSection(0, “” * drm_size, drm_ptr)
# kill the drm pointers
self.patchSection(0, “\xff” * 4 + “” * 12, 0xA8)
# clear the crypto type
self.patchSection(0, “” * 2, 0xC)
# decrypt sections
print “Decrypting. Please wait…”,
for i in xrange(1, records+1):
data = self.loadSection(i)
extra_size = getSizeOfTrailingDataEntries(data, len(data), extra_data_flags)
self.patchSection(i, PC1(found_key, data[0:len(data) – extra_size]))
print “done”
def getResult(self):
return self.data_file
print “MobiDeDrm v0.02. Copyright (c) 2008 The Dark Reverser”
if len(sys.argv)<4:
print “Removes protection from Mobipocket books”
print “Usage:”
print ” mobidedrm infile.mobi outfile.mobi PID”
else:
infile = sys.argv[1]
outfile = sys.argv[2]
pid = sys.argv[3]
data_file = file(infile, ‘rb’).read()
try:
file(outfile, ‘wb’).write(DrmStripper(data_file, pid).getResult())
except DrmException, e:
print “Error: %s” % e
July 4, 2008 at 7:11 pm
Can you repost you version 0.03 with indentation and normal quotation marks, in order to be “paste ready” for a python script?
Thanks
NB: if you can’t put spaces, replace them with special chars to be replaced by spaces.
July 5, 2008 at 2:50 am
Line 24 has two open parenthesis and four close parenthesis
wkey.append(ord(key[i*2]) 8)) ^ byteXorVal) & 0xFF
July 8, 2008 at 12:21 am
I ran across a problem with the ereader2html script and footnotes. As mentioned previously, the footnotes are not being decrypted with the rest of the book. I think this might have something to do with PML. From what I gather, the text of the footnotes are at the end of the pml file, but are ignored as part of the actual book text. I’m guessing that the self.num_text_pages (line 290) is only returning the information up to the first footnote. So, Darkreverser, any chance you’d take a look at this? (please?)
Thanks.
July 8, 2008 at 9:46 pm
8) B) just testin something
July 8, 2008 at 9:47 pm
8) 😉 ); just testin something
July 11, 2008 at 8:57 pm
Any of the mobi files at: http://lds.org/handheld/newarchive/0,18495,344-81-2,00.html
e.g. http://lds.org/handheld/pdafiles/pocketpc/famproc.zip
fail to convert, however the book has NO drm and works fine on any mobi reader.
The huff decode says: Error: The book is encrypted. Run mobidedrm first
But mobidedrm can’t easily be run as the book is not encrypted as far as I know. If I provide the PID of my ebook reader I get: Error: no key found. maybe the PID is incorrect
My only guess is thta maybe it is drm’d to a generic key that all readers have…?? Any clues on this?
August 5, 2008 at 8:52 am
The ‘0.03’ version given above references my first set of patches, but doesn’t fix all problems. Since then I’ve got more used to Python and found out a lot more about the Mobipocket format.
I now recommend some different patches for the 0.02 MobiDeDRM code. Detailed instructions on how to apply the patches can be found at
http://www.mobileread.com/forums/showpost.php?p=222142&postcount=125
HTH.
Paul
August 10, 2008 at 5:39 pm
Thanks for eReader2HTML!
I buy all my books legally, but cannot read them on my expensive BeBook without your help.
Still struggling to get mobi2HTML to work – with out an Kindle PID.
August 15, 2008 at 10:32 am
hmmm, this code does not work with copy/paste
must check8)it
test 8) test
September 4, 2008 at 1:03 am
Ah, thanks for the files! Sorry, this is a really basic python question since I am unfamiliar with it, but how do I get it to run the script?
September 5, 2008 at 7:56 am
Came in just to say a big THANK YOU to Dark Reverser for his eReader2html.py script. I had a few dozens of eBooks I bought over the years for my Palm and now I can read them on my N800, or wherever I want. Thanks again!
PS: I’ve shared my copy of the above scripts in the Donkey P2P network, search for “4DeDRMfiles.zip”.
May the good code live forever 🙂
September 17, 2008 at 4:17 pm
Hello
Please can you update the links ?
What is the latest version of mobidedrm.py ? and where can I find it ?
Thanks
September 24, 2008 at 5:30 pm
Thanks so much DarkReverser! I can now read my legally bought eReader eBooks on my Sony PRS-505!
I have re-uploaded the scripts to pastebin, as I had a hell of a time trying to find them.
eReader2html.py 0.03 : http://pastebin.com/f140eea7f
mobihuff.py 0.03 : http://pastebin.com/f35777523
mobidedrm.py 0.02 : http://pastebin.com/f2a681132
September 24, 2008 at 9:14 pm
Hi Dark Reverser,
Excellent work, judging by all the posts.
Unfortunately I had trouble with a book I bought and tried to deDRM to subsequently convert into pdf to read on my new Archos.
I got error messages with all three versions of mobiddrm, v0.01, v0.02 and the patch 0.04:
MobiDeDrm v0.01. Copyright (c) 2008 The Dark Reverser
Traceback (most recent call last):
File “MobiDeDRM001.py”, line 148, in ?
file(outfile, ‘wb’).write(DrmStripper(data_file, pid).getResult())
File “MobiDeDRM001.py”, line 117, in __init__
found_key = self.parseDRM(sect[drm_ptr:drm_ptr+drm_size], drm_count, pid)
File “MobiDeDRM001.py”, line 75, in parseDRM
pid = pid.ljust(16,”)
TypeError: ljust() takes exactly 1 argument (2 given)
MobiDeDrm v0.02. Copyright (c) 2008 The Dark Reverser
Traceback (most recent call last):
File “MobiDeDRM002.py”, line 176, in ?
file(outfile, ‘wb’).write(DrmStripper(data_file, pid).getResult())
File “MobiDeDRM002.py”, line 144, in __init__
found_key = self.parseDRM(sect[drm_ptr:drm_ptr+drm_size], drm_count, pid)
File “MobiDeDRM002.py”, line 101, in parseDRM
pid = pid.ljust(16,”)
TypeError: ljust() takes exactly 1 argument (2 given)
MobiDeDrm v0.04. Copyright (c) 2008 The Dark Reverser
Traceback (most recent call last):
File “MobiDeDRM004.py”, line 181, in ?
file(outfile, ‘wb’).write(DrmStripper(data_file, pid).getResult())
File “MobiDeDRM004.py”, line 149, in __init__
found_key = self.parseDRM(sect[drm_ptr:drm_ptr+drm_size], drm_count, pid)
File “MobiDeDRM004.py”, line 103, in parseDRM
pid = pid.ljust(16,”)
TypeError: ljust() takes exactly 1 argument (2 given)
Thanks again,
Sassie
September 24, 2008 at 9:18 pm
Oh, I forgot to mention, I’m using preinstalled Python 2.3.4 on Linux at the command line.
September 29, 2008 at 3:48 pm
I downloaded a fresh copy of Mobireader onto my PC and just downloaded a mobi book via booksonboard, when I use the PID fro Mobireader I get the following error
python mobidedrm1.py deaduntildark4.prc deaduntildark.mobi AAAA11A$1A
MobiDeDrm v0.01. Copyright (c) 2008 The Dark Reverser
Error: invalid PID checksum
(key changed to A for letter 1 for number)
I tried using v1 and v2 of mobidedrm and got the same result… did Mobi update keys?
September 30, 2008 at 10:57 am
Kier, you have to escape the $ sign. Like this:
python mobidedrm1.py deaduntildark4.prc deaduntildark.mobi AAAA11A\$1A
That should do the trick.
September 30, 2008 at 6:33 pm
And I’m a python programmer, I should have seen that, doh! worked like a charm!
October 1, 2008 at 8:45 pm
I would like to use ereader2html to convert a book that I myself compiled back into HTML. The book is unencrypted. Is there something I should use for the “name” and “credit card number” values to reflect this? It doesn’t seem to work if I just leave them blank.
October 2, 2008 at 10:42 pm
You have to have Pyhton 2.5 installed and the code posted here is useless as Pythin requires indents. 0.03 will not work properly as it has a bug in the updated code. 0.04 that I have has fixed the bug and works. But, you’ll have to have a proper 0.02 and then go to MobileRead and find the updated code for 0.02 to convert that into 0.04.
October 4, 2008 at 1:38 am
Many thanks Dark Reverser, JS Wolf and Paul Durrant.
Since there are so many posts, I just wanted to be explicitly clear how to make version 0.04:
you take the original version of mobidedrm 0.02 (eg from demonoid) and then simply apply the latest patch instructions referred to above by Paul Durrant on August 5 (and shown at mobileread) and voila you have version 0.04
Otherwise, some (including me) may get confused in thinking that you had to apply the patch to an updated 0.02 rather than the original 0.02
October 17, 2008 at 10:34 am
Help! If I try the following:-
python mobidedrm1.py name_of_file.prc name_of_file.mobi P3DA11A\$1A
The checksumPID function returns a PID but with the last two characters after the “$” different from the orginal. Therefore, the script fails with
“invalid PID checksum”.
October 29, 2008 at 7:20 pm
I got an error with mobidedrm 0.02 – I m decrypted 20 books or so but 3 or so books came back with the same error:
C:\Books>mobidedrm “A Forest of Stars.azw” “A Forest of Stars.prc” 3ASBJCF*KF
MobiDeDrm v0.02. Copyright (c) 2008 The Dark Reverser
Decrypting. Please wait…
Traceback (most recent call last):
File “C:\Python25\Tools\scripts\mobidedrm.py”, line 176, in
file(outfile, ‘wb’).write(DrmStripper(data_file, pid).getResult())
File “C:\Python25\Tools\scripts\mobidedrm.py”, line 159, in __init__
extra_size = getSizeOfTrailingDataEntries(data, len(data), extra_data_flags)
File “C:\Python25\Tools\scripts\mobidedrm.py”, line 74, in getSizeOfTrailingDataEntries
num += getSizeOfTrailingDataEntry(ptr, size – num)
File “C:\Python25\Tools\scripts\mobidedrm.py”, line 64, in getSizeOfTrailingDataEntry
v = ord(ptr[size-1])
IndexError: string index out of range
November 1, 2008 at 8:20 pm
I’ve used the ereader2html script successfully on nearly 300 books. However, I’ve got a couple of dictionaries that I’ve had no luck with.
Back in February, Kurt commented:
“Thanks for the eReader2Html script. It even works with dictionaries, simply they have different id: PDctPPrs.”
Can anyone help me figure out how to apply that info to make it work for me?
I also found this recommendation elsewhere:
With the ereader2html script it’s possible to convert a secure eReader pdb file into plain html. The original version doesn’t work with dictionary files but you can open the script in an editor, look for the “Invalid file format” error and change the raise into a print:
print ValueError(’Invalid file format’)
I tried that with both dictionaries, but to no avail. I’d really appreciate any help anyone can provide.
Many thanks.
November 1, 2008 at 8:38 pm
Regarding my previous comment, I was finally able to de-code both of my dictionaries using ereader2html. I had to change the script as noted (change raise ValueError to read print ValueError), and I had to use the original downloaded versions of the books, not copies. In my experience, Mobipocket books had required use of originals for de-coding, but not ereader.
Thanks to all for sharing your info and knowledge.
November 20, 2008 at 1:08 am
I bought a book from mobipocket that I want to get into my Sony 505.
I have tried to use the python script and have gotten close, but no go. I have been using MobiDeDRM.py with the command python MobiDeDRM.py body.prc body.mobi device_pid and get an error stating Error: invalid PID checksum.
Likewise I have tried to run Removes protection from Mobipocket books Usage: mobidedrm infile.mobi outfile.mobi PID. I want to kept the file a PRC file so that I can convert it to the Sony format in the Calibre software. I would be most helpful, by book has been taken hostage.
Thank you,
December 19, 2008 at 6:36 am
Latest MobiDeDRM.py v0.05 posted here:
http://pastebin.com/m57062830
Changes by Paul Durrant:
http://www.mobileread.com/forums/showthread.php?t=34190
December 21, 2008 at 5:27 am
Where do we get the PID from?
January 10, 2009 at 2:39 pm
This is a great script, thanks very much darkreverser. I have successfully stripped many files. However I have one that it doesn’t work with; I get a message saying “Invalid file format”. This file is readable using Mobipocket software, and Calibre recognises it as a valid mobi file. Any ideas on how I can investigate further to solve the problem?
January 10, 2009 at 9:39 pm
I am using ereader2html and cannot get it to work. I don’t know if I am entering the command wrong or what. I am using version .03 linked above in Bob’s post.
Thanks so much for any help you can give me.
John
January 10, 2009 at 10:35 pm
OK, I finally figured it out and it works great. Thanks so much for this great tool.
John
January 13, 2009 at 6:36 pm
Hi to all – have used the ereader2html script to convert a good number of Ereader files that I purchased legitimately. However, having left it for a few weeks when I try to run the script I am getting the Python error message that “ereader2html.py returned exit code 0”. Clearly I’m doing something slightly different from before and any hints or suggestions would be very welcome
January 13, 2009 at 6:42 pm
Worked it out muself. For other users who may come this way I was putting ereader2html in the arguments box – you only need the book name etc
January 19, 2009 at 9:57 pm
Hi,
I could use some help. While running the MobiDeDRM script I get the following error:
data_file = file (infile, ‘rb’).read()
NameError: name ‘file’ is not defined
Do you have any suggestions as to what might be going wrong? I’ve tried versions 1, 2, and 3 on several ebooks. All with the same result.
January 22, 2009 at 2:11 am
Went to get latest version and it is not there could someone help? need 0.05
January 22, 2009 at 4:54 am
Sorry I have absolutely no idea about computer stuff appart from clicking buttons in windows programs. Could someone give me an idiot’s guide for getting from the ereader2html/txt file that I download to the python script file that I run? I have Activepython installed, but other than pasting the text in to the python interactive shell (which doesn’t work) I have no idea! I have found instructions on running the ereader2html.py file from the command window, but how to get to the ereader2html.py is a puzzler!
Thanks, Jodie.
January 22, 2009 at 1:43 pm
OK I have the latest version I think, but here is my issue. I convert to a DRM free ebook and I have no images within the ebook. Am I doing something wrong?
January 29, 2009 at 6:14 am
MobideDRM.py 0.05 — http://pastebin.com/f31e88df6
January 29, 2009 at 6:15 am
MobiDeDRM.py 0.05 — http://pastebin.com/f31e88df6
This is the latest version.
February 2, 2009 at 2:52 am
This is so great. Now if only there were a version that worked with Topaz files so I could get rid of the ridiculous fonts Amazon forces on Kindle users with all those Kindle books they only sell in Topaz format … 🙂 Come on, Darkreverser, we have faith in you!
February 3, 2009 at 10:02 am
Been using ereader2html successfully for ages. Have now just bought my first non-DRM ereader book (unintentional) and came up with following message:
Error: incorrect eReader version 10 (error 1)
Help…
February 6, 2009 at 6:02 am
i ran 05 version in python 3.0 (should we only use 2.5?)
C:\python30> python mobidedrm05.py file.prc new.prc KKKKKKKK\$KK
i get this error
file “mobededrm05.py”, line 22
print “Bad key length!”
^
SyntaxError: invalid syntax
what am i doing wrong?
i got this same error for versions 1 and 2.
thanks
February 6, 2009 at 2:51 pm
leave out the \
February 7, 2009 at 8:47 pm
FOSSIL: I’ve got the same problem. It just stopped working today. If I comment out the (error 1), I get an (error 2) message.
February 7, 2009 at 10:10 pm
Update: ereader2html.py 0.03 is still working. However, by some act of major irony, UN-encrypted eReader books are providing an endless string of (error 1) wrong version messages. 🙂
Any suggestions?
February 7, 2009 at 10:55 pm
If the eReader file is not DRM laden, then eReader2HTML.py will not work to convert the eReader file to HTML.
February 8, 2009 at 6:58 am
Turned out not to be a problem. The old “pdbshred” program generates a WTF? message, but turns it into entirely readable PML. All I need to do now is turn the PML into HTML and I’m gone.
Thank you.
February 14, 2009 at 8:53 am
I’m using “MobiDeDrm v0.05” and can’t get it to work. I don’t have much experience with this. I’m trying to get a file which opens on my Windows Mobile phone in Mobipocket to run on my XO/OLPC running FBReader. It says it needs a PID so I’m giving it the one from my phone and getting the error below. Am I doing something wrong? It looks like it should be rather straightforward…
C:\books>mobi.py Lion.prc Lion2.prc **********
File “C:\books\mobi.py”, line 22
print “Bad key length!”
^
SyntaxError: invalid syntax
Also, I just noticed, this error comes up because of a check for len(key)!=16. The key I am giving it is only 10 characters long because that’s what MobipocketReader gives me. Is that the wrong one?
February 14, 2009 at 9:13 am
Just wanted to say how great your scripts are, very much appreciated. Let’s me read my books on whatever device I want, instead of being stuck. Thanks.
February 16, 2009 at 2:12 am
Thank you for your wonderful scripts. I use them to convert books I buy so I can read them on my kindle. Without your scripts, I would buy fewer books – I would be restricted to one seller.
February 16, 2009 at 12:18 pm
10 is the correct length for a Mobipocket PID and the copy of 0.05 that I posted works fine with a 10 character PID. Maybe try downloading it again?
February 19, 2009 at 3:41 am
Looks like Adobe’s ePub DRM may have been broken
http://i-u2665-cabbages.blogspot.com/2009/02/circumventing-adobe-adept-drm-for-epub.html
February 19, 2009 at 8:46 am
Will encryption type 1 documents ever work? I just found out I deleted a DRM’d file, which was meant for my PDA. (I still got the one that works on my previous PDA…). I can’t DeDRM it, as it is a encryption type 1…
February 28, 2009 at 3:16 am
Phil, your problem could be that you might be using Python 3.x. MobiDeDRM.py does not work with Python 3.x. It only works with Python 2.5.x or 2.6.x.
March 12, 2009 at 5:22 pm
Okay. The perennial question: How does one use ereader2html for ereader, PDB books that do NOT have DRM? I have a bunch of free, non-DRM ereader files that I would like to use with FBReader, but FBReader can’t read ereader files. It will work with HTML files, mobi files, epub files, and some others, so I need to convert them somehow. There IS a pdb2txt utility, but that doesn’t always work for me, and when it does, it doesn’t retain the formatting.
So, IS there a way to get ereader2html to work with non-DRM ereader files? Any other ideas?
Thanks!
March 15, 2009 at 5:00 pm
I have some VERY OLD (peanut press) files that I would like to read again. They seem to be too old for ereader2html.py (version 259 and earlier). Any ideas?
March 17, 2009 at 7:58 pm
Looking for the tool mobidedrm and clicking on the link above for your latest script seems to bring up a broken/deleted page?
March 17, 2009 at 8:51 pm
Look at my post on January 29, 2009.
March 18, 2009 at 4:20 pm
I have created a program that incorporated the mobidedrm (v1.0) and the kindlefix scripts into a gui and made the whole thing a windows exe. No python needed. Right now it only converts .azw drm books and does the mobi conversion so that a mobi drm can be read on the kindle. If someone is interested in helping me test it further I’d need a SN from a kindle and a .azw drm’d book that was bought with it. I also need a .mobi drm book to test the other function.
March 18, 2009 at 4:22 pm
Sure, I can provide everything you need. E-mail me to tell me how to get them to you.
March 18, 2009 at 4:40 pm
chorpler,
Send mail to the temp address msg4bill-temp@yahoo.com and we’ll talk
March 20, 2009 at 2:00 am
Thanks so much for providing these scripts.
Just wondering if you’ve made any progress on the footnote issue that others have mentioned above.
Great work, and thanks again.
March 29, 2009 at 6:38 pm
i am a total idiot when it comes to computer stuff. so a little patience, please.. so i can’t get python to run mobidedrm v.01. i typed in the script and then what? do you type something in to get it to run? how exactly do i get this to work? i just want to remove the damned DRM’s on my mobipocket books to convert it onto my sony reader. thanks
March 29, 2009 at 6:41 pm
oh and where would i actually go to start removing DRM’s from the books?
March 30, 2009 at 5:47 am
I got .05 pieced together, worked like a charm. Seems like the pieces/parts are all over the place to put this together, so I put together an archive that includes all the related files in one place:
http://rapidshare.com/files/215202363/Mobipocket_DRM_Tools.zip
This puts together Igor’s files including kindlepid.py, Darkreverser’s MobiDeDRM.py .05, and the Applescript GUI from mobileread.com forums (non-handicapped, MobideDRM already installed). Also updated the various readme files to help get the newbie up and running more quickly.
March 31, 2009 at 4:00 pm
I’ll be the first to admit I’m new to this and not at all familiar wiht programming. I do have rather a lot of ereader books that I’d like to use on a Kindle. I’ve installed Python 3.0 for windows. when I run python I get a terminal window. I then type ereader2html.py c:\destination folder\ ‘My Name’ ‘number’. I get “invalid sytax” every time. I don’t know what that means. I’ve tried changing the syntax somewhat or added python to the command line, but still no luck. I wonder what I’m doing wrong. I’ve tried moving all the files to the same directory, etc. Can anyone give me a tip? It all seems so simple when reading about it. Thanks for any help you can give me.
March 31, 2009 at 4:03 pm
Sorry, I do also include the name of the book as well with the pdb ending.
April 3, 2009 at 6:30 pm
Thanks for the version 5 of drm. I incorporated it into a VisualBasic program that I created. If anyone would like to try and run it send email to nodrm.50.shootmel8ter@spamgourmet.com. You can use this on a windows system without loading python.
April 7, 2009 at 9:48 pm
mm… funny.
April 8, 2009 at 2:13 am
Jim, I know absolutely nothing and can’t get the script to work, either. But I do understand that you should try to use Python 2.5 or 2.6, not 3.0. Hope this helps. (And if it does, and you get it to work, maybe you can help me!!)
April 10, 2009 at 6:12 am
Thank you for a lot of PRC files I have converted.
But I cannot use the Mobiderm code to decrypt the following .PRC file.
http://www.mediafire.com/?e3hbragmmmk
April 11, 2009 at 9:21 am
Jane, can you tell me what’s your exact problem ?
Contact me on this email : xr17hkup7ldli3n@jetable.com ,and I will help you
April 13, 2009 at 7:08 am
If you check out this thread over at Mobileread:
http://www.mobileread.com/forums/showthread.php?t=34322
You’ll see that Calibre now supports the ability to use something like MobideDRM as a plugin. Can anyone post some instructions here to pluginize MobiDeDRM, or better yet just post a link to a plugin?
April 14, 2009 at 4:00 am
Dang, the last 7 hours and I still cannot be sure how this works! Can anyone email me at pilotrite AT aol to give me some guidelines? This is so frustrating. I just want to be sure I can read my books I’m paying for if the kindle 2 ever goes belly up. And just in case text to speech disappears, id like to be able to convert and still use. HELP!
I finally gave up tonight after getting the following error… Error: invalid encryption type: 0. UGGGHHHHH! I’m so frustrated! HELP! (Oh, and is there a fix for topaz or azw1 files tooo yet? Of course I can’t get the azw one to work yet!)
April 20, 2009 at 2:50 am
Does anyone know where to get mobidedrm05? Thanks!
April 20, 2009 at 2:53 am
Nevermind I found it a few posts above mine 🙂 I used mobidedrm04 all the time before but now I get a bad key length error and it doesn’t run… 02, 04, and 05. Help? Thanks!
April 20, 2009 at 3:02 am
Figured it out, doesn’t work with Python 3, so I downloaded Python 2.6. All systems go 😀
April 30, 2009 at 4:50 pm
Wow, meriweather. you da person. thanks.
May 14, 2009 at 4:10 am
Meriwether that is one heck of a AppleScript. Thank you very, very, very much. Without the help of you and the others I would have returned my Kindle a few days after buying it and understanding the DRM issue. I have purchased now about 30 books from Amazon and Fictionwise. But I would have been stupid to have bought any if DRM could not be stripped. As an early adopter of technology I am pretty darn sure my next ereader will show up in the next year or eighteen months and will not be Kindle (might be but I doubt it). Amazon and the other companies and sites are being silly to try to block this. If I wanted to pirate there are twenty torrent sites I could use. But I’m not a pirate. But neither do I want to be a patsy and have to buy the same books over and over and over again as years go by.
May 16, 2009 at 7:34 am
Thanks, but not my Applescript – thank pdurrant from the mobileread forums. I just packaged it all together.
May 17, 2009 at 12:34 pm
Hello,
I’m a PRS-505 owner. I bought yesterday 3 books at mobipocket store, thinking that it’would be easy to convert them to lrf with Calibre. I didn’t realize that it would come with DRM.
Then looking on the web for a solution I finally arrived here. I installed pyton and dowloaded the 4DeDRMfiles . My problem is that I’m a complete newbie in this kind of things and after spending a long while trying differents ways, I don’t understand at all how it works. Does-il exist somewhere a tutorial explaining step by step how to proceed?
Thank you very much & and have a nice day!
May 17, 2009 at 7:39 pm
Hello again,
after many investigations, I finally found “the way”.
Thank you again 😉
May 23, 2009 at 6:33 pm
I have recently purchased “Foundation (Isaac Asimov)” in the Kindle edition, only to discover the TTS is disabled. I use TTS a lot in the car, as I have a long commute. Copyright law being on my side, I can use the mobi2mobi program with the “–delexthtype 404” option to remove the disable TTS flag. When I put this file BACK on the Kindle (still containing the DRM) it works fine, and even Whispersync still works between my K2 and my iPhone.
However, I like to use mobidedrm on ALL my AZW purchases and convert them to plain mobi for back up, in the event that the whole Kindle thing goes belly up. In effect, I refuse to acknowledge that I’m only LEASING an eBook for Amazon. I paid for it, I think it’s fair use to strip it and convert it to something I can read on my Sony or Stanza or whatever.
Here is the problem. Mobidedrm creates a corrupt file from Foundation.AZW…it crashes Mobipocket reader, mobi2oeb (Calibre), etc. It does this when I use mobidedrm before the TTS removal, or after…makes no difference. It’s almost as if Amazon has found a way to defeat mobidedrm.
Anyone have a clue about this?
May 23, 2009 at 6:41 pm
Answering my own post here, I found an OLDER version of mobidedrm (version 1, actually compiled into an exe mobiddrm01.exe) and it removed the drm on Foundation.AZW just fine.
June 4, 2009 at 1:02 pm
Have you been able to create a mobidedrm script for AZW1/Topaz files. If not do you anticipate their being one any time soon.
June 5, 2009 at 9:38 am
Hi,
I recently came across some mobi e-books that have TEXtREAd as file format. v.0.5 isn’t able to remove the DRM. I tried editing the code to let it accept TEXtREAd files and that worked, but it still can’t handle the file. I read elsewhere that it was once possible to remove DRM from this kind of files with an edited tool from the iRex devices, but it’s impossible to get that now. Do you know of a way to remove the DRM of TEXtREAd mobi files? Perhaps it’s very easy to write code for it, but my python skills aren’t really great 🙂 Thanks for any help!
Sander
June 23, 2009 at 10:17 pm
I purchased a *.mobi e-book recently, that cannot be cleaned from the DRM infestation with any available mobidedrm version; mobi2mobi sees the following metadata:
./mobi2mobi my-precious-ebook.prc
Database Name: XXXXXXXX
Version: 0
Type: BOOK
Creator: MOBI
Seed: 7307789
Resdb:
AppInfoDirty:
ctime: -1968466898 – Fri Aug 16 19:13:01 1907
mtime: -881913230 – Tue Jan 20 17:06:10 1942
baktime: -2082844800 – Thu Dec 31 23:34:39 1903
—————————————————
FIRST IMG Record Id: 7307783
—————————————————
Image record index: 519 (90 x 120)
START IMAGE INDEX: 519
COVER IMAGE INDEX: 519
PDHEADER Version: 2
PDHEADER Length: 1059157
PDHEADER NRecords: 518
PDHEADER Recsize: 2048
PDHEADER Unknown: 190892
MOBIHEADER ciflg: 65535
MOBIHEADER ciptr: 65535
MOBIHEADER doctype: MOBI
MOBIHEADER length: 228
MOBIHEADER booktype: 2 – BOOK
MOBIHEADER codep: 1252
MOBIHEADER uniqid: 2853551425
MOBIHEADER ver: 4
MOBIHEADER exthflg: 64
MOBIHEADER language: 9 – 9 – 0 – ENGLISH –
MOBIHEADER xtradata: 27651 0x3237363531
COVEROFFSET: 4294967295
THUMBOFFSET: 4294967295
EXTH doctype: EXTH
EXTH length: 84
EXTH n_items: 5
EXTH item: 201 – CoverOffset – 4 – 0xffffffff
EXTH item: 202 – ThumbOffset – 4 – 0xffffffff
EXTH item: 203 – hasFakeCover – 4 – 0x0000
EXTH item: 2 – drm_commerce_id – 14 – EBOOKMALL_0306
EXTH item: 3 – drm_ebookbase_book_id – 5 – 18668
LONGTITLE: XXXXXXXX
LASTID: 7307786
Were there any changes to mobidedrm since the v0.05 release?
June 30, 2009 at 3:49 am
Hint about ereader2html.py:
Footnotes each have there own page. The first page is at byte 44 of the header (like the first image page is at 24). Number of footprints / sidebar items is at byte 46.
Encoding is same as text.
From this it should be easy for any of you python coders to add this feature.
This only works for type 272 files, of course.
July 3, 2009 at 10:09 pm
Trying to find ereader2html.py but the link in the main blog page no longer works. Having no luck with Google. Anyone have a current link to it?
July 7, 2009 at 7:25 pm
Thanks for this site and this scripts which had given a second life to my ebooks bought from Mobipocket.
I use a MacBookPro and a iPhone with the Stanza reader.
Today, I have bought 5 ebooks from Mobipocket for the first time since two years.
The applescript works fin with 2 of them.
But I have a problem with the three others, I think recently published on Mobipocket site.
The applescript seems work fine, but when I try to open the unlocked book with Stanza, I get this message :
Could not load book
java.lang.ArrayIndexOutOfBoundsException : 2182
It’s the same with the three others scripts Mobidedrm 1, 2 and 5, on terminal windows on Mac.
What’s the matter ?
Is it a new defense or new format of ebooks from Mobipocket ?
Thanks for your response.
July 15, 2009 at 2:54 pm
Oh Dark Reverser — Please, please please advise as to AZW1/TPZ files. Anything available or in the works?
July 20, 2009 at 7:05 am
@Babar, Stanza is pretty buggy with lots of Mobipocket titles (nothing to do with MobideDRM), try using Calibre with the decrypted files instead. If you’re using Stanza on the iphone you can use Calibre’s content server to send books to the device.
For Mac users also using their iphone to purchase kindle title from Amazon, there is s new tool out there for you (not mine):
http://github.com/tradewinds/TradeWinds/tree/master
This automatically extracts the books from your iphone/ipod backup directory, simultaneously decrypting them.
July 21, 2009 at 1:31 am
Found and have been using ereader2html and the ER application OK on my PC. Stripped a bunch of books I had purcahsed (I do pay — once — for books). But today Barnes and Noble opened their new ebook store. The ER application fails on the books I have from them. Does anyone know how it could be made to work or why it is failing? PLEASE reply, thanks!
July 21, 2009 at 1:41 am
Further Tradewinds referenced above seems to have been taken down already. I am batting zero taody. Damn DRM anyway. Is this Tradewinds thing available anywheres else? Couldn’t find any place else, also doesn’t help that “tradewinds” means so many things.
July 21, 2009 at 1:59 am
Oooops. I apologize — I was making the same, stupid typographical error every time I was entering a book title in the script. The good news is that Ereader2html DOES work on the new Barnes and Noble ebooks. I feel like a bit of an idiot but would rather correct my error then post misinformation here!
July 22, 2009 at 5:37 am
The actual download link for Tradewind’s application is:
http://cloud.github.com/downloads/tradewinds/TradeWinds/TradeWinds.dmg
July 28, 2009 at 11:18 am
@ meriwether
Thanks for your answer
I tried Calibre, but it can’t open these books entirely. It fall 3/4 of the book.
Now, I have the same problem with the quadrilogy of Twilight, bought by Nulmerilog, one of the 4 book is unreadable by Stanza, after the decruyption. The three others are OK ?
Thanks for the link for TradeWinds, i’ll try it.
August 3, 2009 at 3:40 am
Just thought I’d drop a line to those interested, I made an easy, GUI, batch-capable wrapper for the mobidedrm python scripts. Nothing fancy, but it’s got its uses. http://www.mediafire.com/file/dlmkdzzzm2n/eBookUtility0.2.zip This version has an exe wrapper and is pretty windows specific but if anyone wants it for a *nix platform I can put out a platform independent jar. I know some other work has been done around this that’s independent of the python scripts, this is nothing like that, mostly just a convenience / user friendliness issue. I created it for family members who own kindles and aren’t quite so tech savvy, and I like to use it for batch stripping. This is pretty much my first time releasing it to an audience beyond immediate family, so feedback is welcome.
August 6, 2009 at 12:43 am
I’ve created a patch for eReader2html003.py to enable decryption of “version 259” ebooks.
Only had one example to work with so can make no guarantees, but it’s worth a try right?
Major props to DarkReverser for laying the groundwork and to the author of palmdrm.txt for providing a vital clue.
eReader2html003-to-003b.patch:
http://pastebin.com/f147cc228
remember, authors have to eat too. so if they’re still alive, buy their ebooks…
..AND THEN SET THEM FREE!!!!
December 10, 2009 at 5:03 am
Hi, I’m trying to use this for my 259 error but keep getting a syntax error on line 1. I downloaded the top section of code from pastebin. Here is the error:
C:\eReader> ereader2html003b.py AgathaRaisin.pdb “c:\ereader\AgathaRaisin.pdb””myname” 1234123412341234
File “C:\eReader\eReader2html003b.py”, line 1
*** eReader2html003.py 2009-08-05 20:22:19.000000000 -0400
^
SyntaxError: invalid syntax
Can you please help?
December 10, 2009 at 5:04 am
Above – the arrow is pointing to the second asterisk on line 1 on the actual command screen.
August 7, 2009 at 9:17 pm
I see that there are several people reporting Mobipocket files that won’t decode properly, or that decode with earlier versions but not 0.05.
I thought that with 0.05 I had got the decoding working on all Mobipocket DRMed files.
I would be interested in seeing examples of any file that won’t decode with 0.05.
pdurrant (not the Dark Reverser)
August 9, 2009 at 9:11 pm
Hey, revx, thanks for the kinder, friendlier code! Do you have tips for how to change the default paths the .exe is checking? My python install isn’t on C, etc. Thanks!
August 10, 2009 at 2:36 am
@Paul Durrant et aliis
I’m grateful for everyone’s heroic efforts here. Paul, I today bought a dictionary from Mobipocket that won’t pass through 0.05. (My first purchase from Mobipocket. The program worked fine with an .azw file created from a .prc from another vendor.) I’d be happy to send it to you, but it’s a big file. Tell me if you want it.
The error is either “no key found. maybe the PID is incorrect,” when I pass my kindlepid.py-generated PID to the program or “invalid PID checksum,” if I pass it the PID without the “*” or if I substitute a “$” for the asterisk.
As previously reported by many, Mobipocket stripped the asterisk from a Kindle PID when I registered it for the download (as I set up my account). Presumably, though, it embedded some kind of key in the book as it DRMed it. Can we assume that key is based on either that 9-character PID (original minus the asterisk) or on a 10-character PID it created from that original PID plus some other character substituting for the asterisk? Can’t someone figure out what manipulation they perform on the mutilated Kindle PID?
I’m working on a Mac, so the desktop PID route is not open to me.
August 11, 2009 at 7:00 pm
@egret
Yes, good point on the Python check, I’ve simply removed that check for now, I’ll reimplement it later when I’ve got more time. In the mean time, here’s an updated version and it just assumes you’ll treat it nicely and have a working version of python installed. (i.e. there’s an association so if you type python at command line it’ll open python)
http://www.mediafire.com/file/m4mhin2wmg2/eBookUtility0.3.zip
August 12, 2009 at 12:10 am
Oh, thank you! I had the association working and was head desking pretty hard after that. 🙂
Where the heck does it expect to find the PID? I don’t have a kindle, just a mobi PID…
August 12, 2009 at 12:24 am
Let me report a further odd occurrence in case it provides a clue to someone who understands the guts of these programs better than I do.
(Recall: I’m faced with the now banal–but not trivial–problem of MobiDeDrm 0.05 not running to completion because of Mobipocket stripping the “*” out of kindlePID-generated PIDs at purchase.)
While waiting for some sort of encouraging reply, I was flailing around trying to do something. I was examining the not very imaginative hypothesis that Mobipocket was substituting another character for the asterisk when generating the key for the .prc file. So I was substituting characters myself, hoping I’d get lucky and one would work. No such luck, of course.
But I did find two other characters which passed the checksum test, though they went on to fail whichever next test returns the error “No key found. Maybe PID is incorrect.” The asterisk–ASCII 2A–passes the checksum. But so do ASCII 2F (forward slash) and ASCII 8D (unassigned). I can’t imagine how such different values can all pass the checksum. Maybe it’s just an arithmetic fluke. (For what it’s worth, 2F is one-third of 8D, but nether has any evident arithmetic relation to 2A.) But maybe, since it seems anomalous, it’s a clue to someone. So I pass it along. We live in hope.
August 12, 2009 at 12:55 am
@Chris
Shouldn’t be too hard, here’s all you need to do, I believe.
Create a file in the ebookutility folder called ebook.props (plain text file) and put the following in it:
PIN=
serial=
August 12, 2009 at 12:57 am
Woops, my last comment got a clipped a little, to clarify:
PIN=YourPID (i.e. PIN=8KL*KDJF)
serial=RandomJunk (You don’t have a kindle, so don’t worry about it just give it something bogus, 1234 for instance.)
August 12, 2009 at 1:00 am
Danke!
August 12, 2009 at 1:59 am
Hi Revx — I’ve been using the cripts on my Mac and also the Applescript utility without problems. Today I saw that you had a GUI for Windows and downloaded the ebookutility program hoping to get my Windows machines working on it also. However, the ebookutility program will not recognize my Kindle’s serial number. When I enter it the program gives an invalid serial number error and will not generate a PID for the DRM functionality to work. However, using the same serial number with kindlepid.py on my Mac generates a valid PID and on the Mac no problem stripping the DRM. Any ideas why that serial number is failing in ebook utility? Thanks!
August 12, 2009 at 2:10 am
@Neil
Not sure why it’s not working, but all it’s doing is making a call to kindlepid.py, so if the one on your mac works for your serial, pull it off and throw it in the ebookutility directory and see if it things works, if so, you should be in business. Also, if that still doesn’t work for some bizarre reason you can follow the directions I gave a couple posts above and just feed it your generated PID manually. Not pretty but it would work.
August 12, 2009 at 11:04 pm
FWIW, I used the unexpected arithmetic behavior regarding checksums that I mentioned in my last post to some advantage today. I went to another vendor for the DRMed dictionary I wanted. This vendor didn’t strip the asterisk from my Kindle PID. It simply wouldn’t accept it as a valid PID. (Nor would it accept it if I changed the asterisk to a dollar sign.) Because I was stuck, I used one of the other characters which passed the checksum test in place of the asterisk, and the vendor accepted it! When I then ran mobidedrm v. 0.05 on the resulting .prc file I downloaded from the vendor, it successfully stripped the drm from it. I have no idea whether this strategy would work across the board, nor whether it would have worked successfully with Mobipocket.
August 14, 2009 at 2:53 am
Can someone please repost the scripts as pastebin no longer has them. Use gist.github.com or something. thx.
August 17, 2009 at 12:06 am
Thanks for the awesome GUI revx. Makes life much easier. Now if we could only dedrm topaz files……
August 19, 2009 at 7:20 pm
To all who are interested, I release a new version of eBookUtility, the link is here
Change people care about: after hearing a few rumblings about it I allowed PRC files to be used as source files as well as AZW, it makes perfect sense to do so and probably should have been done in the first place, but I only had the kindle store on the brain when I first wrote it. 🙂
Things that might break: I reimplemented the python check in a much less idiotic way, if it gives you trouble this time around, however, I’ve included directions in the readme which explain how to bypass it.
August 20, 2009 at 10:01 am
I’m probably stupid but it is not working for me, I put my mobipocket reader PID and a serial (one junky because I don’t have Kindle) saved it as book.props.txt but it still asks Kindle serial number. What am I doing wrong
August 20, 2009 at 10:57 am
You have to save the file as “ebook.props”, read the ReadMe.txt file that comes in the zip, it gives you directions for that.
August 21, 2009 at 3:13 pm
Can’t find a copy of eReader2html.py v3 anwhere- no longer on pastebin
August 24, 2009 at 9:10 pm
Thanks for the help, tried it as suggested and works like a charm.
August 25, 2009 at 10:07 pm
I’ve just installed Mobipocket Reader on my PC (not having a portable reader of any sort yet), and purchased a .MOBI book (not Amazon). The MobiDeDRM python scripts fail to work on it. In looking at the script, it LOOKS like it’s using “ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789” as the legal characters in a PID, and the PID I get from the Mobipocket Reader program has a ‘$’ in it. Perhaps that’s why the scripts fail? Maybe Mobipocket has modified it’s latest versions to break the DeDRM scripts?
August 26, 2009 at 1:40 am
No, several of my PIDs have had dollar signs in them too, and they work fine for decryption. You might have to put the PID in quotes or single-quotes … are you using version 0.05 of mobidedrm.py?
August 26, 2009 at 5:12 am
Yes, I’ve tried versions .05, .02 and .01, tried with and without quotes. All give the same … err…. ummm….
shit.
I swear, I tried all three .py scripts, all three gave the same “PID checksum error” (or whatever). I tried it with quotes and without quotes, single quotes and double quotes, I tried rebooting Windows, I tried running the scripts on Vista and on XP, I tried it sitting down, standing up and standing on my hands. All gave the same error.
Then I tried it again just now, to get the exact error message… and it worked.
Sigh.
Never mind.
August 29, 2009 at 7:33 pm
I’m releasing a new version of eBookUtility with a serious bug fix which prevented the program from decrypting entire directories correctly. (Thanks to all who provided feed back) It also fixes a number of other minor bugs, such as correctly reporting when it can’t find python. The next version (0.5) will have more advanced threading and concurrency controls.
You can get the bug fixed version of (0.4a) at http://www.mediafire.com/file/mykmlzwz3gm/eBookUtility0.4a.zip
August 30, 2009 at 6:46 am
For those of u who successfully strip the DRM from your mobi file, can you please explain in steps by steps what to do? I have no idea how to do this and would greatly appreciated if someone can help me out. I probably need python and mobidedrm right? Where do I get those?
August 30, 2009 at 8:07 pm
The ereader2html003b.py code is on Pastebin here:
http://pastebin.com/f629041a4
Is anyone working on DeDRM for Amazon’s TOPAZ files?
They are a real pain in my butt…..
Cheers
August 30, 2009 at 8:21 pm
Superstitious:
First go your Mobireader, HELP|ABOUT and write down your mobi PID, It will be something like:
xxxxxxx$xx
Then put the mobi or prc file you want to decrypt in the same directoty as the dedrm script. I am assuming that you have python 2.6 installed in the c:\python26 directory.
Type the following:
c:\python26\python.exe mobidedrm05.py infile.mobi outfile.mobi xxxxxxx$xx
If all goes well, you will have an exact copy of the original (DRM’d) file with the name of outfile.mobi, but with no DRM.
Cheers
August 31, 2009 at 4:12 am
http://www.mediafire.com/file/mtwndjtzgzl/eBookUtility0.4b.zip Pushing out another major bug fix.
September 3, 2009 at 2:43 pm
Perfect. Thankyou. Precisely what I needed.
September 6, 2009 at 10:09 pm
Anyone know of a plugin for Calibre that makes use of these scripts?
September 7, 2009 at 12:35 pm
I’ve found one or two Mobipocket ebooks where 0.05 decrypts, but with some corruption in the text.
The fixes to make 0.05 into 0.06 are as follows:
Change the the print line (line 174) to read
print “MobiDeDrm v0.06. Copyright (c) 2008 The Dark Reverser”
Replace lines 76 to 82 with:
[tab]testflags = flags >> 1
[tab]while testflags:
[tab][tab]if testflags & 1:
[tab][tab][tab]num += getSizeOfTrailingDataEntry(ptr, size – num)
[tab][tab]testflags >>= 1
[tab]if flags & 1:
[tab][tab]num += (ord(ptr[size – num – 1]) & 0x3) + 1
[tab]return num
Add to the comments at the top:
# 0.06 – And that low bit does mean something after all 🙂
Note that [tab] should be replaced by actual tab characters. Python is sensitive to the amount of indentation on each line.
September 7, 2009 at 2:49 pm
Help. I ran out of dumb luck. I downloaded another drmed mobipocket from the vendor I used in my Aug. 12 (11:04 p.m.) report. But this time, when I ran mobidedrm 0.05 on it, I got the error message: “No key found. Maybe the PID is incorrect.” When I used the PID as I’d altered it to get the vendor to accept it–which involves substituting a forward slash for the asterisk–mobidedrm ran, but hung (as I discovered when I control-C’ed out of it) at line 185.
Also sinister: when I tried running kindlefix.py on it, that too failed, reporting back:
“Encryption: 2
“Mobi publication type: 2
“Mobi format version: 6
“PID doesn’t match this file”
That was the first time I’d seen any mention of a “Mobi format version 6.” Is there anything that can be done about this? Thanks for any help.
September 9, 2009 at 11:27 pm
Hey Paul, having a problem:
error: compiling ‘MobiDeDRM06.py’ failed
SyntaxError: invalid syntax (MobiDeDRM06.py, line 79)
I posted the code here, can you take a look?
http://pastebin.com/f6cbeb3f9
Cheers
September 9, 2009 at 11:30 pm
Sorry, line 80 not 79, made a typo.
Cheers
September 9, 2009 at 11:46 pm
Never mind, got it working. Some weird invisible characters in the code I pasted from your post. I retyped those lines by hand and now it compiles.
If anyone want’s it the full MobiDeDrm06 script is here:
http://pastebin.com/f6590ef1d
Cheers
September 10, 2009 at 7:30 am
There are two faults in the code brutusbum posted on 9/9 at 11:56:
Line 145 that reads
extra_data_flags, = struct.unpack(‘>L’, sect[0xF0:0xF4])
should read
extra_data_flags, = struct.unpack(‘>H’, sect[0xF2:0xF4])
and line 77 that reads
flags >>= 1
should be deleted – it shouldn’t be there at all.
Paul
September 10, 2009 at 10:16 pm
That’s the problem with posting just updated lines, the code is only as good as what you begin with. I wondered about line 77, but both versions I found of version 0.05 had extra lines near the beginning of the file. I will update and repost.
Version 0.0.6 Dedrm with the above corrections here:
http://pastebin.com/f29c0eef0
Cheers
September 13, 2009 at 6:52 pm
I tried the latest script and this is the result with the correct PID:
MobiDeDrm v0.06. Copyright (c) 2008 The Dark Reverser Error: no key found. maybe the PID is incorrect.
September 13, 2009 at 7:17 pm
@Drmed
It’s not a problem with the script. Unless you’ve previously had success with earlier versions?
If your PID comes from a Kindle, then some eBook sellers won’t accept the PID (with a $) as valid for Mobipocket books.
Use a different PID (e.g. from Windows Mobipocket Reader), re-download, and all should be OK.
September 13, 2009 at 8:13 pm
Thanks that worked 🙂
September 14, 2009 at 9:15 pm
Just wanted to say THANK YOU for this excellent program. Total python n00b but was able to figure it out with help from some forums. Looking forward to having my books available on Android, at least until ereader puts out an Android-compatible reader. Thanks again!
September 18, 2009 at 5:17 pm
After a few weeks of development, I’m releasing version 0.5 of eBookUtility, in both a Windows exe wrapped version and a platform independent jar.
Windows:
http://www.mediafire.com/file/kwiwzwmntgj/eBookUtility0.5-win.zip
Platform Independent:
http://www.mediafire.com/file/wjmr1j2zioh/eBookUtility0.5.zip
Changes:
eBookUtility 0.5 – 09/18/2009
—————-
– Complete revamp of the underlying threading system and external process call methods
– All selection method bugs appear to be squashed
– NEW: Thread monitoring tab with progress bar
– NEW: Only dedrm errors are output instead of all output
– NEW: 5 threads run concurrently to process selected book(s) (assuming you’ve selected 5+ books)
September 18, 2009 at 7:33 pm
Revx — Just wanted to thank you for version .5 of the Ebook Utility and to let you know that the .jar version works great on my Intel iMac. I wasn’t sure if it was MobideDRM version 6 or not included so replaced it just to be sure but am betting that was an unneeded step.
September 18, 2009 at 7:57 pm
@Thothamon
I hadn’t actually gotten around to patching my copy of MobiDeDRM to 6 yet, so that was indeed a needed step. I’ll get it patched for the next release. 🙂
September 19, 2009 at 4:40 pm
Using the ereader2html003b.py code my version 259 file was decrypted but I got the following warning:
eReader2html.py:11: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
import struct, binascii, zlib, os, sha, sys, os.path
I don’t know if this is important or not…
Thanks for making this work for version 259!
October 9, 2009 at 6:12 am
I have successfully used ereader2html003b.py, but the book I’m trying to convert is translated from Spanish, and therefore has a number of accented characters. Unfortunately, these aren’t coming through correctly. Many of them come through as Cyrillic characters.
Any suggestions?
October 9, 2009 at 6:16 pm
Thanks for the great programs.
I have used ereader2html v3 for my legit ebooks without any problem until this file. File is freebie from BN, have stripped other files from BN & FW with no problem. Using ereader2html with python25. Re-downloaded file in case orig was bad but get same message. File opens & reads fine in BN reader.
This is the error message I get with this file: (re-typed as I don’t know how to copy from command prompt)
File “e2hv3.py”, line 490, in
convertEreaderToHtml
File “e2hv3.py”, line 477, in convertEreaderToHtml
file.write
File “e2hv3.py”, line 417, in process
r = self.next
File “e2hv3.py”, line 349, in next
c = self.s[p+1]
IndexError: string index out of range
My python skills are just at the beginning level so I’m not quite sure where to start first. It’s a free book so if it’s corrupt somehow, I’m not too upset. Out of 100+ books, it’s the only file that I’ve not been able to convert.
Thanks for any help
October 22, 2009 at 3:10 am
I’ve downloaded the 0.03b version of the ereader2html python script, but I’m having some trouble getting it to work. When running the script I’m getting the error “incorrect ereader version 10 (error 1).” Unfortunately I don’t know what this means, nor what information to provide to help in the trouble shooting. I’ll be happy to answer any questions.
October 22, 2009 at 7:29 pm
Thanks for MobiDeDRM! It works great on my purchased books!
October 23, 2009 at 9:03 pm
I have two Amazon files (azw) that give me an error stating that it has encounter invalid encryption: type 0.
I am using Active Python 2.6, and both mobidedrm005 and mobidedrm006 with no differences.
I would be happy to provide these files (and the PID) to someone who wants to try to figure out what the heck is going on.
Many thanks,
October 23, 2009 at 9:09 pm
Encryption type 0 means no encryption. Not all eboks in the Kindle store have DRM applied to them.
October 26, 2009 at 4:50 am
Is there a script available that would just remove the drm from ereader .pdb books without converting to html?
October 26, 2009 at 5:50 pm
I’m releasing the newest version of the program. I’ve updated the ReadMe to include in detail instructions on how to solve the Python path issue, please read it if you’re having problems!
Windows:
http://www.mediafire.com/file/m0mmu4mfvzn/eBookUtility0.6-win.zip
Platform Independent:
http://www.mediafire.com/file/ymnn1zm2xmj/eBookUtility0.6.zip
Changes:
eBookUtility 0.6 – 10/26/2009
—————-
– Not much new in this release, I haven’t had any reports of bugs, and only one feature request.
– NEW: Added a checkbox to allow overwriting of the source files — many people didn’t like the ‘-stripped’ name addition, and it was easy enough to implement, so now you have it. 🙂
– NEW: The DeDRM scripts provided with the utility are now 0.06 so there’s no need to upgrade.
October 26, 2009 at 8:37 pm
Thanks to some prompt feedback I was able to catch a severe bug in the 0.6 release and pulled it immediately. I’ve reuploaded it and the new files can be found http://www.mediafire.com/eBookUtility here.
October 26, 2009 at 10:37 pm
Thanks for the update!
October 29, 2009 at 9:48 pm
Hi–anyway to do batch conversions with the mobi script?
P.s:
Note that for non-drm conversions, in response to a question, Amber Palm Converter, is a simple point and click solution for prcs and pdbs…
October 29, 2009 at 11:33 pm
Question–with the ereader script…it keeps telling me the sha module is deprecated, use hashlib instead.
Alas, I have no idea what this means, let alone how to fix it. Python 2.6 and 3.1 both installed…Any help would be appreciated.
October 30, 2009 at 1:17 am
A sincere thanks to revx for eBookUtility.
October 30, 2009 at 10:53 am
@Mitch
My utility is specifically designed for doing batch stripping with the MobiDeDRM scripts, check a couple posts up for the info.
As to your second question, it’s saying that a method used in the script is deprecated, meaning that newer functionality has been released in the Python language that replaces the functionality that the script currently uses. It’s not an error, it’s a warning. (i.e. when the script was written — perhaps I don’t know this for a fact — the sha module was probably current and what was available to coders and in later versions of Python hashlib was released)
October 30, 2009 at 11:40 am
Ok, thanks….is there a way to fix this? I have both Python 2.6 and 3.1 installed
October 30, 2009 at 7:31 pm
@Mitch
You’d have to install an old version of Python to get rid of the warning, or rewrite the script using the new module. Regardless, it’s just a warning. It’s not stopping it from working, it doesn’t need fixing. It’s purely informational.
October 31, 2009 at 11:01 am
HELP!!!
I have been reading your blog for 2 days and have tried everything that you have suggested to your readers. I have tried the advise and step by step of other posts. I have never tried something like this and am pretty sure that somewhere along the way. I have done something wrong or am leaving out a step. Please help. I have 249 ebooks from Mobi and would love to be able to convert them to a different format.
Thanks
October 31, 2009 at 11:42 am
Ah…Revx thanks..unfortunately am still doing something wrong. Sorry to bother you,but I suspect this is something simple in the syntax as it keeps telling me the creditcard or name is wrong.
This seemed to me as it was in the script
ereader2html.py book.pdb \”Joe Blow\” 12341234″
Also tried the usage syntax that appears on the screen at times and not in the script of
ereader2html.py book.pdb “Joe Blow” 12345678
None of those seem to work. On one occasion I got it to actually create a directory (Joe) but then it told me the name and credit card number were wrong. Re-downloaded file to make sure of that, same result.
I’m sure this is simple, unfortunately something is not clicking. Sorry for the time waste.
October 31, 2009 at 11:54 am
@Cyndi
Try joining Mobileread and asking nicely there. I’m sure someone will send you a private message and help out.
Don’t forget to mention what operating system you’re using (Windows, Mac or Linux)
October 31, 2009 at 1:19 pm
Thanks for the hints on eReader2html and footnotes. I would be happy to take a look at this one and try to add support.
I have added:
self.num_fnote_pages = struct.unpack(‘>H’, r[46:26+2])[0]
self.first_fnote_page = struct.unpack(‘>H’, r[44:24+2])[0]
and modified getText to simply dump any footnotes out after the main text
if self.num_fnote_pages > 0:
for i in xrange(self.num_fnote_pages):
r += zlib.decompress(des.decrypt(self.section_reader(self.first_fnote_page + i)))
and in a vain attempt to get internal anchors and links for footnotes done the following:
def ilinkPrinter(link):
return ‘‘ % link
and added to html tags this entry
‘Fn’ : (ilinkPrinter, ‘‘),
but I really can’t test and fix anything without a DRM encoded eReader file to see more.
I would be happy to buy a cheap or free one from someplace if anyone knows of one that definitely has footnotes.
October 31, 2009 at 1:22 pm
Hi,
In case anyone wants to try my changes above, please correct the following:
self.num_fnote_pages = struct.unpack(‘>H’, r[46:46+2])[0]
self.first_fnote_page = struct.unpack(‘>H’, r[44:44+2])[0]
and of course you have to add the proper tabs as well
October 31, 2009 at 4:56 pm
Okay, I have ereader2html spitting out the full text of the footnotes now.
The problem is the 1st record of the footnotes which is 63 bytes in length (in my particular case) is not a multiple of 8. That means that the decrypt routine won’t touch it.
So I tried just zlib decompressing it but I get a zlib header error.
I have dumped the first record but it seems to be gibberish or encoded somehow. Given what Mobi Wiki says, it is supposed to be a null separated list of footnote names but I can’t see it.
Any hints on what needs to be done to the first record of the footnotes would be greatly appreciated.
November 2, 2009 at 2:38 am
Paul,
I appreciate your response to my posting on 10/31. I took your advice and paid a visit to Mobileread. I did a search for past and present conversations pertaining to the topic of DRM removal and all results point back to mobidedrm, python, and that I should do a Google search of these topics. Which lead me back to here.
I have once again read through each posting on this site looking for assistance from the responses given to others that have had similar problems like my own. I have tried the solutions that have been given and unfortunately I can not get this to work for me. If you have the time I would greatly appreciate any advice or assistance you would be willing to give.
I am trying to run mobidedrm06 with windows (Vista) operating system.
Thank you,
Cyndi (cysomers68@att.net)
November 2, 2009 at 4:47 am
Hi Dark Friend,
Thanks again for your hint about handling footnotes. I am now able to get all of the footnotes in readable fashion. The problem is the first page of the footnote information is 63 bytes long (in my test case) and actually is not encrypted. By watching the code run in gdb, I was able to see that the page 1 footnote data is exclusive ored with a sequence of 32 entries from a table so make this useful. I simply can not see where this table of values is coming from. pdbshred hints at a fourth key based on the first 8 bytes of the sha digest of the “title”. I am not sure what he/she means by title.
Any hints or ideas of where the table to clarify the footnote page 1 info would be appreciated.
All of this just to get my copy of Freakonomics to work with my new Sony reader. I guess I could just read it on my computer or my ipod touch but I really prefer the Sony reader so …
Thanks for any hints.
November 4, 2009 at 1:37 am
Kevin — http://pastebin.de/1196
November 4, 2009 at 1:42 am
Kevin – http://pastebin.de/1196
November 4, 2009 at 2:56 am
Hi 123 and Kevin Hendricks Friend,
Thanks that is a huge help!
I figured out that they are walking an XOR table modulo some set length starting at an initial offset into the table (which depends on the relative page number within each section). It took me forever to find that XOR table actually sitting in data of record 1 **before** it is unencrypted and unshuffled. Once I had that I could dump the first footnote ids record and make it understandable, as well as all of the links records (allowing me to determine where they start and the count, and ditto for the chapter records, expanded text sizes record, metainfo records (all for type 272 only).
If you want any of that let me know.
Now I can actually read the footnotes in Freakonomics after converting it to ePUB and reading it on my brand new Sony ereader 600B!!
Take care and thanks for bringing back “fair use” and “common sense” to the messed up ebook industry. There are many people (myself included) that will *never* pirate anything and just want to read the damn things on their platform of choice and be able to archive it for the future when many of these devices are long gone.
November 4, 2009 at 4:48 am
See
http://pastebin.de/1199
November 5, 2009 at 2:40 am
Kevin
Can you provide your final working version of the script? Is there any difference in yours when compared to either of 1166 or 1199 mentionedearlier
November 5, 2009 at 3:38 pm
I looked at both.
1166 is the script to start with since it has many improvement over version 0.03b of ereader2html.py including footnote support (which unfortunately does not handle footnote record 0 properly)
1199 was based off the older 0.03 version but adds features that properly handle footnote record 0 to get a list of the actual footnote ids and using that same approach, is able to properly output chapter offset records, link records, and etc.
Both of these are for type 272.
So people need to start with the generally more advanced 1166 and join with it the new pieces from 1199 (XOR table, properly decoded footnote ids, etc, etc). They could also easily mimic the use the footnote changes to properly handle sidebar changes as well. Note: sidebars may map to the same first record as footnotes but they have a count of 0 if they are not being used and visa versa).
I hope some kind soul will paste a properly merged version to pastebin soon.
November 5, 2009 at 3:51 pm
Whoops!
Everywhere it says 1166 it is supposed to say 1196
November 5, 2009 at 9:55 pm
I’ve combined 0.04 and 0.05 to make a new 0.06.
I’ve made it put the images in a sub-folder, so that you can just drop the PML file onto DropBook and make a new eReader file.
I’ve improved the HTML output a bit, although really you get better results making an unencrypted eReader file using DropBook and converting with Calibre.
I’ve tested on the few encrypted eReader books I have, and it seems to be working well. I haven’t been able to test any of the fancier features.
November 5, 2009 at 9:56 pm
And I’d better mention where to find 0.06!
http://pastebin.com/f76582f9b
regards,
Alf.
November 5, 2009 at 11:30 pm
Here is a slightly different version with 0.04 and 0.05 merged that handles some of the new stuff a bit better and adds sidebar support (sidebars still need to be tested but it should work – theoretically)
http://pastebin.de/1278
Now we need to merge this with the html and image folder fixes from 0.06!
November 6, 2009 at 12:55 am
Okay, I fixed an indentation issue with the newest version 0.05, removed some of the extra debug info from it, and added in the image folder and html fixes and things from 0.06, so we now have a 0.07 if anyone wants it, that should have everything to date in it.
http://pastebin.de/1280
November 6, 2009 at 4:51 am
Does anyone know of python or perl (or even java) code that is smart enough to split html files intelligently into many smaller html files (and by intelligently, I mean not splitting in the middle of a tag, properly updating any internal links to point to the achor that may now resides in another html file, etc.
I really don’t like the code generated by calibre going from html to ePub. It seems to miss all toc entries, and many internal links are not being updated, and there are so many styles added the resulting xhtml is almost unreadable which makes hand editing hard.
So I thought I would look to see if there is something better and thought people here might know of something.
Thanks,
Kevin
November 6, 2009 at 8:12 am
For hand-editing (X)HTML to make ePubs, take a look at Sigil.
http://code.google.com/p/sigil/
The GUI for HTML markup is still in the early stages, but you can easily edit the (X)HTML code directly, and it does a very good job of generating correct ePubs.
November 6, 2009 at 3:42 pm
Thanks for that sigil url. I just tried it and it took about an 45 minutes to manually set chapter breaks and table of contents, and meta information and then it generated a very nice epub file.
It seems ereader2html does not automatically add around chapter titles so Sigil will not recognize any TOC entries. Nor does Sigil parse the meta tags that are already in the html input file. So I had to reenter them manually.
Other than that, the epub it created was very very good. Sigil very intelligently handled all internal links (even those for footnotes) when it split the files. Manually unzipping the created epub file shows a well structured set of files and directories.
I am very impressed!
Sigil will now replace calibre for my html to epub conversions.
Thank you.
November 6, 2009 at 10:41 pm
Hi RevX
I have downloaded the latest eBookUtility. I have Java and Python installed on my computer. When I double click on eBookUtility it asks for the serial number (of my Kindle, I assume).
I enter the serial number, taken directly from the back of my Kindle, and it says that it “Cannot generate PIN. Invalid Serial Number?”
Any advice? My Kindle is a Kindle 2 International. The Serial number is definitely correct. I even know its PID.
Many thanks. I want to use your application to strip a Mobipocket ebook of its DRM so I can read it on my Kindle 2.
Cheers, Will
November 6, 2009 at 10:51 pm
It seems that eBookUtility doesn’t yet understand International Kindle Serial numbers. The decode just like otherKindle serial numbers, but they start with B0003. (Is that the right number of zeros? anyway – where Kindle and Kindle 2 have a 1 and a 2 the Kindle International has a 3)
November 6, 2009 at 11:05 pm
I see. Thanks Paul. You’re almost correct re the serial number, its B003.
I’ve also tried the command prompt kindlepid.py and kindlefix.py scripts with a PID that I know is correct for my Kindle. However the kindlefix won’t recognise the PID that kindlepid.py generated, that i know is correct. I’ve been confused as to why, but perhaps your response explains it now. I guess living in Australia means I’m not going to be able to strip these mobi files to be read on my Kindle. Disappointing, but thanks for the information.
November 6, 2009 at 11:16 pm
You don’t need eBookUtilty to strip DRM from Mobipocket files. Just an Installation of Python 2.6.4, and the MobiDeDRM 0.06 script. Python you can get from python.org, and the script is probably available from one of the links in previous comments.
November 6, 2009 at 11:21 pm
Thanks Paul, but the MobiDeDRM script doesn’t recognise my PID…even though it definitely is the PID of my Kindle (I’ve double checked it through the MobiPocket Reader 6.0).
Perhaps, as you’ve indicated in your previous post, this is because my Kindle is an international version.
November 6, 2009 at 11:30 pm
MobiDeDRM does work with International Kindle PIDs.
Most likely, you just need to surround the PID by single quotes or double quotes on the command line (I forget which ones work on Windows).
November 6, 2009 at 11:34 pm
It does huh? Well I wonder what is going on. I have tried the single quotes and double quotes trick before on the command line. But alas, no joy, it still says the PID is not correct. Thanks for your advice, i’m resigned to the fact that MobiDeDRM just isn’t going to work for me.
November 6, 2009 at 11:43 pm
Feel free to email me off-blog if you’d like to. I hate to see someone stuck with DRMed ebooks. See my website.
November 7, 2009 at 2:59 am
Thanks so much Paul. Just sent you an email.
November 7, 2009 at 5:12 am
This is a response to Will’s message earlier.
If I understand things correctly, you have purchased a DRM protected ebook in mobi AKA prc or mobipocket format. If this is the case then the ebook was encrypted with a PID that was previously registered with the ebook vendor. mobiDeDRM should work with the PID registered with that vendor. The PID associated with your Kindle is irrelevant and in fact, mobiDeDRM will report back to you that your Kindle PID entered into the script in incorrect.
I think you mentioned in one of your earlier posts that you have the MobiPocker reader installed. I’ll bet the PID associated with your instance of that program is the one that should be entered into mobiDeDRM.
Again, referencing an earlier post, the double quote markes surrounding the PID won’t hurt but should not be necessary unless you have some special characters that are meaninful to DOS
November 7, 2009 at 7:26 am
Yes, Will, the Kindle PID will be useful for stripping the DRM from Mobipocket files purchased from Amazon for reading on the Kindle. It won’t be useful for reading books from other sources on the Kindle, unless that’s the PID you registered with the vendor when you purchased the file (vendors who sell DRMed Mobipocket ebooks, and libraries who check out Mobipocket ebooks, make you enter the PID of your reader or device before you can download the book; that PID is used to infest the file with DRM the file and the DRMed file is what you can download.)
November 9, 2009 at 5:15 pm
You’re my hero! Now I can read those books that I bought for my Palm on my much nicer Cybook! Thanks.
November 10, 2009 at 9:12 pm
Please see
http://pastebin.de/1362
This is a trial version 0.09
This one tries to improve support for pml to html conversion by borrowing from calibre’s pml to html conversion approach as a starting point
– uses appropriate xhtml tag pairs everyplace
– replaces deprecated html with css pieces
– adds an internal only pseudo tag \h to handle the case where the \t tag is used to generate hanging indents
– properly handles the remaining \t tags which do block indents
– uses html comments to mark ChapterTitle Starts which makes it easy to generate Tables of Contents using Sigil especially when the toc entries are not found by Sigil automatically
Pretty much everything is reasonably handled except for:
– the \T=”40%” tags which are really tab stops that can not be easily incorporated into html without setting up tables of some sort (which are not supported by most ebook readers)
– the normal and standard fonts tags
November 10, 2009 at 9:40 pm
These sound like really useful improvements for those just interested in the HTML output.
However, I’m not sure that it’s worth spending much more time on the html conversion part of the script. It’s simple (from 0.06 on) to create an unencrypted eReader file from the output using DropBook, and then do any necessary conversions using that file through Calibre.
I don’t mean to be negative about improvements to the script, but might it be more use to more people to spend the time on improving the Calibre eReader import routines?
November 10, 2009 at 10:28 pm
Hi Paul,
The Calibre ebook-convert routines simply converts pml to html (Calibre’s intermediate form) and then from html to epub (or whatever version you want). Unfortunately, it makes html code that is very very hard to read at times and does not gracefully handle table of contents and the like, so you end up with epub files that are sub-optimal.
So even though you use something like Dropbook and then Calibre, all you are really doing is converting the exact same pml to html inside a python script inside of Calibre and then html to whatever other format you want.
I would much rather skip Calibre completely (which is way overkill for doing html to epub conversions), and use Sigil to convert html to epub.
It simply does a better job os slicing the html into correct portions, creating a better table of contents, etc, without all of the overhead of
Calibre.
As for long-term archiving, I would much rather have an html version than a pml version. They way you use it, ereader2html is really DeDRMereader but for me archiving pml is not a long term option.
Therefore, I would like a much better pml to html converter and that is what I tried to do above.
My 2 cents …
November 10, 2009 at 10:58 pm
Fair enough. You’re putting the work in to make the tool you want, and being kind enough to share it. Thanks.
November 11, 2009 at 10:22 am
thanks, guys, you are all really great and this “Reply section” must be the most useful one in any blog I know and care 🙂
I’m even starting to learn Python now, after all those technical discussions you had here leaving me wanting to understand more about all this new programming languages (I was once a profissional programmer, but with some old languages like Cobol, Pascal and Fortran 🙂 – I know some VB and C++, but I want to “modernize” myself 🙂 )
so, thanks again for all the work on this tool!
November 17, 2009 at 7:35 pm
Thanks for all the great work on these scripts – I couldn’t survive without MobiDeDRM!
I’ve recently been playing with the eReader2html scripts and want to ask a question that’s been raised before:
(1) I want to convert some non-DRM eReadr files to html for use in other editors (like BookDesigner to make FB2). Is there a good tool to use (on Mac OS X if possible, but Windo$e is OK too)?
(2) Is there any way for the eReader2html script to handle non-DRM files? I would assume you could put in a check to bypass the decryption process and go to the PML/HTML conversion steps, but I’m not sure if the code is structured that way.
I feel bad asking – the effort on the scripts is greatly appreciated, as the comments on this and other forums indicate. but this little tweak would be great (unless there is a better method to extract the data – see (1) above).
Cheers!
November 17, 2009 at 7:48 pm
For non-DRMed eReader files to HTML, I’d recommend using Calibre. It’s cross-platform.
November 20, 2009 at 10:37 pm
Hi
Just in case anyone is interested, I have made some additional fixes for the pml to html conversion being done in “ereader2html”
These fixes change things so that more proper XHTML 1.0 is created that after passing through “Tidy” will actually pass XHTML 1.0 Strict (xhtml1_strict.dtd) with flying colors.
Paul is right, this really is not useful at all to people unless they really want very strictly clean xhtml as the final output (which I use for archiving) or for later conversion to epub format.
So I would be happy to post a completely separate python script that just takes any pml file as input and outputs xhtml that can be further fed into “Tidy” to generate quite clean and strict xhtml.
Just reply here if anyone is interested, and I will post something.
November 21, 2009 at 4:15 pm
Thank you so much for this effort. I now stip the DRM from Kindle books as soon as I purchase them.
November 23, 2009 at 4:25 am
In answer to the kind offer from some_updates
I would be interested in seeing both the additional fixes in eareader2html and also the separate script that will do a good translation of any pml file into xhtml. Both could be very useful.
November 23, 2009 at 3:09 pm
Sure thing.
The original file has been split the original file into two pieces. There are two reasons. The first is that the DRM piece now stands alone as it will be updated much less frequently. And the second reason is that the pml to xhtml piece will be free to use by anyone anywhere since it now has nothing to do with DRM (and can be updated more often more easily).
1. erdr2pml.py which simply generates the pml file, any images, and bookinfo.txt file and this can be used with Dropbook as Paul recommended. Its syntax is that of the original file.
See: http://pastebin.de/2391
2. pml2xhtml.py which takes a pml file, and bookinfo.txt file and generates an xhtml file. It uses python subprocess to invoke tidy via the command line. Command line versions of tidy build out of the box on Linux, and MacOSX (actually it is already installed on MacOSX machines) and is available for Windows as a binary from http://int64.org/projects/tidy-binaries
Its syntax is:
python pml2xhtml.py infile.pml [bookinfo.txt] outfile.html
See http://pastebin.de/2392
I plan on improving the pml2xhtml version further as time permits.
November 23, 2009 at 6:32 pm
I was looking at the EXTH header in Kindle ebooks, and I noticed a record code 208.
Does anyone have any idea what this record is used for? It isn’t listed in the mobileread wiki.
November 23, 2009 at 6:58 pm
This field appears to be a way of storing descriptive or metadata information. Take a look at this site
http://kindlejunkie.livejournal.com/5510.html
and this
http://www.kindleboards.com/index.php?topic=5410.25
November 23, 2009 at 7:57 pm
Thanks for the links. But unless I missed something, they all seem to be about EXTH 501 (cdetype). I was curious out EXTH 208.
I’ve now split out the EXTH record type in the wiki into its own table. If anyone has the time to help fill it out, that would be good.
http://wiki.mobileread.com/wiki/MOBI
November 24, 2009 at 10:20 am
I’ve been meaning to send out an update of my ereader2html code for a while. While I don’t totally agree with splitting the files apart, I can play ball. I have attempted to merge most of my changes into the latest script, but I’ve only tested the merged script on a single book so I’d appreciate feedback.
The biggest user-visible change is a switch to paragraph elements and a CSS styling that more closely matches that of a printed book. There are also numerous bug fixes.
v0.18 builds off pml2xhtml.py (http://pastebin.de/2392):
http://pastebin.de/2420
November 24, 2009 at 12:54 pm
At http://david.woodhou.se/mobidedrm_plugin.patch I put a patch which applies to mobidedrm006.py and turns it into a Calibre plugin. It’s based on a similar plugin I found elsewhere, which was invoking mobidedrm.py as a separate process (and which would fail to import non-DRMed files because it didn’t have any error handling).
After applying the patch, you then have to rename the file to mobidedrm_plugin.py and put it into a zip file on its own, then you can add that zip file as a plugin.
There are some improvements which would be nice — the ability to store multiple PIDs, checking the PID checksum at the time the user enters it rather than at conversion time, showing a error dialog when asked to decrypt a file for which it doesn’t have a valid PID (with ‘abort import’ and ‘import without stripping DRM’ options).
November 24, 2009 at 3:52 pm
Hi Logan,
Thanks for posting that. I think our goals are slightly different. Your goal is to make something look more book-like and my goal is to replicate exactly what the official eReader shows with that same pml file (at least on my own machine!)
Your idea of keeping more state information is a very good one!
It can be extended to prevent block elements from being used inside of inline elements and prevent a log of the errors that tidy has to deal with right now and even to handle the case of conflicting inline elements which happen a lot since pml has no overall structure.
Also your fixes for browsers with empty anchors, real use of smallcaps, and changing the \T output make things even nicer.
I will definitely be testing a version of your changes and let you know how they work.
The one real benefit of splitting things out into a separate file like pml2xhtml.py is that now that the “questionably illegal in some countries” code has been removed we can actually set up a public repository for the pml2xhtml.py code and even a versioning system if needed.
We could also easily have erdr2pml.py invoke pml2xhtml.py directly so that the user see no real changes in how they work.
Thanks for sharing your improvements!
November 25, 2009 at 1:44 am
@DavidWoodhouse – I am not sure how to apply a patch. I’ve tried to google for it, but haven’t found any instructions. Is it just a pasting of your code into the mobidedrm.py or is it a cmd line prompt?
tried patch -mobidedrm.py <mobidedrm_patch.py
November 25, 2009 at 2:50 am
Using eBook Utility 0.6 and my own ebook.props file with the PID that works with ModiDeDRM.py cmd nets me this:
Error for C:\Documents and Settings\xxx\My Documents\My Dropbox\Conversions\B002VGSXBY_EBOK.prc File “MobiDeDRM.py”, line 22 Error for C:\Documents and Settings\xx\My Documents\My Dropbox\Conversions\B002VGSXBY_EBOK.prc print “Bad key length!” Error for C:\Documents and Settings\xxx\My Documents\My Dropbox\Conversions\B002VGSXBY_EBOK.prc ^ Error for C:\Documents and Settings\xx\My Documents\My Dropbox\Conversions\B002VGSXBY_EBOK.prc SyntaxError: invalid syntax
November 25, 2009 at 3:28 am
In answer to my own question, you must use Python2.6. Place the contents of the ebook zip file and put it in the python2.6 folder.
November 25, 2009 at 8:32 am
Hi, I know people will be irritated with my lack of knowledge, but I’ve having a horrible time trying to figure out how to do anything you’re talking about, but I desperately want to use my ebook on my phone’s ereader (android based). I have a mac, and I’ve downloaded python. Also put my .pdb ebook into a folder with what I think is the ereader file (although I cannot see any file named ereader2html.py) and I’ve run the command through terminal:python ./ereader2html.py Book.pdb desktop/ereader2 “Jane” CC# but nothing. My error message is: “can’t open file ‘./ereader2html.py’: [Errno 2] No such file or directory”, which I’m interpreting as meaning I don’t have the ereader2html.py file afterall, but I can’t for the life of me figure out where to download it from.
Sigh. Help!
November 25, 2009 at 9:14 am
Slebet… I don’t know what you asked Google but if I search for “how to apply a patch” most of the first few results give the answer.
Also, patches are human-readable. You can just follow the instructions in an editor. Lines starting with a ‘-‘ mean that you remove the matching line from the original file. Lines starting with a ‘+’ mean that you add the line. Lines which start with a space are just “context”, helping you find where in the file you should be looking. The lines with @@ separate different “hunks” of the file, and they tell you the line numbers at which you should be looking.
Or just use http://pastebin.com/f6f3531d9
December 6, 2009 at 4:59 am
Thank you. I didn’t realize that the patches were human inputted. I will remember that for the future. Thank you for creating the plugin.
November 25, 2009 at 9:23 am
To clarify, for now I just want to use my mac to remove the stupid drm from the .pdb ebook file I have so that I can use calibre to convert to .epub, and then I can use it on my phone. So if anyone can please please please help walk me through how to download whatever I’m supposed to download to remove the drm, I’d be greatly appreciative.
November 25, 2009 at 9:38 am
@Jane: Send me an email.
November 25, 2009 at 10:29 am
@Paul You are awesome. Thanks for the help! Thanks too to DarkReverser. You are all great!
November 25, 2009 at 4:11 pm
Gee, I feel stoopid. Weeks ago (Sep 7) I posted that I was having trouble getting a file de-DRMed. I thought the issue was the asterisk in the Kindle PID (which a vendor wouldn’t accept) and the character I substituted for the asterisk (which the vendor accepted, but which seemed to hang MobiDeDrm). A few weeks ago I got a PID for my PDA and used it with the vendor to download a different copy of the DRMed book. Same problem. I thought I was doomed.
I’d check this site from time to time to see if there was a new version that would solve the problem. Yesterday I saw there was a version I hadn’t picked up yet. I tried it. Same problem. No. Wait. MobiDeDrm “hung” at a different line. Tried it again and decided to let it run in the background while I did other work. I checked again about an hour later. Book was de-DRMed. I don’t think I ever really had a problem. It was just a HUGE book (a 18 mb dictionary) that took a longer time to complete than I ever expected.
Emily Litella moment: never mind. Except to say Thanks, once again, for all the efforts the programmers on this site are putting in. Very much appreciated!
November 25, 2009 at 5:31 pm
For those with an INT’L Kindle trying to successfully use kindlepid.py – all you need to do is add one line to the script which is very, very easy. I did this with the arrival of my DX which, at that the time, also had a s/n not recognized by kindlepid. (began with “B004”) It works like a charm.
Did Will (above posts) successfully extract his Kindle PID?
November 25, 2009 at 8:16 pm
Hi Logan,
Okay, I tested your version of pml2xhtml.py and found a few minor problems which I cleaned up with this patch
http://pastebin.de/2478
– The comments in the style caused problems with some software which extracts the style sheet into its own file
– there were some extra tabs which should be removed
– both the \t and the pseudo \h tags are actually Block-Explicit paired tags
– hr with width tags are not allowed in xhtml strict so I had to put back what you had commented out
Once those changes were made it seemed to work just fine (no double lists entries when enclosed by \t, etc. Although the spacing was set so tight I really could not easily read anything. I guess it would be wonderful for printing but no easy to read on a small screen reader.
I have a slightly different version running which conflicts with your version in that it uses state-information to prevent block elements being used inside inline elements. Unfortunately, your added tags never get added to in_tags and it makes keeping track of when things are inline and when they are block inside the in_tags list almost impossible.
Once I figure out how to merge the two successfully, I will post a new version.
Thanks for sharing your improvements.
Kevin
November 25, 2009 at 8:56 pm
Awesome, Kevin.
The comments weren’t really necessary, anyway.
Sorry about the tabs. My original modifications used tabs, but I tried to match your style when merging. It doesn’t surprise me I missed something.
I never added support for \t (I’ve never run across it), but I really liked your \h hack. 🙂 I do, however, have a handful of books that use \T (both at the start of, and in the middle of, a paragraph), and I’d like to add something like \H (or change \h to accept an attribute). I have actually wanted to implement hanging indents for some time, but I wanted to continue processing the document linearly. Creating a new tag wasn’t even something that had occurred to me.
As for style widths, that is strange. I can see “width=” attributes not being allowed, but CSS style ‘width’? CSS definitely applies to elements, and I am fairly certain that it is a block element and therefore required to support ‘width’. Furthermore, it works in every web browser I’ve tried, iSiloX, and it passes tidy. It doesn’t work in ADE, though, and I mainly prefer semantic elements for backward compatibility.
I’m not sure I have a file where block elements are added inside of inline elements with PML. I did see your code when I was merging, but it was difficult for me to work through the intent without a concrete example. If you have a snippet of code, maybe I can offer advice and avoid the problem in the future.
Thank you for being so friendly.
November 25, 2009 at 10:11 pm
Hi Logan,
I didn’t see the style part of the hr. I will go and put it back and see if my validator barfs on me under strict or not. I just assumed it was the same thing I had tried earlier.
I ran into one other problem. It seems some pml files allow \X0 – \X4 to be nested in \c (centered) tags. So when you look at what element just closed to set the state back to None it actually messes up since it is still inside the \c (or \r or \t or \h) tags.
I think the right approach is not to use the “state” at all but to derive the current state from the in_tags list which keeps track of the history of all open tags.
For that matter, I think you could safely add html paragraphs tags around all text that is not in a heading of some sort (ie. not in \X* or \x tags) so that paragraphs would then exist inside large blocks of text aligned center, indented, right, or hanging.
So perhaps instead of checking state we examine the last element of in_tags to see what cmd we are in to determine the state.
Then in makeText we output all of the lines with paragraphs tags as long as they are not being used inside heading tags (where we default to ). If there is some left-over text at the end with no line break, we start it with a paragraph tag and then append to append to in tags[] a new pseudo tag of some sort that remembers the fact we are in the middle of a self-introduced paragraph.
We would of course have to handle that new pseudo cmd in in_tags as well since the next real tag may or may not terminate the paragraph.
I think something along those lines might work.
As for \T=”50%” tags. They are horrible. They are actually tab stops with percentages as positions. When they start a line of text (ie. \T=”5%”) they are easy to handle as paragraph indents (as you did). The problem is they can be used to create simple tables by acting like tab stops.
\T=”10%”Address:\T=”50%”Main Street, Walt Disney World\n
Unfortunately, there is no way to know what to do with tabstops in html without a full table but tables are not supported by most readers as far as I know. Also, given how small some reader screens are, using them like tabstops to do simply columns of text is very dangerous.
Horrible really. The state of pml can be so poor that it is really hard to make it into something structured properly.
I have a number of pml files that have in-line elements (bolds, large font size, italics, etc) that are never proeprly closed before introducing a new block element (a div, etc, etc). They all have to be terminated before handling the new div and then inserted (reopened) to allow the code to continue properly.
I have a whole new version I am working on that I will try to merge your work and mine together to handle both but I think I will need to use the in_tags[] approach to keep track of state since nesting of block elements can occur.
Hope this helps.
November 27, 2009 at 2:25 pm
Hi Logan,
I have a new version that merges your stuff and mine and that keeps a separate style tag stack to make the html that much cleaner.
I will post it over the weekend, once I have had a chance to test it on more cases.
Kevin
November 28, 2009 at 6:05 pm
I’ve bought a lot of books from eReader.com over the years and any book that I’ve purchased after 9/29/05 decrypts great with the erdr2pml.py script posted on 11/23/09.
I can’t get any book that I purchased before 12/19/04 to decrypt though.
The following messages spit out:
Processing…
Decoding File
Traceback (most recent call last):
File “C:\Python26\erdr2pml.py”, line 638, in sys.exit(main())
File “C:\Python26\erdr2pml.py”, line 624, in main convertEreaderToPml(infile, name, cc, outdir)
File “C:\Python26\erdr2pml.py”, line 560, in convertEreaderToPml er = EreaderProcessor(sect.loadSection, name, cc)
File “C:\Python26\erdr2pml.py”, line 414, in __init__ logging.debug(‘self.num_footnote_pages %d, self.first_footnote_page %d’, self.num_footnote_pages , self.first_footnote_page)
AttributeError: ‘EreaderProcessor’ object has no attribute ‘first_footnote_page’
Am I doing something wrong?
November 28, 2009 at 6:18 pm
No, there’s just a few typos in the script. search for “first_footnote_pages” You’ll find a section of lines that have similar first_something_pages being set to -1.
Unfortunately, that extra s at the end is wrong. they should all be first_something_page without the s.
Change them, and the decoding should work fine. Don’t change the num_something_pages variables – they should have the s.
November 28, 2009 at 6:31 pm
Worked great! Thanks a bunch!
November 28, 2009 at 9:38 pm
Hi Logan,
As promised, here is a merge of your stuff and mine, with things borrowed from pml2html.pl converter and good ideas from others (mainly John from Calibre) that have really helped.
This one does a nice job on every book I have as well as some nasty test cases.
To Do Items:
1. Add back in your automatic parsing of the header comment line for meta information which I took out
I found many books that indented paragraphs simply using spaces and so tried to handle that based on the approach of the very nice pml2html.pl script.
Please let me know if it is an improvement.
http://pastebin.de/2702
Take care,
Kevin
January 18, 2010 at 7:46 pm
some_updates, why is it your pml2html.pl is not that but links to kiddie porn?
http://pastebin.de/2702
The above is your link.
January 18, 2010 at 8:15 pm
’cause he is using pastebin.DE
that service is awful. there are 19 references to pastebin.DE and except for one they are all either empty or useless junk.
use pastebin.COM *not* .DE.
November 30, 2009 at 5:29 am
David Woodhouse:
Could you do the same plugin treatment to ereader2html.py
Your plugin version of mobidedrm is pure genious.
The latest version I can find is at http://pastebin.de/1362
Thank you!
November 30, 2009 at 10:25 pm
To Brutusbum…
There seems to be typos in that version that prevent it from working on some older books (see Paul Durrant’s most recent post on how to fix these).
Also there exist much improved html conversion routines that have been split off from ereader2html.py. You might want to think about re-integrating them or using the latest versions of erdr2pml.py and xpml2xhtml.py instead.
See recent posts after the one you noted.
December 1, 2009 at 8:18 pm
Can someone repost the eBookUtility 0.6 zip file? It’s no longer available for download. Thanks!
December 1, 2009 at 10:46 pm
OK, I corrected the typos and reposted version 0.09 of ereader2html.py here.
Merging the features of the newest forked versions is beyond me I am afraid. I would be grateful is someone would convert this to a plugin for calibre.
http://pastebin.com/f7be19a99
B.
December 3, 2009 at 4:35 pm
It was a pain finding a standalone (non-Calibre plugin) version of MobiDeDRM 0.06; most of the pastebin entries have been deleted/expired. So, here it is again: http://pastebin.com/f17f1a8ad
December 3, 2009 at 7:12 pm
Thanks, Michael. You know what would be a wonderful Xmas or Holiday Gift for everyone here? If anyone who is good at all this (not me I am afraid) could take the time to list in one message ALL of the various scripts referenced on this blog as to their latest revisions and download locations. I find that, right now, it’s becoming a bit like a weird puzzle or adventure game trying to figure which of the many, many iterations of each script is the latest, is working, and can be had. I would love to update everything. Any help from anyone who would like to play Python Claus for us all??
December 6, 2009 at 4:55 am
I’ve added the Calibre plugin but I don’t ever get prompted to add to the PID.
December 6, 2009 at 5:01 am
Answering my own question, go to Preferences/Plugins/File Type Plugins/
Select the MobiDeDRM and click “customize”. It will ask for the PID.
December 6, 2009 at 1:44 pm
Ideally, the plugin should be improved so that it stores _multiple_ PIDs, and will try them in turn until it finds one that works… and will prompt for a new one if none of the existing ones work.
Feel like learning a little python and implementing that? It shouldn’t be particularly hard…
December 6, 2009 at 4:14 pm
Hi Logan,
The latest version of xpml2xhtml.py is available at:
http://pastebin.de/3639
– greatly improved footnote support (but it requires true xml style footnotes and sidebars in the pml file) including return links, single footnote/sidebar to a page when viewed in an epub reading device, etc
if you want your copy of erdr2pml.py to output proper xml style footnotes and sidebars please see the code snippets at:
http://pastebin.de/3444
– ability to import all meta data directly from the pml file (extends your code to include more meta fields)
– fixes for html corner cases
– more robust conversion of bad pml (now handles case of non-nested block level markups)
– can automatically output Sigil style chapter breaks to make using Sigil much faster (command line switch –sigil-breaks)
– uses command line tidy to make the code even nicer (indented, etc) all controlled by command line switch (–use-tidy)
– overall much more complete and correct xhtml created
Hope this helps,
KevinH
December 6, 2009 at 11:50 pm
Great blog, lots of useful information. This blog has gotten me through many hurdles.. Here a new one!
Amazon has just released a beta version of the “Kindle for the PC”. Allows you to read and purchase Kindle ebooks on your PC.
How do I get the PID off of my PC?
December 8, 2009 at 5:30 pm
I am soooooo frustrated! I am trying to remove the DRM on a mobipocket reader file .prc from the library to read later. I have no idea what I’m doing, but ain’t no dummy neither ; ). I’ve installed python 2.x, python 3.1, and activepython all to no avail. I’ve extracted the mobidedrm files to the same directory as the file to be processed. I’ve opened both msdos terminal and python terminals. The dos terminal says python not defined. The python terminal says invalid syntax. I am typing in : python mobidedrm.py filein.prc fileout.prc ********, for the dos command line and the same without the word python for the python terminal command line. What am I doing wrong!!!!!!!!!!!!!!!!!!!!!!?????????????????????????????? HELP, I am going nuts!
December 10, 2009 at 2:37 am
LG – I type in mobidedrm.py fileinprc fileout.mobi ‘xxxxxxx’
I have to use single quotes and my out file is always mobi. Good luck.
December 10, 2009 at 2:38 am
I have a new error that I have not received before and I use the script all of the time. Anyone seen this one with advice on what to do?
eReader2Html v0.03. Copyright (c) 2008 The Dark Reverser
Processing… Error: incorrect eReader version 10 (error 1)
I bought the book from fictionwise but I think I inadvertently bought the wrong “secure ereader” as it did not have the 5% bonus listed.
Thanks for any help you can offer.
Stacy
December 10, 2009 at 5:49 pm
I updated the version of kindplepid.py included with eBookUtility so that it now supports Kindle 2 international serials, nothing changed other than that. There are no outstanding bugs that I’m aware of and I haven’t received any feature requests that I have to time to implement — one was basically asking to wrap other popular ebook scripts as well, which I may do at some point, but not right now.
You can find eBookUtility 0.6a at http://www.mediafire.com/eBookUtility
Please report any bugs you find and send any feature requests you have.
December 10, 2009 at 6:01 pm
Stacy,
Error: incorrect eReader version 10 (error 1)
Means you are trying to remove DRM from an ereader book that was NOT DRM encoded. It is just a regular .pdb book.
Calibre can read these and convert them to whatever format you might like very well.
There is also a perl script running around that will output the pml file and you can convert that to html easily as well.
KevinH
December 10, 2009 at 6:17 pm
Thanks Kevin – worked like a charm in calibre. I didn’t even think it would be that easy 🙂
Stacy
December 11, 2009 at 6:16 pm
KevinH, I’m having some trouble with a PDB using the latest xpml2xhtml. It starts processing and then bombs out with this error:
WARNING:root:Unknown tag: d-None
Traceback (most recent call last):
File “xpml2xhtml.py”, line 843, in
sys.exit(main())
File “xpml2xhtml.py”, line 818, in main
html_src = pml.process()
File “xpml2xhtml.py”, line 501, in process
final += getTag(pair, False)
File “xpml2xhtml.py”, line 387, in getTag
r = self.html_tags[cmd][end]
KeyError: ‘x’
This PDB has been problematic all along – earlier ereader2html scripts also failed with a traceback call. Any ideas what I should be looking for in the file?
December 11, 2009 at 9:53 pm
Hi,
The unknown tag warning can be safely ignored (the \d tag happens with a lot of the books (I think it is a placeholder for where the software inserts “This ebook belongs to Kevin”.
The error message is that the hash table (dictionary in python terms) html_tags has been presented with a key it does not understand. The key is the “x” cmd which is a perfectly legitimate key to be found in a pml file but one that should have been replaced already.
Earlier in the code the all \x pairs are replaced with \p\X0 pairs so we should not have any \x commands when we reach this point.
See this line in the code.
s = convert_x_to_pX0(s)
which invokes the following routine
def convert_x_to_pX0(src):
# converts all \x \x to \p\X0 \X0 make later code simpler
p = re.compile(r’\\x(.*?)\\x’)
m = p.search(src)
while m:
(b, e) = m.span()
src = src[0:b] + ‘\\p\\X0’ + src[b+2:e-2] + ‘\\X0’ + src[e:]
m = p.search(src)
return src
For some reason we seem to still have one – so that routine is broken somehow.
The simplest workaround is to uncomment the last line in the code in the html_tags key/value table (and remove the text after the comma).
In other words, change the following …
html_tags = {
‘v’ : (‘‘),
‘c’ : (‘\n’, ‘\n’),
‘r’ : (‘\n’, ‘\n’),
‘t’ : (”,’\n’),
‘h’ : (”,’\n’), # pseudo-tag created to handle hanging indent cases
‘X0’ : (”, ‘\n’),
‘X1’ : (”, ‘\n’),
‘X2’ : (”, ‘\n’),
‘X3’ : (”, ‘\n’),
‘X4’ : (”, ‘\n’),
‘q’ : (LinkPrinter, ‘‘),
‘Fn’ : (FootnoteLinkPrinter, ‘‘),
‘Sd’ : (SidebarLinkPrinter, ‘‘),
‘Ft’ : (Footnote, EndFootnote),
‘St’ : (Sidebar, EndSidebar),
‘I’ : (‘‘, ‘‘),
‘P’ : (”, ‘\n’), # pseudo tag indicating a paragraph (imputed from pml file contents)
#’x’ : (”, ‘\n’), handled via recoding
}
so that it looks like this:
html_tags = {
‘v’ : (‘‘),
‘c’ : (‘\n’, ‘\n’),
‘r’ : (‘\n’, ‘\n’),
‘t’ : (”,’\n’),
‘h’ : (”,’\n’), # pseudo-tag created to handle hanging indent cases
‘X0’ : (”, ‘\n’),
‘X1’ : (”, ‘\n’),
‘X2’ : (”, ‘\n’),
‘X3’ : (”, ‘\n’),
‘X4’ : (”, ‘\n’),
‘q’ : (LinkPrinter, ‘‘),
‘Fn’ : (FootnoteLinkPrinter, ‘‘),
‘Sd’ : (SidebarLinkPrinter, ‘‘),
‘Ft’ : (Footnote, EndFootnote),
‘St’ : (Sidebar, EndSidebar),
‘I’ : (‘‘, ‘‘),
‘P’ : (”, ‘\n’), # pseudo tag indicating a paragraph (imputed from pml file contents)
‘x’ : (”, ‘\n’),
}
And then it should be properly handled.
Also, please tell me the name of the book so that I can buy it and download it and figure out why that replace routine is not working for your case.
Hope this helps,
Kevin
December 11, 2009 at 9:55 pm
Whoops, all of the html tags have been stripped out.
I will post a new version on pastebin and post the link here.
Kevin
December 11, 2009 at 10:00 pm
Hi,
As a workaround, please try this version:
http://pastebin.de/4538
Once I get a testcase I can figure out why this is happening.
My guess is your pml file has a single \x tag that is never properly terminated by another \x . This would be an error in pml.
If so, I am not sure how to deal with that since the pml spec clearly states that ever \x must be terminated by another \x.
If that is the case, the new version I just posted may not work perfectly.
But you can alwys look at the pml file and try to find the \x tag that is not properly pared and simply add it and then everything should work.
Hope this helps,
Kevin
December 11, 2009 at 10:22 pm
Kevin – thanks for the speedy reply! I’ll give it a whirl later tonight and let you know how it goes.
December 12, 2009 at 5:54 am
Well, no go so far – the updated script you posted errors out with:
Traceback (most recent call last):
File “xpml2xhtml_edit.py”, line 843, in
sys.exit(main())
File “xpml2xhtml_edit.py”, line 818, in main
html_src = pml.process()
File “xpml2xhtml_edit.py”, line 477, in process
r = self.next()
File “xpml2xhtml_edit.py”, line 180, in next
c = self.s[p+1]
IndexError: string index out of range
I haven’t been able to find any unclosed \x tags in the pml file, either.
This may just be down to problems B&N is having as they integrate the nook – the reason I was trying to convert this file in the first place is because it wouldn’t unlock on the nook (though I had no problem in the desktop/iPhone B&N Readers). Customer service said they’re having trouble with some files, so maybe an updated version of the file will fix it. (The book is Voyager, book 3 of the Outlander series by Diana Gabaldon, btw – http://search.barnesandnoble.com/Voyager/Diana-Gabaldon/e/9780440335153/?itm=1).
December 12, 2009 at 2:08 pm
Hi,
As a Canadian, I can’t buy that book from Barnes and Noble. Would you please open the pml file in a text editor and do a “find” for “\x” and simply count how many times it is found in the entire file. My guess is it will be an odd number (which means the pml file is bad and should be fixed).
That error above is caused by the same thing. I wrote a simple bad pml file with an odd number of \x tags and it errors out in exactly the same way as yours did in both versions.
Sorry but I can’t be more help here without some way of reproducing the problem.
Please do double-triple check for an unmatched \x tag) (look for a spurious one near the beginning or end someplace).
Kevin
December 12, 2009 at 4:09 pm
Stilll no luck. There are 146 \x tags in the file – I went through them all and I don’t see anything that would cause them to be parsed incorrectly. Guess I may just have to write this off as a bad file. Thanks for the help, though – the pml conversion script should come in very handy in the future.
December 12, 2009 at 4:28 pm
Hi,
I would really like to get this fixed so that it does not happen for other pml files.
The only thing I can guess is that there must by at least one \x tag that is embedded inside a comment or some other way of hiding it or making it not “count” so to speak.
Is there any way to zip up the pml and post it for me to run tests on and then delete?
The only other thing I can think of is if I can buy that book from ereader.com (Fictionwise owns them and Barnes and Noble owns Fictionwise) do you think it will be the exact same file as the one on Barnes and Noble’s site itself?
KevnH
December 12, 2009 at 4:49 pm
Hi,
I thought … what the hell, it looks like a good book and if I like it I may want to read the entire series (I like time travel stuff) so I bought it from the ereader.com (Fictionwise site who is owned by Barnes and Noble) and tried xpml2xhtml.py and it worked flawlessly. There was not one \x in the file at all. It was all \X0 tags used for chapters.
So B&N is not selling the same ereader versions of the exact same books across its own subsidiaries!!! I never would have dreamed.
I would ask for my money back and get a different copy of the book (try ereader.com / fictionwise) it seems to be well done.
Sorry I can’t be more help here.
KevinH
December 12, 2009 at 7:57 pm
Well, there are some funky things going on with B&N’s ebooks right now, with the release of the the nook and the impending switchover of their catalog to epub-only. The customer service rep I talked to said there are known issues with some files (she referred to them as “corrupted,” although this one works fine on the desktop and iPhone software). She indicated they would be posting fixed versions, so I’ll probably just wait and see if that works. I did go through the file pretty thoroughly and for the life of me I can’t see anything that would be masking one of the \x tags, but my pml is pretty rusty so I may be missing something.
Not that big of a deal – I actually bought the book for my wife to put on the nook I bought her for Christmas, so I’ve still got a couple of weeks to either get a working file or just buy it from Fictionwise.
I appreciate all of your help – the Sigil chapter breaks switch in your script are going to save me a considerable amount of time and effort.
December 12, 2009 at 8:26 pm
Hi Booda,
That is why the xpml2xhtml.py code replaces the \xChapter Tiltle Here\x pml markup to \p\X0Chapter Title Here\X0 which explictly includes the page break.
For every explicit \p page break in the pml, the xpml2xhtml.py code inserts a chapter breakpoint at the proper un-nested (only enclosed by the body tags) level because Sigil (or anyone else’s) chapter breaks are good split points for splitting html files to make epubs.
Glad you find that useful. After hand doing a 150 chapter book once in Sigil, I came to the conclusion there had to be a better way!
Take care,
KevinH
December 13, 2009 at 7:39 am
B&N’s gone over to ePub, at least as of yesterday. Anybody got any clues how to deal? The ePub file itself isn’t encrypted per se, it’s pretty much a standard .zip file including directories. However, the book content itself IS encrypted, and I’m going nuts trying to locate a key.
December 13, 2009 at 3:45 pm
Check out Adobe Adept DRM removal . Look for ineptepub.pyw, but it requires some work to get the proper adept key from your system.
December 15, 2009 at 6:42 am
Tried finding the key. No luck so far. I’ve got all the python scripts involved, though, AFAIK.
December 13, 2009 at 7:51 am
OK, first let me say thanks to all those that have contributed to the mobidedrm project and all its surrounding permutations!
Unfortunately, I am have a problem with my very first DRM-ed .PRC file that I am trying to free from DRM.
I’m using Mobipocket Reader on my PC. I have Windows 7 x64. I’ve installed Python 2.6 (freshly downloaded to day from python.org). After having all kinds of issues even getting the python program to run, I finally got some results.
However, I get one of two errors everytime I try to use the script:
Error: invalid PID checksum
or
Error: no key found. maybe the PID is incorrect
I’ve checked and double, triple and quadruple checked my PID to make sure it is right. Sure enough it is right. It’s got a $ in it. So I’ve tried it with double quotes, single quotes, a < in front (suggested on the forums) and even a \ in front of the $. Nothing seems to work. I'm starting to think that I don't have the right PID.
So, can someone tell me where to find my PID? The one I am using is in Mobipocket Reader, under Reading Devices it shows only my PC and has an Edit Properties button there. When I click it, it shows a PID of G2ZZIPJ$KB.
Is that the right place to get the PID? If so, does anyone have any suggestions about how to make this work? I'm getting really frustrated. I'm no n00b to computers. I've been doing computer work since the 80's and even programed in FORTRAN, Pascal, BASIC, and now Java and Perl. So this is really baffling to me.
Thanks in advance,
MRB
December 13, 2009 at 7:53 am
Just in case anyone is wondering, I purchased the ebook through the Mobipocket Reader interface from eBook Crescent.
December 13, 2009 at 8:03 am
And I’ve tried versions 0.02 through 0.06 of MobiDeDRM.py and I get the same results with each.
December 13, 2009 at 9:27 am
Well, I got it to work using eBookUtility 0.6a (the .jar version). Thank you REVX!!
The problem was that the PID used to encrypt the file was not the PID of my reader. The site I bought the book from used a username and password set to encrypt it. So, I followed the directions from this web page:
http://www.makeuseof.com/tag/how-to-strip-mobi-and-prc-ebooks-of-encryption/
And connected my Blackberry to my PC, transferred the ebook in Mobipocket Reader to the Blackberry. Then looked up the PID for the Blackberry, copied the .prc file from the Blackberry (manually) to my PC and used the Blackberry PID to decrypt it. However, it is worth mentioning that I still couldn’t get mobidedrm.py to work from the command line. But eBookUtility worked like a charm!! I don’t know why it worked, but it did.
Yeah!
Thanks,
MRB
December 15, 2009 at 6:36 am
Can you help with the error below?
C:\Users\Marcy\Documents\My Library>python mobidedrm002.py fordra.prc fordra2.pr
c 4GPMZVT$D6
MobiDeDrm v0.02. Copyright (c) 2008 The Dark Reverser
Decrypting. Please wait…
Traceback (most recent call last):
File “mobidedrm002.py”, line 176, in
file(outfile, ‘wb’).write(DrmStripper(data_file, pid).getResult())
File “mobidedrm002.py”, line 159, in __init__
extra_size = getSizeOfTrailingDataEntries(data, len(data), extra_data_flags)
File “mobidedrm002.py”, line 74, in getSizeOfTrailingDataEntries
num += getSizeOfTrailingDataEntry(ptr, size – num)
File “mobidedrm002.py”, line 64, in getSizeOfTrailingDataEntry
v = ord(ptr[size-1])
IndexError: string index out of range
If I use v0.01 or v0.05 I get a supposedly de-drm’d file, but it can’t be opened or viewed, either with Calibre or Mobipocket Reader. The above error was using v0.02.
Any help would be appreciated!
-Marcy
December 15, 2009 at 9:33 am
Try with 0.06
If that doesn’t work, I’d like to have a copy of the Mobipocket file. I thought 0.06 handled all the weird Mobipocket files.
December 16, 2009 at 7:07 am
I encountered the same problem as Marcy using version 0.06 of the mobidedrm.
MobiDeDrm v0.06. Copyright (c) 2008 The Dark Reverser
Decrypting. Please wait…
Traceback (most recent call last):
File “/home/Michael/bin/mobidedrm-0.06.py”, line 183, in
file(outfile, ‘wb’).write(DrmStripper(data_file, pid).getResult())
File “/home/Michael/bin/mobidedrm-0.06.py”, line 166, in __init__
extra_size = getSizeOfTrailingDataEntries(data, len(data), extra_data_flags)
File “/home/Michael/bin/mobidedrm-0.06.py”, line 82, in getSizeOfTrailingDataEntries
num += (ord(ptr[size – num – 1]) & 0x3) + 1
IndexError: string index out of range
December 16, 2009 at 9:51 am
I am very glad that this DRM Removal code has been written.
It’s very usefull so my many thanks to the authors.
I already used mobidedrm many times in the past and the bookutility works like a charm. ConvertLit likewise. However i was obliged to buy a book in eReader format and I wanted to try out the erdr2pml.py and xpml2xhtml.py on this book.
However for a very strange reason I cannot access pastebin.de. So can somebody please upload both of the latest files to pastebin.com instead.
Thank you,
Cavelion
December 17, 2009 at 12:57 am
Here is the latest
xpml2xhtml.py
http://pastebin.com/m5f77fec8
December 17, 2009 at 1:00 am
And I think the other one is at:
http://pastebin.com/m61e281c9
December 17, 2009 at 10:34 am
Thank you for the links.
I will put them to good use.
Cavelion
December 16, 2009 at 5:43 pm
There’s a problem with MobiDeDRM 0.06 when decoding some older Mobipocket files. It turns out that the Extra Data Flags aren’t present when the MOBI header is exactly 0xE4 bytes long. Most of the time, those bytes are zero, so it doesn’t matter. But sometimes they’re not, and then things go wrong.
I’ve posted 0.07 to pastebin and rapidshare, although the rapidshare copy has limited downloads.
http://pastebin.com/f7be270a9
http://rapidshare.com/files/321716489/MobiDeDRM_0.07.py.html
regards,
Alf.
December 16, 2009 at 6:01 pm
Corresponding v0.07 Calibre plugin at http://pastebin.com/f78c5e474
December 16, 2009 at 11:33 pm
Thank you. I’ll try version 0.07 tonight.
December 16, 2009 at 11:34 pm
Thank you. I’ll try the standalone and calibre 0.07 versions tonight.
December 17, 2009 at 7:30 am
Version 0.07 runs without error on the same file I was having problems with. However the resulting output file causes calibre to print this error and fail to display or convert the file.
Python function terminated unexpectedly
string index out of range (Error Code: 1)
Traceback (most recent call last):
File “site.py”, line 103, in main
File “site.py”, line 85, in run_entry_point
File “site-packages\calibre\utils\ipc\worker.py”, line 90, in main
File “site-packages\calibre\gui2\convert\gui_conversion.py”, line 21, in gui_convert
File “site-packages\calibre\ebooks\conversion\plumber.py”, line 736, in run
File “site-packages\calibre\customize\conversion.py”, line 208, in __call__
File “site-packages\calibre\ebooks\mobi\input.py”, line 22, in convert
File “site-packages\calibre\ebooks\mobi\reader.py”, line 295, in extract_content
File “site-packages\calibre\ebooks\mobi\reader.py”, line 679, in extract_text
File “site-packages\calibre\ebooks\mobi\reader.py”, line 674, in text_section
File “site-packages\calibre\ebooks\mobi\reader.py”, line 666, in sizeof_trailing_entries
File “site-packages\calibre\ebooks\mobi\reader.py”, line 654, in sizeof_trailing_entry
IndexError: string index out of range
December 17, 2009 at 8:37 am
This is a bug in Calibre (essentially the same one as was in MobiDeDRM). It’s been reported on the Calibre bug tracking system, so hopefully a new version should fix it shortly.
If you just can’t wait, open the unlocked file with a hex editor, locate the four bytes ‘EXTH’ (ie 45 58 54 48 in hex), and then set the two preceding bytes to zero.
Don’t do this on your original file, of course!
December 17, 2009 at 8:15 pm
Thank you. Out comes the hex editor.
December 17, 2009 at 7:01 am
hi, many thanks for so hard work! I’m going to try this new 0.07 version also 🙂
but I’m looking now for fresh news about finding k4pc pid… I know that at least two people already have a method to get it but couldn’t find the complete method descripted anywhere (only a post somewhere where some crucial part was already edited out…. I know some of you know what I’m talking about 😉
so, anyone can give me a hint how to find some more info on that? thanks in advance!!
soalla
December 17, 2009 at 8:00 am
At http://pastebin.com/m31062a97 I posted a version of 0.07 which works both as a Calibre plugin and standalone, in the same file.
December 18, 2009 at 2:27 am
I just wanted to say: THANK YOU SOOOOO MUCH! I just got the DRIOD and unfortunatley couldn’t read any of my mobi books on it. Now I just load them via Calibre w/ plug-in and VIOLA!!
Thanks again!!!!!!!
December 18, 2009 at 7:18 pm
Sabeen, what program are you using to read on the Droid? Just got one and still learning all the tricks in my small dab of spare time.
Back to the topic, thanks for re-posting all the scripts. I love being able to read my purchased book on any device.
December 19, 2009 at 7:59 pm
If he is converting the Mobi to ePub via Calibre, then he is most likely using Aldiko on the Droid. Aldiko is very good software.
December 19, 2009 at 4:27 am
Can some one please explain in simple english. Where do I find the file I need to remove DRM easily. I value gui if possible. Specifically I DO need to be able to remove the tag that stops tts on the kindle.
But I also want to have DRM free all the books I own.
Please tell me… First get this….. link.
Then do THIS with it… I know there is some kind of assembly required for these scripts but I don’t know how to do it. Do I copy something and rund something?
Then use the complied program to do the folloowing…. (instructions please)
Thanks in advance zadrielATaolDOTcom
December 19, 2009 at 6:45 pm
I just downloaded MobiDeDRM07 with the Calibre plugin and added to Calibre; It put it under File Type Plugins and I highlighted it and hit customized and it prompt me for my PID which I entered and it saved it. I put a DRM .PRC book in Calibre and I’m getting the DRM error but I ran the standalone script using the same PID and I was able to remove the DRM. What am I doing wrong in Calibre? I was trying to convert the DRM PRC book to LRF within Calibre without having to go through MobideDRM.
December 19, 2009 at 10:06 pm
Did you remember to customize the plugin with your PID? There’s a button that says “Customize Plugin” toward the bottom of the Calibre Plugin preference pane. Click it and it’ll ask for your PID.
Once I did that, it worked like a charm.
December 19, 2009 at 11:31 pm
Thanks, for the help. I did all that already. But now the problem is fixed. It seems that if you add the book before you add the PID, it does not work. I deleted the book and added again to the Calibre Library and BAM, it worked!
December 20, 2009 at 1:20 am
Okay, for all the people who stumble upon this board, then read and read and scratch their heads and read again… I ASK…
Can some one please explain in simple english. Where do I find the file I need to remove DRM easily. I value gui if possible. Specifically I DO need to be able to remove the tag that stops tts on the kindle.
But I also want to have DRM free all the books I own.
Please tell me… First get this….. link.
Then do THIS with it… I know there is some kind of assembly required for these scripts but I don’t know how to do it. Do I copy something and rund something?
Then use the complied program to do the folloowing…. (instructions please)
Thanks in advance zadrielATaolDOTcom
December 20, 2009 at 5:18 pm
Zad,
It isn’t that easy. There are a number of different book formats with their own drm and the steps to remove that drm only exist for a subset of them, and depend on which format.
So, first ask yourself the following – Why am I trying to remove the DRM? If it is to make anything other than personal use backup copies or to convert it so that your specific reading device can read it, then no one here will be willing to help.
Assuming you just want to convert for your specific reader and for fair use backup then …
1. most of the conversion tools are written in python and the version of python you need plus the libraries depend on platform. I would look for python 2.5 or 2.6 and get the appropriate version of python for your platform and install it and make sure it works. There are links to where to find platform for your platform (for Mac OSX it is already installed) all throughout this blog and in other places on the web.
2. then download and install calibre (which is written in python) and it will handle almost any conversion you need once the drm has been removed (google is your friend – so just search for it.
3. Then determine which format of e-book you have:
A) If it is Mobipocket with drm, you will need to get the latest version of the python script that handles that:
See http://pastebin.com/m31062a97
Download that file and rename it to mobidedrm.py
When properly run with your correct PID, it will remove the DRM from the book and allow you to use Calibre to convert it to whatever format you want.
B) If it is a eReader .pdb book with drm, then use
http://pastebin.com/m61e281c9
Download that file and rename it to erdr2pml.py
This will output the raw pml and images. You can then use Dropbook to convert this back to a non-drm pdb file or you can use:
xpml2xhml.py to convert it to html.
You can get xpml2xhtml.py from
http://pastebin.com/m5f77fec8
Download the file and rename it to xpml2xhtml.py
C) If it is an epub file with Adobe Digital editions DRM (normal kind not the new B&N DRM hich no solutions exists yet), then you must use a program to determine your adeptkey.der file (after registering with Adobe Digitial Editions website) and once you have that you can remove the drm.
You need to search the web for ineptkey.pyw and ineptepub.pyw and find those scripts and figure out how to run them. Many DRM epubs have faulty zip archives built so you might need to fix that as well.
D) if you have the old Sony DRM files, then you are pretty much out of luck cause they were never broken (and probably never will be since Sony has given up on them in favor of Adobe DE epubs). Your best bet is to get on the new Sony store and download epub versions of those files if you still can.
E) if you have a Kindle book , then it gets even more complicated. If you have an original Kindle up through the International Kindle Version then you need to search the web for kindlefix.py and kindlepid.py and the mobidedrm.py. If you run Kindle on the ipod/iphone, I am not sure if anything works here at all? If you have Kindle for PC, then there has been a recent breakthough (search for Cabbages) on the web and look for a blog similar in style to this one. That said, no one yet knows how to remove the drm from .azw1 or Topaz style e-books, so if you do try this, then you may still end up with a book that you can not remove the drm from.
F) Barnes & Noble has a new form of Social DRM for epubs, I heard that someone had a solution for that case but I have no idea where it is or what it is called.
G) There are still other file formats (Lit, etc) that exist and there may or may not be drm solutions for those formats. I have pretty much fouced only on the current popular ones above. I am sure others can point you in the right directions.
As you can see, the various players in the ebook world, have made a huge mess of things and these companies will now fight a 2 to 3 year war to see who will win in the end. This is the fun part to watch how long it takes the publishing industry to realize that the whole paradigm has to change and that they need to move with it and not fight it and make the change first!
Most of the tools are not easy to use if you are not familiar with using the command line (although some nice people have packaged up some of these into more user friendly versions).
No one is going to provide you with step by step directions most probably. You must be willing to go out and experiment and try things.
Please understand, that in some countries, removing the DRM is against the law (or at least the publishers claim this). Others view DRM removal as allowed under “fair use” to make backups and to move your books to other formats. But this has never been tested in court and no one here wants to be a test subject …
SO DON’T USE THESE PROGRAMS TO STEAL.
That is the overview anyway. Now re-read this blog (especially as it pertains for how to run the tools) and give it a try and then come back and ask for help when you get stuck and hope that someone will help you.
February 5, 2010 at 6:52 am
I’m not sure where you are from but in the U.S. (and most major countries) there has been, since the early 20th century, an additional limitation of copyright besides fair use called “first sale” http://en.wikipedia.org/wiki/First_sale. People legally obtaining paper books routinely exercise their fair use rights to sell, lend or other wise dispose of them. Publishers have used ToS and EULA methods to circumvent first use and many, many e-book users think its unwarranted, unfair and (if they can have anything to say about it) unenforceable.
Do you and other people on the list oppose using the software hacks here to enable first sale rights?
December 20, 2009 at 5:04 pm
Hello everyone, First, thanks, a lot of great info and simplified for those of us who never caught onto programming code or script.
I’ve read through just about everything on this page, makeuseof, and a bit of stuff on mobileread.
I am struggling also to get a usable PID for my newly purchased .prc ebook. I bought it from eBookMall last night and am very new to ebooks, but have run mobidedrm.py through the latest 007 as well as calibre with the 006 and 007 plugins.
I always receive the same “invalid PID checksum” error or the “no key found…” I do not have a mobile reader, i.e. a palm blackberry, etc. to try to get a different PID from, only laptops. Admittedly, it would have been a lot easier to just buy the PDF version, i didn’t think it was going to be such a hassle converting it.
Any help in how to get the “correct” PID to use would be much appreciated. Again, I’m not too savy, but get around ok. I too have tried the tricks listed above, started with mobidedrm (seems to be working but not accepting my PID), tried mobipocket creator (couldn’t figure out how to make that even open a prc), tried calibre as mentioned above, tried eBookutility… Same problem it apears MRB was having, but running eBookUtility gave similar results:
“Error: no key found. maybe the PID is incorrect”
I’ve also changed the “ebook.props” for eBookUtility to have the PID with single quotes around.
I’ve tried chopping off the last two digits, tried placing a \ before the $.
I’ve tried so many things, I can’t even remember them all. 🙂 I really don’t have as much time as I’m allowing for this, but i’d like to conquer it.
Computer’s running XP. Not sure what alse could be wrong, other than just plain not having the correct PID, but mobipocket reader opens it fine and, again, don’t know how else to get the “correct” PID.
Any help would be much appreciated!!!
December 20, 2009 at 7:40 pm
Did you have to enter your Mobipoocket Reader’s PID into your account at the ebook store you bought the ebook from?
If not, then your ebook is getting the PID encoded in it by Mobipocket reader, and a copy of the file you downloaded, with the PID encoded in it, is being made in the Mobipocket Reader’s library directory.
You will NOT be able to decode using the file you downloaded, as it doesn’t contain your PID. You will have to find the encoded copy that Mobipocket reader made, and run MobiDeDRM on that.
December 20, 2009 at 9:31 pm
Thanks for getting back quickly, thought I’d be waiting a day.
In fact, no information was given to the ebook store regarding the PID, upon opening the ebook in Mobipocket reader I had to give the user name and password given by the store in a popup and it will only install the ebook when I have an internet connection up to validate.
Is there a way of getting the PID I need out or am I stuck? How would I find the encoded copy made by Mobipocket reader?
December 20, 2009 at 10:50 pm
The PID is the PID of your copy of Mobipocket Reader. You can find this by looking in the About dialog of the program. (Click the ? in the blue circle at the top right of the mobipocket window, and choose “About…”. The PID will be displayed.)
You should find the modified copy of your book in My Documents/My eBooks
December 20, 2009 at 11:47 pm
Paul, you have been a blessing from above, thank you!
The key to my problem (I feel silly, but I really didn’t know better) was in your statement, “You should find the modified copy of your book in My Documents/My eBooks.”
I had been working with the original prc and this has been my bain all day, (and last night). Upon copying the modified copy to the folder and running it through mobidedrm007.py, I was able to remove the DRM. Then I ran it through calibre and tadah! I have my pdf.
Again, thank you and I thank each of you for your work here, I work out of country and can’t readily go to the local book store anymore, but can buy ebooks.
God’s Richest Blessings!
December 21, 2009 at 12:15 am
I am having a problem with the Calibre plugin. I saved the file as a plugin in a zip file, added it to Calibre and added my PID. But if I try to convert a book, it says I must remove DRM first. I tried saving a book to my desktop and adding it to Calibre again, as Lilly suggested, but no luck. I would really like some suggestions on how to add ineptpdf and ineptepub as well. I renamed them as plugins and saved to zip files, but they won’t import as plug ins.
I would love to have Calibre manage my entire library, and cut out the step of removing the DRM before I convert and move it to my reader! Thanks, Debra
December 21, 2009 at 1:39 am
I understand your need for qualification. Frankly if I were looking to simply steal reading material, I wouldn’t be fooling around on this site. There are plenty of PDF’s for any material I wanted to read.
No, I’m sorry I forget there are other kinds of readers out there. Basically what I want is step by step for removing DRM from Kindle books.
Why? Well frankly two reasons. 1. I want to be able read my books on other devices if I ever move from kindle. For now, they may stay on my book in drm format. I can read them fine. HOWEVER…
2. I need to remove DRM because I want to be able to have TTS read my book to me regardless frankly of what the publisher thinks of it. Some of these books I have were TTS enabled… THEN amazon decided to send me the update that didn’t do anything other than read a TTS flag and take some of those away from me!
So that is the two reasons I, and I think most, want the tool to function for KINDLE.
Your instructions were very well written and I thank you. Its just that they appear to miss the biggest up coming format… Kindle.
So if someone could add step by step for KINDLE… I’d be most appreciative.
Thanks again… (oh, and if you knew what I do for a living, you might find it funny that you had to clarify the “honest reasons” for DRM removal.)
z
December 21, 2009 at 2:08 am
As I wrote above (if you read it all of the way through)
For Kindle:
E) if you have a Kindle book , then it gets even more complicated. If you have an original Kindle up through the International Kindle Version then you need to search the web for kindlefix.py and kindlepid.py and the mobidedrm.py.
(see the link in my original mail for mobidedrm.py)
If you run Kindle on the ipod/iphone, I am not sure if anything works here at all?
If you have Kindle for PC, then there has been a recent breakthough (search for Cabbages) on the web and look for a blog similar in style to this one.
That said, no one yet knows how to remove the drm from .azw1 or Topaz style e-books, so if you do try this, then you may still end up with a book that you can not remove the drm from.
So your next steps are to:
– install python (as in my original e-mail) and I think pycrypto module is needed as well.
– then search the web for kindlefix.py and kindlepid.py (I don’t think they are hosted on this site at all but you might find a link to them somewhere in this blog, I am not sure
If you have Kindle for PC (and not a stand-alone Kindle) then search the web for Cabbages latest program that should help there.
The kindlefix.pid will take as input your serial number (of the kindle) and output a PID. You will need to use this pid.
Someone else here may have something prepackaged that is eaiser to use, but either way you need to install python and pycrypto for your platform to make any of this work.
All of this way already in my earlier reply. Please read it and start trying and when you get into trouble, ask for help.
December 21, 2009 at 10:07 am
Zad,
I just got back and saw your post. I had some of the steps written down for a co-worker but I walked her through most of it by phone, so I added more to help you. I e-mailed you a step by step directions and a zip file of all the scripts you will need. Good Luck
December 24, 2009 at 3:13 pm
Re: step by step for k4PC. Could you email me the step by step and zip file with all the scripts.
I use WinXP
Felice Anno Nuevo,
TIA , Dan
December 24, 2009 at 3:13 pm
Re: step by step for k4PC. Could you email me the step by step and zip file with all the scripts.
I use WinXP
Felice Anno Nuevo,
TIA
December 24, 2009 at 3:41 pm
I too would like K4PC step by step emailed to zadrielATaolDOTcom, I don’t have the software yet, but plan to down load it eventually. Thanks
December 21, 2009 at 2:09 am
whoops, I should have said kindlepid.py will take in your serial number and output your PID.
December 21, 2009 at 2:23 am
Zad,
Here are two other links that might help make things clearer:
http://klungvik.com/index.php/2008/how-to-remove-drm-from-ebooks/
http://igorsk.blogspot.com/2007/12/mobipocket-books-on-kindle.html
December 21, 2009 at 2:58 pm
An addition to the Kindle information. You can also do this with the Kindle for iPhone/IPOD Touch.
1. Buy the book, download it within the Kindle for iPhone application. Then open iTunes, back-up your iPhone.
2. Download a copy of iPhone back-up extractor. Start it up and have it analyze your back-up.
3. Navigate through Application>com.amazon.lassen>Documents>eBooks
4. Choose a book with a PRC extension, then choose next. Then it will download the book to your computer.
5. Then follow the mobidedrm steps stated above.
Note:The book can’t be AZW1/TPZ of course.
December 23, 2009 at 8:48 am
Any help for “universal” encrypted books? For example, the Cybook manual available here:
http://bookeen.free.fr/bin/CybookGen3/UserManual/CybookUserGuide_1_0.prc
Does not decrypt with mobidedrm. Calibre reports it as encrypted, but the Mobipocket desktop reader software reads it just find without registering any devices or entering any PIDs.
Does anyone know if there’s a special “universal” PID that the mobipocket reader tries on these books?
December 23, 2009 at 11:08 am
Replace line
temp_key = PC1(keyvec1, pid, False)
with
temp_key = keyvec1
replace line
if verification == ver and cksum == temp_key_sum and (flags & 0x1F) == 1:
with
if verification == ver and cksum == temp_key_sum:
And that’ll do it. A bit of a crude change, since you still need to supply a PID on the command line, even though it’s not used, but those are the two essential changes.
December 29, 2009 at 4:56 pm
That worked great! Thanks!
December 24, 2009 at 6:35 am
Here is a modified version of the mobidedrm Calibre plugin that can handle multiple PIDs. http://pastebin.com/f243608c4
Install as normal, PIDs must be separated by only a comma, no spaces.
12334323,14334243
December 24, 2009 at 6:54 pm
Hi all and Happy Holidays! I need some help with the mobidedrom plug-in for Calibre. I downloaded it and renamed it to mobidedrm_plugin.py and put it into a zip file. However I could not find where to put my PID. But I loaded it into Calibre where it does appear as an active plug-in. I expected that not having a PID in it would then prompt me for the PID when loading a mobi (or azw??) file. But I don’t get any prompts, it loads the file as a DRM ebook just as if there was no plugin. So I guess my questions are 1) Where in the script exactly do I put my PID(S)? 2) Should it have prompted without a PID in place? and 3) will it work on azw files as well as mobi or do I need to first change the extension. Thanks for all your ongoing help!
December 24, 2009 at 7:16 pm
Yay, I did a search on “calibre” and found the post here that mentioned the customize button in Calibre’s plugin panel. That did let me enter the PID(s) and I have it working now. I still have a question tho…. Why are all the mobidedrm-based scripts so demanding that the filenames end in mobi as the azw books are also mobibooks? So when I have a azw book, to make any of the mobidedrm stuff work I first have to manually change the extension to mobi. Not a big deal but if azw and mobi are equivalent why can’t the scripts be made to automatically recognize azw as well as mobi? To clarify I have a book called, well, let’s say book.azw. If I run the Calibre plugin on it I wind up with a DRM protected ebook. If I first change the extension so it is book.mobi and I run the Calibre plugin then I get a non-protected book in the library. Any light to shed?
December 24, 2009 at 7:54 pm
Hi, I’ve been trying to get Cabbages kindle dedrm script to work this morning, with no luck. I’ve worked from both his v6 and v3 scripts; when I save the file as Unswindle.pyw and double-click, nothing happens. If I save the script as Unswindle.py, a console window flashes open, but then immediately closes. In either case, the script doesn’t seem to execute. I’m running XP with Python 2.6.4. I’ve used the Mobidedrm and Ereader scripts in the past with no problem. Can anyone make any suggestions? A list of steps for the Kindle script would be hugely appreciated!
December 25, 2009 at 5:26 am
I’m using the one that can handle multiple PIDs, but I have to change the file extension to .mobi in order for it to work. It clearly says all 3 file types, but it doesn’t work on azw.
December 25, 2009 at 5:41 am
It also works on .PRC, no it does not seem to work on .AZW. I had to change my Kindle books to .PRC before it would convert.
December 25, 2009 at 8:53 pm
There is a typo. It is AWZ rather than AZW. Edit with Idel and it will work beautifully!
December 26, 2009 at 5:09 pm
That is to say look for awz in the script and change it to azw.
December 27, 2009 at 7:01 pm
That was pretty incomprehisible of me, wasn’t it!
December 25, 2009 at 8:54 pm
Anyone working on Calibre Plugins for ineptpdf and ineptepub? I just love the multiple PIDs – don’t have to remember which one I used!
December 25, 2009 at 10:08 pm
[…] book that you want to extract and close the application, unswindle grabs the book and fires up the mobidedrm script to strip off the DRM. I looked at my bought content, as an experiment, and it turned out that out […]
December 26, 2009 at 5:11 pm
Your article is wrong actually.
It turns out that the native kindle format (azw) is in fact just the Mobipocket DRM format with a few extra flags/bits set (this has long been known). So if you know the PID of your Kindle (which you can get from the serial number in most cases) you can decrpyt them. So there is already a script that converts AZW books to mobi format and then of course the mobidedrm script to decrypt them.
The only format that has not been unravelled yet is the Topaz (.tpz).
See
http://igorsk.blogspot.com/2007/12/mobipocket-books-on-kindle.html
for the original work (notice the date as well). A more recent work that handles Kindle On PC which uses a session based PID and not a simple one based on the unit serial number can be found by searching for Cabbages.
In case you want to fix your blog article linked above.
December 26, 2009 at 5:06 pm
I just did mobidedrm one of my recently purchased AZW from Amazon. My kindle works great with the original purchase, of course.
Since I know my own PID, I tried to dedrm it. (with 0.07 version) The result was strange. The first half of the book was decrypted OK, but the other half was just a garbage strings when viewed by calbre’s in-built viewer.
On my kindle, the decrypted book just showed one half the the book. If you do location jump to end of the book, it does not do anything at all.
The book contains some pictures in it, maybe that caused this problem?
By the way, I know mobidedrm isn’t perfect (…yet and it’s a wonderful piece still) so I am just reporting an experience, not complaining. 😉
December 26, 2009 at 5:09 pm
Which book? It would be interesting to see what’s going wrong with the decoding. It’s probably a simple fix. As the Mobipocket format isn’t documented, some of the choices in the script are guesses.
December 26, 2009 at 5:12 pm
Er, you want the title of the book?
It’s called Lost-to-the-West-The-Forgotten-Byzantine-Empire-That-Rescued-Western-Civilization.azw
December 26, 2009 at 5:16 pm
And the decrypted part was without original’s pictures. No cover images, no maps although the original awz has them.
December 26, 2009 at 5:54 pm
I tried again within the WindowsXP virtual mode. (Previously I was in Windows7 64bit OS and got partial decryted)
Done it through Ilovecabbage’s new unswindle v6 script and same mobidedrm script.
I worked perfectly.
I wonder what was the problem. hmmm…
December 26, 2009 at 6:17 pm
I think it’s a calire-related bug. Not mobidedrm.py
December 26, 2009 at 7:04 pm
I hope this is my last comment on this. 🙂
Further tests showed that mobidedrm.py version 0.07 does not produce a good copy of deDRMed content from the source I mentioned. It does remove DRM but with it all the pictures are gone too, though all the text part is intact.
However through unswindle.pyw & kindle for PC & same mobidedrm.py combination works.
Calibre and the same mobidedrm plugin does produce a half broken .mobi
December 26, 2009 at 7:12 pm
How very interesting. Thanks for the title – I was going to buy if it wasn’t too much, but $14.30 is more than I’m willing to pay.
I can’t imagine what’s going on here. If it works with the copy downloaded for Kindle for PC, it ought to work with the copy downloaded for your Kindle, since you obviously have the correct PID.
Would you be willing to send me the copy that doesn’t work and your kindle’s PID? paul at durrant.co.uk
December 26, 2009 at 9:20 pm
Sorry Paul, please understand that I wouldn’t be willing to go that far. The book is worth the money though. 🙂
I feel bad about mentioning the title. It just happen to be my most recently purchased kindle book that is not topaz.
I didn’t mean to encourage anything by mentioning it, except, of course, a legal purchase.
December 26, 2009 at 10:38 pm
Don’t worry – I’m not in the least offended.
The reason it’s useful to have the name is that it’s a lot easier to find a problem with a specific example.
So far, 0.07 has worked on all my mobipocket books. Perhaps I’ll take a look at the one you mention anyway.
December 28, 2009 at 8:37 am
I have now taken a look at the book mentioned. I have been unable to reproduce the problem. I can only think it’s some peculiarity of your system, or a corrupt file.
I can do no more without a sample file that does show the problem.
December 31, 2009 at 12:21 pm
Thanks for the confirmation.
I will try again and report. Since it worked with unswindle script, I know it should work. 🙂
December 31, 2009 at 5:06 pm
It works. As I suspected my mobidedrm007 script might be broken so I got one from other source. It works perfectly. Sorry for the confusion and thank you again for your help, Paul.
December 26, 2009 at 7:51 pm
Responding to TEDD
Further tests showed that mobidedrm.py version 0.07 does not produce a good copy of deDRMed content from the source I mentioned. It does remove DRM but with it all the pictures are gone too, though all the text part is intact.
However through unswindle.pyw & kindle for PC & same mobidedrm.py combination works.
Calibre and the same mobidedrm plugin does produce a half broken .mobi
Were your tests with MpbiDeDrm 07 done in the Windows XP virtual mode or was itr just your testing with unswindle.pyw? I would hazardf a guess that you version of Python 2.6.4 is not quite compatible with your 64 bit OS but that it works correctly in 32 bit mode.
December 26, 2009 at 9:04 pm
Both OS produces same result.
December 29, 2009 at 6:55 pm
I recently converted about 12 ebooks from .prc files with DRM to .mobi files without DRM using eBookUtility0.6a which uses mobidedrm v0.06. I was further converting them into RTF so I could easily combine them into a single omnibus collection ebook file. Anyway, while cleaning up the RTF version of the first file, I discovered some garbage characters embedded within the text. Here’s an example:
“Every kettle in the kitchen abrupt Ckitly boiled over, steam flushing out”
You can see that ” Ckit” was injected into the word abruptly between the t and the l. Different garbage characters are embedded throughout the ebook. When I open the original .prc file in MobiReader, it is correct. When I open the de-drm’d file (.mobi) in Calibre, the garbage is present. In the 12 files that I converted (I bought them at the same time and converted them right afterwards.), only a few contain the garbage.
Could this be an issue with mobidedrm v0.06? Anyone have any ideas?
Thanks in advance,
MRB
December 29, 2009 at 7:06 pm
Try with MobiDeDRM 0.07.
There’s a feature in some Mobipocket files where some text at the end of a block is repeated at the start of the next block (for decoding efficiency reasons), and there are some special bits to identify when this happens.
Unfortunately this is all undocumented.
If 0.07 doesn’t do the trick, I’d like to received a copy of the DRMed file and the PID used to decode it.
December 30, 2009 at 10:19 pm
OK, now I’m very confused. I just tried DeDRM-ing that same file, using version 006 and 007 and compared them. They are both perfect now – no garbage characters in either file.
Now I really wonder what happened last time?!?
December 29, 2009 at 10:52 pm
Someone seems to have decoded the Topaz format. I haven’t tried it myself, but a python script is now available.
Someone else has worked out the per-book encoding used by Kindle for PC. C++ source and windows binary are available.
See the comments on this page: http://www.openrce.org/forums/posts/1199
December 30, 2009 at 1:29 am
Wow! Where’s the Topaz script?
This is AWESOME if true!
December 30, 2009 at 2:58 am
I found what appears to be the script http://www.pastie.org/pastes/759636 here, and it doesn’t work with my Topaz books so far. 😦 It keeps telling me “Invalid Header” during the parseTopazHeaderRecord() function.
December 30, 2009 at 3:07 am
I think the topaz script you may want is here if I read that blog properly.
I can not try it since I do not have a Kindle myself.
http://www.pastie.org/760591
December 30, 2009 at 3:13 am
Hope that topaz script works. Can anyone confirm it? It looks like it is for Kindle4PC, am I right?
December 30, 2009 at 3:09 am
Unfortunately that script is identical to the one I posted a link to, and neither one seems to work. Hmmm.
December 30, 2009 at 3:15 am
No the one you posted is the unswindle script. the pastie one is supposedly for topaz
December 30, 2009 at 3:17 am
Oh yeah, right, sorry about that. I was looking at both of them (since they both contain the word “topaz”) and got confused. The Topaz script is definitely the one at http://www.pastie.org/760591 and is the one that doesn’t work for me so far, on any of five Topaz books. They all say “Invalid Header.”
December 30, 2009 at 3:18 am
… and they say “Invalid Header” regardless of whether I download the Topaz book directly or with Kindle for PC.
December 30, 2009 at 3:19 am
Same script, same error.
December 30, 2009 at 5:32 pm
New script is out that fixes invalid header message.
But the script itself will only dump one record of the topaz file after decrypting it.
It is more a demonstration of how to do it, and not a full fledged program to actually handle things (yet – hopefully).
I have been looking at the code and I think there may be a way to modify it to have it dump all records it finds but I am not sure yet and really don’t even have a Kindle to try things on.
People who do, may want to give the code a look over and try to extend it to drop all records.
December 30, 2009 at 8:12 pm
Awesome, a new version 2.0 is up here:
http://pastie.org/pastes/761657
Looks like it will output a whole book.
December 30, 2009 at 8:30 pm
The new version has two deliberate errors. You need to uncomment the line after the “uncomment the next line” line, and you need to change:
opts, args = getopt.getopt(sys.argv[1:], “vir:o:p”)
to:
opts, args = getopt.getopt(sys.argv[1:], “vir:o:p:d”)
to get it to recognize the -d option.
Then you run:
cmbdtc.py -v -d -o
Way awesome. I just don’t know what format the output file is.
December 30, 2009 at 8:44 pm
The file opens with “TPZ”. and the rest is (mostly) binary, although the book title and some other material at the end is in the clear.
December 30, 2009 at 8:49 pm
Right, it seems the output is (which should have been obvious) a decrypted Topaz file, and the Amazon Kindle for PC app will open it just fine. Unfortunately, that’s not much of an improvement since it’s ONLY good for piracy, not for format shifting, since the Kindle app will open legal Topaz files anyway.
So now we need a little decoding of the file format so we can figure out how to convert to other formats — if the information about Topaz being a variety of ePub is correct, hopefully it won’t be too much work…
December 30, 2009 at 9:18 pm
You might want to check out the source code for the “Social DRM” pieces the author added.
It looks like he is either encoding or just adding your login name to the file to be written.
Interesting.
He does not seem to understand that no one is interested in piracy, and that instead the goal is to convert it to your preferred format.
December 30, 2009 at 9:33 pm
In addition, his book decryption code invokes getBookPayloadRecord which handles the decryption but does not zlib.decompress the data if it is compressed (and from the sounds of things , many of the records are).
If you look at extractBookPayLoadRecord later in that file you can see how to check for compression and to uncompress it if needed using zlib.decompress.
So with a few small changes to getBookPayloadRecord you should be able to handle decompression if needed, to reset the compression flag, for that record, and output uncompressed data in each record.
You should then be able to quickly see if you have an epub like structure or not.
Sorry but I don’t have a Kindle and so can’t help modify the code itself without one to check my changes.
Hope someone takes a look at this. I really would like to see if the underlying structure really is epub-like.
December 30, 2009 at 9:47 pm
You shouldn’t need a Kindle — you should be able to download Kindle for PC and run that, if you have a Windows-based machine.
December 30, 2009 at 9:51 pm
I already have Kindle for PC but every damn Topaz style book I have tried to buy to play around with has been unable in my country “Canada”.
So unless someone can point me at a cheap, known to be Topaz book that is available in Canada, I am kinda waving my hands at things from afar.
December 30, 2009 at 10:48 pm
What about this free one? Is it available in Canada?
December 30, 2009 at 10:46 pm
Depends what you mean by cheap. I know Piers Anthony’s Two To The Fifth is tpz and available in the US, so it should be available in Canada.
Stew
December 30, 2009 at 11:28 pm
Thank you chorpler for the script fixes but can you tell me what you typed to decrypt the book?
I have tried combinations of the options but it doesn’t seem to work.
is it cmbdtc.py -p myPID -d -o bookname_free.prc bookname.prc
btw i’ve downloaded the book via iphone so they are all in the .prc extension.
December 31, 2009 at 1:43 am
I bought one and have decoded it using the fixed script.
The cmbtc script actually confuses which is the compressed size and which is the decompressed size but this does not matter if nothing is uncompressed and instead just unencrypted (as the -d switch does).
I had to fix this to see the decompressed files but they seem to be just a huge dictionary of terms followed by binary information and lots of images and page descriptions.
The format makes me think of PDF that can be reflowed but I really can’t tell.
I can see over 226 page descriptions all starting with __PAGE__ and 34 images (jpegs) all starting with JFIF from the book I selected to play with. But the page descriptions are not readable in human form.
It appears that some common string based compression scheme is being used but I really can’t tell.
I will play around with it more just for my own curiosity.
December 31, 2009 at 1:58 am
What cmd line command are you using? I used cmbdtc.py -v -d -o filename, and it gave me a PID. Not sure where to go from here, but will keep tinkering with it.
Stew
December 31, 2009 at 2:09 am
The command line I used after entering the code fixes, including an improperly indented line, was:
cmbdtc.py -v -d -o fileout.prc ebookfile.prc
December 31, 2009 at 2:13 am
Holy crap! That worked. Thanks. Not sure what to do now until a way to convert it to my Sony Reader format.
December 31, 2009 at 2:27 am
Actually, the program should be named “lapidary”, since we’re dealing with jewelry. 🙂
December 31, 2009 at 2:46 am
I wonder if someone else can open it (the prc file) in another Kindle App now that it has been so called liberated?
December 31, 2009 at 2:54 am
Yes, it can now be read on any Kindle or Kindle app, which is why the guy who developed it is so concerned about piracy: currently, the only thing you can practically do with the un-DRMed Topaz file that you couldn’t do with the original, is send it to others so they can read it on their Kindle or with their Kindle for iPhone or Kindle for PC app Since Kindles and Kindle apps will *already* open legally-purchased Topaz files, it could be seen as encouraging piracy.
However, the real benefit here is that now we can focus on figuring out the Topaz format (as in comment 450 above, at https://darkreverser.wordpress.com/2008/02/13/new-blog/#comment-450 ) and somebody will figure out how to convert to other formats. Once the format itself is decoded, then we’ll hopefully have the option of format shifting, so those of us who want to read our Topaz books on, say, a Windows Mobile phone will be able to do so.
And by the way, the author of the script is so worried about piracy that, as “some updates” posted above, he encodes your Kindle login info in the un-DRMed file, to make sure you won’t be sharing it with anybody.
December 31, 2009 at 1:53 pm
Yes. Now conversion! I was able to un-drm a topaz book after commenting out the section here:
#
# Read the encrypted database
#
### Commented out to allow operation without Kindle4PC installed.
# try:
# kindleDatabase = parseKindleInfo()
# except Exception as message:
# if verbose>0:
# print(message)
as I do not have Kindle4PC installed and it seemed to work fine. I was able to read both the original and de-drm’d files on my kindle2
December 31, 2009 at 2:55 am
The decrypted output displayed fine in a Kindle for PC.
December 31, 2009 at 3:36 am
In my “test”, the liberated file couldn’t be opened by a friends Kindle For PC app. Have to be careful with “tests” as my Kindle Login info is there embedded (thanks chorpler for that piece of info).
December 31, 2009 at 4:45 am
@Chorpler: Thanks for posting the script fixes and the command line arguments to use the script. After making the adjustments to the script using IDLE, I was able to decrypt one of my Topaz eBooks. Worked like a champ.
December 31, 2009 at 6:27 am
I can now confirm that the DRM-strippage works. The file could be accessed by two completely different IDs on two different machines. All that remains is to get the content into shape that will let it be format-shifted.
December 31, 2009 at 6:51 am
I’m getting 1KB for my output file. It’s a free Topaz book from Amazon. Here is the link http://www.amazon.com/Confronting-Challenges-Participatory-Culture-ebook/dp/B0030DFWZM/ref=sr_1_1?ie=UTF8&s=digital-text&qid=1262242107&sr=1-1
Can someone try to decrypt it so I can see if I’m doing anything wrong?
December 31, 2009 at 12:15 pm
I can’t either.
It gives me a
“KeyError: ‘AbaZZ6z4……”
One thing to note – I don’t have a Kindle for PC installed. (I have installed it, but could not get it to start. It always crashes.)
The script looks like it uses some info from Kindle PC version, not sure.
December 31, 2009 at 2:04 pm
See my post above about commenting out the section that reads the K4PC info.
December 31, 2009 at 2:04 pm
Lilly,
It produced a 1.23MB file. Make sure you made the corrections to the script that chorpler kindly provided above.
Then run it like this: (I renamed it to test.prc to make it easier)
cmbdtc.py -v -d -o test1.prc test.prc
That’s all it took.
Stew
December 31, 2009 at 3:04 pm
Thanks. I made the changes already but I needed to indent the For line after the uncomment change. Once I fixed that it was great.
December 31, 2009 at 12:37 pm
@Tedd, I was having the same problem with K4PC. I stopped a lot of programs that always start up at boot. For example Active-Identity, Unlocker, all Apple programs and Quicktime. I’m not sure which one fixed the problem, but I stopped all that I didn’t need to have running and K4PC started working fine. In fact the same thing was happening to the Sony Library software also, which started working also.
I am sure you need K4PC installed before running that script, but not 100% sure.
December 31, 2009 at 5:00 pm
Thanks for the tip. Sony program works fine for now and it’s been confirmed that uninstalling MS Office 2007 (non-US version that is) makes it work by several people sharing this problem.
I haven’t tried other eliminations though. I will give it a try.
December 31, 2009 at 3:32 pm
kennyc, I made the changes for it to run without K4PC. Are you using your Kindle PID to run the script? The following works for K4PC books cmbdtc.py -v -d -o test1.prc test.prc but not Topaz books that I copied directly from my Kindle.
December 31, 2009 at 5:18 pm
It works with K4PC version of TPZ.
So if you want to run the script without K4PC then you have to manually put your kindle PID in the script, right?
Could you please point out how to do this?
January 1, 2010 at 12:50 am
That’s the question I’m posing to Kennyc. I too am stuck on how to do this. He does not have K4PC loaded on his computer and yet the script works for him. I’ll let you know once I find out.
January 1, 2010 at 3:25 am
Answered the question below how to run the script using a book from your Kindle.
December 31, 2009 at 4:33 pm
I was wondering about that also. Aren’t the files that come off of the Kindle a azw1 or tpz file extension? Will the script work with other than PRC files?
December 31, 2009 at 5:58 pm
Hi,
Would someone please try the following with there Topaz file and cmbtc’s code?
cmbtc_v1.1.py -v -r dict:0 -o dict.txt BOOKNAMEHERE
or
cmbtc_v2.0.py -v -r dict:0 -o dict.txt BOOKNAMEHERE
What I am seeing in dict.txt is just a huge list of all of the words in the book. The first 2 bytes are in fact the 7 bit encoded number for the number of strings in the file.
Then try the same for a few of the early pages:
cmbtc_v1.1.py -v -r page:0 -o page0.dat BOOKNAMEHERE
cmbtc_v1.1.py -v -r page:1 -o page1.dat BOOKNAMEHERE
cmbtc_v1.1.py -v -r page:2 -o page2.dat BOOKNAMEHERE
cmbtc_v1.1.py -v -r page:3 -o page3.dat BOOKNAMEHERE
It seems page*.dat files are the “template” for each page and somehow words from dict.txt must be properly inserted in the template to make the proper page.
I have looked at this I I can’t see how this mapping is being done.
A fun puzzle nonetheless!
December 31, 2009 at 6:37 pm
When I ran the first command, the txt file was all gibberish. I looked the the dat file with notepad and the word page is the only clear word. I will try to look at the other pages.
December 31, 2009 at 6:45 pm
Hi,
Perhaps there is more than 1 dictionary?
So please try both of these and see if anything is in either dict0.dat or dict1.dat that looks readable.
cmbtc_v2.0.py -v -r dict:0 -o dict0.dat BOOKNAMEHERE
cmbtc_v2.0.py -v -r dict:1 -o dict1.dat BOOKNAMEHERE
If not, then the formats themselves must be highly variable!
December 31, 2009 at 7:10 pm
I got an error when trying to run the dict1 command, and the file could not be written.
When doing the dict0 command it looked alot like the dict.txt file that I produced earlier.
December 31, 2009 at 7:11 pm
Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.
C:\Users\Scott>cmbdtc.py -v -r dict:0 -o dict0.dat twotofifth.prc
Device PID: PM4D53KT
Book PID: NQX1A8IG
Book key: 7bafdfa08c45255c
Wrote record to file: dict0.dat
C:\Users\Scott>cmbdtc.py -v -r dict:1 -o dict1.dat twotofifth.prc
Device PID: PM4D53KT
Book PID: NQX1A8IG
Book key: 7bafdfa08c45255c
Could not find record
Traceback (most recent call last):
File “C:\Users\Scott\cmbdtc.py”, line 893, in
sys.exit(main())
File “C:\Users\Scott\cmbdtc.py”, line 879, in main
extractBookPayloadRecord(recordName,int(recordIndex),outputFile)
File “C:\Users\Scott\cmbdtc.py”, line 437, in extractBookPayloadRecord
raise CMBDTCFatal(“Could not write to destination file”)
__main__.CMBDTCFatal: Could not write to destination file
C:\Users\Scott>cmbdtc.py -v -r dict:1 -o dict1.dat twotofifth.prc
Device PID: PM4D53KT
Book PID: NQX1A8IG
Book key: 7bafdfa08c45255c
Could not find record
Traceback (most recent call last):
File “C:\Users\Scott\cmbdtc.py”, line 893, in
sys.exit(main())
File “C:\Users\Scott\cmbdtc.py”, line 879, in main
extractBookPayloadRecord(recordName,int(recordIndex),outputFile)
File “C:\Users\Scott\cmbdtc.py”, line 437, in extractBookPayloadRecord
raise CMBDTCFatal(“Could not write to destination file”)
__main__.CMBDTCFatal: Could not write to destination file
C:\Users\Scott>
December 31, 2009 at 7:51 pm
Hi,
That means you only have 1 dict page (the second one could not be found) just like I do.
But sadly it seems your dict0.dat is very different in structure to my dict.dat file.
Mine is a huge list of string lengths followed by strings that make up all of the words in the book.
If yours does not look like that, then the formats must be highly variable.
I am beginning to this this is a bitmapped file format that draws words on the screen (must like PDF) and that we will never easily be able to recode it.
Oh well … I will try again later when I feel more motivated.
Thanks for checking that.
December 31, 2009 at 8:00 pm
I tried the dictionary option with both of the Topaz books I used to try the script and indeed the output files contain words from the books – I can recognize phrases, settings but it seemed a sort of short hand for each page, more like snippets from the text – it also covers the whole book since the last words are from the last page
December 31, 2009 at 8:20 pm
Yes,
That is exactly like mine. I don’t know why stewball’s is so different form the ones like you and I are seeing.
From what I can tell. The dict contains each word in the book exactly were in the book it is first used. So you can think of a big array holding all of these strings and then a base offset into the array for that page is given and then the words are selected out to recreate that page.
I think that the page*.dat files are in fact those templates for how to select out the words that make up the page but I simply can’t figure out how yet.
Please take a look at page0.dat (see above) , page1.dat, etc and see if you can tell how the words from dict.dat are mapped into each page.
December 31, 2009 at 8:04 pm
My test book only has one dictionary, dict:0.
December 31, 2009 at 8:27 pm
I opened dict0 in Ultra Edit and I could read alot of the book. I was opening the dat file in notepad before and was getting gibberish.
It’s amazing how much of the book is visible. And I believe its the whole book in my one dictionary.
Stew
January 1, 2010 at 3:21 am
Thanks to KennyC, I now know how to run the script using a book that I copied directly from my Kindle 2. You have to use your Kindle PID, leaving out the last 2 digits (stop at the *). I used a Command Prompt to cd to the folder were I copied the book and the cmbdtc script and ran the following:
cmbdtc.py -d -v -o test.prc -v -p 8ZY7ABC* -v “Confronting the Challenges of Pa-asin_B0030DFWZM-type_EBOK-v_0.azw1”
January 1, 2010 at 3:30 am
This was the result!
C:\Mobi Books\Unswindle>cmbdtc.py -d -v -o test.prc -v -p 8ZY7ABC* -v “Confronti
ng the Challenges of Pa-asin_B0030DFWZM-type_EBOK-v_0.azw1”
DSN: 6Cnhn6t07xP5SkPft5tEPb6dP7t37fnr7YnS54t4
Device PID: ILYP7F4B
Account Token: 9d088fa821483dbf576516d065d9cb5dd06170c9
Book PID: kHMS9Evx
Book key: 621c89eb4e862f1b
Decrypted book saved. Don’t pirate!
C:\Mobi Books\Unswindle>
January 1, 2010 at 10:59 am
wow. thanks.
By the way, Gurus are all speaking higher language
I wish I could understand. 🙂
January 2, 2010 at 4:09 pm
Cool! Glad you got it going Lilly!
January 1, 2010 at 8:42 am
Looking at the dictionary file for a popular astronomy book, I think it almost looks like badly-OCRed text, featuring entries like “solar,yslen,” (presumably “solar system), “space,huttlc” (presumably “space shuttle”), and “rainhow” (clearly supposed to be “rainbow”). This could be a bad sign, since it might mean that Topaz books only contain the information of a scan and an OCR, not a real text format that would allow us to export it to another format.
These results do match the quality of this book, however, because this particular book displays very poorly, with large gaps in the letters themselves (like the book was scanned with the brightness setting too high, so the thinner parts of the letters were washed out). So if the dictionary file contains OCRed (and not even proofread) versions of the words on each page, well, this book isn’t going to be very exportable. If we could reconstruct the page images we’d be able to re-run our own OCR and proofread it, but what a pain.
The one thing that makes me wonder about this conclusion is the fact that you’re supposed to be able to search Topaz files, right? But on my Kindle you can only search via an index of all the books, and the Kindle for PC app doesn’t seem to have a way of searching at all that I can find. So I can’t test to see if searching for “solar,syslen,” actually comes up at the place that says “solar system” in the graphical text.
Anybody know what format the glyphs are stored in?
January 1, 2010 at 8:50 am
Oh, and incidentally, this meshes well with what a guy who worked for a publisher told me about why they sometimes used Topaz instead of Mobi for kindle books: he said they used Topaz when they didn’t have a handy electronic version of the book, so they’d just send a copy of the book to Amazon and Amazon would whip up a scanned copy really quick for them in Topaz format. I don’t know if that’s true, but it certainly seems true if the actual text in the Topaz file really is as bad as it seems with this astronomy book.
January 1, 2010 at 1:47 pm
There’s a new CMBDTC out, Version 2.1; this one actually compress the file, making your output the same size as your input. Here is the link.
http://www.pastie.org/763115
January 1, 2010 at 3:17 pm
Thanks Lilly. The script author said that would be his last version.
Don’t forget to make the script corrections.
January 1, 2010 at 5:42 pm
chorpler: which astronomy book did you use to get solar,syslen?
January 1, 2010 at 6:40 pm
“Death by Black Hole.”
January 2, 2010 at 3:39 am
I posted an update version of skindle (04) that handles both mobi and topaz files. It has an option to output decompressed topaz files if you want to have a clearer view of the file format. No corrections to be made and no “social drm”
http://rapidshare.com/files/329044305/skindle-04.tgz
January 2, 2010 at 4:24 am
Awesomeness!! Skindle, you have my undying gratitude.
January 2, 2010 at 4:35 am
Thanks Skindle. I haven’t used this before so I am installing Cygwin to test it out.
January 2, 2010 at 5:02 am
Skindle, when I start up the executable, it shows up, then disappears. What could I be doing wrong?
Thanks
January 2, 2010 at 5:16 am
Skindle,
Nevermind. I am a knuckle head. I ran it with the command prompt and it worked great. Thanks again.
Stew
January 2, 2010 at 9:20 am
Updated skindle takes additional command line args
http://rapidshare.com/files/329149409/skindle-05.tgz
January 2, 2010 at 12:21 pm
Great!
Does resulted tpz file suppose to be a bit short of original in size? About 600 bytes?
January 2, 2010 at 8:46 pm
yes because the decryption key blob is not written to the output file
January 2, 2010 at 4:52 pm
Rapid share is almost impossible to get it to download unless I have a premium account. Uhg!
January 2, 2010 at 9:04 pm
free downloads should ask you to wait not very long at all
January 2, 2010 at 5:10 pm
Er, I got bellow message, but the resulted file cannot be opened in K4PC
I used -p flag to provide my Kindle PID as downloaded tpz was for the device.
Using VolumeSerialNumber = “26177781xx”
Device PID: DQD6TVxx
PID for Murder Takes The Cake is: Vr4Txsxx
Didn’t find PID magic numbers in record
Didn’t find PID magic numbers in record
Didn’t find PID magic numbers in record
Found a DRM key!
95edd88c0d2c10xx
Success! Enjoy!
January 2, 2010 at 6:06 pm
My bad. Clicking refresh button in K4PC makes them all fine.
January 2, 2010 at 9:50 pm
updated skindle to remove accidental cygwin dependency
http://rapidshare.com/files/329403401/skindle-06.tgz
January 2, 2010 at 11:30 pm
Thanks Skindle. I was placing the cygwin1.dll file in a folder with the Skindle files to satisfy the programs need for it. I tested v6 without it and it worked great.
Stew
January 3, 2010 at 12:04 am
I was able to **imperfectly** decode page0.dat from my file and see the following (it was the cover page)
_PAGE__
0x5f 0x3 0x3d 0x74 0
glyph
0x74 0
links
0x74 0
page 6
type fcvr
h 13488
pageid 1
pagelabel cvr1
w 8578
startID 0
container 0x32 0x5 0x35 0x39
h 13488
w 8578
x 0
y 0
src
img 5
h 13488
w 8578
x 0
y 0
src
0x0 0x0
Notice the x and y positioning and text container and height and width settings.
The next page has the following:
ocrText
So it was done by optical character recognition software. It does appear to be a page layout language very much like the original pdf.
The bottom line is that if I spent a long period of time I could reverse engineer the page*.dat files, But I am not sure how of if we could ever do anything with them. Because of the x, y positioning, word order need not even be linear. So even just getting the text from the file would be a pain and imperfect at best.
So I think it is pretty much a lost cause.
The answer is to simply NOT buy TOPAZ books. They are really not convert-able to any other format, are painful to read, slow, etc.
This is pretty much why I stay away from PDF as a book format as well.
Oh well.
I am moving on to other things, sorry I can’t be more help.
January 3, 2010 at 12:14 am
Before signing off, could you explain exactly how you decoded the page? I agree it looks bad, but I’d still like to see exactly what you had to do to get the page info decoded.
January 3, 2010 at 12:27 am
Oh, one thing in particular I’m interested in is what the glyphs represent. Do the Topaz books display each page by putting glyphs that are letters right next to each other, or do they store one glyph for each word? I think it must be letters, numbers, symbols, etc., because my average-sized “Death by Black Hole” book has only 160 glyphs in it, and another book only has 134 glyphs.
January 3, 2010 at 12:29 am
Oh, and not just what the glyphs represent (since it seems highly probable they represent letters) but what format they’re stored in. Is there some easy way to modify the cmbdtc.py script to uncompress any records that are compressed in the file? Though if the glyphs are just bitmaps of letters, they may not be compressed with a standard compression algorithm at all…
January 3, 2010 at 12:13 am
Thanks to all of you for all your hard work. Some updates, what you found was pretty much what I was afraid of. I’d cuss, but it wouldn’t do any good. Ok, guess we’ll be doing this the hard way….I refuse to have a book I can’t format shift to meet my needs. And I can’t avoid Topaz books, a bunch of stuff I want to read is not available any other way. Screen shots and OCR, here I come… *sigh*.
January 3, 2010 at 12:17 am
zanf, supposedly copistar is working on extending their app to do the screenshots and saving-as-images part automatically, assuming they can get it to work with the Kindle for PC app. That may end up being the best way to do things. Of course, with a book like “Death by Black Hole,” where the original scan is of marginal quality, even that may not work well (though I think I could still do a much better job than Amazon did).
January 3, 2010 at 12:32 am
chorpler, I must admit I’m glad to hear it. I just tried it with screenshots from K4PC into Word, and they come out TINY. So I hope copistar manages to find a way to do it more efficiently.
January 3, 2010 at 1:10 am
Well, I’ve confirmed that the Topaz books often use crappy OCR.
After erasing my Kindle, copying “Death by Black Hole” over, and letting it re-index the book, I searched for “rainhow” and found it on an index page (which, for whatever reason, would not change the font size from “ultra tiny” no matter what I tried to change the font to, almost as though it were just a page image — the same thing happens on Kindle for PC; both the “name index” and “subject index” appear to just be page images, rather than resizable text … although they don’t show up in the extracted images, so they must be made of glyphs or something somehow …) under the listing for “rainbow seven.”
On the other hand, as I said, the font (or rather, word size, since it isn’t actually text, from what I can see) on those index pages is truly tiny; maybe the OCR is better on other pages. That might be the single bright spot.
January 3, 2010 at 1:22 am
By the way, did anybody happen to save a copy of the Topaz file exporation info that used to be up at this blog:
http://www.latenightcode.com/devblog/kindle-topaz-file-format-explorations-part-i/
It’s gone now, but it might have some interesting additional information for us.
January 3, 2010 at 1:55 am
Hi,
I have posted my poor decode attempt at:
http://pastebin.com/m73d11ed9
To use it you will need 2 pieces:
First is the dictionary of all words in the book
cmbtc_v2.1.py -v -r dict:0 -o dict.dat YOURBOOKHERE
Second you need a page to test with – the simpler the better so maybe the book cover or title page
cmbtc_v2.1.py -v -r page:0 -o page0.dat YOURBOOKHERE
Then by studying what I know to be on certain pages and what the binary page files looked like in a hex editor (and having the entire dict printed out), I was able to determine some very rudimentary things about the format.
These I put in the program decode_page.py
Please note, that if anything gets out of sync, the file produces garbage.
So all you need is one tag that needs an argument that I haven’t seen yet, and it pretty much blows up.
You can run the decode_page.py code as follows
decode_page.py dict.dat page0.dat
Here is what it shows on my machine:
python decode_page.py dict.dat page0.dat
__PAGE__
0x5f
0x3
0x3d
0x74:0
glyph
0x74:0
links
0x74:0
page 6 type fcvr
h 13488
pageid
1
pagelabel
cvr1
w 8578
startID 0
container 50 5 53 57
h 13488
w 8578
x 0
y 0
src
img 5
h 13488
w 8578
x 0
y 0
src
0x0
0x0
If anyone recognizes any of the syntax, please let me know. Knowing which strings need arguments and how many would make decoding this much much easier.
Have fun and see if you can fix and extend it to understand which string tokens need arguments and which don’t and how the other commands (all in the 0x70 range) work.
January 3, 2010 at 2:31 am
Thanks man. Hey, if you want to discuss this in more detail, e-mail me at vadir89 on gmail.
January 3, 2010 at 11:18 pm
Hi,
I have advanced it to the point where it will drop the ocrText’d that is at the top of the page on pages with text (before it barfs on something it does not understand). The text has line break indicators but not layout information.
So it will be eventually possible to recover the text and all of the images from a topaz file for conversion to another format but I am not sure how much of the formatting will be salvageable.
I will work a bit longer to try and figure out a few more things and then post my version so that others can try and add to it and improve it.
January 4, 2010 at 10:32 am
Wow, very impressive. I’m eagerly awaiting your next version!
January 3, 2010 at 3:24 am
Hi,
I tried to decode a few more pages and ran into typos and thinkos.
I will work a bit more and then send you something better at some point.
January 3, 2010 at 4:10 am
A non Topaz comment. 🙂
I ran into a problem with mobidedrm today. I was using version 0.07 and noticed that the output mobi file (without drm) was missing a few pages. So I compared it to the original file in the proprietary reader and discovered that about 120 pages were missing. There did not appear to be a pattern to the missing pages. The missing text begins within a sentence – it is not on a page boundary. The text picks up between 10 and 20 pages later again in the middle of a sentence. This happens about 8 times in the book. Just for fun I tried version 0.06 and all of the pages were present. So there appears to be a problem in the 0.07 script. Please let me know what information you need to debug
Thanks.
January 3, 2010 at 6:24 am
I ran into similar problem. Not really a solution but, try to get a different mobidedrm007 from some other source. That’s what I did and worked. I believe there is some mobidedrm007 that is broken floating around.
January 3, 2010 at 9:09 am
If you could send the original file, your PID, and your copies of 0.07 and 0.06, that would be a great help.
January 4, 2010 at 2:17 am
Hi Folks, I have tried to remove the DRM of four dictionaries. The whole process seems to succeed (MAC OS X 10.6, python 2.6.1):
MobiDeDrm v0.07. Copyright (c) 2008 The Dark Reverser
Decrypting. Please wait… done
But I’m not able to open the files. I tried on Stanza and on my Kindle.
Some one got better luck on similar files?
January 4, 2010 at 2:43 am
This was the stack trace on stanza (FYI):
stanza.util.Log FINE 03:42:13: stanza.util.FormatErrorException: java.lang.ArrayIndexOutOfBoundsException: -1
stanza.util.FormatErrorException: java.lang.ArrayIndexOutOfBoundsException: -1
at stanza.util.BookFormatService.load(BookFormatService.java:150)
at stanza.util.BookFormatService.loadBook(BookFormatService.java:108)
at stanza.container.swt.Stanza$16.run(Stanza.java:3591)
Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
at stanza.format.mobi.MobiPocketFormat.getSizeOfTrailingDataEntry(MobiPocketFormat.java:469)
at stanza.format.mobi.MobiPocketFormat.getSizeOfTrailingDataEntries(MobiPocketFormat.java:456)
at stanza.format.mobi.MobiPocketFormat.decompressHuffdic(MobiPocketFormat.java:245)
at stanza.format.mobi.MobiPocketFormat.decompress(MobiPocketFormat.java:227)
at stanza.format.pdb.PDBFormat.decompressRecords(PDBFormat.java:302)
at stanza.format.pdb.PDBFormat.readFile(PDBFormat.java:225)
at stanza.format.pdb.PDBFormat.load(PDBFormat.java:60)
at stanza.format.pdb.PDBFormat.load(PDBFormat.java:56)
at stanza.util.BookFormatService.load(BookFormatService.java:146)
… 2 more
January 4, 2010 at 8:22 pm
I’ve been looking at a file that’s problematic with all MobideDRM versions back to 0.02. It’s the pesky Extra Data Flags again. I think I have a solution, but there have been a couple of other reports here of problems with ModiDeDRM that I haven’t been able to reproduce.
Sample files (with PIDs) demonstrating the problems much appreciated. If you can find a free file on the Kindle store that exhibits the problem, that’s even better.
January 5, 2010 at 3:25 pm
Looks like everyone has moved on to the topaz problem.
I’ve got an issue with the latest versions (I think) of ereader2htnl.py (0.09) and erdr2pml.py (0.11).
The extracted image file names are not matching the generated pml & html code. The images file names in the img directory contain all lower case letters, while the pml & html code are looking for a file name with a mixture of upper & lower case.
I’ve searched the code and the blog, but am unable to locate what to change. It’s not a big deal to change the image file names in the img directory to match what the pml and html code expects, but it can be annoying at times.
I expect the image file names in the img directory are being sanitized in some way, while the markup language file names are not?
Can someone point me to the right place in the script(s)? I’m not a python guy.
January 5, 2010 at 3:55 pm
Yes,
Your analysis is correct and a problem. As far as I understand things, the file names should NOT be touched except to strip out characters that would mess up on some operating systems.
Right now in the code you can see:
def getImage(self, i):
sect = self.section_reader(self.first_image_page + i)
name = sect[4:4+32].strip(”)
data = sect[62:]
return sanitizeFileName(name), data
The problem is sanitizeFileName(s) does the following:
def sanitizeFileName(s):
r = ”
for c in s.lower():
if c in “abcdefghijklmnopqrstuvwxyz0123456789_.-“:
r += c
return r
It converts everything to lower case. I am not sure that is correct. It seems to be a bug if the pml itself is looking for mixed case file names.
Their are two solutions:
1. add the upper case chars to the sanitize string code
or
2. stop using the sanitize string code at all.
I am not sure what is best but I would try to change sanitizeFileName to look like the following:
def sanitizeFileName(s):
r = ”
for c in s:
if c in “ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_.-“:
r += c
return r
In case that get messed up, I am doing the following:
1. delete .lower() from the s.lower() line in sanitizeFileName
2. adding the A-Z capital letters inside of the quotes to the list of lower case letters in the “if c in” line.
This should do the trick.
January 5, 2010 at 4:24 pm
Worked like a charm on the several books I ran it on.
Thanks for the assistance!
January 6, 2010 at 6:43 am
I was having the same problem with ereader2htmlv9 and erdr2pmlv11. I couldn’t get dropbox to work on any of the resulting pml files due to problems with the image file. I’m glad that you figured out the glitch, I had given up & moved back to ereader2html/pmlv6 as that was the last version that I could get to work. Time permitting, I’ll go into the saved pml files & update the image names.
DiapDealer, which solution from someupdates did you end up using?
Thanks!
January 6, 2010 at 3:20 pm
Unless you have a whole lot of images, it’s easier to change the image names than the pml code.
I used the last part of someupdate’s post to correct it:
1. delete .lower() from the s.lower() line in sanitizeFileName
2. adding the A-Z capital letters inside of the quotes to the list of lower case letters in the “if c in” line.
Line 321 in ereader2html v9
Line 325 in erdr2pml v11
Just make the sanitizeFilename function look like:
def sanitizeFileName(s):
r = ”
for c in s:
if c in “ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_.-“:
r += c
return r
January 6, 2010 at 8:18 pm
I’ve just put MobiDeDRM 0.08 up on pastebin
http://pastebin.com/f523cbb17
This version incorporates the code to make it a Calibre plugin, as well as a stand-alone script. (i.e. it’s both – thanks for the code, David Woodhouse. And yes the awz/azw typo is fixed in this version.)
It also has a fix for a problem with a very few Mobipocket files, that caused corrupt characters to be inserted into the file every so often (at block boundaries). These files involved are Mobipocket verison 5 files, which seem to be very rare, at least from my sources of ebooks.
regards,
Alf
January 7, 2010 at 2:47 am
Thank you for the update. I am not much of a programmer so I have no clue what to do, but do you think you could do the plugin treatment to ereader2html?
The latest version I can find is 0.09 here:
http://pastebin.com/f7be19a99
Thank you in advance.
B
January 7, 2010 at 5:44 am
Thank you! For some reason version 7 never worked as a Calibre plugin for me but version 8 is working very smoothly. I just bought a book from the Kindle store and now Calibre just reads in the file and converts it right away without any problem. It’s good to know that now I have a backup of a ebook or a file I can convert to another format if I get a different brand reader! Thanks again! Without this site I would never, never, never have spent more than $300 already on ebooks. I truly think that if publishers keep this DRM thing going it’s likely to kill the whole field.
January 6, 2010 at 8:30 pm
[quote]corrupt characters inserted into the file every so often.[quote]
mobihuff 0.03 seemed to do the same thing with one book I used it on. Same problem? Is there a newer version?
January 6, 2010 at 10:01 pm
MobiHuff 0.03 has very poor code for detecting extra data at the end of records. Changing that bit of code to match the code in 0.08 would probably be a big improvement.
January 6, 2010 at 9:25 pm
No improvement for me in ver 08 😦
January 6, 2010 at 10:03 pm
Stanza may have problems reading the trailing data. But if the dictionaries don’t work on a Kindle, there does seem to be a problem.
Can you point to a source for the problem files?
January 7, 2010 at 8:36 am
Thanks, I’m not sure what you mean by ‘the source for the problem files’ the files I’m trying to read are Dictionaries enclosed on a prc file that I obtained from ‘the mobi****t shop’.
MobiDeDrm’s output:
MobiDeDrm v0.08. Copyright (c) 2008 The Dark Reverser
MOBI header length = 228
MOBI header version = 6
Decrypting. Please wait… done
I took the source code for MobiDeDrm from http://pastebin.com/f523cbb17
After I run the script against a prc file I still can’t open the dictionary in my Kindle, I Used Stanza too because there are better chances to actually get information from errors in it.
I’ve also checked and the file generated by v7 and v8 are exactly the same:
MD5 (undrm_v07.prc) = 41be145c43693a98c6066d83813a010f
MD5 (undrm_v08.prc) = 41be145c43693a98c6066d83813a010f
Have you got any success yourself to clear out the DRM from any dictionary files so far?
January 7, 2010 at 9:16 am
I’ve certainly been able to decode Mobipocket dictionaries.
Chambers (10th ed): https://www.fictionwise.com/ebooks/b70285/The-Chambers-Dictionary-10th-edition/Chambers-Harrap-Publishers/?si=0
Pocket Oxford: https://www.fictionwise.com/ebooks/b14825/Electronic-Pocket-Oxford-English-Dictionary/Oxford-University-Press/?si=0
and I’ve successfully installed the Chambers as default dictionary on my Kindle.
By “source” I meant which dictionary, and where, so that I could obtain a copy to test. Or you could email me one of the troublesome dictionaries with your PID.
It certainly looks like the decoding worked correctly. Stanza is not a good test of Mobipocket decoding, as it has bugs in its handling of mobipocket books.
Have you tried in the Mobipocket Windows reader?
January 7, 2010 at 10:10 am
Hi Again, I just tried on mobipocket windows reader, it also crashed 😦
The sources I’ve tried are Collins English to french, Collins French to english (http://www.mobipocket.com/en/eBooks/eBookDetails.asp?BookID=68810), PONS German-English and PONS English-German (http://www.mobipocket.com/en/eBooks/eBookDetails.asp?BookID=121466).
Both in mobipocket reader and in Kindle I’m able to see the title of the dictionary, it appears as an accesible item but when I try to open it it fails.
Does it make sense for you to have a look at the dump of mobipocket reader? (The affected files are: MobiPalmSync.dll, msvcr80.dll, pdf2xml.exe, reader.exe; and the mobipocket version is 6.2)
January 7, 2010 at 1:02 pm
I wish you’d pointed to just one of the French/English; English/French – I didn’t realise the combined book was just the two individual books. I could have saved myself $6!
However, the good news is that I can reproduce your problem. Hmmm….
January 7, 2010 at 2:35 pm
Reproducible bugs are always good news to me! (Thanks and sorry for the extra expenses ;-))
January 8, 2010 at 9:50 pm
Hi Paul, I still didn’t manage better results using v09 on the dictionaries mentioned above.
Did you manage better results so far? I might be doing something different from you!
January 8, 2010 at 9:55 pm
Yes, on the Collins English/French French/English dictionaries, 0.09 decodes them fine for me.
Please email me (see website) and perhaps then we can get to the bottom of the problem.
January 9, 2010 at 9:39 am
Thanks Paul!
But I have a very basic question, which website shall I look for? Is your mail the one in the Huffdic compression root post?
January 9, 2010 at 9:41 am
Oh! Sorry ignore my previous e-mail I found your website (-: !
January 9, 2010 at 10:09 am
Hi Again and thanks a lot for your compromise towards our community, it finally worked!
The only difference from before is that before (when it was not working) I had two different PID registered on mobipocket.
Then I removed one of them. And after downloading the file I’ve seen that its md5 was different than before!
I tried MobiDeDrmv09 again and it worked, might it be related to multiple devices registered on the DRM?
Thanks again!
January 9, 2010 at 10:33 am
I’m gald to hear that your problems seems to be solved. I don’t think that the different number of PIDs had anything to do with it though.
January 7, 2010 at 4:31 pm
It’s been pointed out to me that I was wrong about MOBI headers of 0xE4 not having extra data flags. It seems that only version 5 and earlier mobi headers of 0xE4 don’t have them. version 6 mobi headers can be 0xE4 or 0xE8 in length, and both have the extra data flags.
0.09 fixing this bug is now at
http://pastebin.com/f696ea728
regards,
Alf
January 7, 2010 at 7:38 pm
Thanks. Any chance you could convert ineptpdf and ineptepub to Calibre plug-ins?
Don’t mean to sound ungrateful – thanks for all your hard work! You make my kindle so much more enjoyable.
January 8, 2010 at 1:01 pm
DR, Paul, et al,
Awesome work, v008/v009 seems to fix the corruption in files such as “Around the World in 80 Days” by Jules Verne from Amazon for Kindle. This appears to be a MOBI v5 file and your fixes worked well!
Unfortunately, the similar type of corruption (and truncation) still exists in books like “Atlas Shrugged” by Ayn Rand also from Amazon for Kindle. This appears to be a MOBI v6 file and the corruption isn’t affected at all by the new version of MobiDeDrm, sadly.
I suspect that Amazon will not be using this version 6 type of encoding on free books for testing, but I’ll continue to look for them. In the mean time, I am willing to donate $9.99 to support your efforts if you need to purchase a copy of A.S. for yourself for testing, etc. Just let me know how to get it to you/you’all.
Best,
Tagny
January 8, 2010 at 1:34 pm
As it happens, I got all of Ayn Rand’s works from Fictionwise when they were on a very special offer. So I have an encrypted Mobipocket version of “Atlas Shrugged”. It does seem to be a version 6 Mobipocket file, but as far as I can see, the fictionwise one decodes perfectly. There are a distressingly large number of OCR errors in it, however, that should have been picked up easily by a spell-check.
It might be that the Kindle one is converted differently, or I might be missing the errors. Could you give some examples of where in the text your version decoded with MobiDeDRM 0.09 is corrupted, so that I can check I’m not missing something.
January 8, 2010 at 8:04 pm
I’ve been in correspondence with Tagny.
The upshot is that Stanza doesn’t read Mbipocket files very well. 0.09 decoded the Ayn Rand book fine.
I don’t know of any Mobipocket books that don’t work with 0.09. If you think you have one, post a comment on this blog.
January 8, 2010 at 9:41 pm
Paul is quite correct here. I have since dumped Stanza (for the mac) and replaced it with Calibre and all of my DeDRM’ed files are perfect. And, it does a great job of working with Stanza for the iPhone (which is the best eReader I’ve found on the iPhone) by converting the PRC files into EPUB files very nicely and hosting a server for the iPhone app to use to download the books.
Many thanks to Paul!
January 8, 2010 at 5:37 pm
For those of you interested in Topaz conversion:
There is good news and bad news. The good news is that I have a “test” python script that takes the output of cmbtc_v2.0 when used to dump individual records that creates snippets of pseudo xml that includes the ocrText and some stylesheet info and the like.
The bad news is that yet another program has to be written to make use of the xml pieces, handle the xml injection and output something html-like.
But I have run out of free time. So see
http://pastebin.com/m6ad70be5
for the decode_page.py script.
It pretty much documents everything I know so far about the conversion so that others may take over. It is enough for me to change by hand my 1 topaz book into a form that will eventually load into an epub for my Sony Reader
To use the script – first generate a sampling of files from the book to work with. You must get at least dict0.dat since it is needed for everything else and you probably want to get a sample of page*.dat files and other0.dat file to see what they look like decoded.
Here is how to get a few files to play around with from your book
(you can use cmbtc_v1.1.py as well)
cmbtc_v2.1.py -v -r other:0 -o other0.dat YOURBOOKHERE
cmbtc_v2.1.py -v -r dict:0 -o dict0.dat YOURBOOKHERE
and then a selection of pages
cmbtc_v2.1.py -v -r page:0 -o page0.dat YOURBOOKHER
cmbtc_v2.1.py -v -r page:10 -o page10.dat YOURBOOKHERE
…
repeat for other selected page values
You can also get selected images (so far all have been jpeg) by doing
cmbtc_v2.1.py -v -r img:0 -o img0.jpg YOURBOOKHERE
and similar
Then once you have these decoded record files (I have a modified version of cmbtc that actually dumps all of the files in the book to a directory)
Your can begin to play around with decode_page.py
You use it as follows:
decode_page.py dict0.dat other0.dat > stylesheet.txt
decode_page.py dict0.dat page0.dat > page0.txt
…
decode_page.py dict0.dat page10.dat > page10.txt
Then you can look at the decodeed txt.
The other0.txt file will be the stylesheets and other info used to process the the page*.dat files more fully
The page*.txt files will all have garbage at the top since they are missing pieces of other xml files that need to be injected in the right place to handle styles and things.
No guarantee it will work for anyone else, but it is enough to do my book.
Hope this gets others going on converting this into html.
January 8, 2010 at 10:00 pm
Hi,
I found even a few more thinkos, but I think this version is the last.
I won’t bother to post it unless someone actually wants to play around with it.
Please reply to me here if anyone wants the latest version of decode_page.py
January 8, 2010 at 10:02 pm
I’d appreciate a copy of your latest. I might not get around to playing with it much at the moment, but there’s no point in re-inventing the wheel later. Thanks.
January 8, 2010 at 7:38 pm
Hi,
I found a few more thinkos and fixed them.
Here is a better version of decode_page.py
http://pastebin.com/mb29d493
January 8, 2010 at 10:18 pm
I wanted to poke around with this, but apparently cmbtc_v2.1.py was written for a Windows Python environment.
I tweak… but porting to linux is beyond me. 😦
January 8, 2010 at 10:43 pm
Sure Paul,
Here it is.
http://pastebin.com/m47526cf8
January 9, 2010 at 3:39 pm
Is anybody else tinkering with this?
I’m able to duplicate all of someupdate’s results with his latest decode_page.py except the decoding of the other0.dat file (the one that represents the stylesheets). It fails with the following error(s)
Traceback (most recent call last):
File “decode_page.py”, line 598, in
sys.exit(main())
File “decode_page.py”, line 593, in main
pp.process()
File “decode_page.py”, line 537, in process
print self.decodeCMD(v, ‘number’)
File “decode_page.py”, line 481, in decodeCMD
return self.doLoop72(argtype)
File “decode_page.py”, line 393, in doLoop72
result += self.procToken(self.dict.lookup(val))
File “decode_page.py”, line 119, in lookup
print “Error – %d outside of string table limits” % val
TypeError: %d format: a number is required, not NoneType
Great stuff so far! And it gives me hope that this is indeed doable.
January 9, 2010 at 6:23 pm
Hi,
That error means for some reason you have read past the end of the file.
There is an undocumented debug switch the will produce a lot of output.
Were there any ‘***’ pieces of text in the output – these indicate unknown entities.
I have now posted yet another version, fixing a very few minor things, but it may help.
http://pastebin.com/mc7d58b9
Please try it first. If it dies, then try using the undocumented debug switch as follows:
decode_page.py -d dict0.dat other0.dat > output.txt
That should tell us more.
The more people that try decode_page.py and report errors, the faster we can lock down the final fixes and start on the next phase.
January 9, 2010 at 7:36 pm
I get the same error as above when running the newest decode_page.py on the other0.dat file. And it produces an empty stylesheet.txt file.
I tried it with the debug switch and the output does indeed contain some unknown entities/tokens.
The output file produced with the debug switch is 106Kb. Do you want me to post some snippets of that file, or do you want me to get it to you somehow?
January 9, 2010 at 7:45 pm
Hi,
Look in the text file (open it with any text editor) and search for the first ‘**’ or ‘Unknown’ or ‘UNKNOWN’
That is the problem pieces.
Everything before that is fine.
Everything after that is messed up since things may be out of sync caused by the unknown item
Make a copy of other0.txt and edit it down to remove everything but the last few lines before the “Unknown” or UNKNOWN” and then keep the next few lines after the first unknown.
That should help me to identify what tag exists that is not in my book but does exist in yours.
Posting it here or better yet
Take your web browser to http://pastebin.com
Simply paste in your source and press the send button.
Then post the link to that page.
Thanks
January 9, 2010 at 8:48 pm
This is the first occurrence of ‘Unknown’ in the Output0.txt.
Right about the middle.
http://pastebin.com/m6da6aff0
(I hope it automatically converts urls to links)
Thanks for your time!
January 9, 2010 at 9:08 pm
Hi DiapDealer,
Please try this version. You have things my other0.dat does not and it seems to mess up the parsing and it chokes on a null byte (tries to interpret the null byte as a token to be processed.
http://pastebin.com/m5e53e665
If this does not work, then I will send you privately my e-mail and you can send me a zip file of just other0.dat and dict0.dat and I can debug it from my end.
Thanks for helping with this.
January 9, 2010 at 9:22 pm
some updates,
Looks like it’s still trying to parse a null byte as a token with that newest version. Let me know what I need to do to get you these files.
You obviously need dict0.dat and other0.dat… do you want any of the page.dat files as well?
Thanks for your work so far!
January 9, 2010 at 10:15 pm
If you have an account on MobileRead simply pm KevinH with your preferred e-mail address and I will contact you.
Once we are in contact you can just e-mail me the zip archive.
Thanks again for helping to debug this.
I actually have a version of decode_page.py that can parse the glyphs*.dat files a bit as well.
January 9, 2010 at 10:41 pm
Good deal. I’ll wait to hear from you on MobileRead then.
January 10, 2010 at 2:58 pm
Hi,
Thanks to DiapDealer’s help. We now have a new version of decode_page.py that even will decode the glyphs*.dat files.
See:
http://pastebin.com/m18e6df91
There is still work that needs to be done … given that there are two undocumented loop styles (modes 4 and 5) that need to be figured out but we are getting closer.
Right now it is possible to take this info and write a quick and dirty python program to dump out all of the text, dehyphenate it, add in line breaks as indicated, add in paragraphs markers, make links, drop image files, put in roughly correctly placed image tags, and even add in style classes.
The problem is the glyph data itself is where all of the font, font size, font weight, font style info is and you literally have to look at the glyphs formed on screen itself to figure out what is happening on that front.
The glyphs are literally just a set of vertices (x,y pairs) that draw the outline of the glyph. There are no names, no weight info, no anything, just the raster outline of what should appear there.
So this part will really take some work.
Hope this helps.
January 13, 2010 at 1:59 am
Regarding your loop modes, here is what I know from looking at k4pc.
Take the mode byte that you are reading. The least significant bit dictates whether a bias value follows or not (you are reading the bias in your loop 1 and 3 handler, it is also present in any odd number loop type such as 5) The remainder of the mode value is shifted right one bit (dropping bit 0). The resulting value is a loop count that tells you how many times you need to perform the summation loop that you have in modes 2 and 3 (1 in those 2 cases) Here is a stab at general purpose mode loop (my python is not good). This should capture your cases 0-3 as well as the cases you have not covered
def doLoop76Mode(self, argtype, cnt, mode):
result = ‘{ ‘
adj = 0
if mode & 1:
adj = readEncodedNumber(self.fo)
mode = mode >> 1
x = []
for i in xrange(cnt):
x.append(readEncodedNumber(self.fo) – adj)
for i in xrange(mode):
for j in xrange(1, cnt):
x[j] = x[j] + x[j – 1]
for i in xrange(cnt):
result += self.formatArg(x[i],argtype)
result += ‘}’
return result
January 14, 2010 at 1:57 pm
Is there any way you could post that code on pastebin? There’s no easy way of knowing what goes with the first ‘if” clause without indentation.
January 15, 2010 at 4:26 pm
The code above can be found at
http://www.www.pastebin.com/f5aae9bad
January 17, 2010 at 8:12 pm
Hi,
Just found this. I have figured out modes 4, and 6 and even have a shot at 5 (I look for cases when this mode is used to process ocrtext, and from knowing what should be on that page and where in the dictionary it is, I reverse out the mode.
Future versions of the topazscripts.zip will instead incorporate your code.
Thanks!
January 11, 2010 at 6:43 pm
Ereader Question:
syntax for pml2xhtml.py is : python pml2xhtml.py infile.pml [bookinfo.txt] outfile.html (Is that correct?).
My real question is what is the syntax on running the initial erdr2pml.py script? Thanks
January 13, 2010 at 9:39 am
Anyone?
January 12, 2010 at 2:55 am
Thank you so much for all of the wonderful help given here. I really hate the ereader software for use on my Droid. Now with all of the help here I am able to ditch ereader and go with Aldiko which is far superior. I have converted many pdb and mobi files so far with no problem. I am very new to all of this and prior to reading this blog had never even heard of python or drm. I have come a long way but am still a baby at all of this.
I was hoping some of you could help shed some light on some recent troubles I have had converting two pdb books. The conversion with erdr2pml seems to run just fine, but when I go to calibre to convert to epub I get the message failed to read metadata from the following: What am I doing wrong?
January 12, 2010 at 5:21 pm
For those interested in converting Topaz to some other format:
Here is the very latest version of decode_page.py.
It has a number of bug fixes and can actually decode the glyphs*.dat files as well as the other0.dat and page*.dat files.
It should be very close to complete.
http://pastebin.com/m2675b7c8
Help is needed for the next steps:
1. Someone to start with decode_page.py and create a decode_glyphs.py file that takes the glyphs*.dat files and converts each one into an SVG standalone font file that can be referenced from the html.
To help, I can explain what I have figured out about how glyphID’s are used to determine which glyph*.dat file to grab the character from. I can also explain how relative font size can be calculated. Beyond that, more work is needed here.
2. Someone to start with decode_page.py and figure out how to handle the insertion of xml snippets properly to create an actual xml description of the page.
3. Someone to write code that processes the xml from the full xml page generated in step 2, extracts the ocrText, handles the dehyphenation, insert font based style tags, insert the links and images references, handle html tag generation for paragraphs, and in general maps things into html.
With those 3 pieces I think we can safely convert from Topaz format to another format.
This is a lot of work. Simply too much for one person. I have done what I can and I am now ending my involvement unless some help from other developers (preferably in python) is offered.
If not, then I hope someone picks ups these pieces and runs with them.
January 12, 2010 at 7:12 pm
I hate to see this die at this point. Unfortunately, my python scripting skills are limited to modifying other people’s code to suit my needs.
I have managed to mangle the original cmbdtc.py so that it is platform independent – just supply your kindle pid (in the script or on the command line) to decrypt books or extract records (social drm free). The dat files produced still work with some update’s decode_page.py anyway.
I’m more than willing to supply dict*/other*/page*/glyph*. dat files if it will attract other developers who may not have access to these type of books or don’t have their own kindle pid.
January 12, 2010 at 8:03 pm
DiapDealer –
I am interested in your platform-independent version of cmbdtc.py – can you post it somewhere?
January 13, 2010 at 3:24 pm
I’ve written a quick script to dump the glyphs in a decoded glyph XML file into SVG files. It’s in perl, because my religion strictly forbids python.
http://pastebin.com/f4fbb1976
You might need to use the -r switch which will calculate the height and width of the glyphs based on the data instead of the reported sizes from the XML. This is likely because topaz specifies weird widths to do correct kerning/tracking. However those sometimes weird widths leave half a letter when translated to SVG.
January 14, 2010 at 7:59 pm
Hi clarknova,
Wonderful!
I have slightly modified the output format to decode_page.py (it now does not use the { and }) so I will modify your code to handle this case.
One question:
Is is possible to use something like FontForge or some other program to take your svg glyph files and group them to create one ttf or type 1 font from each glyphs*.dat file?
I don’t know the legal max for the number of glyphs in a font, but each page uses char values that do not overlap so, it might be possible to collect all of the *.svg files and convert them into one font file.
I know nothing about glyphs really so any input on if this is possible would be useful.
I have kept working and figured out to properly inject snippets to create full xml descriptions of each page, and I have even started work on using that to convert the ocrtext to something useful.
Your approach of making a full svg description of each page might be the best solution since so much of what is there relates to placing glyphs at exact positions on the drawing surface.
Great work!
January 12, 2010 at 5:30 pm
[…] is exactly what I am saying. Go here and read the post made by some updates on December 20th, 2009 at 5:18 pm. Word of caution, they are […]
January 12, 2010 at 8:43 pm
grokker,
Nothing fancy, it works the same way as the original (v 2.1)… just doesn’t need Windows/K4PC.
http://pastebin.com/m3beb356c
January 12, 2010 at 11:36 pm
Thanks DiapDealer, much appreciated.
January 13, 2010 at 1:59 pm
My PC(Windows7) can’t run Kindle for PC, don’t know why. I had to use Vmware for XP to use K4PC. Such a troublesome process that was. Thanks.
January 13, 2010 at 3:37 pm
@test011,
I had the same problem on Windows 7. Go Run>msconfig. The go into the start up boot programs and turn everything off you do not need. For example itunes, quicktime, the Microsoft Office Groove-something or another, Active Client, etc. This fixed my problems. IT was also causing Sony’s new reader software to freeze up, or not start.
Restart your computer, and this might fix it.
Hope this helps,
Stew
January 14, 2010 at 12:34 pm
Thanks for the tip. I just tried that with several different options without a success. The problem probably is registered dlls. Since many local forum people have the same problem and only commonly installed application is MS Office 2007 non-us version. This might be caused by a non-us language specific dll.
January 13, 2010 at 10:28 pm
Wow. These 500something comments were a 5 hour fun read! Wow! Just WOW.
It’s amazing how many bright people are working for that one goal, to remove DRM “protection” from already purchased e-books. There seems to be a strong sense of ethics here, not to do this for cheap reasons; in fact every one has bought a book he wouldn’t have otherwise just to be able to tinker around.
I tip my hat to all of you!
January 14, 2010 at 2:44 pm
Is there any solution for ebooks created with EBX_Handler (etd) method?
January 14, 2010 at 6:59 pm
Problem: For all Pdfs, which can be read by Digital Editions the solution is ineptpdf.py. Unfortunately Digital Editions 1.7.1 does not download eBooks with format(ebx.etd). A lot of books, protected with Ad Digital Right Management, can be read by Reader 7, 8 or 9, but only time-restricted. I am looking for a method to remove drm from .etd files.
Can you help?
January 14, 2010 at 10:52 pm
@Loverboy,
Can you print pdf’s in Adobe Reader?
I used Lizardtech virtual printer for pdf’s protected by fileopen plugin, no quality loss.
If not I would try to save pdf as a set of images.
Check out which options are available for you.
August 15, 2010 at 6:02 pm
Use ineptpdf 8.4.31 (or higher) for accurate FileOpen support.
January 15, 2010 at 7:57 am
I have a problem with the 0.09 mobidedrm calibre plugin. It doesn’t work with non-DRM files. It seems to corrupt these files on import – I get error messages:
ERROR: ERROR: Unhandled exception: TypeError:coercing to Unicode: need string or buffer, NoneType found
Traceback (most recent call last):
File “site-packages\calibre\gui2\dialogs\metadata_single.py”, line 138, in add_format
File “site-packages\calibre\gui2\dialogs\metadata_single.py”, line 152, in _add_formats
TypeError: coercing to Unicode: need string or buffer, NoneType found
)
and the metadata is missing from these files.
Anyone can change the plugin, so it leaves non-DRM files alone?
January 15, 2010 at 8:56 pm
Help I feel like an idiot. I want to get rid of drm from a mobi book from the library with a time limit so my wife can read it on here jetbook lite. When I try to run the script I have in python it just chokes on the book name. Maybe I have the wrong sytax or maybe I need to put things in a different directory. Can you please give some step by step instuctions to a guy trying to help his wife. Thanks
January 17, 2010 at 3:51 am
Converting Topaz to HTML
This is experimental and it will probably not work for you but…
ALSO: Please do not use any of this to steal. Theft is wrong.
This is only meant to allow conversion of Topaz books
for other book readers you own.
Here are the steps:
1. First you must use the python scripts in topazscripts.zip to do the translation from Topaz to HTML
The files you should have after unzipping are:
cmbtc_dump.py – (author: cmbtc) unencrypts and dumps to files all of the sections, properly numbered and named
decode_meta.py – converts metadata0000.dat to human readable text
convert2xml.py – converts page*.dat, other*.dat, and glyphs*.dat files to their “pseudo” xml descriptions.
flatxml2html.py – converts a “flattened” xml description to html using the ocrtext and markup as its basis.
stylexml2css.py – converts stylesheet “flattened” xml from other0000.dat into css (as best it can – mainly supporting paragraph style classes)
genxml.py – main program to convert everything to xml
genhtml.py – main program to generate “book.html”
2. You must remove the DRM from the Topaz book and build a
directory of its contents using the following commands:
cmbtc_dump.py -d -o TARGETDIR [-p pid] YOURTOPAZBOOKNAMEHERE
This should create a directory called “TARGETDIR” in your current directory.
It should have the following files in it:
metadata0000.dat – metadata info
other0000.dat – information used to create a style sheet
dict0000.dat – dictionary of words used to build page descriptions
page – directory filled with page*.dat files
glyphs – directory filled with glyphs*.dat files
3. You should convert the files in “TARGETDIR” to their xml descriptions
Please note, this python program uses “decode_meta.py” and “convert2xml.py” so don’t move them.
genxml.py TARGETDIR
4. Next attempt a conversion to html where “TARGETDIR” is the directory
that was created in step 2. Please note, this python program uses “decode_meta.py”, “convert2xml.py”, “flatxml2html.py”, and “stylexml2css.py” so don’t move them.
genhtml.py TARGETDIR
Once it completes:
You should have created the file “book.html” inside of TARGETDIR
You should also have created the directory xml inside of TARGETDIR
which has the full xml descriptions of the pages and glyphs for later
(better) conversion attempts.
You can’t post a zip on pastebin.com, so we really need someplace/someone to host these. If that is something you are willing to do, pm me on Mobileread and I will get the scripts to you.
One warning … this is not the best long-term solution because much of the layout is only really correct if drawn to the screen (as an svg). Until that solution exists, this should get you something that you can load into Sigil and clean up and make an ePub that you can then convert to other formats
January 17, 2010 at 4:12 am
some more updates: Where can I download the scripts involved. I’ve only got some of them. Thx.
January 17, 2010 at 4:47 am
Hi,
All have been modified since earlier postings. They are not posted anywhere yet. I am looking for a volunteer to host them. If no one volunteers, then we will have to figure something else out.
The 7 different python scripts needed would be too hard to keep up and in sync via pastebin.com.
Ideas welcome?
January 17, 2010 at 6:50 am
Yes, please, if a host site can be found, would you please provide a link?
Derek
January 17, 2010 at 6:51 am
Using a distributed version control system would make most sense, so it’s easy to see what’s changed, and for you to pull in contributions that other people make.
Send me an email; I can give you an account on git.infradead.org so that you can publish a git repository. I can also set up a mailing list for discussion.
January 17, 2010 at 12:30 pm
Using a public version control system would be great, but you’ll have to find one that won’t be taken down by a DMCA notice or worse – distributing DRM software is illegal in the US, which is where git.infradead.org.
January 17, 2010 at 5:28 pm
Where would you like it hosted? Right now I can do .ca, .uk, .us and .it.
The USians think their silly laws apply to the whole world anyway.
January 17, 2010 at 6:18 pm
The topazscripts.zip also seem to be here:
http://www.mediafire.com/?qmzjmt25yzf
January 20, 2010 at 7:32 am
ok, using the new version 1.5 (I tried it with the binaries posted above – thanks brutusbum, they worked perfectly) the “region type vertical” error I was getting was there, now as a warning, but the process was not aborted and I got the book.html file this time. Here is the message I got running genhtml.exe:
page0343.dat
page0344.dat
page0345.dat
page0346.dat
page0347.dat
Warning: Unknown region type vertical
Treating this like a “image” region
page0348.dat
Processing Complete
ok, book.html created, some very minor things to correct (there was achange of order on the title and subtitle….) but it looks rather good 🙂
some_updates, do you want me to still post that xml page?
and thanks again!!
January 20, 2010 at 1:39 pm
No need.
It can stay as a warning. There really wasn’t anything inside that region I could deal with in any way.
January 21, 2010 at 6:38 pm
Actually, it cannot be found there any longer.
January 21, 2010 at 6:40 pm
Not any longer. That link is dead.
January 21, 2010 at 7:35 pm
JWolf, both links are still ok, just tried them with success…
anyway, I uploaded the file to box.net, maybe it is easier for you:
http://www.box.net/shared/gqukrbp0js
hope this helps 🙂
January 21, 2010 at 8:36 pm
Thank you Soalla!! That was very helpful and I, too, appreciate it.
January 27, 2010 at 10:30 pm
Am I correct that none of these scripts (or others) will work on epub files? The new iPad is said to use the epub format. I am hoping that Apple, realizing that removing DRM from all their music hardly hurt them,will post ebooks in epub without DRM. But, I doubt it. I do want to get an iPad though (ok, ok, Mac fanatic here) and would like to be able to strip DRM from the files it supports. Failing that I guess I can always buy books from Amazon, strip them, use Calibre to turn them into epubs and feed them to the iPad.
January 27, 2010 at 10:35 pm
you search for ineptkey and ineptepub you should find scripts that will remove Adobe DRM from ePubs.
There’s no word yet on which (if any) DRM Apple is going to have on the ePubs in their ebook store.
January 28, 2010 at 8:42 am
To all the hard working people whose work culminated into this set of scripts: Bravo!!
I just spent 3 hours going through all these comments, feeling the suspense… did they succeed? How are they going to do it, trying some of the steps, skipping some… I must admit that at some point I went quicker, only skimming the text to see what happened rather than trying to understand all the technical details. Then I reached the place were “some updates” declared “Since the bug reports have stopped, here is the (hopefully) final version (2.0) of topaz scripts.” What a rush! I downloaded them and followed the instructions… just one little hiccup… that [-p pid] which I didn’t realize was optional, and then, delightful surprise: I got my book into HTML format, which I quickly converted to a pdf file which I loaded immediately in my reader. It works!
This is totally awesome! Thank you so much to all of you!!
I wish I could reward you somehow. Money? You guys should open a Paypal account for donations. Anything I can help with (I am a C++ software engineer)? Please let me know.
January 28, 2010 at 9:09 am
Hello,
Can you help me to know how I get my PID if I just have Kindle4PC?
The script kindlepid.py requires a serial number that I don’t have…
Thanks a lot
Riccardo
January 28, 2010 at 9:18 am
kindle4PC doesn’t have a single PID – every book downloaded to it has its own PID. You need the unswindle.py script
http://i-u2665-cabbages.blogspot.com/2009/12/circumventing-kindle-for-pc-drm.html
January 28, 2010 at 9:27 am
Thanks a lot, Paul.
Unswindle was my first option.
The problem is that Kindle4PC is showing some new bugs.
The 1.0.1 Beta version (that is now being distributed by Amazon) will NOT open, out of the blue, ebooks that it had always opened. It is a known error to many users.
The precondition for unswindle to work is that it can open Kindle4PC; that you open manually the book and only THEN does unswindle call your magical script.
However, if the ebook can’t be opened (because of this Kindle4PC bug that Amazon developers can’t resolve), then mobidedrm.py cannot be applied by unswindle.
That’s why I was hoping that I could apply mobidedrm DIRECTLY to the file so to be able to open it with some better reader 😉
I will then check unswindle to understand how it gets the PID out of each single file.
Thanks again,
Riccardo
January 28, 2010 at 2:07 pm
Try using skindle. This doesn’t require the K4PC app to be open like unswindle, it just grabs the key info directly from the kindle.info file.
http://rapidshare.com/files/329403401/skindle-06.tgz
January 28, 2010 at 2:44 pm
Hi Clarknova,
Thanks a lot for your help. Something’s wrong even for Skindle :-S
I’ve launched skindle by command line with the same (administrator) account I had used to download the .prc Amazon file, but not even skindle can get the right PID. I get an error that is not even reported in the whole blog :-S
Attempting to locate kindle.info
Found kindle.info location
Using UserName = “PIPPO01”
Using SystemDrive = “C:\”
Using VolumeSerialNumber = “508290077”
Device PID: F8KDI5FX
PID for Implementing-8_Distribution is: b/dn5p3w
No valid pids available, failed to find DRM key
An error occurred, unable to process input file!
Is it possible Amazon is distributing a different encryption?
I bought the book on Dec. 21, 2009
Thanks again!
Riccardo
January 28, 2010 at 3:37 pm
There’s no new encryption. If the file doesn’t open in K4PC, then there’s something wrong with it.
Try this:
Open K4PC, go to the home page, right-click the book and select “remove from device.”
The go to your archived items and re-download the book. Then try drm stripping again.
January 28, 2010 at 3:45 pm
Yes,
I guess something’s wrong with the file, but I have already removed it and re-downloaded it many times.
I have even un-installed Kindle4PC and re-installed it (as suggested by Amazon callcenter), then re-downloaded the file (which is an SAP manual).
The only change I think did to the PC was changing the network password for company policies (not the User, though).
All the other files open without any problem.
I wanted to decrypt the .prc file because I thought the problem was with the Kindle4PC… Now I am starting to think that the problem bears with the file.
What baffles me is that it had worked for two weeks and then it didn’t work anymore.
I’ll try to have the money back for the malfunctioning file from Amazon… Who knows :-S
Thanks anyways, Riccardo
Just one question: what’s with the .mpb files that come with the .prc? nothing useful in them?
January 28, 2010 at 6:21 pm
Hi, in the end, everything worked with unswindle.
The problem is that – when you reinstall Kindle4PC – the kindle.info file is not deleted and this messes everything up.
I deleted both the “My Kindle Content” and the “Amazon” folder in Local Settings (after proper backup), reinstalled Kindle4PC, re-downloaded the book.
Only now could I open the book and convert it to non-drm.
DRM really sucks, my friends!
Thanks everybody
Riccardo
January 29, 2010 at 4:53 pm
The latest script package is here (version 1.6):
http://www.megaupload.com/?d=Z0RH24XI
This contains Windows binaries as well as the source code for all of the following except for UNSWINDLE which remains python only:
Contents:
Adobe DeDRM scripts:
INEPTKEY
INEPTEPUB
eReader DeDrm scripts:
eRdr2Pml012
ereader2html09
xPml2XHtml018
KINDLE-MOBI fixing scripts:
KINDLEPID
KINDLEFIX
Mobipocket DeDrm Scripts:
MOBIDEDRM09
MOBIHUFF
PC Kindle DeDrm tool:
SKINDLE
UNSWINDLE (Python Script only)
Amazon.com’s Topaz DeDrm Scripts:
CMBTC_DUMP
CMBTC_DUMP_NONK4PC
CONVERT2XML
GENHTML
GENSVG
GENXML
Barnes and Noble Epub DeDrm:
IGNOBLEEPUB
IGNOBLEKEY
IGNOBLEKEYGEN
Reply
February 2, 2010 at 5:42 pm
I’m not finding the precompiled binary for skindle.exe in v.1.6?
TIA
February 2, 2010 at 7:00 pm
Oooops!!!
Here is a nnew version of the archive (Version 1.7) that contains the following: (Including the SKINDLE binary):
This contains Windows binaries as well as the source code for all of the following except for UNSWINDLE which remains python only:
Contents:
Adobe DeDRM scripts:
INEPTKEY
INEPTEPUB
eReader DeDrm scripts:
eRdr2Pml012
ereader2html09
xPml2XHtml018
KINDLE-MOBI fixing scripts:
KINDLEPID
KINDLEFIX
Mobipocket DeDrm Scripts:
MOBIDEDRM09
MOBIHUFF
PC Kindle DeDrm tool:
SKINDLE
UNSWINDLE (Python Script only)
Amazon.com’s Topaz DeDrm Scripts:
CMBTC_DUMP
CMBTC_DUMP_NONK4PC
CONVERT2XML
GENHTML
GENSVG
GENXML
Barnes and Noble Epub DeDrm:
IGNOBLEEPUB
IGNOBLEKEY
IGNOBLEKEYGEN
http://www.megaupload.com/?d=ASHGYRMT
I am removing the older V 1.6 archive for clarity’s sake.
B
February 2, 2010 at 7:41 pm
“Unfortunately, the link you have clicked is not available.
Reasons for this may include:
– Invalid link
– The file has been deleted because it was violating our Terms of service.” on the new archive.
February 2, 2010 at 7:47 pm
Sorry, found an error in one of the scripts.
Updated to version 1.71 here:
http://www.megaupload.com/?d=AMLZYT0J
B
February 14, 2010 at 6:27 pm
When running CONVERT2XML I get “Indexerror: list index out of range”. Any clues? Thanks!!
February 14, 2010 at 8:56 pm
Hi Blas,
I need more than that. Please copy the entire error message and explain what you were trying to do at the time so that I have some idea of what might be wrong.
You may have run into a whole new tag that no one else has seen.
What version of topazscripts are you using?
February 14, 2010 at 9:00 pm
Here is:
tools_v1.1.zip
http://www.mediafire.com/?ju3qzwmmjgy
The only change from tools_v1.0.zip is a change to allow very strange ordering in topaz xml not previously seen.
February 15, 2010 at 6:52 pm
With asian characters in path (“Topaz eBook input file” field or “Output Directory” field) breaks the front-end script.
February 15, 2010 at 6:54 pm
some_updates, could you please fix that,?
February 15, 2010 at 9:37 pm
Hi Tedd,
Sorry, no idea how. Using unicode with Tk widgets and in subprocess command lines is not something I know anything about.
None of the strings are specifically non-unicode that I know about, although I have no idea if something special is needed to get Tk to support unicode strings in Python 2.6.
Simply work around it by creating a directory without asian characters in the path and copy your book to it and set utf-8 as your encoding. All should work well from there if the underlying code in cmbtc_dump.py can handle it.
If anybody does know of a solution, patches welcome.
February 15, 2010 at 11:12 pm
Hi Tedd,
I checked and Tcl/Tk widgets only support unicode-16 (or utf-8) and not unicode-32.
Also the subprocess call (used to invoke the cmbtc_dump.py program) can not handle unicode characters at all right now. There is an open bug for it on the python development website (and it has been open for over a year now)
http://bugs.python.org/issue1759845
They have a workaround for windows systems (I assume you are using windows) but nothing has been implemented in the internal subprocess routine yet so even if Tk widgets are doing the right thing, it would not work when it reached the subprocess invocation.
So the only thing to do is try the workaround and create a directory off of C:\ that has no asian characters in it. Copy your book there and try. Also, I have no idea if any of the html Topaz conversion tools will work on a unicode book, if that is what you have.
Sorry I can’t be more help here.
February 16, 2010 at 12:19 am
Thanks some_updates,
I’ve asked for some help from a local forum. I didn’t show them any code though. Since this kind of character handling is so common a problem here, people here might have some solution. If anything comes up, I will post.
BTW, the book I tried was English right from Amazon.com. Never seen a Asian-languaged TPZ yet. I am using Windows7 on utf-8, I think unicode-32 is never used in practice. not in a general purpose code anyway. 😉
February 16, 2010 at 12:43 am
Hi Tedd,
In case this post got lost somehow …
Hi Tedd,
There is one thing you could try. Please edit TopazExtract_Kindle4PC.pyw and right **before** this line:
p2 = Process(cmdline, shell=True, bufsize=1, stdin=None, stdout=PIPE, stderr=PIPE, close_fds=False)
You could try adding the following line (with the same level of indentation).
cmdline = cmdline.encode(sys.getfilesystemencoding())
If that works, you can add the same thing to all of the other pyw scripts in the Topaz_Tools directory and they should work too.
This is of course assuming the problem is in subprocess and not in Tk itself.
If that works please let me know and I will add that to all of the tools and create a tools_v1.2.zip.
February 16, 2010 at 6:10 am
It works!
I guess similar encoding is also needed for printing initial “os.getcwd()”.
(The Chinese new year holidays just ended yesterday, the workload is killing me. Happy new year 🙂 )
February 16, 2010 at 11:12 pm
Hi Tedd,
Once you get a free moment will you try the following for me:
Find the first instance of the following line:
self.tpzpath.insert(0, os.getcwd())
and replace it with the following set of lines (all with the same indentation level as the original)
cwd = os.getcdwu()
cwd = cwd.encode(“utf-8”)
self.tpzpath.insert(0, cwd)
and see if that fixes the issue. Notice I have changed the command from os.getcwd() to os.getcwdu() which should return a unicode path – but I am not sure if utf-16 or utf-8, so we encode it to force it to utf-8.
If this fixes the os.getcwd problems (at least for the first input box) then I cna use that info to fix all of the gui tools to work better with other languages.
Thanks,
February 17, 2010 at 11:29 am
To all of you that contributed, thanks for the wonderful work. Tools v1.1 ereader seems to work flawless. I’ll try others later!?!
Anyone set up a new board for us on this topic? Something that can be more easily followed?!
February 17, 2010 at 4:02 pm
Thanks for the trial code. It works though there was a typo. (os.getcdwu / os.getcwdu 😉 )
#sself.tpzpath.insert(0, os.getcwd())
cwd = os.getcwdu()
cwd = cwd.encode(“utf-8”)
self.tpzpath.insert(0, cwd)
and
#self.outpath.insert(0, os.getcwd())
cwd = os.getcwdu()
cwd = cwd.encode(“utf-8”)
self.outpath.insert(0, cwd)
and
cmdline = cmdline.encode(sys.getfilesystemencoding())
in the right place works quite well.
February 17, 2010 at 7:48 pm
Hi Tedd,
Sorry about the typos, I have way too many of them and can never see them for some reason until I have already submitted things.
I have now modified all of the tools I wrote front-end guis for to hopefully deal with the issue of non-latin1 based names in directory paths and filenames.
I will post these versions as tools_v1.2.zip when I finally get a free moment. I would appreciate you testing them when you get a free moment to make sure they work for you, as I can’t recreate the problems on my own machine.
Thanks
February 18, 2010 at 4:09 pm
No problem! Glad to be any help.
February 20, 2010 at 11:49 am
Some_updates had posted this above, and with the weird wordpress system, it got lost in the mess, so, posted from above (all credits to some_updates).
—————-
Here is the latest update to scripts and gui tools:
tools_v1.3.zip
http://www.mediafire.com/?yjumy0dn0hy
changes:
– update to mobidedrm version 12
– fixes for all of the gui tools to work on asian based machines
– fixes for more out of order xml in the Topaz Tools
– now includes ineptkey_v4.3.pyw for use on Windows for ADE 1.71,
and 1.72
– now includes mac version of ineptkey
– new versions of mobiunpack.py and mobiml2html.py that work with normal (unencrypted) mobi tools to unpack them so that small changes can be made easily and then use MobiCreator to recreate them, plus a tool that converts them to xhtml for archival purposes
—————-
Thanks some_updates and everyone else for your hardwork!!!
February 21, 2010 at 9:26 pm
Thanks – this mostly works without error. I found a book whose last page is an empty glyph file, generating the following error:
Traceback (most recent call last):
File “gensvg.py”, line 398, in
sys.exit(main(”))
File “gensvg.py”, line 294, in main
gp = GParser(flat_xml)
File “gensvg.py”, line 31, in __init__
self.count = len(self.guse)
TypeError: object of type ‘NoneType’ has no len()
The book in question is the sample for the following:
While just deleting the empty glyph file works, I instead added some sanity checking to GParser’s __init__ routine in gensvg.py. The new routine can be found here:
http://pastebin.com/f7e987cb9
Might be a better way of handling this case – I’m a programmer, but haven’t done anything with Python before. The original Topaz book does have that empty page, so I think that it’s right to have it, but I could understand how someone might prefer to just strip it out instead.
February 21, 2010 at 9:27 pm
Sorry – this was supposed to be a reply to comment #1057. Though I guess it works as a general comment…?
February 21, 2010 at 9:29 pm
Also, I hate WordPress.
February 25, 2010 at 8:29 am
From some_updates, lost in the mess above:
—————————————————–
New tools release
tools_v1.4.zip
http://www.mediafire.com/?nwzjmmkyzdy
Changes in version 1.4 include:
– bug fix by “Chaos Incarnate” in Topaz Tools for gensvg.py
– add support for new “empty_text_region” tag to Topaz Tools
– eRdr2Pml.py new version 0.14 now supports –make-pmlz option
code contributed by calibreuser
– new gui tool: eReaderPDB2PMLZ.pyw to use the new eReader option
—————————————————–
Thanks for the work guys!
March 3, 2010 at 12:56 am
Gone already. Uploaded any place else?
March 3, 2010 at 1:46 am
If you are looking for the tools
The newest version of tools (v1.5) is available from:
http://www.mediafire.com/?njoxq4lnzqn
March 3, 2010 at 4:25 am
Got ’em, thanks!
March 10, 2010 at 7:38 am
i had a problem with the TopazFiles2HTML script. it fails on books where the table of contents references pages that don’t exist (eg some samples from the kindle store, like this one: http://www.amazon.com/Shooter-Autobiography-Top-Ranked-Marine-ebook/dp/B001C30JLO).
if you add a check for such a case @ line 471 of flatxml2html.py then the sample goes through fine. eg see the patch here: http://pastebin.com/hR3368FR
March 26, 2010 at 7:24 am
Hi can you repost a link to download the latest version of the tools? The mediafire link is broken. Thanks
March 26, 2010 at 12:50 pm
Here
http://www.mediafire.com/?jjyvnumnjzy
March 28, 2010 at 3:48 pm
FYI, with Python 32 bit installed, all scripts worked as advertised (so to speak). I deDRM’d my topaz file (John Hart’s The King of Lies) by (1) running TopazExtract_Kindle_iPhone, (2) then running the TopazFiles2XML .pyw script, (3) then running the TopazFiles2SVG .pyw script, (4) then running the TopazFiles2HTML .pyw script and (5) then opening the resulting book.html file in Word and using spell check to edit a number of errors (words like “my” and “me” often showed up as “mv” and “nie” respectively — also a lot of hyphens dropped, etc.).
I mention all of the above just in case it helps someone else out — although the GUI interface is wonderfully helpful in its own right.
Thanks again for your help, Some_Updates!
March 29, 2010 at 6:55 am
I’m getting an Invalid or Deleted File message. Thanks some_updates
March 29, 2010 at 11:25 am
Hi Victory1,
That is because you are using old links. Older versions are always deleted to prevent confusion.
The latest link is still valid. You just have to find the latest link which is hard given this forum seems to sometimes post out of order.
So, the best way to read this list is to use the RSS 2.0 feed available at:
feed://darkreverser.wordpress.com/2008/02/13/new-blog/feed/
From there you can see things in reverse chronological order and find the link to the latest things including tools_v1.6b.zip
Also you can check out Apprenctice Alf’s Blog and its comments to find links to the most up-to-date pieces.
tools_v1.6b.zip.
http://www.mediafire.com/?mn3vmttbwrt
April 5, 2010 at 2:25 am
thank you so much for this, ive bought a number a books on my kindle iphone app but prefer using stanza to read ebooks, and now i can, this was exactly what i needed . thanks again
February 2, 2010 at 7:02 pm
This archive has been deleted. See bottom of blog for latest archive.
B
January 21, 2010 at 7:36 pm
Very much alive… just downloaded it from there.
January 21, 2010 at 8:36 pm
Here also:
http://www.megaupload.com/?d=4SEFQLOT
January 31, 2010 at 12:28 pm
OK, found this and am trying to use ereader2html09.exe. Placed the eReader book in the same directory with the executable and pointed to the same directory for the outfile. Got error message “LoadLibrary(pythondll failedThe specified module could not be found.
Any ideas?
January 31, 2010 at 4:42 pm
You need to extract the complete archive to the same driectory (keep the directory structure).
These compiled scripts require the dll files that are in the archive.
B
January 17, 2010 at 12:31 pm
I mean, of course, distributing DRM-removal software…
January 17, 2010 at 6:37 am
use : 4shared.com easy to maintain.
January 17, 2010 at 10:57 am
Found your blog on Yahoo and was so glad i did. That was a warming read. I have a tiny question.Is it OK if i send you an email???…
January 17, 2010 at 5:54 pm
Apparently the file can currently be found at:
http://rapidshare.com/files/336800633/topazscripts.zip.html
January 18, 2010 at 2:10 am
I tried the package with the 6 topaz books I have; one worked perfectly and the book looks good in html, while the svg rendering is excellent
On the others genxml chokes on the occasional page.dat (about one in ten since once it crashed on one, I tried removing it, and then it works for 10 more page.dat or so and then it crashes..); actually convert2xml crashes with the error message being list index out of range
January 18, 2010 at 1:49 pm
that means the parser got very confused and tried to look something up in dict0000.dat that was outside the bounds of the file and it barfed as a results.
I can fix this but I am going to need your help.
First copy convert2xml.py intto your TARGETDIR.
Then copy from the page subdirectory one of your page*.dat files that is the problem case back up to the TARGETDIR so all of the pieces are in the same place.
Then run the following command replacing #### with the page number of one of the pages that seems to cause the problems.
convert2xml.py -d dict0000.dat pageNNNN.dat > debug.txt
The -d switch turns on some debugging information.
The “> debug.txt” should redirect the debug output into a file called debug.txt
Somewhere in the file will hopefully be something that caused the problem that I should be able to track down and fix.
Look in the debug.txt for the first string “Unknown” and copy a bunch of lines before and after that first occurrence and post it here for me (or send send the entire debug.txt file to me).
I am guessing your book simply has something in it that none of the other books so far have had, and we simply need to account for it.
Thanks
January 18, 2010 at 3:26 pm
It chocked up on mine at page0009.dat. When I run the debug command, I get this. C:\Users\Scott>C:\Users\Scott\TARGETDIR\convert2xml.py -d dict0000.dat page0009
dat > debug.txt
Traceback (most recent call last):
File “C:\Users\Scott\TARGETDIR\convert2xml.py”, line 821, in
sys.exit(main(”))
File “C:\Users\Scott\TARGETDIR\convert2xml.py”, line 807, in main
dict = Dictionary(dictFile)
File “C:\Users\Scott\TARGETDIR\convert2xml.py”, line 107, in __init__
self.fo = file(dictFile,’rb’)
IOError: [Errno 2] No such file or directory: ‘dict0000.dat’
January 18, 2010 at 6:42 pm
Done as above; got a pretty long debug.txt file with two occurrences of “unknown” – after the first, the script continued for a while, but last is where it stopped and the last line below is the last line of debug.txt
debug.txt:
………………………………
Snippet: 9
Processing: paragraph
subtags: paragraph has 1
Processing: paragraph.class
Loop for 2 with mode 0 :
Snippet: 10
Processing: word
subtags: word has 2
Processing: word.type
Processing: word.class
Snippet: 11
Processing: hang
Unknown Token: hang
Snippet: 12
Processing: book
subtags: book has 0
Snippet: 13
Processing: img
subtags: img has 5
Processing: img.h
Processing: img.w
Processing: img.x
Processing: img.y
Processing: img.src
Loop for 0 with mode 0 :
…………….
Loop for 1 with mode 0 :
Processing: paragraph
subtags: paragraph has 3
Processing: paragraph.class
Processing: paragraph.firstWord
Processing: paragraph.lastWord
Loop for 0 with mode 0 :
Mina Loop: Unknown value: 0
Injecting Snippets:
(end debug.txt, crash convert2xml.py)
January 18, 2010 at 4:04 pm
Hi,
That means it can’t find your dict0000.dat file which should be right where you are.
This can not be the true error since other pages worked.
Please make sure that all of these are in the some location (i.e. side by side inside of TARGETDIR
convert2xml.py
dict0000.dat
pageNNNN.dat
where NNNN is the number of the problem page*.dat file
Then make sure you have cd to the TARGETDIR and then run
convert2xml.py -d dict0000.dat pageNNNN.dat > debug.txt
where again the NNNN is the number of the page file that does not work.
Then look in debug.txt for “Unknown” or any other warning or error message and let me know what it says around that poin in the debug.txt file.
Thanks,
January 18, 2010 at 5:26 pm
I got it work. Before I did this I removed all the page.dat files that it could not process, 22 out of 302 and got a great looking HTML. So thanks for all your hard work. Here are the only two incidences of unknown (token or value) that I could find in page0009. I am going to look at some of the others now.
Processing: word
subtags: word has 2
Processing: word.type
Processing: word.class
Snippet: 14
Processing: PC-FR-V1_right_2_1
Unknown Token: PC-FR-V1_right_2_1
Snippet: 15
Processing: book
subtags: book has 0
Processing: paragraph
subtags: paragraph has 3
Processing: paragraph.class
Processing: paragraph.firstWord
Processing: paragraph.lastWord
Loop for 0 with mode 0 :
Mina Loop: Unknown value: 0
Injecting Snippets:
January 18, 2010 at 5:33 pm
Here are 3 debug files that I have done so far. They all appear the same as far as unknown errors goes.
http://drop.io/dlx9m8d
I hope I am helping some.
Stew
January 18, 2010 at 6:49 pm
Additional Info:
I noticed that when running the genxml.py, them 22 files/pages are not created. They are missing from the xml folder.
January 18, 2010 at 7:29 pm
Hi stew,
What you are doing is helping a lot!. It seems your book has an unknown snippet type that I have never seen before. Based on your debug.txt files, I was able to guess as to what it may look like.
I was able to build a test version of convert2xml.py
This may not fix the problem but it should help to figure out what else might be there, I need to handle.
Please grab it from:
http://pastebin.com/m6b9273e7
Then recreate the debug output and and any resulting xml files and post them for me to make sure I have not made things worse!
Thanks for testing things.
January 18, 2010 at 7:40 pm
Whatever you did it worked as far as I can see. It created all of the xml files, and then I was able to create the html without any errors. I will check the book over to see if it is complete.
Thanks again.
January 18, 2010 at 7:47 pm
Here is page0009.dat after running debug. It has alot more data and the errors are gone.
http://drop.io/joasdvg
January 18, 2010 at 7:47 pm
Hi stew,
That “word” tag must mean something. Please be careful that an entire word is not missing on those particular pages (compare the html word by word with the svg output for that page and let me know if there was something I should have done other than read it in.
Thanks
January 18, 2010 at 8:01 pm
There are some text errors in the html. Is there a place where I can send you what I am working on so you can see the whole product? I can send it on MobileRead.
January 18, 2010 at 7:52 pm
Can someone please post all the current various programs, where to get them, what versions, and what they do? Thanks.
January 18, 2010 at 7:55 pm
Hi stew,
From looking at the debug output, the code in question is somehow trying to insert and inline image right into a paragraph.
This could be a graphical symbol or mongram or it could be a character that is not handled by a regular font and needs special handling.
So please double check against the book for a missing or funny extra image someplace on the page.
If something is wrong …
I think I can modify the html conversion code to look for and handle this case if you are willing to test it.
January 18, 2010 at 8:09 pm
Hi Conrad,
Please try replacing your version of convert2xml.py with the one from here this one as well and let me know how it goes.
http://pastebin.com/m6b9273e7
You are having the same problem as stew is having.
My guess is that every chapter begins with an ornate graphical letter as an image that is freaking things out. That is why it only happens every 10 pages or so.
The version above will not output the ornate first character of the chapter (yet) but should not choke on it either
January 19, 2010 at 12:31 am
Great – all five that choked previously worked like a charm with the new script getting a nice html in each case; I did not do all the svg’s yet since I am not sure I really need them, but I expect they will work nicely too – the svg’s for the first two looked great, really like a well scanned page, but the html is good enough
You are absolutely right about the ornate letter – that’s precisely the case in the books I tried and the pages the script used to choke on – they are chapter beginnings and the weird letter is missing in the html though it appears in the svg; the pictures – covers, maps.. work nicely though in the html too
Many, many thanks
January 19, 2010 at 1:39 am
grab the latest version of topazscripts_v1.3.zip from the link one or two up from the bootom and your html should now have the letters above.
make sure it is version 1.3 though. The blog seems to have some messages stuck at the end that nothing new is getting past.
January 18, 2010 at 10:22 pm
I have been trying to remove a drm from a book I purchased yesterday from Amazon, and receive the following msg.
MobiDeDrm v0.08. Copyright (c) 2008 The Dark Reverser
MOBI header length = 228
MOBI header version = 4
Error: it seems that this book isn’t encrypted
If I try to use it with Calibre i receive the following:
ERROR: Could not convert: Convert book 1 of 1 (Karma Girl) It is a DRMed book. You must first remove the DRM using third party tools.
I have been able to get it the MobedeDRM script to work with other Amazon books. Renaming it to .mobi gives the same result.
Any suggestions would be helpful.
January 18, 2010 at 10:42 pm
MobiDeDRM 0.08 only gives that message if the encryption type in the mobipocket header is zero. So I don’t know why Calibre is saying that it is encrypted. If you look at the ebook’s page on Amazon, what does it say next to “Simultaneous Device Usage? If “Unlimited”, as for my ebook for Kindle, http://www.amazon.com/dp/B0034KZ0LC/ , then the book doesn’t have DRM.
January 18, 2010 at 10:45 pm
Oh – just noticed you gave the name, Karma Girl, presumably http://www.amazon.com/dp/B001CUUNJU/ ?
In which case, it does look like it’s encrypted.
If you send me your copy of the book, I’ll take a look to see what’s up. No need for the PID – it’s not getting that far in the decoding. You can find an email address on my website.
January 18, 2010 at 11:17 pm
Hi 🙂
some_updates, thanks for all your hard work!!
I tried your pack of scripts, first using the old version of convert2xml.py and getting similar errors to those got by Stew, and the using the new version of convert2xml.py with success creating the xml folder.
then I used genhtml.py and got a nice html file with some minor errors as when starting a chapter with an image or due (I believe) to some typos on the original topaz file… and also the original italics are gone! but it’s an excellent work and a readable file – I’ll try to use it later on Calibre and convert it to epub for my prs-505 🙂
also, I got this errors when using genhtml.py:
Processing …
metadata0000.dat
other0000.dat
page0000.dat
Unknown region type synth_fcvr.center
Warning: skipping this region
Unknown region type synth_fcvr.center
Warning: skipping this region
Unknown region type synth_fcvr.center
Warning: skipping this region
page0001.dat
page0002.dat
page0003.dat
do you know what it means?
January 19, 2010 at 12:04 am
Hi,
That error means the program that converts from xml to html has come across a region type it does not know about.
In this case, someone has synthesized a “cover” for the book, which may contain graphics or words or both.
I know of someone else with that same problem and they sent me a test case so this will be fixed before the next posting.
Also, as for all the italics being gone. That is correct and you will need to put them back manually (see the program Sigil) as all we have are the characters that the optical character recognition sees, and not if they are bold or italic. By looing at headings and things I can set some of the bolding , but the only wayt to see the itals is to use the ./gensvg.py program that clarknova developed.
With it, you can see the exact image of the page.
January 19, 2010 at 7:35 am
yep, that’s right, version 1.3 corrected those problems!! Thanks!!
now it’s time to go and reformat it (well, adding bold and italics where due) and then convert to epub 🙂
thatnks again to you and all the group!!
January 19, 2010 at 1:05 am
Hi,
There is a new version of topazscripts (topazscripts_v1.3.zip) that fixes many of the issues talked about recently on this list. It also includes a much improved gensvg.py program written by ClarkNova that now shows nicely smoothed fonts!
This is the work of many people. So please read the included file “readme.txt” and abide by its plea for no use of this for “theft”.
http://www.mediafire.com/?ik1yjndyccj
There should also be a link via rapidshare coming soon.
January 19, 2010 at 1:41 am
The new 1.3 scripts are also at:
http://rapidshare.com/files/337496753/topazscripts_v1.3.zip.html
January 19, 2010 at 3:25 am
Hi,
Here’s a debug.txt with an Unknown tag. These seem to be the numbered list of notes, at the end of the book. Let me know if you need more info. Thank you for your excellent work.
—- debug0218.txt —-
Loop for 0 with mode 0 :
Snippet: 21
Processing: region
subtags: region has 5
Processing: region.type
Processing: region.h
Processing: region.w
Processing: region.x
Processing: region.y
Loop for 1 with mode 0 :
Snippet: 22
Processing: paragraph
subtags: paragraph has 3
Processing: paragraph.class
Processing: paragraph.firstWord
Processing: paragraph.lastWord
Loop for 0 with mode 0 :
Main Loop: Unknown value: 0
Injecting Snippets:
bmatter
12273
222
198
7454
73070
listitem
461
5630
1075
1612
List 1
0
14
14
listitem
734
5635
1070
2136
List 1
14
50
listitem
672
5625
1075
2932
List 1
50
73
January 19, 2010 at 2:30 pm
Yes, it does appear to be a list of items (endnotes?) or an index of terms?
Generally, when you see Loop for 0 with Mode 0 followed by a 0 then the page is done. Anything beyond that is something else that has to be processed. Or there is really a new type of “snippet” I haven’t seen before and the parser is just out in the woods someplace.
The only way to deal with this is for me to have a test-case so that I can see what is going on and fix it. It looks like a number of new tags are being used (bmatter, listitem, List 1, etc).
So if you want this fixed, I will need the dict0000.dat and the pageNNNN.dat that causes the problem (look at the page*.dat printed last on the screen right before this error message). That should be enough to figure this out.
If you zip them up and post them for me, someplace, that would be a big help.
January 19, 2010 at 9:40 pm
http://www.easy-share.com/1909047757/sb-endnotes.tgz
pages 216 and 217 are there. Page 216 is where the notes start, and page 217 causes the program to crash.
Let me know if you need the next pages also, as the book have several pages of endnotes.
Thanks!
k
January 19, 2010 at 4:18 am
some_updates: thank you SO much for all your work on this. i realize there was group effort, but you clearly stepped up and led the charge.
I’ve tried working on two books. The first stops after page 1; the second doesn’t output any pages. Although an XML folder is created, I don’t get the book.html file as expected.
Here are the messages thrown while running the scripts on one of the books. I’m barely able beyond a layperson in this realm, so I’m not able to provide a lot of assistance. Thought I’d share the experience though, in the event that it helps you all with the end result.
C:\Documents and Settings\ShiTrain\My Documents\My Kindle Content\topazscripts_v
1.3>genxml.py TARGETDIR
Processing …
metadata0000.dat
other0000.dat
page0000.dat
Traceback (most recent call last):
File “C:\Documents and Settings\ShiTrain\My Documents\My Kindle Content\topazs
cripts_v1.3\genxml.py”, line 121, in
sys.exit(main(”))
File “C:\Documents and Settings\ShiTrain\My Documents\My Kindle Content\topazs
cripts_v1.3\genxml.py”, line 102, in main
xmlstr = convert2xml.main(‘convert2xml.py ‘ + dictFile + ‘ ‘ + fname)
File “C:\Documents and Settings\ShiTrain\My Documents\My Kindle Content\topazs
cripts_v1.3\convert2xml.py”, line 757, in main
xmlpage = pp.process()
File “C:\Documents and Settings\ShiTrain\My Documents\My Kindle Content\topazs
cripts_v1.3\convert2xml.py”, line 681, in process
snippet = self.injectSnippets(self.snippetList[0])
File “C:\Documents and Settings\ShiTrain\My Documents\My Kindle Content\topazs
cripts_v1.3\convert2xml.py”, line 537, in injectSnippets
aso, atag = self.injectSnippets(asnip)
File “C:\Documents and Settings\ShiTrain\My Documents\My Kindle Content\topazs
cripts_v1.3\convert2xml.py”, line 529, in injectSnippets
name = tag[0]
IndexError: list index out of range
C:\Documents and Settings\ShiTrain\My Documents\My Kindle Content\topazscripts_v
1.3>genhtml.py TARGETDIR
Processing …
metadata0000.dat
other0000.dat
Traceback (most recent call last):
File “C:\Documents and Settings\ShiTrain\My Documents\My Kindle Content\topazs
cripts_v1.3\genhtml.py”, line 128, in
sys.exit(main(”))
File “C:\Documents and Settings\ShiTrain\My Documents\My Kindle Content\topazs
cripts_v1.3\genhtml.py”, line 108, in main
cssstr , classlst = stylexml2css.convert2CSS(xmlstr, fontsize)
File “C:\Documents and Settings\ShiTrain\My Documents\My Kindle Content\topazs
cripts_v1.3\stylexml2css.py”, line 221, in convert2CSS
csspage = dp.process()
File “C:\Documents and Settings\ShiTrain\My Documents\My Kindle Content\topazs
cripts_v1.3\stylexml2css.py”, line 141, in process
ems = int(val)/scale
TypeError: int() argument must be a string or a number, not ‘NoneType’
January 19, 2010 at 2:38 pm
Hi,
It sounds like you have 2 problems.
First is something on page0000.dat which is most probably a synthesized book cover of some sort, is not being understood and it is making things barf.
The way around that is to simply move page0000.dat out of the page directory and see how much further it gets.
The second issue is with trying to read in your stylesheet and convert it into css for the html.
I would need the following filesin order to figure out what is up here:
If you look in the xml/subdirectory you should see a “stylesheet.xml” created before genxml.py barfed on page0000.dat. That will tell me why the stylesheet does not have a value associated with one of its items that should have one.
Please post the stylesheet.xml for me someplace and send me a link and I will take a look at it and try to figure out why/how it is different from the rest.
January 19, 2010 at 2:46 pm
I sent you some debug info on another test book to the other site. It trips up during convert2xml on page 1. But here is what I get from page0001 from the debug text.
Processing: info
subtags: info has 2
Processing: info.glyph
subtags: glyph has 3
Processing: info.glyph.glyphID
Loop for 0 with mode 0 :
Processing: info.glyph.y
Loop for 0 with mode 0 :
Processing: info.glyph.x
Loop for 0 with mode 0 :
Processing: info.word
subtags: word has 3
Processing: info.word.ocrText
Loop for 0 with mode 0 :
Processing: info.word.firstGlyph
Loop for 0 with mode 0 :
Processing: info.word.bl
Loop for 0 with mode 0 :
Set of 2 xml snippets. The overall structure
of the document is indicated by snippet number sets at the
end of each snippet.
Snippet: 0
Processing: page
subtags: page has 6
Processing: page.w
Processing: page.type
Processing: page.h
Processing: page.pageid
Processing: page.pagelabel
Processing: page.startID
Loop for 1 with mode 0 :
Snippet: 1
Processing: empty
Unknown Token: empty
Main Loop: Unknown value: 0
Processing: used
Unknown Token: used
Main Loop: Unknown value: 0
Main Loop: Unknown value: 0
Main Loop: Unknown value: 0
Injecting Snippets:
January 19, 2010 at 2:59 pm
I have narrowed it down to page 1, 2, 10, 516, and 518. I am almost done so there may a couple more, or no more.
January 19, 2010 at 4:08 pm
Hi Shister,
The second problem, has not bee fixed for a future release since someone else had a similar problem. If you can’t wait, let mw know and I will post a new version of styl2xml2css.py that will handle it.
The first problem, remains an issue and I will need something more to debug it.
January 20, 2010 at 3:05 am
Version 1.5 worked like a charm.
You guys totally rock. I cannot believe how cool this is!
January 19, 2010 at 4:23 am
Trying the topaz 1.3 scripts and cannot get past the cmbtc_dump.py script. Here’s what I’m getting:
C:\Data\eBooks\Topaz>cmbtc_dump.py -d -o Intro Introv.prc
dict [[856, 81260, 43341]]
dkey [[1, 51, 0]]
glyphs [[78970, 5948, 2734], [81713, 5774, 2534], [84256, 5017, 2755], [8702
869, 2719], [89748, 4375, 2856], [92613, 5876, 2718], [95340, 5861, 2789],
*
…many number lines deleted for brevity here…
*
Traceback (most recent call last):
File “C:\Data\eBooks\Topaz\cmbtc_dump.py”, line 865, in
sys.exit(main())
File “C:\Data\eBooks\Topaz\cmbtc_dump.py”, line 814, in main
kindleAccountToken = getKindleInfoValueForKey(“kindle.account.tokens”)
File “C:\Data\eBooks\Topaz\cmbtc_dump.py”, line 263, in getKindleInfoValueForK
ey
return getKindleInfoValueForHash(encodeHash(key,charMap2))
File “C:\Data\eBooks\Topaz\cmbtc_dump.py”, line 255, in getKindleInfoValueForH
ash
encryptedValue = decode(kindleDatabase[hashedKey],charMap2)
KeyError: ‘AbaZZ6z4a7ZxzLzkZcaqauZMZjZ_Ztz6’
What am I doing wrong?
Felix
January 19, 2010 at 2:20 pm
It appears you don’t have K4PC installed. Or the book was not purchased with your K4PC.
Open cmbtc_dump.py… find line 757 and comment out the following 5 lines of code:
try:
kindleDatabase = parseKindleInfo()
except Exception as message:
if verbose>0:
print(message)
Run the script again and provide your device pid (first 8 characters) on the command line with the -p option:
cmbtc_dump.py -d -p -o Intro Introv.prc
January 19, 2010 at 2:27 pm
It seems to have mangled the last line of my instructions.
Should read:
cmbdtc_dump.py -v -d -p 12345678 -o Intro Introv.prc
January 19, 2010 at 7:13 pm
Well, I purchased using the K4PC, and am able to read it on there. Also I can read it on my itouch too.
I can try and comment out those lines, but how can I find out what my PID is? I have seen this question raised here several times, but no answer on how to do that for us non-Kindle device users. I tried to use kindlepid.py but it returns “unrecognized serial number”, so where do I get the serial number to put in so I can extract the PID?
January 19, 2010 at 11:32 pm
Thanks, that did it (at least it got past the errors and output the files).
I was able to use another source to find out how to get the serial number from itunes:
http://www.tuaw.com/2008/08/08/iphone-101-find-udid-with-a-single-click/
and put that into kindlepid.py to get my PID.
Then commented out the 5 lines are ran again. Success!!
January 19, 2010 at 7:27 pm
some_updates
using version 1.3 I successfuly converted two Topaz files to html, again with small errors I’ll correct later.
I tried it again on a third file and got this error when running genhtml.py (the two previous steps had no errors):
page0340.dat
page0341.dat
page0342.dat
page0343.dat
page0344.dat
page0345.dat
page0346.dat
page0347.dat
Warning: Unknown region type vertical
Treating this like a “fixed” region
Traceback (most recent call last):
File “C:\+++ Books\+ Kindle for PC\Topaz decrypting and converting\topazscripts\genhtml.p
y”, line 128, in
sys.exit(main(”))
File “C:\+++ Books\+ Kindle for PC\Topaz decrypting and converting\topazscripts\genhtml.p
y”, line 118, in main
htmlstr += flatxml2html.convert2HTML(flat_xml, classlst, fname)
File “C:\+++ Books\+ Kindle for PC\Topaz decrypting and converting\topazscripts\flatxml2h
tml.py”, line 417, in convert2HTML
htmlpage = dp.process()
File “C:\+++ Books\+ Kindle for PC\Topaz decrypting and converting\topazscripts\flatxml2h
tml.py”, line 394, in process
htmlpage += self.buildParagraph(pclass, pdesc, ‘middle’, regtype)
UnboundLocalError: local variable ‘pdesc’ referenced before assignment
then it stops and I got no book.thtml file…
then I used gensvg.py without problems, gettiing all pages.
do you know what may be the problem?
thanks again for all the work!
January 19, 2010 at 7:44 pm
Yes,
a combination of an unknown region type and a type bug in flatxml2html.py that only triggers when an unknwon region type is hit.
Please run
genxml.py TARGETDIR
and then look in the xml subdirectory and send to me the pageNNNN.xml that is the one that has this new unknown region type. I only need the bottom part of that page (starting with “page”) if you want to post just that bit on pastebin.com and send me a link to it.
January 19, 2010 at 11:20 pm
thanks for your reply 🙂
It’s a little late for me already, but I’ll try to do it tomorrow.
thanks again!
soalla
January 19, 2010 at 11:49 pm
I hit a similar error but with the listitem unknown region.
My error was:
Warning: Unknown region type listitem
Treating this like a “fixed” region
page0038.dat
Warning: Unknown region type listitem
Treating this like a “fixed” region
Traceback (most recent call last):
File “C:\Data\eBooks\Topaz\genhtml.py”, line 128, in
sys.exit(main(”))
File “C:\Data\eBooks\Topaz\genhtml.py”, line 118, in main
htmlstr += flatxml2html.convert2HTML(flat_xml, classlst, fname)
File “C:\Data\eBooks\Topaz\flatxml2html.py”, line 417, in convert2HTML
htmlpage = dp.process()
File “C:\Data\eBooks\Topaz\flatxml2html.py”, line 394, in process
htmlpage += self.buildParagraph(pclass, pdesc, ‘middle’, regtype)
UnboundLocalError: local variable ‘pdesc’ referenced before assignment
I have put the page0038.xml portion below page on pastebin at:
http://pastebin.com/m5dc9e85
January 19, 2010 at 11:58 pm
There is a new version of topazscripts available:
topazscripts_v1.5.zip
http://www.mediafire.com/?gcwomttmnim
Changes in version 1.5
– completely reworked generation of styles to use
actual page heights and widths
– added new script getpagedim.py to support the above
– style names with underscores in them are now properly
paired with their base class
– fixed hanging indents that did not ever set a left margin
– added support for a number of not previously known
region types
– added support for a previously unknown snippet –
– corrected a bug that caused unknown regions to abort
the program
– added code to make the handling of unknown regions
better in general
– corrected a bug that caused the last link on a page
to be missing (if it was the last thing on the page)
January 20, 2010 at 12:07 am
Should be fixed in version 1.5, link below
January 20, 2010 at 12:26 am
Using the version 1.5 I get the following error when generating the html.
c:\Python26\dist>genhtml 1
Processing …
metadata0000.dat
other0000.dat
Using font size: 101
Using page height: 11520
Using page width: 7742
page0000.dat
page0001.dat
page0002.dat
page0003.dat
page0004.dat
Traceback (most recent call last):
File “genhtml.py”, line 144, in
File “genhtml.py”, line 134, in main
File “flatxml2html.pyc”, line 435, in convert2HTML
File “flatxml2html.pyc”, line 350, in process
File “flatxml2html.pyc”, line 101, in getParaDescription
AttributeError: ‘NoneType’ object has no attribute ‘lower’
Ideas?
Thanks
January 20, 2010 at 12:31 am
The book directory as it is is here in a RAR file, in case you want to take a look at the files.
http://www.megaupload.com/?d=VGKV6UBP
B
January 20, 2010 at 12:40 am
Just on the chance my compiler messed something up I ran the script directly with python. Still get the same error, just a little more details:
c:\Python26>python genhtml.py 1
Processing …
metadata0000.dat
other0000.dat
Using font size: 101
Using page height: 11520
Using page width: 7742
page0000.dat
page0001.dat
page0002.dat
page0003.dat
page0004.dat
Traceback (most recent call last):
File “genhtml.py”, line 144, in
sys.exit(main(”))
File “genhtml.py”, line 134, in main
htmlstr += flatxml2html.convert2HTML(flat_xml, classlst, fname)
File “c:\Python26\flatxml2html.py”, line 435, in convert2HTML
htmlpage = dp.process()
File “c:\Python26\flatxml2html.py”, line 350, in process
(pclass, pdesc) = self.getParaDescription(start,end)
File “c:\Python26\flatxml2html.py”, line 101, in getParaDescription
pclass = pclass.lower()
AttributeError: ‘NoneType’ object has no attribute ‘lower’
January 20, 2010 at 1:21 am
Of course, right after I release a new version … ;-(
It seems your book is unique in that it does not have paragraphs style classes for some of the paragraphs. I will have to either, rewrite the code to allow this case or, when I detect the missing class name – set it to some class name that is meaningless, but that you could easily add a style for to create an indent, etc.
I have chosen the first and have posted a new version of flatxml2html.py on pastebin.com for you to test.
http://pastebin.com/m61c95774
Please let me know if you still have troubles.
January 20, 2010 at 1:59 am
That worked like a charm!!!
I have compiled the version 1.5 scripts to windows binaries for those that do not wish to install python. No changes, same command lines, just run the exe files.
http://www.megaupload.com/?d=3ZN1S536
B.
January 20, 2010 at 5:22 am
Version 1.5 is also available here:
http://rapidshare.com/files/338082163/topazscripts_v1.5.zip
January 20, 2010 at 11:51 pm
Since the bug reports have stopped.
The final version of topazscripts (version 1.6) can be found here.
http://www.mediafire.com/?dyutgutm20t
The changes in version 1.6 include:
Changes in version 1.6
– support for books whose paragraphs have no styles
– support to run cmbtc_dump on Linux and Mac OSX provided
you know your PID of your ipod or standalone Kindle
(contributed by DiapDealer). (see cmbtc_dump_mac_linux.py)
This version will also work on Windows provided you are not
using Kindle 4PC and instead have a standalone Kindle or ipod
and know your pid.
If you do have Kindle4PC and use it to get your books, you must use the original version of cmbtc_dump.py
January 21, 2010 at 12:31 am
Version 1.6 is up here:
http://rapidshare.com/files/338521889/topazscripts_v1.6.zip
January 21, 2010 at 7:34 am
Downloaded my first topaz book. I tried version 1.6 of the topazscripts.zip and got this reslt for 1 page of the book. The other 360 pages went fine.
Traceback (most recent call last):
File “genxml.py”, line 121, in
sys.exit(main(”))
File “genxml.py”, line 102, in main
xmlstr = convert2xml.main(‘convert2xml.py ‘ + dictFile + ‘ ‘ + fname)
File “C:\Temp\topaz\convert2xml.py”, line 760, in main
xmlpage = pp.process()
File “C:\Temp\topaz\convert2xml.py”, line 670, in process
self.doLoop72(‘number’)
File “C:\Temp\topaz\convert2xml.py”, line 462, in doLoop72
snippet.append(self.procToken(self.dict.lookup(val)))
File “C:\Temp\topaz\convert2xml.py”, line 411, in procToken
subtagres.append(self.procToken(self.dict.lookup(val)))
File “C:\Temp\topaz\convert2xml.py”, line 411, in procToken
subtagres.append(self.procToken(self.dict.lookup(val)))
File “C:\Temp\topaz\convert2xml.py”, line 126, in lookup
print “Error – %d outside of string table limits” % val
TypeError: %d format: a number is required, not NoneType
January 21, 2010 at 2:49 pm
Arrghh!
Something on that page has confused the parser so that it could not convert it to xml.
The only way I can fix this is if you post for me (on one of the free download services) a zip or rar archive of the dict0000.dat file and pageNNNN.dat file where NNNN is the number of the page that has the xml conversion error.
Then post a link to it here for me and I will figure out what is on that page and how to make the program deal with it.
January 21, 2010 at 8:20 pm
Another thing you could try is the following:
copy the following files all to the same place:
convert2xml.py
dict0000.dat
pageNNNN.dat (where NNNN is the number of the file that causes the error)
Then run the following command:
convert2xml.py -d dict0000.dat pageNNNN.dat > debug.txt
Then with any text editor look at the file and search for all occurrences of “Unknown” plus anything near to where program aborts and paste it here for diagnosis.
January 22, 2010 at 2:36 am
I have little experience with uploading files to public servers for sharing so I’ll post the debug output first.
Snippet: 18
Processing: span
Unknown Token: span
Snippet: 19
Processing: version
subtags: version has 252
Processing: version.260
Unknown Token: 260
Processing: version.lastWord
Processing: version.margin-right
Unknown Token: margin-right
Processing: version.
Unknown Token:
Processing: version.
Unknown Token:
Processing: version.span
Unknown Token: span
Processing: version.version
subtags: version has 252
Processing: version.version.213
Unknown Token: 213
Processing: version.version.lastWord
Processing: version.version.margin-right
Unknown Token: margin-right
Processing: version.version.
Unknown Token:
Processing: version.version.
Unknown Token:
Processing: version.version.region
subtags: region has 5
Processing: version.version.region.type
Processing: version.version.region.h
Processing: version.version.region.w
Processing: version.version.region.x
Processing: version.version.region.y
Loop for 1 with mode 0 :
Processing: version.version.paragraph
subtags: paragraph has 3
Processing: version.version.paragraph.class
Processing: version.version.paragraph.firstWord
Processing: version.version.paragraph.lastWord
Loop for 0 with mode 0 :
Processing: version.version.region
subtags: region has 5
Processing: version.version.region.type
Processing: version.version.region.h
Processing: version.version.region.w
Processing: version.version.region.x
Processing: version.version.region.y
Loop for 1 with mode 0 :
Processing: version.version.paragraph
subtags: paragraph has 3
Processing: version.version.paragraph.class
Processing: version.version.paragraph.firstWord
Processing: version.version.paragraph.lastWord
Loop for 0 with mode 0 :
Processing: version.version.
Unknown Token:
The debug.txt file ended at the above line.
Thank you for looking at this. Let me know if you need me to upload the page and other files.
January 22, 2010 at 4:26 am
Ran into the same issue, I can post mine if needed as well.
January 21, 2010 at 3:44 pm
Please find some more reliable place to host things. Rapidshare is almost useless.
January 21, 2010 at 4:25 pm
It’s only useless is you don’t have a Premium account. They are a business that makes money from selling access.
January 21, 2010 at 3:10 pm
I have tried out the various Tpaz scriptas quite successfully with my one and only Topaz encrypted ebook and thought that I would point out one niggling problem and my workaround. The final book.html file is littered with instances of the string   as in “> 
This string corresponds to the original page numbers that were in my original Topaz encrypted ebook but of course make no sense in an ebook where there are no fixed size pages. My workaround is to edit the html file with a text editor (wordpad in Windows) and re[place all instances of the string with a null. i.e. I changed “> 
to “>
After that the ebook displays very nicely on my book reader.
January 21, 2010 at 3:53 pm
Hi Yecam,
You could be referring to either of two things:
1. The first is a chapter break indicator
<div style=”page-break-after: always;”> </div>\
These should be found before the beginning of each chapter and when converted into an ebook that supports xhtml (ie. epub, etc) these will force a page break after each one so that the new chapter begins on a new page even if you do not decide to split the file at that point.
You a can also use them in regular expressions in Sigil (or any other html editor) as split points for the file into chapter size hunks.
They should not be inserted anyplace in the document unless there was a region with type “chapterheading” indicating the beginning of a new chapter.
or
2. Second is that each page needs to have a link anchor point so that links that refer to pages (ie. the table of contents) can link to the right points.
These anchors should only be inserted in-between regions (paragraphs or images) and they look similar to this
<div id=”pageNNNN” class=”page_text”> </div>
These could be what you are referring to.
If that is the problem, then I could add that the display is hidden for that tag or try and convert it to something else.
Is it the second thing that is causing the problem?
January 21, 2010 at 4:50 pm
I think I see the problem. I think what you are seeing is a bunch of number 2 items (anchors) that are adding vertical whitespace between paragraphs that make it look less than wonderful.
I have now changed this to make sure these anchors have visibility; hidden and have 0 height and 0 width. I tried display: none but then it would not work as a link anchor anymore on some browsers.
The layout does look cleaner.
After a few more bug are reported< I will release a new "final" version (1.7).
If you need a new version of flatxml2html.py earlier, just ask and I will post it on pastebin.
January 22, 2010 at 12:44 am
I believe I am seeing a little of scenario number 1 at th ebeginning of the book and scenario number 2 all the way through the book. The &string appears in both en ebub and a mobi formatted ebook converted from the html file by Calibre. If you can meke these strings invisible it would certainly clean things up a lot.
Weirdly enough, the mobipocket erader software can import the html file and convert it to prc file without the &strings appearing but instead I see the strin /div.
January 23, 2010 at 3:11 am
I just tried out version 1.7 of the scripts on my one Topaz file and am very pleased with the results. The page break string is now completely invisible the extra white space at the end of each chapter is looking very nice and clean. In my opinion, the converted file looks a lot better than the original Topaz file on my Kindle.
Great work!
January 21, 2010 at 3:41 pm
Please do not use Rapidsahre. Can someone put up Topazscripts1.6.zip on some other place that actually works?
January 21, 2010 at 3:54 pm
If you actually read the post immediately before the rapidshare post, you will see an alternative site.
January 21, 2010 at 3:55 pm
So try the mediafire link and see if it works better for you. Almost every release has been posted in two places.
January 21, 2010 at 5:15 pm
What’s the problem with Rapidshare? Works fine for me without a premium account.
January 21, 2010 at 6:39 pm
Rapidshare is constantly busy and it’s annoying to have to keep trying over and over and over.
January 21, 2010 at 6:59 pm
Just checked and it is on the mediafire site. Make sure you are looking at the link for the latest version: 1.6
All older versions have been removed.
January 21, 2010 at 7:34 pm
JWolf, both links are still ok, just tried them with success…
anyway, I uploaded the file to box.net, maybe it is easier for you:
http://www.box.net/shared/gqukrbp0js
January 21, 2010 at 8:35 pm
I have just created an archive that contains Windows binaries as well as the python sources for the following:
cmbtc_dump.exe v1.6
cmbtc_dump_mac_linux.exe v1.6
genhtml.exe v1.6
gensvg.exe v1.6
genxml.exe v.16
eRdr2Pml011.exe
ereader2html09.exe
xPml2XHtml018.exe
kindlefix.exe
kindlepid.exe
MobiDeDrm09.exe
MobiHuff.exe
These are the latest version I can find.
The file is here:
http://www.megaupload.com/?d=4SEFQLOT
January 31, 2010 at 11:52 am
Tried to download the brutusbum archive from Megaupload and got a message that it had been taken down for violating the terms of service. Anyplace else I can get it? I’ve just started looking into migrating my eReader books to ePub and it looks like this is where I need to start.
Thanks.
January 31, 2010 at 4:50 pm
See here:
January 21, 2010 at 9:34 pm
I’ve tried 1.3, 1.5 & 1.6 of the Topaz scripts and so far when I run cmbtc_dump it doesn’t actually create a TARGETDIR. The script seems to run for a few seconds, but nothing actually happens.
Any ideas?
I’ve had no problems running any of the other scripts out there on this machine (K4PC, mobidedrm, ereader2html, inept, B&N, etc.).
January 21, 2010 at 9:53 pm
Hi Oak,
Try with the -v switch and see if it tells you why (it might say you have the wrong PID).
If you bought and downloaded these books for Kindle for PC then cmbtc_dump.py should work. Did you use the -d switch? Did you use -o to set the output directory? Are you using the same machine/account as you did when you used Kindle for PC to buy the book?
If instead you have a standalone Kindle and or use an ipod/iphone version of Kindle, then you need to know your PID and use the -p switch.
To get your PID for a standalone Kindle or ipod, use the kindlepid.py script links to which can be found on this site.
Once you have the PID, you should use cmbtc_dump_mac_linux.py program instead and add -p YOURPIDHERE to the command.
Yes, I know it says mac and linux but it will work just as well on Windows and is the version to use if you have a book you bought for a standalone kindle or ipod/iphone.
Decide which one to use and try it and let us know if you still are having problems.
January 21, 2010 at 9:58 pm
It’s a stand alone K file.
The command I’m trying is from the instructions…
cmbtc_dump.py -d -o TARGETDIR [-p “My K’s PID”] book.azw
January 21, 2010 at 10:06 pm
Sorry that should read book.azw1 not book.azw
January 21, 2010 at 10:31 pm
Make sure you’re only using the first 8 digits of your pid.
January 21, 2010 at 11:28 pm
@Oak,
Use the cmbtc_dump_mac_linux.py script (even on Windows). The other scripts appear to only work with K4PC files.
The command is:
cmbtc_dump_mac_linux.py -d -o TARGETDIR -p xxxxxxx* bookname.tpz (or .azw1)
And just like DiapDealer said, use the first 8 digits of the PID only.
Hope this helps.
Stew
January 21, 2010 at 11:54 pm
Thanks a bunch to all of you. Following stew’s instructions worked perfectly.
^_^
January 28, 2010 at 11:17 am
Hey,
i’m trying to do what Oak was doing here, with v2.0 scripts from below
I have my book file “book.azw1” in my same directory as the .exe
i try to run the command:
cmbtc_dump_nonK4PC.exe -d -o samebook -p [my first 8 of PID] bookname.azw1
When I run it (also with the .py and mac_linux) nothing happens, it just goes back to my directory that I started in. When I run the .py file, it says “not recognized as an internal or external command”.
Any help would be awesome, or if willing, please email
eodnhoj87 at gmail dot com
Thanks
January 28, 2010 at 3:01 pm
Hi John,
Please add the -v switch to the command so that any error in the script that dumps contents of the topaz file can be seen.
Post what the output of the error message to this list.
Also you seem to be using “samebook” as your output directory.
Please look to see if you have a directory “samebook” to see if it ran at all.
January 29, 2010 at 8:58 am
Hey Some_updates,
I will try this tomorrow. Sometimes this blog jumbles the order of posts, and it’s hard to find where you’re at!
i looked for the directory samebook, and it did not even get created. I’m not sure what’s happening.
I’ll report back.
January 30, 2010 at 8:54 am
Hey some_updates,
I just tried this again, to no avail.
I’m using Win Vista 64 bit. I have Python 2.6.4 installed and I have PC Crypto installed. I have all the files from Brutusbum’s v1.6 zip, unzipped to the location| c:\scripts |
I follow these steps:
1. Open the command prompt
2. Type in: cd c:\scripts
It changes to that directory
3. I type this exactly: cmbtc_dumb_nonK4PC.exe -d -o -v samebook -p “my kindle PID first 8 starting with B003 not in quotes” bookname.azw1
4. I hit enter…it does nothing….prompt shows this: c:\scripts
Also, if I do this: cmbtc_dumb_nonK4PC -d -o -v samebook -p “my kindle PID first 8 starting with B003 not in quotes” bookname.azw1
Nothing happens, just goes back the scripts directory prompt?!
I’m very new to all of this. I have successfully ran ereader2html09.exe from Brutusbums zip, as well as all of Cabbage’s unswindle and ignoble scripts.
Please help.
If needed, please email: eodnhoj87 at gmail dot com if you need more detail, thanks!
January 30, 2010 at 9:13 am
Your Kindle’s PID is NOT its serial number. You need to first convert the serial number to the PID using the kindlepid script.
January 30, 2010 at 10:29 am
Well, sh*t. Thanks Paul, I guess I’m an idiot, and missed that part!
January 30, 2010 at 10:38 am
Hey Paul,
Using the kindlepid command, I entered my serial and then it output a PID, something like 7 letters, an *, and letter number
ex: ABCDEFG*H1
I put that into the above code, that I mentioned above (both ways) and it just goes back the prompt.
January 30, 2010 at 11:05 am
Well, at least you’ve got kindlepid to work correctly.
Looking closer at your command line, you’ve put the -v flag between the -o flag and the output directory name. I’m sure it won’t be very happy with that. Try leaving out the -v, or switch them around. Also, I didn’t notice the file name in your command line. Looking at other messages, try something like
cmbtc_dumb_nonK4PC.exe -d -v -o outdirname -p 1234567* encryptedfile.tpz
where
outdirname is the name for the output directory (which will be created)
1234567* is the first eight characters of the PID generated by KindlePID.
encryptedfile.tpz is the name of your encrypted topaz file.
January 30, 2010 at 11:20 am
Hey Paul,
Thanks for your response.
I tried what you said, and left out the “-v” command option.
No difference. cmbtc_dump_nonK4PC.exe does nothing, but return me to the directory.
Also, the book name is like this: Word Word of Word Word Word – First Last.azw1
I don’t know if the name is what is causing it or not. The file type azw1 or tpz should both be Topaz. I know it’s Topaz, because when I tried unswindle it said “Book is Topaz”.
Thanks for your continued help!
January 30, 2010 at 11:39 am
Just a quick update…
I think maybe the filename was screwing it all up. I made a copy of the book, and copied it. I renamed it to something simple, Same.azw1.
That ran a ton of numbers, then in my directory, I saw the folder, with files in it!
Maybe that’s they key? It doesn’t like spaces and dashes?
January 30, 2010 at 11:50 am
Paul/Same_updates,
Thanks for the help…I did this:
Using Win Vista 64 bit. I have Python 2.6.4 installed and I have PC Crypto installed. I have all the files from Brutusbum’s v1.6 zip, unzipped to the location| c:\scripts |
I follow these steps:
1. Open the command prompt
2. Type in: cd c:\scripts
It changes to that directory
3. I type this exactly: cmbtc_dumb_nonK4PC -d -o samebook -p my kindle PID bookname.azw1
***Note: Bookname, CANNOT have spaces and dashes (or, maybe just dashes).
After it ran correctly.
4. Ran genxml samebook
5. Ran gensvg samebook
6. Ran genhtml samebook
Now, I have book.html, and it appears it WORKED! I just have to check what it looks like on the Kindle/Nook. Thanks to all of you for your help, and your hard work on these files!!!
January 30, 2010 at 12:25 pm
After getting into Calibre and converting, it works! It has a few format issues, but it works!
Thanks again!
January 22, 2010 at 3:11 am
Hi Keeska,
It seems to be out in the weeds before the first line on part of the debug.txt file you showed. There is no tag “span” that I know of only “_span”.
So could you post the 20 or so lines that come **before** what you just posted so that I can see what might be happening.
Thanks,
January 22, 2010 at 3:14 am
Keeska,
Actually the best thing would be to post those two files someplace for me and then post a link. I use http://www.mediafire.com since they will let you post things without making an account. Then go to the download and copy the url and paste it here for me.
Thanks
January 22, 2010 at 4:33 am
http://download792.mediafire.com/m4m1vd1nm0yg/zdqzg5ztqzj/dict0000.zip
This is two pages that bugged out on my same error %d not a number, in fact the python genxml gives a bit more information saying %d outside of string table limits.
Hope this helps!
January 22, 2010 at 5:12 am
Hi Alex,
Your error is the same as Keeska’s. It seems that “span” is a tag that exists in your xml whereas before all that existed was “_span”.
Perhaps a newer version of the topaz format exists and some slight changes have been made.
I added support for it.
Please grab the latest version 1.7 from here:
http://www.mediafire.com/?z45mi4nnf2g
And let me know if it helps or not.
January 22, 2010 at 6:16 am
Fantastic man, it is running great process completed without problems. Few warnings on the tables but thats to be expected. I appreciate the hardwork you did man. And I’ll let you know if I run into anything.
January 22, 2010 at 4:18 am
outputs all the files, wont genxml my book. It runs into a formatting issue on more than a few pages. %d format: a number is require dhere, not NoneType.
Happens in line 126 of convert2xml.pyc in lookup. This toolset is developing rapidly though I am looking forward to it getting better and better!
January 22, 2010 at 5:09 am
New version 1.7 of topazscripts:
Download topazscripts_v1.7.zip from:
http://www.mediafire.com/?z45mi4nnf2g
Changes in version 1.7
– gensvg.py has been improved so that the glyphs render
exactly (ClarkNova)
– gensvg.py has fixed a render order “bug” that allowed some
images to cover or hide text. (ClarkNova)
– change generated html to use external stylesheet via a
link to “style.css”
– add missing tag
– make xhtml compliant doctype and minor changes to write
correct xhtml
– make divs that act as anchors be hidden visually and to take
up 0 height and 0 width to prevent any impact on layout
– added support for new version of the tag called
– added support for “vertical” region type
– added warning message about how messed up “table” regions
can be an recommend snaphsot of table from svg
January 22, 2010 at 9:55 am
Again, the Topazscripts 1.7 are mirrored at rapidshare:
http://rapidshare.com/files/339215953/topazscripts_v1.7.zip
January 23, 2010 at 2:02 am
Do you think you could ad a version string when the scripts run? I did that for the version I am hosting (as of 1.7)
Cheers
B.
Version 1.1 of the complete scripts package, updated with the latest (1.7) topaz, latest ereader and mobi scripts.
Contains windows binaries and source.
http://www.megaupload.com/?d=12IOFJX1
January 23, 2010 at 2:56 am
Hi brutusbum,
Do you mean printing a version number to the output as it runs, or having a version number in the name, or having a version number that only appears in the usage message?
Also are we talking about version numbers for each script? The files are matched so I would rather keep 1 version number for the entire set.
Thus I added the version number to the name of the zip archive for the entire set and I remove all of the older versions as soon as a new version is released, to prevent out of sync matches of the 9 scripts – – that would drive me batty hunting down the bugs that would cause.
January 23, 2010 at 7:25 am
Thank you! Version works perfectly for me. All pages convert and the resulting html is excellent. I converting it to epub and am reading the book on my Sony reader.
January 23, 2010 at 6:52 pm
I meant a single version number for all scripts in the same release. I think we a re at 1.7 now?
It really does not matter if it prints or not, just something in the file to be able to visually id the things.
The files I posted to megaupload, I added the version 1.7 string to the usage message.
Cheers
B.
January 23, 2010 at 2:04 am
Version 1.1 of the complete scripts package, updated with the latest (1.7) topaz, latest ereader and mobi scripts.
Contains windows binaries and source.
http://www.megaupload.com/?d=12IOFJX1
January 23, 2010 at 3:45 pm
Thank you!!
January 23, 2010 at 6:53 pm
you are welcome!
I’ll keep doing this as long as time permits.
🙂
B.
January 23, 2010 at 2:48 am
Thanks for all your hard work! Version 1.7 of the scripts worked to convert a Topaz book I accidentally purchased to HTML. A little cleanup and I’ll be able to convert it to epub and read it on my Sony.
Thanks again, your work is much appreciated.
January 23, 2010 at 11:43 pm
Can you create a plugin for Calibre to do the same for DRM’ed .PDVs as you did for Mobi ?
January 24, 2010 at 2:32 am
FYI: A version 1.8 is coming tonight or tomorrow with some major changes to how tables are treated (now as images) and a greatly improved gensvg.py program that now creates a wonderful xhtml version of the book to page though.
I added a comment as the second line of each file that states what version of topazscripts it was meant for. Hope that is good enough for you.
I will post the a link later after some more testing.
January 24, 2010 at 4:01 am
Topazscripts Version 1.8
topazscripts_v1.8.zip can be found here:
http://www.mediafire.com/?nomqxfdjamt
Changes in version 1.8
– gensvg.py now builds wonderful xhtml pages with embedded svg
that can be easily paged through as if reading a book!
(tested in Safari for Mac and Win and Firefox)
(requires javascript to be enabled)
– genhtml.py now REQUIRES that gensvg.py be run FIRST
this allows creation of images on the fly from glyphs
– genhtml.py now automatically makes tables of words into svg
based images and will handle glyph based ornate first
letters of words
– cmbtc_dump_mac_linux.py has been renamed to be
cmbtc_dump_nonK4PC.py to make it clearer
when it needs to be used
Please see the readme.txt for explicit instructions on how to use these scripts
January 24, 2010 at 4:19 am
Thank you so so so so much!!
January 24, 2010 at 5:30 am
Version 1.8 mirrored on Rapidshare at:
http://rapidshare.com/files/340180820/topazscripts_v1.8.zip
January 24, 2010 at 2:11 pm
some_updates,
I sent you a sample of one I ran through the new scripts. On this one it did not pull the images from the svg files except for the front and back cover.
Thanks,
Stew
January 24, 2010 at 4:03 pm
http://i47.tinypic.com/14avgvc.png – K4PC
http://i50.tinypic.com/25tu8n9.png – Output Of Scripts
I had figured these blocks of code were some type of table and that they would be formatted better in the new version; however, this doesnt appear to be the case. Why does it not convert them formatted correctly? Is it just due to how the styles are done in the book? By no means is it a deal breaker since honestly its still readable. I am just interested. Do you think they just do it like a style or paragraph tag that the scripts are picking up on?
January 24, 2010 at 4:20 pm
Hi Alex,
If you look at the xml for that page (it can be read in any text editor) you will see what we have to work with. The ocrText field is all of the words that make up the data. But all whitespace is gone. Later on you will see a page region that lists which words fit together to make a paragraph and you can see the actual x, an y positions for each glyph that makes up each letter but the glyphs are not letters they are just pictures that form somethign that looks like a letter (ie. there is no way to take a glyph and figure out which letter it represents without actually looking at it.)
So the only way to make that look better to pull up the svg for that page and take a screen snapshot of that code example and then edit the html to remove the code and instead insert a link to the snapshot image.
Even if the whitespace was included, html ignores duplicate whitespce unless you use the
tags to indicate preformatting has been done (or other shenanigans).
In the xml for that page look at the part that represents the code example and see if there is any way to distinguish it from regular text. Does it have a specific paragraph class or anything that might tell us that we should handle it differently.
If so, I might be able to whip up a version of flatxml2html.py that can replace those pieces of code with their graphical equivalent (and svg) automatically but that would take some way of pointing out the need for special handling.
Sorry I can’t be more help here.
January 24, 2010 at 9:47 pm
Hi Alex,
I was thinking that if a specific class name is used with the paragraphs that are code examples, it would be quite easily to turn those into graphics (which would keep all formatting).
So please do look at the xml for the page in question and at other pages with code examples and see if a particular class name is used for the code examples.
If you see something that might make a good trigger, post a couple of sample pageNNNN.dat files, the dict0000.dat, and the other0000.dat so that I have something to test with.
January 24, 2010 at 10:40 pm
They all look similar to this
fixed
379
2875
2467
6763
F-L-V1
207
216
They are a fixed type and ya they do use the same class the F-L-V1. I don’t know much about it so I am probably outageously wrong, but if something is of type fixed would it not indicate that it probably has some type of white space? How does the kindle know the whitespace? It seems as if on fixed you could just replace the section with its graphical representation everytime? Or is there a reason this isn’t done? Like I said I dont have much of a clue how the graphics and such work so I am just taking a guess based on the xml.
PS heres an archive with a few pages and such http://download665.mediafire.com/jjdj3xxdnb0g/yyzzyygkzkz/page0105.zip
January 24, 2010 at 10:42 pm
Err forgot about the markup being messed up
[region]
[type]fixed[/type]
[h]384[/h]
[w]3081[/w]
[x]1560[/x]
[y]11726[/y]
[paragraph]
[class]F-L-V1[/class]
[firstWord]152[/firstWord]
[lastWord]160[/lastWord]
[/paragraph]
[/region]
January 24, 2010 at 11:11 pm
Hi Alex,
The kindle does not keep the whitespace. It instead simply draws glyphs at specific x, y positions.
I can’t trigger off region “fixed” since then almost every page would be a large image. You might as well just use the xhtml svg embedded pages.
If you have included the other and dict files and some page files, I will take a look to see if that paragraph class F-L-V1 can be the trigger or not.
January 25, 2010 at 12:00 am
Alex,
Please grab a test version of flatxml2html.py to replace the one in topazscripts_v1.8 from here:
http://pastebin.com/m593618bd
Please let me know if it does what you want or not. Please remember, that every time we create an image of a page, we lose the ability to reflow it properly and to search it. So this may create too many svg images that will slowdown things but it is worth a try.
January 25, 2010 at 3:56 am
Well, I have a prc file that was origionally a kindle topaz. Has already had the drm stripped from it andI can open it in my kindle for pc. I have been trying for days to convert it to anything else to get on my sony reader to no avail. Can someone help?
January 25, 2010 at 1:37 pm
Hi Amber,
Unfortunately, the prc with the drm already removed really only works on a Kindle.
If you have access to the original topaz book (in its original format **with** the DRM), and if you are okay with running python command line tools, then grab topazscripts_v1.8.zip (links to which are posted near here on this page).
The readme.txt file inside should lead you through the remainder of the translation to html.
If you run into trouble, please post the error message here and someone will help.
January 25, 2010 at 7:14 pm
Here is the latest collection (V1.2) of scripts, including Kindle, Mobi, Ereader and TOPAZ
http://www.megaupload.com/?d=0U6STI1F
Included are source and windows binaries, updated with the Topaz 1.8 scripts.
January 25, 2010 at 8:25 pm
Okay, I’ve tried a couple of versions of the topaz script and I can’t get it to work. Whenever I try to open cmbtc_dump, it opens for a few seconds and before I can read anything, it closes. My only experience with python is with unswindle, which I find very user friendly. I’m hoping that someone can help me because I really want to read topaz books on my Sony. I will be downloading a book with K4PC. Right now, I want to use a sample that is in the topaz format. I just thought of something, could not having a full book on HD be the problem with opening cmtbc_dump?
January 25, 2010 at 8:40 pm
Becca, try the windows binaries here:
http://www.megaupload.com/?d=0U6STI1F
Open a dos command prompt in windows and run them there and you will be able to see the screen output.
B
January 26, 2010 at 12:06 am
I’m using Vista. How do I open the dos command prompt? And once open, what do I do then?
January 26, 2010 at 12:37 am
start menu-programs-accessories-command prompt
Once there, change directories to where you extracted the archive with the exe files. Then run the one you want without options and it will give a screen with the proper switches.
IE:
extract your archive to c:\scripts; open the command prompt window, type: cd c:\scripts, then type at the command prompt:
cmbtc_dump
that should give you usage directions which would be something like this:
Assuming your book filename is XXXXXXXX.prc, the command would be:
cmbtc_dump -d -o outdir XXXXXXXX.prc
This should create a directory called outdir, with a bunch of other files and directories in it.
Now run gentxml like this:
genxml outdir
then :
genthtml outdir
At this point, if you had no errors processing the files, you should have, in the OUTDIR directory, a file called book.html
That’s your book that you can now convert with calibre or some other converter to any format you wish.
January 26, 2010 at 12:48 am
small error above, run GENSVG before GENHTML
B
January 26, 2010 at 4:09 am
Thanks for the help so far. However, I’m getting an error.
File “cmbtc_dump.py”, line 866, in
File “cmbtc_dump.py”, line 804, in main
File “cmbtc_dump.py”, line 163, in openBook
__main__. CMBDTCFatal: Could not open book file: Rumor.prc
January 26, 2010 at 5:23 am
Nevermind my previous post regarding the error. I suddenly thought that the book needed to be in the same file as the scripts. Once I moved the book, then ran the scripts it worked fine. Again, thank you for your help, brutusbum. And thank you to everyone who worked on getting these scripts. Now I’m off to buy more books from Amazon.
January 26, 2010 at 2:12 am
Decryption of kindle prc file for kindle for pc worked perfectly!!
Many thanks!!
Used swindle for two books and the topaz files for the remaining book. Syntax was sometimes different from readme file.
I have the old US version of kindle 2 and no US credit card. This way I can get kindle content with the US kindle via a second kindle for pc account.
January 26, 2010 at 3:02 am
er, assuming you live outside US, why not just reigster your old K2 US to your second K4PC account? You can’t do it via Whispernet on your Kindle2 US, but you certainly can do that with your web browswer at http://www.amazon.com.
And I don’t think you need a US-issued CreditCard to purchase contents from Amazon now, well, I didn’t anyway. Have you ever tried?
January 26, 2010 at 5:25 am
Hey Brutusbum I tried but having a problem. First, keep in mind I do all my python scripts on a Mac and am not very comfortable in Vista but I fired up the laptop to try this on the one Topaz book I have. I put all of the programs and files in a folder named TOPAZ at the root level of the C:drive. I changed dir to the TOPAZ folder and then ran cmbtc_dump -d -o book.tpz and it looked like it was working. It spit out all sorts of file names and data. But here’s the thing I cannot find the outdir directory anywhere. I figured it would be in the TOPAZ folder but it is not. Given I ran cmbtc_dump within the TOPAZ folder with the above command line where might outdir be???? Thanks!
January 26, 2010 at 1:42 pm
@Thothamon,
If you used the command that brutusbum gave you it will be in the root of the C drive, or depending on your OS, in the root of your Users folder. Start up the command prompt. Mine says C:\USERS\SCOTT>. In the folder labeled SCOTT is where my outdir folder would be if I ran the command line the way he gave it to you.
Hope this helps.
January 26, 2010 at 5:26 pm
Nope it was in neither place. I have an awful feeling that what might be happening is the intrusive permissions routines of Vista are interfering with the folder being created. Is there any way to run a program in administrator mode from the command prompt? Has anyone here running Vista had this work for them? Would love to be able to do this! Thanks.
January 26, 2010 at 6:28 pm
I am running in W7 and it creates the outdir in the same directory where the scrip is. In my case my scripts are in c:\python26\dist and the outdir directory is in c:\python26\dist\outdir
The only time I think you would see a problem is if your scripts are in the program files directory. Vista and W7 place a lot of restrictions on there.
One think I notices if the outdir name “BOOK.TPZ” ? try it with just outdir or book or something like that, not extension.
B
January 26, 2010 at 6:31 pm
I just notice that you command line is wrong:
cmbtc_dump -d -o book.tpz
instead try:
cmbtc_dump -d -o outdir XXXXXXXX.prc
Seems you forgot the output directory name.
January 27, 2010 at 12:52 am
For some reason no reply butoon on Brutusbum’s note below…. But the typo was here not in my actual use of the command line. I do indeed use the outdir parameter in the command line.
January 26, 2010 at 11:00 pm
Since the bug reports have stopped, here is the (hopefully) final version (2.0) of topaz scripts.
Rapid development of these scripts has now stopped. Perhaps a kind soul outside the US will host all of the DRM liberation scripts that are available so that people can jointly further develop them and fix bugs as needed.
topazscripts_v2.0.zip
http://www.mediafire.com/?meug1jejn1e
Changes in version 2.0
– gensvg.py now accepts two options
-x : output browseable XHTML+SVG pages (default)
-r : output raw SVG images (useful for later conversion to pdf)
– flatxml2html.py now understands page.groups of type graphic
and handles vertical regions as svg images
– genhtml.py now accepts an option
–fixed-image : which will force the conversion
of all fixed regions to svg images
– minor bug fixes and html conversion improvements
Please see the readme.txt for explicit instructions on how to use these scripts.
January 26, 2010 at 11:07 pm
Thank you !!!!
January 26, 2010 at 11:30 pm
There are here now also. This archive contains the TOPAZ 2.0 scripts and also the MOBI, KINDLE and EREADER scripts.
Contains source scripts as well as Windows binaries for those that don’t want to install python.
B.
http://www.megaupload.com/?d=MK0LGYGS
January 27, 2010 at 1:32 am
Thothamon,
I have seen something similar happen with cmbtc_dump.py when the book in question was not actually purchased and downloaded to Kindle for PC.
Please add the -v switch to see exactly what the error message if any is:
cmbtc_dump.py -v -d -o mybook YOURTOPAZBOOKNAMEHERE
If it comes back and says that it can’t find the right PID or something along those lines, then perhaps you purchased and downloaded the book for a standalone Kindle or iPhone/iPod and if so you should be using the other script.
cmbtc_dump_nonK4PC.py -v -p 12345678 -d -o mybook YOURTOPAZBOOKNAMEHERE
where 12345678 are the first 8 characters of your Kindle’s PID.
By the way, if you do have a standalone Kindle or iphone/ipod, then you can run the scripts from your Mac OSX side quite easily. I am a Mac OSX user and do this all of the time.
To make it work easily on MacOSX simply prefix each command with the command “python” and provide the path to the python script you want to run and for any arguments the python program takes.
python ./genhtml.py mybook
Hope something here helps.
January 27, 2010 at 6:42 am
OK, I see that I was indeed using the wrong cmbtc file as the book I am trying to decode was purchased for a Kindle and I was using the PC cbmtc file. The reallly really really stupid thing I did though was even bothering to use the exe files on a PC. I did not understand that the actual python scripts are in the Source folder. I thought that there was no way to use the Mac as I do on all the mobidedrm stuff. Tomorrow I’ll try using the actual scripts with the Mac and I am betting it will work right away for me. Thanks very much!
January 27, 2010 at 7:03 am
OK, couldn’t go to bed until I tried it on the Mac with the scripts and it worked! The only problem is that each chapter in the book begins with a large, initial cap. In the converted files the initial caps take up a whole page by themselves then all other pages are OK. MORE than just acceptable! Thanks again for the scripts and all of the help!
January 27, 2010 at 3:35 pm
Thothamon,
It seems your page/book must have a higher dpi than every other one we have seen so far, so we end up not scaling the opening letter image properly and so it ends up too large.
If you look in svg/ and open the same page (.xhtml) with Safari, is the first letter improperly scaled there as well. If not, then the scaling is something I can probably fix in the program.
If you know html and you know how big (in pixels) yu want those letters to be, you can simply add the height and with to the img src tag to get it to display exactly the way you want it.
January 27, 2010 at 3:59 pm
The initial caps look OK in the html pages. They are really graphics in that they are about 30% gray and ornamental. Do you want me to email you the book file or any of the pages?
January 27, 2010 at 5:38 pm
Thothamon,
To debug this, I would need the following pieces – please zip them up and post them someplace (like the mediafire site used for topazscipts) and post the link to them.
dict0000.dat
other0000.dat
svg/glyphs.svg
page/pageNNNN.dat
where NNNN is one or two of the offending pages.
Also, if the “too-big” graphic is an img/img*.jpg file then including it as well would help. If it is an img/pageNNNN_MMMM.svg file then those are actually created by the code on the fly and so I won’t need them.
Thanks for helping me track this one down and get it fixed.
January 27, 2010 at 6:15 pm
OK it is all at this link as “debug folder.zip”:
http://www.mediafire.com/?mmzuygkqnml
Good luck!
I’m just hoping Apple announces today an ereader, and iTunes support with DRM-free ebooks.
January 27, 2010 at 7:55 pm
Hi,
I looked at the page generated by the scripts and it looks almost identical to the xhtml page. img0012.jpg is a large sized gray “U” that starts out the first word “Until” and that looks exactly as it does in the xhtml (except that the alignment between the image and the text is better in the xhtml).
Nothing is big enough to fill any page.
I did notice that the html included a link to img0011.jpg which you did not include in what you posted to me.
Perhaps that is the culprit? Please post img0011.jpg for me and I will take a look at it and try to figure out how to determine the proper height / width for the image from the page description itself.
Thanks
January 27, 2010 at 10:22 pm
OK, there is NOT as big a problem as I thought. What happened is that I was viewing the book in Calibre’s “view” window. That was resulting in the initial caps being so large. However, I thought before we went further I should at least try it on the Kindle 2 which I did. On the Kindle 2 the problem is almost completely not there. The initial caps are the size that they should be but they float just a tiny bit too high and to the left from where they should start the paragraph. NOT something I would have even reported and I should have looked at it on the Kindle rather than in Calibre. So, I think you can actually consider this translation to be a success. Sorry to have been a little inaccurate with the first report(s) but that it could be Calibre rather than the book didn’t occur to me until after you said the pages looked OK. Thanks again!
January 27, 2010 at 8:53 pm
Just got a new Kindle 2, and it looks like kindlepid.py doesn’t work with the serial number from that machine.
Has anybody got any way to figure out what is going on here?
January 27, 2010 at 9:10 pm
At a guess you have an old version of the script – what’s your Kindle’s serial number start with?
January 27, 2010 at 9:18 pm
Pardoz:
I suspect that I do have an old version of the script. The new K2 is an international version, and the serial number starts with B003 which I believe denotes an international kindle.
I have been trying to find an updated copy of kindlepid.py, but so far to no avail. Any ideas where I can get one?
Thanks for the quick response…
M
January 27, 2010 at 9:47 pm
Mike,
With any text editor edit kindlepid.py and look for the following:
elif serial.startswith(“B002”):
print “Kindle 2 serial number detected”
Add this immediately after the lines above (and indentation is important) it should look like the lines above in the original script
elif serial.startswith(“B003”):
print “Kindle Intl serial number detected”
That should handle it.
January 28, 2010 at 1:35 am
Can someone help me. I am trying to remove DRM from an ereader .pdb file.
I have installed python and downloaded the topaz scripts (which include the ereader files).
At a windows cmd prompt I am running the following command (where 12345678 are the last 8 digits of my CC) from the directory I extracted the topaz files to (I had to move ereader2html09.py to the same directory as ereader2hml09.exe):
c:\ereader\ereader2html09.exe ereader2html.py AngelsDemons.pdb c:\ereader\temp “My Name as on Credit Card” 12345678
this is what gets returned:
eReader2Html v0.09. Copyright (c) 2008 The Dark Reverser
Converts DRMed eReader books to PML Source and HTML
Usage:
ereader2html infile.pdb [outdir] “your name” credit_card_number
Note:
if ommitted, outdir defaults based on ‘infile.pdb’
It’s enough to enter the last 8 digits of the credit card number
Can anyone tell what I am entering incorrectly?
January 28, 2010 at 1:50 am
You are running the exe file on the script?
OK, you either use python to run the ereader2html09.py script or you run the ereader2html09.exe executable, not both.
Try this:
ereader2html09.exe bookname.pdb outdir “Your Name” 123456789
In my experience, sometimes using just the last 8 digits of the CC sometimes does not work, so I make it a practice to use all 16 digits.
Best bet is to copy your book to the same directory where the ereader2html09.exe file is.
If you run the above command, replacing your book name, “Your Name” with the name used to buy the book (Keep the quotes “”) and your cc number, you should end with a directory called “outdir” that will contain your book in html format.
B
January 28, 2010 at 1:53 am
Also, you do not need the *.py files if you just run the executables (.EXE) files. The .PY scripts are for people that run linux, macs or have Python already installed.
The exe files are compiled from the PY scripts, but are standalone files that do not require python to run.
I included the source folder, in case people wanted to modify the things.
Cheers
B.
January 28, 2010 at 7:03 am
@ Brutus bum or whoever can help me?
I am trying to get rid of some DRM on a PDB book. I downloaded Brutusbum’s file, with all of the exe files.
I try to run the file, or the script for that matter, and all that happens is the command prompt pops up for a split second, then goes away?
Do they need to be put in a separate or distinct directory, or can they be anywhere?
Please help, I am VERY new to this whole ebook/script/python thing.
If able, an email to eodnhoj87 at gmail dot com would be amazing !!
Thanks
January 28, 2010 at 4:23 pm
these are command line tools. You can not run them by simply double clicking them. Open a command box, and run then from the command line. I included a nice HOW-TO text file in my archive.
B
January 28, 2010 at 2:53 am
I have updated my archive with the latest KINDLEPID script. The one in there before only worked with the original kindle. This one handles all kindles and iphones serial numbers.
This archive contains the TOPAZ 2.0 scripts and also the MOBI, KINDLE and EREADER scripts.
Contains source scripts as well as Windows binaries for those that don’t want to install python.
http://www.megaupload.com/?d=QT4LU9R9
B.
January 31, 2010 at 11:54 am
This version has also been taken down by Megaupload. Anyplace else I can get it?
Thanks,
FrQuixote
January 31, 2010 at 4:48 pm
See the near end of this page. I deleted the older archives, no sense in having the older versions around.
The latest one is 1.6.
January 28, 2010 at 3:20 am
Hello everybody, thanks for the good work.
I have Kindle4PC (no kindle at all). I have installed Python 2.6. I would like to convert an amazon .prc file to a drm-less one.
If I launch
python mobidedrm.py name1.prc file.mobi,
the program tells me it needs the PID.
To get a PID, I launch python kindlepid.py.
Yet this script asks for a Kindle Serial Number / iPhone UDID which I don’t have, since I just downloaded the Kindle 4 PC…
How do I get the PID? Thanks a lot, Riccardo
PS: unswindle won’t work because the Amazon file is defective and Kindle4pc won’t open it…
January 28, 2010 at 8:19 am
Files for Kindle4PC must be decoded using unswindle, because each book has a book-dependent PID.
I’m not sure why you expect any script to be able to read a corrupt file. If you can’t read it on your genuine Mobipocket device (Kindle, Kindle for PC, Mobipocket Reader, etc), then you’re not going to be able to decode it.
January 28, 2010 at 4:26 pm
Try skindle. It’s what I use for amazon stuff. TOPAZ books also end in PRC, so if after running skindle you can not open the book in mobipocket reader, then probably it’s a topaz file. The you would have to use the CMBTC scripts to remove the drm and convert to html.
B
January 28, 2010 at 3:24 am
Hello,
How do I get my PID if I just have Kindle4PC? script kindlepid.py requires a serial number that I don’t have…
Thanks a lot
Riccardo
January 28, 2010 at 6:40 am
@ Brutus bum or whoever can help me?
I am trying to get rid of some DRM on a PDB book. I downloaded Brutusbum’s file, with all of the exe files.
I try to run the file, or the script for that matter, and all that happens is the command prompt pops up for a split second, then goes away?
Do they need to be put in a separate or distinct directory, or can they be anywhere?
Please help, I am VERY new to this whole ebook/script/python thing.
If able, an email to eodnhoj87 at gmail dot com would be amazing !!
Thanks
January 28, 2010 at 8:38 am
To all the hard working people whose work culminated into this set of scripts: Bravo!!
I just spent 3 hours going through all these comments, feeling the suspense… did they succeed? How are they going to do it, trying some of the steps, skipping some… I must admit that at some point I went quicker, only skimming the text to see what happened rather than trying to understand all the technical details. Then I reached the place were “some updates” declared “Since the bug reports have stopped, here is the (hopefully) final version (2.0) of topaz scripts.” What a rush! I downloaded them and followed the instructions… just one little hiccup… that [-p pid] which I didn’t realize was optional, and then, delightful surprise: I got my book into HTML format, which I quickly converted to a pdf file which I loaded immediately in my reader. It works!
This is totally awsome! Thank you so much to all of you!!
I wish I could reward you somehow. Money? You guys should open a Paypal account for donations. Anything I can help with (I am a C++ software engineer)? Please let me know.
January 28, 2010 at 2:41 pm
Hi Clarknova,
Thanks a lot for help. Something’s wrong even for Skindle :-S
I’ve launched skindle by command line with the same (administrator) account I had used to download the .prc Amazon file, but not even skindle can get the right PID. I get an error that is not even reported in the whole blog :-S
Attempting to locate kindle.info
Found kindle.info location
Using UserName = “PIPPO01”
Using SystemDrive = “C:\”
Using VolumeSerialNumber = “508290077”
Device PID: F8KDI5FX
PID for Implementing-8_Distribution is: b/dn5p3w
No valid pids available, failed to find DRM key
An error occurred, unable to process input file!
Is it possible Amazon is distributing a different encryption?
I bought the book on Dec. 21, 2009
Thanks again!
Riccardo
January 28, 2010 at 4:03 pm
Hi Brutusbum,
I downloaded your archive and compared the source from your erdr2pml.py and xpml2xhtml.py with mine and noticed one small late change you seem to be missing in your erdr2pml.py.
This fix needs to be added to prevent lowercasing of image names when the pml code itself uses a different case in the link name.
You can find posts related to this change earlier in the archive. You should probably increment the version number to 12 just to make it clear this is the newer version.
Here is the diff of the two files:
def sanitizeFileName(s):
r = ”
– for c in s.lower():
– if c in “abcdefghijklmnopqrstuvwxyz0123456789_.-“:
+ for c in s:
+ if c in “ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_.-“:
r += c
return r
January 28, 2010 at 4:44 pm
changes made. Thank you.
B
January 28, 2010 at 6:06 pm
I have updated the scripts package. Version 1.5 of the Ereader-Topaz-Mobi–ePub-Scripts package is available here:
http://www.mediafire.com/?mmzwaxmjmld
It contains the latest of the following scripts and tools in windows binaries as well as sources for those that want to compile themselves or just need to run them in python 2.6
Adobe DeDRM scripts:
INEPTKEY
INEPTEPUB
eReader DeDrm scripts:
eRdr2Pml012
ereader2html09
xPml2XHtml018
KINDLE-MOBI fixing scripts:
KINDLEPID
KINDLEFIX
Mobipocket DeDrm Scripts:
MOBIDEDRM09
MOBIHUFF
PC Kindle DeDrm tool:
SKINDLE
Amazon.com’s Topaz DeDrm Scripts:
CMBTC_DUMP
CMBTC_DUMP_NONK4PC
CONVERT2XML
GENHTML
GENSVG
GENXML
Cheers
B.
January 28, 2010 at 6:18 pm
Hi,
Please consider adding IHeartCabbages latest Barnes and Noble epub DRM removal scripts as well to your archive. They are all python scripts.
They can be found here:
http://i-u2665-cabbages.blogspot.com/2009/12/circumventing-barnes-noble-drm-for-epub.html
The scripts to get are:
ignoblekey (version2)
ignoblekeygen (version 1)
ignobleepub (version1)
Also he has posted …
unswindle v6-rc1 for Kindle 4 PC
which you might want to add for completeness from here
http://i-u2665-cabbages.blogspot.com/2009/12/circumventing-kindle-for-pc-drm.html
That way all of the most recent version of all of the goodies can be found in one place.
Thanks,
January 28, 2010 at 10:56 pm
done!
in version 1.6
B
January 28, 2010 at 11:01 pm
The latest script package is here:
http://www.megaupload.com/?d=Z0RH24XI
This contains Windows binaries as well as the source code for all of the following except for UNSWINDLE which remains python only:
Contents:
Adobe DeDRM scripts:
INEPTKEY
INEPTEPUB
eReader DeDrm scripts:
eRdr2Pml012
ereader2html09
xPml2XHtml018
KINDLE-MOBI fixing scripts:
KINDLEPID
KINDLEFIX
Mobipocket DeDrm Scripts:
MOBIDEDRM09
MOBIHUFF
PC Kindle DeDrm tool:
SKINDLE
UNSWINDLE (Python Script only)
Amazon.com’s Topaz DeDrm Scripts:
CMBTC_DUMP
CMBTC_DUMP_NONK4PC
CONVERT2XML
GENHTML
GENSVG
GENXML
Barnes and Noble Epub DeDrm:
IGNOBLEEPUB
IGNOBLEKEY
IGNOBLEKEYGEN
January 29, 2010 at 6:35 am
A couple of ignorant questions –
(1)Is any program better or worse for opening rar files? any opensource programs?
(2)I currently use python2.5 & the command line on windows vista, is there a read-me that explains what to have installed or updated to run the programs.
(3)I’ve not been able to get any ereadertopml version since 6 to run consistently, is that because I’m using python2.5?
Thanks.
January 29, 2010 at 6:45 am
Some questions – (1)is there a program that’s better or worse for using rar files? anything opensource?
(2)I’m currently using python2.5 in the command line on windows vista, does the read-me or equivalent explain what programs need to be installed to run the scripts?
(3)I haven’t gotten ereadertopml to work consistently since version 6, is that due to using python 2.5 rather than 2.6? Version 6 has worked for the most part, a couple of pml files that dropbox disliked and a couple of pdb files that ereader handled ok but that calibre disliked.
Thanks
January 29, 2010 at 2:18 pm
Hi Elizabeth,
Some answers:
1. No Idea. I use a Mac and even though rar is a windows file format, I simply double-click it and it is unpacked nicely for me 😉 so someone else will have to answer that.
2. python 2.5 is fine but you do need to know something about running a command line program — ie. typing at the prompt, cd to directories, how to add options and arguments to the command line, etc. I am sure you can find nice web tutorials on how to pull up the command line and do basic things on the web even under Vista. If you do not like python, Brutusbum has “compiled” the scripts so that they can be run without python installed. Check this list near the end for a link to his latest version.
3. Some of the in between releases have broken a few things for the older file formats. Right now, I strongly recommend using erdr2pml.py (version 12 is in Brutusbum’s archive) to convert it to pml (where you can use Dropbook if you want) and xpml2xhtml.py to convert it to html if you so desire. From there, Sigil and Calibre are your friends for cleaning up, customizing, and converting to a host of other formats.
Post any problems or errors here and I (or someone else on this list) can help you work through things.
I guess you still need something to decode rar to get to the scripts but …
I am sure someone who uses windows will know the answer to that question.
January 29, 2010 at 2:40 pm
Elizabeth,
Get 7-Zip for rar, zip, and just about any other compressed file out there. It’s free and open-source.
January 29, 2010 at 2:41 pm
Hello Elizabeth, I am a Window user.
1.
To open RAR files, download this executable file at this site:
http://www.7-zip.org/
it is free and faster than winrar in my experience.
2.
To launch Python scripts thru Windows command line, just go to “run”, type “cmd” and move (with the CD command) to the directory where your executable script is. If Python is well installed, you just have to write “scriptname.py (parameter1 parameter2)” and Windows will automatically launch it with the Python interpreter.
January 29, 2010 at 6:47 am
ooops – sorry about the duplicate posting/
January 29, 2010 at 8:21 am
I just have to say that, as the person who designed the Topaz format to begin with, I’m pretty pleased with what you guys are doing here.
http://beesbuzz.biz/blog/e/2010/01/07-interesting_topaz_drm_development.php
I’d be happy to answer questions about the history of the Topaz format, although of course I don’t want to get sued by Amazon by revealing anything about the technical stuff inside (but it looks like you guys have that all pretty well in-hand at this point anyway).
January 29, 2010 at 9:33 am
Awesojme. Awesome, awesome, awesome.
I mean, I’m a little irritated with you for all those Topaz books I have to deal with that I still have to proofread before I can read on my Windows Mobile device ;), but heck, I suppose at worst, now I could read the SVG pages…
Thanks for making that blog entry. It was extremely informative. So is it basically true that Topaz files are created by Amazon from physical books, and that’s why it’s basically an OCRed image file similar to a text-under-image PDF?
January 29, 2010 at 4:41 pm
Yep. There were some pretty significant business constraints which led to those decisions – basically we had a huge back-catalog of books already scanned for Search Inside The Book, and we wanted to leverage that into actual ebooks. However, with only 99.9% accuracy on a good day, OCR just isn’t good enough, and OCR also doesn’t solve some of the other fiddly issues like how to deal with images, funky layouts, page-to-page block flow, or any of the other hundreds of problems that made Topaz take significantly longer to do than anyone expected. 🙂
Topaz primarily covers books that CAN’T be straightforwardly converted to an ebook, because the publisher doesn’t have any sort of master layout file for it (many of them were actually typeset by hand onto copper plates!) and so the whole point to Topaz is to take scanned images and try to approximate a reflowable document from it.
Book conversion is one of those things that is intuitively obvious how to do but actually extremely difficult when you actually think about it.
Oh, and converting Topaz to another format or writing a third-party reader for it or whatever is difficult but way easier than actually making Topaz books to begin with. Okay, you can trivially make Topaz files from structured documents and an existing TrueType font, but what would be the point?
I think the most challenging problem I solved was how to efficiently take all the little tiny images of letters that were scattered around the book and condensing them to a reasonably-small average set without accidentally combining them together too often. I don’t know if my solution ever made it into production, though (the original solution worked pretty well under certain constraints but mine was much more general and led to less image degradation overall). Sadly, while the original one is in the public record via patents, we never got around to doing any sort of public-record writeup of mine. Maybe I’ll write a blog entry about it someday, seeing as how my NDA is long-since expired (although it’s still in the murky legal world of “trade secrets” and whatever).
Oh, also, these two entries show a visualization I wrote of my paragraph styling computations:
http://beesbuzz.biz/blog/e/2006/07/07-visualize.php
http://beesbuzz.biz/blog/e/2006/07/12-visualize_2.php
Those images actually saved the Topaz project from cancellation, when Jeff Bezos was getting concerned about our ability to reasonably (and cheaply) determine consistent paragraph formatting throughout the book, which is another one of those problems which is intuitively obvious but REALLY FREAKING DIFFICULT in reality.
January 29, 2010 at 5:05 pm
I have started to proof-read and polish off my first Topaz conversion to html and the very poor quality of the Topaz OCR is driving me batty (absolutely horrible with mixing up I 1, and /)
So does anyone know of:
1. a good spell checker for text **inside an html*** document (open source / cross-platform hopefully)
2. is there any way to take another ocr program and re-ocr the svg version of the pages and compare it against the topaz ocr?
I guess what I am hoping is that there exists some independence of errors across different ocr software programs (or it can be induced with different software settings) that would allow a comparison of two different ocr versions to help detect errors.
Any hints or info on any of these would be welcome.
Thanks
January 29, 2010 at 5:14 pm
Yeah, Topaz basically only keeps the OCR text for search functionality. It’s not intended to be readable.
January 29, 2010 at 6:12 pm
Luckily most of it seems to be there and in proper order to actually create the bulk of the document properly in html.
Adding in the ability to create svg images for difficult regions of text (on a paragraph or region by region basis) as images in the html makes for a nicely reflowable document without the need for anything much extra.
I just wished the people who processed the book had taken a bit more care in their OCR work and tried harder to keep things in order and correct. Then we wouldn’t have to resort to svg images to create tables and things. They could have actually used basic html as their format with images for the tough spots.
Some of the OCR is visually so bad, it is clear that almost no care was taken with it at all. Word order can be important for good search (especially for specific phrases) so I am actually surprised it is so bad in places.
Not much we can do about it now except clean up afterwards.
January 29, 2010 at 6:58 pm
Yeah, the issue was one of expense. We were trying to process MILLIONS of books for cheap (and Amazon foots the bill for conversion – the point to the project was to convince publishers that ebooks were a good business idea so that in the future they would provide new books in an electronic form to begin with). We couldn’t spend the time or money to have people manually correct problematic OCR. That’s the whole reason Topaz exists in the way that it does to begin with.
January 29, 2010 at 7:45 pm
Oh, also, I put a LOT of work into making it so that the file’s internal structure made it easy to deal with in a linear manner while still being able to recover the original page layout – you’re welcome. 😉
January 30, 2010 at 3:21 pm
Hi John,
Yes, spaces in the filename are the culprit. The command line has to be “parsed” to determine the name of the program, any options, the name of the file and things like that. If you have spaces in the filename and you do not surround the file name with quotes, then the parser can not tell the difference between a space in a filename and a space that ends the name of the file.
That is why quotes are typically used around the person’s name in ereader2html.py (and therefore in erdr2pml) since a person’s name typically has a space between their first and last names.
In the command line environment (called a shell) certain characters also have special meaning and things are needed to turn off that special meaning (like the $, *, etc) if they need to be seen as just a simple character.
Glad to see you got it working.
January 30, 2010 at 6:23 pm
Thanks all for the info. 7-zip worked on the rar files on windows with no problems, end result was a folder with various .exe files, .txt files etc.
I updated from python 2.5 to 2.6 and checked that python still comes up in the command prompt and that it works fine with the version of ereader2html that I have been using.
But, when I try to open any of the .exe files that are in the 1.6 rar file, the file apparently tries to open (very quick glimpse of command prompt) then nothing. I’m currently using windows vista 32bit, have python 2.6.
Is there a program that I need in order to use the files in 1.6? I thought that I just needed to open the .exe file like I would any other in windows – open or double-click.
I even extracted everything in 1.6 to the python26 folder where I already had a pdb ebook file , tried opening/running the .exe file but same result – quick glimpse of the command prompt then nothing.
Baffled this morning, hoping for help so I can use the newest script versions. Thanks Beth
January 30, 2010 at 6:40 pm
Hi Elizabeth,
No you simply just can’t double click them, not even the *.exe files.
All of these programs are python scripts that have been compiled so that you do not need python installed on your machine but you do need to be able to open a command line window and type specific commands to process the files.
There have been earlier posts about how to get to a command line prompt in Windows (one even in response to your list of questions). Once there you must type the command line with the right options and filenames to get it to do the job.
For example, once you get to a command line prompt, you can type
erdr2pml012.exe (or any of the other scripts) and it will print to the display what options it supports and how to use it.
If you try that and still have problems, show exactly what you are typing and what the response is and we can help guide you.
January 30, 2010 at 6:50 pm
Hi. When I run the script cmbtc_dump_nonK4PC.py on a book file from my iPhone, there is no directory created and no error message. When I run in verbose mode, I get an error that says “Book key not found. Maybe this book is not registered with this device”. I have successfully used the PID for my iPhone with the dedrm script, so I’m sure it’s right (and I am using ONLY the first 8 characters of the PID) in the cmbtc script.
The book is of file type .prc but the first three characters of the file when I view in a file browser are TPZ, so I’m thinking it IS a Topaz type format.
Any help greatly appreciated.
January 30, 2010 at 9:08 pm
Hi Level,
That error message means either you did not provide the right pid or the book file itself was never downloaded to you ipod/iphone.
Try downloading the file to your iphone/ipod touch. Open the book up in Kindle for Ipod/iphone and look at it. Close it. Then plug in your iphone/ipod touch to your computer and it should back itself up to itunes.
Once it is backed up, grab any one of the free programs that let’s you extract book files from the backup. On a Mac I use “iPhone Backup Extractor.app” and with it you should be able to extract the newly purchased book (it will be named .prc even if it is a topaz book).
Then you should be able to run cmbtc_dump_nonK4PC.py on the just extracted book.prc file with your ipod/iphone PID.
This approach does work. The key is to actually download the book to the ipod/iphone look at it, and then extract the book from the backup on your mac/pc.
January 30, 2010 at 9:31 pm
I downloaded a Kindle book B002LGGJGU_EBOK.prc intending to remove the encryption so my Sony reader can upload it. When I run Unswindle I get this message:
Error: problem running MobDeDRM:
File “C:\Users\Public\Documents\KindleDecrypt\mobidedrm.py”, line 16
pass
^
IndentationError: expected an indented block.
Any help will be appreciated. By the way the ^ was located under the first 2 in pass, if that is important.
January 30, 2010 at 9:42 pm
OK went back to the original files I downloaded and now the process runs to completion but the book is still encrypted. Again the file is of type .prc and it appears that it simply copies the file from the Kindle folder to my Sony Reader folder. Any help and ideas are welcome. Thanks
January 30, 2010 at 11:04 pm
Paul,
I believe Unswindle is only used for mobi/prc files that you have on Kindle For PC. If it is just a Kindle mobi ebook, just run the mobidedrm.py at the command prompt. You will need your devices PID. brutusbum provided all these scripts, including the kindlepid script to get your PID using your serial number to your Kindle.
I hope this helps.
January 30, 2010 at 11:07 pm
If it is a Kindle For PC file, then open K4PC, make sure the book is in your library, and not in the archive. Close K4PC. Double click unswindle, and it will open K4PC. Open the book you are working on, as if you were going to read it in K4PC. Then close K4PC. a box will open asking you where to save and to give it a file name. Then it will save a non-drm copy of your book to that location you chose. (if it is a topaz file unswindle will not work).
Stew
March 5, 2010 at 3:48 pm
Thanks Steve. I think it’s the way the dialog box is worded “select unecrypted mobipocket to produce” that is causing confusion. If it’s “save the unecrypted mobipocket to a location”, then it is better.
January 30, 2010 at 11:21 pm
Thank you for the quick response.
Ok. I did purchase the book yesterday evening from Amazon on my iPhone, and I opened it yesterday and today on the iPhone in the Kindle app with no problems.
I reran the kindle PID script again just to make sure that I hadn’t mis-keyed anything, and it returned the samie PID . The PID by the way that I get is a 10-digit result in the format: LLLLNNL*NN , where L=letter, N=number and *=asterick. The first two characters of the PID appear on one line and then break to the second for the remaining 8 characters. I tried running script with the first 8 characters and the last 8, just be sure I wasn’t mis-reading the first two characters as part of the PID when they weren’t really.
As far as extraction goes. My first attempt before I initially posted, I SSH’d into my iPhone and copied the book file (the B000xxxxxxx.prc) file into the Python directory. This afternoon I restored it using iPhoneBackupExtractor for windows.
Bottom line, though, is I’m still getting the same result.
Any other ideas or anything I might be missing?
Thanks again.
January 30, 2010 at 11:51 pm
Curious why you just don’t install Kindle For PC, sync all your books. And use Unswindle with the directions in my above post. If you remove it off of the iPhone, which I have done in the past until K4PC came out, then just run the mobidedrm script in the command line window. Do you know if the file is topaz or mobi? Topaz requires the cmbtc_dump_nonK4PC.py script for Kindle and Kindle For iPhone books. Mobi requires the mobidedrm script.
January 31, 2010 at 12:44 am
Stew, Thanks for responding. I *think* it’s a Topaz file — the extension is .prc, but when I open it in a text editor, the first three characters of the file are TPZ, which makes me think “Topaz”. Do you know if that’s the case based on the first three chars?
Thanks.
January 31, 2010 at 12:52 am
Hi Level,
What on earth do you mean by:
> The first two characters of the PID appear on one line and then break to the second for the remaining 8 characters.
This makes no sense. For the script to work you need the first 8 letters of the PID used as follows:
cmbtc_dump_nonK4PC.py -v -p 12345678 -d -o myoutputdir YOURBOOK.prc
where you replace 12345678 with the first 8 characters of your pid which you
generated using kindlepid.py and the UDID of your iphone/ipod.
So I am not sure why your pid would have some chars on one line and some others on another line.
Please confirm this is what you tried. And that it still comes back with wrong BOOK PID message or if it creates myoutputdir as a subdirectory in the current directory.
January 31, 2010 at 6:25 pm
Hi SomeUpdates.
What I meant was that when I generated the PID using the kindlepid script, the output in the command line window started on line, ran out of space in the command line window, and broke to the second line, so that the first two chars were on the first line, and the next 8 were on the next line.
That said, problem solved . . . it turns out I was using the kindlepid script for the non-topaz files, which generated a completely different pid given the same udid. When I used the kindlepid utility that came with the topaz scripts, it generated a different pid number and that one did the trick.
Thanks for the great utility and the support. Sincerely appreciated.
January 31, 2010 at 4:26 pm
Just a guess, here but it could be trying to find precompiled python libraries. Perhaps these are included in the rar. and because you moved ereader2html awy from them, it can not find the dll’s it needs? Brutusbum would know for sure.
Try leaving the .exe files where they were unpacked and simply adding the proper path to your book.
cd to where all of the *.exe files you just downloaded and unrar’d are
Then type:
ereader2html09.exe PATH_TO_YOUBOOK.pdb book “YOURNAME HERE” last8digitsofyourCCnumber
This command should create a new directory called book right near where you *.exe files are
Again, this is a guess since I use the source scripts only. Also, if what you want is html, the fullest conversion of ereader pdb to html, comes by using erdr2pml012.py first to get the pml and images and then using xpml2xhtml.py to convert the pml to html.
If you still run into an error message, then please type the command line exactly as you did into a message (replacing your name, cc number. and bookname with placeholders) and the exact error message so that we can see what might be going wrong.
Also remember to put your name in quotes and even your bookname in quotes if it has any spaces in its name.
Hope something here helps.
January 31, 2010 at 4:44 pm
You are correct. The whole archive needs to be upacked with it’s directory structure intact. Each .EXE file depends on several libraries and the python26.dll file which are included.
B
January 31, 2010 at 5:16 pm
If all else fails, use this tool. Place the Ereader PDB file inside this folder. Run the er.exe program. Fill in the blanks. It probably doesn’t use the newest version of ereader2html, but it still works.
http://drop.io/pnxowwt
It’s good for anyone struggling with command line and Python.
January 31, 2010 at 7:11 pm
Hi Level,
There is not a kindlepid.py for topaz versus non-topaz files. You were probably using an older kindlepid.py and it gave you the wrong pid for you ipod/iphone.
The version that comes with Brutusbum’s rar archive is the very latest version and should be used all of the time. Get rid of any older versions of it you have around to rpevent confusion.
Glad to hear you got it working.
February 1, 2010 at 1:41 pm
I’m trying to use topazscripts 1.6 and while mobidedrm works with the PID I provide, cmbtc_dump_mac_linux.py does not:
beorn:t $ cp /Volumes/Kindle/documents/book1.azw book1.azw
beorn:t $ cp /Volumes/Kindle/documents/book2.azw1 book2.azw1
beorn:t $ python mobidedrm2.py book1.azw book1.mobi ”
MobiDeDrm v0.02. Copyright (c) 2008 The Dark Reverser
Decrypting. Please wait… done
beorn:t $ python cmbtc_dump_mac_linux.py -d -v -o out -p ” book2.azw1
dict [[599, 125726, 66484]]
dkey [[1, 51, 0]]
glyphs [[133072, 5826, 3067], [….]
img [[68551, 163, 77], [….]
metadata [[58, 531, 0]]
other [[67090, 3795, 1453]]
page [[94119, 245, 161], [….]
Book key could not be found. Maybe this book is not registered with this device.
The regular .azw book decrypts fine, but the Topaz book does not seem to like the PID i give. Thoughts?
February 1, 2010 at 1:43 pm
Silly HTML stripped out my santiized pid placeholder… it should be ‘$PID’ not ”.
February 1, 2010 at 2:37 pm
First, Version 1.6 of Topazscripts is old and has been removed. Please try using the latest topazscripts_2.0.zip (links to which can be found in this blog).
Also, when using cmbtc_dump_nonK4PC.py (the new name for the script you tried above), pleasse note, it uses only the FIRST 8 chars of your pid and not the full 10 that MobideDRM uses (the last 2 chars are checksum chars).
Please make sure you are only giving it the first 8 characters of your pid.
February 1, 2010 at 2:58 pm
Removing the last two characters of the fixed it (also grabbed 2.0 of the topazscripts). Thank you!!
January 31, 2010 at 7:18 pm
You know,
given the amount of time consumed with helping people with command line issues, we should probably build one front-end using python TK widgets to collect arguments and invoke any of the programs properly.
That way they just double click it and it presents a pulldown of the of the conversion types it knows how to do.
Once the pull down has been done, it then queries the user to browse to the file to be converted, to select the output directory, etc.
Having one interface like this would certainly help a lot of people and would not require a whole bunch of duplicate code to put front-ends on each program instead.
What do people think? Anyone want to take a shot at it?
If you want to see what the code would be like, take a look at the EPUB ADE removal scripts for an idea of how to use tk to do this to create a pyw (python with tk graphical interface toolkit).
January 31, 2010 at 7:41 pm
I said this a year ago in regards to the mobi scripts, and got laughed at, so I never mentioned it again. I’ve done my share of providing script syntax to people this week as well, via private emails. I think this is a fantastic idea. The thing is with this, we aren’t dealing with scripters, we are dealing with a lot of people who have no idea about command line (I was one of those a year ago when I got my first reader). I love the epub ade removal TK gui. I’m not sure how much help I can be, but if there’s anything I can do, let me know.
January 31, 2010 at 7:50 pm
I was about to dust of my VBasic and whip something up.
I know, I know, but vbasic is the only language (other that turbo pascal, ahh the 80’s bring back memories) that I am even marginally competent in.
Perhaps I should leave this to someone more competent in these matters; I am sure the results of my fiddling would not be pretty.
Anyway, you have my vote!
I am an old command line hand (even dabble in linux 🙂 ), so I have never had a problem myself, bu seems a lot of the younger folks have lost any semblance of the command line lore.
B
January 31, 2010 at 9:13 pm
There is a GUI for KindlePID and KindleFIX on Windows (you do have to swap out to a newer KindlePID.py script). Maybe someone could build on it
http://blog.chinoy.com/archive/2009/03/07/AppAWeekend_KindlePid.aspx.
January 31, 2010 at 11:43 pm
Hi zyzzyx,
I looked at it and it seems to be Windows only (and not cross-platform). I would prefer and ugly cross-platform solution versus a nice one platform only solution if at all possible.
My 2 cents,
February 3, 2010 at 3:22 pm
Wouldn’t it be better to make everything work as Calibre plugins rather than creating a new GUI?
January 31, 2010 at 8:41 pm
Hi Zanf, Brutusbum,
I found the following pure python front end to Tk toolkit that makes very very ugly but very very easy and quick front ends (based on simple function calls – no real event loop needed).
http://easygui.sourceforge.net/
I am going to see if I can use to to put together a brain dead simple but ugly front end that people can easily modify and use themselves.
We can simply add easygui.py as one of the scripts we include (and compile in BrtusBum’s case).
What do you think?
January 31, 2010 at 9:43 pm
Sounds great to me. Pretty isn’t the issue…dead simple, is.
January 31, 2010 at 10:25 pm
Blasphemy!!
Just kidding… it sounds like a good idea – considering the command line seems to be such a challenge for so many. I myself have never gotten the hang of double-clicking icons. 😉
January 31, 2010 at 10:49 pm
Hi,
As proof of concept I have used easygui.py to build a test front-end that works on a Mac to run erdr2pml.py.
Would someone please test it on Windows (I am sure Linux users have no trouble with the command line but it should work there as well)
The program is called frontend.pyw
You can grab it from here:
http://pastebin.com/m494449b
Then right next to it put erdr2pml.py (rename the eRdr2Pml012.py from Brtutusbum’s Sources archive to just erdr2pml.py).
And of course you need a ereader PDB you would like to convert and know your own name and the last 8 digits of your creditcard number.
And lastly you will need easugui.py which I have posted here:
http://pastebin.com/m75448091
please it beside erdr2pml.py and frontend.pyw
On a Mac double clicking frontend.pyw opens a console window and then prompts you for your pdb file, the directory you want to save things to, your name, and the last 8 digits of your cc number, and then it passes these things to erdr2pml.py whose output appears in the same console window.
Ugly as hell but if you look at frontend.pyw in a text editor you can see how dead easy it is to use.
If people think this is too clunky we can go for a normal event loop driven tk python front end but I thought this might be worth a look at.
Let me know what you think?
February 1, 2010 at 12:08 am
Just a quick comment, seems to work on the Mac fine here (I dont have an ereader file right now but calls up the directory box fine) but you refer to the front end in your note as “easugui.py” and, of course, it should be easygui.py. Just a typo — but this is funny — I was so concerned I would make a typo renaming downloads that I was cut and pasting from the message to the get info boxes!
February 1, 2010 at 3:10 am
Quick report of the GUI on Windows Vista:
Double clicking on frontend.pyw does nothing until you change eRdr2Pml.py to all lowercase letters.
Once you rename erdr2pml.py, the gui opens and allows you to select the pdb file. Then it quits and goes no futher.
If you run frontend.pyw from a command line, however everything proceeds as it should (after you rename eRdr2Pml.py to all lowercase).
February 1, 2010 at 3:18 am
Yes, I noted that you must rename eRdr2Pml012.py to erdr2pml.py for this test script to work. That is expected.
I did not expect that under a command line it would work but under regular it fails.
Interesting.
I will have to figure out why (perhaps, Windows does not create an automatic console to collect the output of the print commands used in erdr2pml.py).
I will experiment some more with redirecting stdout to someplace else to see if that helps.
Thanks for trying it.
February 1, 2010 at 3:19 am
CORRECTION: It works by double-clicking on frontend.pwy (from Windows Explorer), but the prompts after the initial file selection open behind the original Windows Explorer window.
Weird.
February 2, 2010 at 5:46 pm
Here is a new model for a generic front end for the command line tools. It no longer uses easygui.py since that had issues and very limited capabilities.
I wanted to build something standalone and cross-platform, that did not require modifying the original scripts in any way, and did not require a heavy fancy graphics toolkit like PyQt.
I designed it to use subprocess calls to invoke the scripts while using threads to create the appearance of non-blocking file io to collect the output of the subprocess stdout and make it appear in the frontends own scrolledtext widget.
The program is called frontend.pyw and it acts as a model gui front end for the erdr2pml.py program. Modifying it for other command line tools is straightforward.
You can grab frontend.pyw from here:
http://pastebin.com/m73c3d96e
**** You need to place it beside erdr2pml.py
*** take the eRdr2Pml012.py from Brutusbum’s archive and rename it to “erdr2pml.py”
I have tested this on MacOSX, and Windows XP (running in a virtual machine on my Mac). I really need people to try it on other Windows boxes (XP and Vista). Linux, other Macs and let me know if it works okay.
BTW: The python Tk toolkit is a mess as you can’t use your own mainloop and therefore can’t easily handle non-gui events once the mainloop has started, and doing any cross-platform non-blocking io without threads is impossible.
Anyway, if you have a windows box and access to a eReader book to test this with, I would appreciate hearing your comments.
If it works, I will modify it to create frontends for the other scripts that do not already have a gui front end.
February 2, 2010 at 7:27 pm
Windows 7 here and it works:
Conversion Log
Please Wait …
eRdr2Pml v0.12. Copyright (c) 2009 The Dark Reverser
Converts DRMed eReader books to PML Source
Usage:
erdr2pml infile.pdb [outdir] “your name” credit_card_number
Note:
if ommitted, outdir defaults based on ‘infile.pdb’
It’s enough to enter the last 8 digits of the credit card number
File successfully converted
February 2, 2010 at 7:38 pm
Spoke too soon, it’s actually spitting out usage info. Something wrong with the command line being passed?
I tried using my name in quotes, with no quotes, my full CC#, the last eight numbers.
Always gives the usage instructions, does not actually convert the file.
From the command line this works:
python erdr2pml.py 1.pdb outdir “MyFirst MyLast” XXXXXXXX
Same info in the frontend gives the above output.
B
February 2, 2010 at 8:14 pm
Hi Brutusbum,
Interesting … Will you please try once more but do NOT add quotes anyplace (even around your name) .
The script itself adds all of the quotes needed to protect your name and the paths which might have spaces in them”.
cmdline = ‘erdr2pml.py “‘ + infile + ‘” “‘ + outdir + ‘” “‘ + name + ‘” ‘ + ccnum
So just using putting your name in the box (no quotes), and the last 8 characters of you CC should enough once you have found the input file and the output directory.
Thanks,
February 2, 2010 at 8:29 pm
Hi Brutusbum,
I have made a new version of frontend.pyw that shows the exact command line being used before launching it.
Please compare what it shows to what you know that works and let me know what I need to change.
http://pastebin.com/m7ac48641
Thanks
February 2, 2010 at 8:29 pm
No go. Here is what I entered:
ebook PDB input file: C:\test\1.pdb
output directory: C:\test\outdir
Name on CC: Firstname Lastname (No quotes)
Last 8 Digits of CC: xxxxxxxx (my actual last 8 numbers)
Here is the output:
Conversion Log
Please Wait …
eRdr2Pml v0.12. Copyright (c) 2009 The Dark Reverser
Converts DRMed eReader books to PML Source
Usage:
erdr2pml infile.pdb [outdir] “your name” credit_card_number
Note:
if ommitted, outdir defaults based on ‘infile.pdb’
It’s enough to enter the last 8 digits of the credit card number
File successfully converted
It did create the “Outdir” folder, but it is empty.
B
February 2, 2010 at 9:08 pm
Weird, weird weird:
Command: erdr2pml.py “C:\test\1.pdb” “c:\test\outdir” “First Last” XXXXXXXX
does not work. Spits out usage info.
HOWEVER!!!!!
python erdr2pml.py C:\test\1.pdb outdir “First Name” XXXXXXXX
works just fine. Hell if I know why.
Why it would make a difference in windows 7 I do not know. Could it be the command line parameters are not passed unless you specifically tell run python with the script name and the rest of the commands?
Taking out the
February 2, 2010 at 9:11 pm
UGH! my comment got all mangled…
Anyway from my command line, if I run:
python erdr2pml.py C:\test\1.pdb outdir “First Last” 12345678
it works.
If I run
erdr2pml.py C:\test\1.pdb outdir “First Last” 12345678
the script runs but only gives out the help message.
B
February 2, 2010 at 9:18 pm
changing line 266 from:
cmdline = ‘erdr2pml.py “‘ + infile + ‘” “‘ + outdir + ‘” “‘ + name + ‘” ‘ + ccnum
to
cmdline = ‘python erdr2pml.py “‘ + infile + ‘” “‘ + outdir + ‘” “‘ + name + ‘” ‘ + ccnum
fixed the problem. Now it runs at it should.
B
February 2, 2010 at 9:26 pm
One minor problem, the default output directory should be other than the script base directory. The PML file got extracted into the same directory where the script file is, unless you enter something in the outdir box (other than the default). I expected the script behaviour to be maintained (no outdir entered, creates folder with the same name as the book)
Ie:
if script is in c:\test then unless you change the default outdir, the output of the script will be in c:\test also. When you select the book, it should change the default outdir to c:\test\bookname or some such.
B
February 2, 2010 at 9:53 pm
Hi Brutusbum,
So it appears under Windows 7 that you need to invoke the python executable much like you have to under Linux and Mac OSX. Yet under Vista and Win XP adding the “python” to the front of the commandline freaks it out unless python.exe is found in the system’s PATH.
Wierd … to say the least.
I am no windows person so this is a stupid question: what if any difference is there between Vista and Winodw’s 7 (or are they the same thing?).
I am not sure how to detect which one of the two command lines should be used and when.
Ideas welcome.
As for working with the command line versions … that may or may not work. It depends on if the compiled version checks for environment variables like python does when running a script.
ie. does it do the right thing when the PYTHONUNBUFFERED environment variable is set.
Please test it by changing the name in the cmdline to eRdr2Pml012.exe and see if it works properly.
To check if it is working properly see if all output in the Conversion Log window just sits there until the entire script has run and completed or if it is updated as the script goes along.
If nothing new appears until after the script has finished, then it is NOT working properly. If stuff does show up before the script finishes then it is working properly.
I am interested to hear what happens.
2.
February 2, 2010 at 10:02 pm
Hi Brutusbum,
As for the output directory, it defaults to the current working directory and yes you must actually select something (hit the … button and browse to where you want the output to go).
The reason I do that is when running a gui app, you do not always know where to find a directory if you did not specify it and it was instead created someplace based on the input file name.
Again, you expected it to default to command-line version because you know the command line interfaces well, whereas this was designed to be used by people who have never run the command line versions.
We still have to figure out when to use “python” in the command line and when not to under Windows machines. There must be some way of testing in advance to see which version of the command line to use.
So something is different about how you have installed python that is not the same as other windows users.
Perhaps you manually added python to your system PATH or did not use the save Active Python installer that most Windows users do?
Any ideas?
February 2, 2010 at 10:03 pm
I am pretty certain that running the python executable with the script and commands as arguments will work in XP, at least I seem to remember so when I had my other laptop with XP.
So the same command line should work fine in both, ie:
python erdr2pml.py C:\test\1.pdb outdir “First Last” 12345678
B
February 2, 2010 at 10:14 pm
Something’s fishy here. If you are getting the usage message, then windows obviously already knows to use the python interpreter. Otherwise you’d get ‘unknown command’ or some such.
Definitely weird.
February 2, 2010 at 10:20 pm
🙂 I believe my comment above was:
“weird, weird, weird”
B
February 3, 2010 at 12:30 am
Hi Brutusbum,
I tried your recompiled unbuf version of erdr2pml.exe and it will work but it is not working correctly. It does the conversion but the output of the conversion should be appearing in the scrolltext window while it is running and not waiting until it has completed. Basically, even through you did “unbuffered as True) the stdout was not unbuffered and it kept all of its output until the end in a buffer and then spit it out but only after the whole things was complete.
So that should have fixed things, but it did not.
I am not sure what to do now. recompiling with unbufferd True should have done the trick but it did not.
hmm …
February 2, 2010 at 8:54 pm
I’m getting your both of these latest versions to work as they should on Vista and Linux.
February 2, 2010 at 8:59 pm
Hi,
Thanks for testing them. I am glad to hear that they seem to work.
I have one positive test on Win XP, one now on Vista, one on Linux, one on Mac OS X, but Brutusbum’s case is going to drive me crazy!
There really is no difference between the two versions except that the second one will show the user the exact command used which should allow Brutusbum (and other testers) to see what is different so they can tell me and we figure out why this is not working in some cases. Strange, indeed.
Thanks,
February 2, 2010 at 9:13 pm
I don’t know if it’s something messed up in my python install or if it’s just a Windows7 thing. Is it possible for the frontend to call the compiled erdr2pml.exe instead?
February 2, 2010 at 10:04 pm
and yes, Python should be in the path anyway. It’s setup like that by default on any install that I have done.
B
February 2, 2010 at 10:18 pm
I am a user, by no description an expert, but from what I understand, the command box in W7 is totally different from vista, what with the new support form virtual XP mode and all that.
Changing the command line to run erdr2pml.exe fails the conversion. Nothing scrolls in the window, just the failed message at the end.
So I guess we’ll have to stick with the scripts.
As for my python install, it’s standard 2.6.4 (I think?) from the active website. And I have not added the path by hand. Of that I am sure.
But it’s easy enough to warn possible users that the PATH must be set before using the GUI.
B
February 2, 2010 at 10:37 pm
Hi Brutusbum,
Yes, the path is the issue.
On my Win XP (Active Python installed) C:\Python26 was not in the path.
If I run python erdr2pml.py blah
it would cause an error. If I ran it without the “python” at the front it would run perfectly.
Next I manually added C:\Python26 to the path and restarted the machine.
If I run the script with “python” at the front, then it runs like a charm.
So it appears the correct version depends on whether python is in the PATH or not.
I guess asking people to add something to their PATH is just about as hard as running a command line.
Arrghh!
BTW: I am not ready to give on on the precompiled versions of the scripts. There must be some way to get them to work.
February 2, 2010 at 10:47 pm
is there a way to parse the path variable and determine if it’s set to the python directory?
B
February 2, 2010 at 11:31 pm
Hi Brutusbum,
I played around with your erdr2pml.exe file and was able to run them with a modified frontend.pyw
It did run but it did not properly update the screen. This is because the compiled python code need to be compiled with the “unbuffered” switch to work properly with the frontend.pyw (this will not impact how the compiled executable runs otherwise).
Here is a link on it that I found on py2exe wiki:
http://www.py2exe.org/index.cgi/ListOfOptions?highlight=%28unbuffered%29
See the example at the end of that page for how to adjust the setup.py to enable unbuffered as True.
When you get a free moment, would you rebuild erdr2pml.exe using this approach so that we can test it and see if it will work properly with a modified front end.
If so, we can rebuild and compile all of the front ends and code and simply include them in your rar so that windows users can use them without having to worry if python is in the path or not.
February 2, 2010 at 11:47 pm
Hi,
Yes, I can read environment variables (and PATH should be one of them) and look to see if python is in the path or not.
It should not be hard to do.
I am hoping we can get the frontend working with your compiled pythong scripts (see my last email on adding unbuffered True in setup.py) and if so, we make the compiled version the official way to handle things for people on Windows who do not like the command line and probably don’t want to install Python in the first place.
On Mac and Linux python is there by default in most cases, so just using the source scripts should work fine for them (assuming the frontend gui exists)
What do you think?
February 2, 2010 at 11:55 pm
Here is a test version, compiled with unbuffered:true as you requested. A trivial change at compile time.
I should be able to compile the frontend also, so we could eventually dispense with the python installs, for windows users anyway.
B
http://www.mediafire.com/?gmzky2lzzjy
February 3, 2010 at 12:03 am
Changing line 266 to:
cmdline = ‘erdr2pml.exe “‘ + infile + ‘” “‘ + outdir + ‘” “‘ + name + ‘” ‘ + ccnum
and I was able to run the unbuffered version just fine:
Conversion Log
Please Wait …
Command: erdr2pml.py “C:\test\1.pdb” “C:\test\outdir” “xxxxx xxxxxx” 11111111
eRdr2Pml v0.12. Copyright (c) 2009 The Dark Reverser
Processing…
Decoding File
Extracting images
Extracting pml
elapsed time: 7.11 seconds
output in C:\test\outdir
done
File successfully converted
I am going to try and compile the frontend and see what blows….
B
February 3, 2010 at 12:42 am
OK, the regular erdr2pml shows the proress, the unbuffered one does not.
I just recompiled both versions and here is what I have working now.
Compiled frontend with unbuffered=true, compiled erdr2html as before.
That combo works well.
Test archive here:
http://www.mediafire.com/?5jzltwjalej
February 3, 2010 at 12:44 am
The screen output appears in the log windows, but seems to lag a bit. When the conversion is done, it all appears in one shot.
But wth, it works!
B
February 2, 2010 at 10:34 pm
I think I have tried just about any way I can think of. In my W7 machine, the path must have the python26 directory set, and the python executable must be used to load the script.
Otherwise it does not work, just spits out the help.
I would be interested to know is this is a W7 behaviour or my python install is weird.
B
February 2, 2010 at 6:30 pm
[…] If so, you’re going to have to do some Python hacking! You’ll need a copy of mobidedrm.py to remove the DRM so that you can read the book on your Kindle. The process is a bit complicated; […]
February 2, 2010 at 7:34 pm
Hi Brutusbum,
I just tried your 1.7 link
http://www.megaupload.com/?d=ASHGYRMT
but nothing is there.
Is this a typo or is there a new link I should be using.
February 2, 2010 at 7:50 pm
Yes, I found a typo in erdr2pml012, so I had to update the thing.
It’s here (version 1.71)
http://www.megaupload.com/?d=AMLZYT0J
B
February 2, 2010 at 7:46 pm
I am having a bad day. Updated the archive to 1.71.
I found a typo in one of the scripts (erdr2pml) so had to update it.
http://www.megaupload.com/?d=AMLZYT0J
Version 1.71
B
February 3, 2010 at 12:58 am
Hopefully Final Version of frontend.pyw
http://pastebin.com/m92d9a64
Should work on Windows (with and without python in the path), Mac, and Linux.
To test it:
1. grab a copy of erdr2pml.py (or take copy eRdr2Pml012.py from Brutusbum’s rar archive and rename it to “erdr2pml.py”
2. grab frontend.pyw from link and place it beside erdr2pml.py
double-click to launch it and with the right info, it should provide the gui to convert an eReader pdb book to pml
Please let me know if there are any problems with it.
If it is okay, we can use it to create gui front ends for all of the other command line tools that don’t already have one.
Thanks,
February 3, 2010 at 1:25 am
Anyone that want to try this, need to download the latest archive (Version 1.71) from here:
http://www.megaupload.com/?d=AMLZYT0J
Previous versions had a BAD eRrd2Pml script.
B
February 3, 2010 at 1:40 am
Worked beautifully for me on Win XP.
February 3, 2010 at 1:48 am
I just modified the command line portion to make it use the compiled .EXE script. Works great for me here (both scripts compiled to windows binaries)
I won’t post the binaries until we have a more complete package of GUI’s ready. Can it be possible to combine them all into one single GUI. Something with big buttons that branch you out to ereader, mobi, epub, etc….?
Just an idea.
Keep up the good work!
B
February 3, 2010 at 3:00 am
Hi Brutusbum,
I am upset that the unbuffered option is being ignored by py2exe. it should not be. I found a patch back in 2004-12 that enabled it internally but from what I can tell, they have never handled the interface details.
I think I can modify build_exe.py so that the option -u can be used with py2exe (like you do for optimize) to force the resulting executable to actually have unbuffered stdin and stdout like it is supposed to now.
If I sent you a replacement build_exe.py, would you be willing to reinstall py2exe from its 0.6.9 code base with the new build_exe.py in place and then install it and rebuild erdr2pml.py to see if it helps?
Let me know if you can and are willing to do that.
Thanks,
Kevin
February 3, 2010 at 1:07 pm
I will give it a try.
B
February 3, 2010 at 3:02 am
Hi Brutusbum,
Imagine that lag when running the topaz scripts which spit out lots of stuff that would never be seen by the user. People would fele it hung and kill it long before it had a chance to complete because not feedback is being given.
That is why getting the unbuffered thing to work is important.
Thanks,
Kevin
February 3, 2010 at 1:13 pm
I agree, I thought it was not working the first time I ran it, took some 15 seconds before I saw any feedback. Send me that new build_exe.py and I will recompile it.
B
February 3, 2010 at 9:03 pm
Hi Brutusbum,
I have found another (easier) solution that involves adding just a few lines to every script. I will take care of doing this for the topaz tuff, and erdr2pml, xpml2xhtml, ereader2html , mobidedrm and any others I see that can easily be modified to fix them (such as properly returning 0 upon success and 1 on failure at the end).
Once I get a few moments over the weekend, I will post for you a set of revised scripts (whose names will all be lowercase for simplicity) with their version numbers properly incremented, the code to auto flush stdout in place and proper return codes as well.
With these new versions, current py2exe will work just fine and we will have a fully python free version for windows users.
February 3, 2010 at 3:47 pm
Hi David,
Well not everyone wants to use Calibre. I try not to use it much since it really hacks up the html, strips styles, and basically does not really try to keep things as close to the original as possible (which was its original stated goal). The stripping of styles drives me crazy. I want to see the book with paragraph indents and fonts as it was in the original not forced one way or the other by setting options.
Secondly Calibre would be a big install for many who just want to archive a book in html.
Thirdly, why would would you need Calibre if all you wanted to do was read a Mobi? The same is true for ADE epubs, B&N epubs, etc. There all you want to do is strip the DRM not convert it to another format and so reason to use or need Calibre unless you have no better way to manage your book collection.
The fourth reason is that the conversion themselves are not perfect. This is especially true for the Topaz stuff because much work needs to be done cleaning up the html because of poor OCR before it can be imported anyplace and so a plugin of that type would make no sense.
And finally, the plugins themselves interfere with using non-drm books that have the same file name extension (given that some of the plugins fail when presented with non drm’d books). What we really need is different filoename extensions for drm’d versus non-DRM’d books, but they do not exist.
So I think there is a real need for a simple gui frontend for all of the tools so that people can use them the way they want. That does not preclude the use of plugins, the gui front end does not actually change the scripts themselves.
My slightly more than 2 cents …
February 4, 2010 at 1:15 am
More on Front-ends:
After closer examinations we just need front-ends for the following 3 sets of programs (all of the others either have their own gui or are imported into other programs and not used directly from the command line.
1. ereader2html.py, erdr2pml.py, xpml2html.py (all eReader PDB)
2. gensvg.py, genhtml.py, cmbtc_dump.py, cmbtc_dump_nonK4PC.py
(all Topaz stuff)
3. kindlepid.py, kindlefix.py, mobidedrm.pu mobihuff.py (all Kindle/Mobi)
So I think generating 3 different frontends (one for each group) would be easiest.
I will start by modifying the current front end to handle all the eReader PDB things. Then create one for Topaz. And Finally for Kindle/Mobi.
Does this sound reasonable?
February 4, 2010 at 3:56 pm
Sounds like a plan. Looking forward to test them.
B
February 4, 2010 at 5:38 pm
Hi Brutusbum,
Here is a link to the 0.13 version of erdr2pml.py
http://pastebin.com/m34fabb33
If you compile this one with py2exe it should function properly with the frontend.pyw code (ie. you will see whatever prints happen as they are made as opposed to only when the script has finished.
Please let me know if this works when compiled. If so, we are in business.
Thanks
February 4, 2010 at 6:54 pm
Test very nicely. The output is in the log window as it happens, instead of one clump at the end. I think we are in business!!
A small request, could you make the log window wider and/or taller?
B
February 4, 2010 at 6:57 pm
By the way, this was using both the binaries versions of both frontend and erdr2pml013.
I was thinking that to make it easier for future upgrades, you could include a config button that would let the user select the binary to run, save the info to an INI file and that way if the erdr2pml script gets upgraded, you could just point it to the newer version and not have to rename the thing. Get’s confusing after a while with multiple versions laying around.
B
February 4, 2010 at 7:55 pm
Hi Brutusbum,
You can use the mouse to make the window bigger. But if you mean bigger at the beginning, I can change the code to do that.
Also, from a naming convension sort of thing, I would much rather remove all version numbers from the file names to prevent them from having to change with every minor change.
All of the scripts will print usage information and the version number if you just launch them and people will see that version number in the front end gui as well.
I think in the long run a good idea would be to do the following:
Show only the gui apps and have everything else hidden in a local ‘lib’
So if you download and unzip eReaderTools_v1.0.zip, you would see only the following:
eReader2Pml.pyw
Pml2HTML.pyw
lib/
Inside of lib/ would be the following:
erdr2pml.py
xpml2xhtml.py
subasyncio.py
scrolltextwidget.py
and of course any other python pieces needed.
So the user would simply unzip and double-click the program of their choice and everything would work from there.
In your compiled version you could do the same, except that all of the .py and .pyw would be .exe and everything would be hidden inside of the lib/ file.
Using a lib directory to hide all of the pieces which might confuse anyone is simple to do in python.
At the top of any python program, just do the following:
import sys
sys.path.append(‘lib’)
That way once we have converted everything we point people at:
eReaderTools_v1.0.zip
Kindle_Mobi_Tools_v1.0.zip
Kindle_Topaz_Tools_v1.0.zip
The Adobe Adept epub stuff and the Barnes and Noble epub stuff already has its own gui built right into them as does unswindle.pyw so nothing need be done with them.
February 4, 2010 at 4:37 am
Adobe Digital Editions has a new 1.7.2 update. If you have been running ineptepub in the past, it will no longer work if you do the update.
ADE will expire in 45 days, and you will have no choice to upgrade.
Just a heads up.
February 4, 2010 at 5:04 am
Well, I am hearing some can still remove DRM, and some can’t.
February 4, 2010 at 7:21 am
Thanks so much for your work!! I am a little confused on versions, however. The link above indicates 1.71, but within the zipped file the topaz changes txt file describes v 2.0. Does that mean the included topaz scripts are 2.0?
Thank you for your efforts – absolutely brilliant!!
February 4, 2010 at 3:57 pm
Yes, version 1.71 of the archive contains the LATEST of all scripts, including the TOPAZ 2.0 version.
B
February 5, 2010 at 12:05 am
Hi,
a collection of gui scripts for ebooks
as a test of the idea, I have uploaded a zip with all of the graphical gui frontends that have been discussed along with all other scripts hidden in lib/. Please note, almost all of the scripts have needed to be updated to some extent so new version numbers abound. Nothing should break anything and all programs should still work well from the command line and as plugins if they were set up to do that before.
tools.zip can be grabbed from here:
http://www.mediafire.com/?5ngybiamztw
This is meant for testing purposes only as only limited testing and debugging has been done (and almost all on a Mac).
I would like people to test these and let me know where problems exist.
Better versions may come later as ClarkNova has been helping me figure out how to simplify greatly the gui front end pieces.
February 5, 2010 at 2:54 am
That was some quick work there some-updates! 🙂 Thanks.
I’m just starting to play with the test GUIs and I noticed that for Topaz you have Extract scripts for the iPhone and Kindle for PC. I don’t see one for Kindles. I have 2 Kindles and don’t really use K4PC or K4iPhone. Am I looking in the wrong place?
February 5, 2010 at 3:00 am
Scratch that last reply, I now see the Extract script is for Kindle AND K4iPhone.
Sorry for the confusion. 🙂
February 5, 2010 at 2:16 am
It is known that AZW has clipping limit in exth 401 (as seen in mobi2mobi)
TPZ has same attribute somewhere. Can it be disabled? (Or is it already disabled by cmbtc?)
(Yes, I do clipping a lot. I do memos and notes all over the books I read. 😉
February 5, 2010 at 6:50 am
some updates, said on 20 Dec 2009, in comment #382
“So, first ask yourself the following – Why am I trying to remove the DRM? If it is to make anything other than personal use backup copies or to convert it so that your specific reading device can read it, then
no one here will be willing to help.”
I’m not sure where you are from but in the U.S. (and most major countries) there has been, since the early 20th century, an additional limitation of copyright besides fair use called “first sale” http://en.wikipedia.org/wiki/First_sale. People legally obtaining paper books routinely exercise their fair use rights to sell, lend or other wise dispose of them. Publishers have used ToS and EULA methods to circumvent first use and many, many e-book users think its unwarranted, unfair and (if they can have anything to say about it) unenforceable.
Do you and other people on the list oppose using the software hacks here to enable first sale rights?
February 5, 2010 at 1:13 pm
Speaking for myself only. I am a book hoarder and I never sell books of any sort. That is why I went digital – I ran out of bookshelf space in my house to keep them (and long before my wife had relegated my books to the basement while her books actually filled up the shelves).
Given zero space and storage costs, I really can’t see why anyone would want to sell an e-book. I would rather they charged me less for the ebook (less than $9.99) in the first place, and one of the arguments in its favour is the inability to resell.
A think a discussion better suited to mobiread.com
February 6, 2010 at 9:01 am
I’m the same for non-fiction/reference books. They are rarely ever sold. If they become obsolete they are usually tossed. Not so with my fiction books. I often lend or give them to friends after one or a few readings.
As for reasons to sell an e-book I think its the same as for paper: to recoup some of the investment.
February 5, 2010 at 4:56 pm
Question … Is there a something like mobihuff.py which outputs the exact mobi html from non-drm’s mobi books but for the older non-HuffMan compressed books?
I tried installing Mobiperl but the number of pre-reqs and the fact that many of them have nothing to do with simply unpacking the mobi file structure, drove me crazy.
I realize calibre will open these but it will change the html on the fly (removing style information again) and I would rather work with the strange version of html 3.2 that mobi books have directly and not a “converted version”.
Is there anything like that out there or do I have to start with the calibre python scripts and remove all of the code that hacks up the html and styles to get something?
If anyone knows of anything, please let me know.
Thanks,
February 5, 2010 at 5:37 pm
You want MobiUnpack.py
It extract the HTML from unencrypted Mobipocket files, both Palm Doc compressed, and Huffman compressed
It was written by a mobileread member:
http://www.mobileread.com/forums/showthread.php?p=654996
I’ve made a couple of changes to the script myself, to match the trailing byte changes in MobiDeDRM, (ant to make the output file name optional in the command line) and I’ve uploadeed that verion to Mobileread in a comment to the thread.
February 5, 2010 at 8:45 pm
Hi Pul,
Thank you!
That is exactly what I wanted. I am going to add a front end to this tool as well for people and add it to the Kindle-Mobi_Tools toolset.
February 5, 2010 at 9:26 pm
Hi Paul,
One question … the MobiUnpack program does not seem to unpack the image files at all. Is there another program that does that?
Or is that something I will have to add myself.
Thanks!
February 5, 2010 at 10:56 pm
I’m just working on adding that in now. It should be up later tonight. Or perhaps tomorrow morning….
February 5, 2010 at 11:07 pm
Now with image output.
http://www.mobileread.com/forums/showpost.php?p=774836&postcount=5
February 6, 2010 at 3:31 pm
Hi Paul,
Thanks! Grabbed it and modified it to work with a gui frontend and will include it in the final release of the gui versions of the tools.
Nice work!
BTW: I plan to extract and hack up the Calibre Mobi reader.py script to create an a standalone version of mobiml2xhtml.py that will try not to mangle the html too much or strip styles. I will post it so that there are tools to create proper xhtml for all of the major formats for use as an archival format (my intention) or the input to other conversions etc.
Thanks.
February 7, 2010 at 9:18 pm
H Paul,
One last question. I have modified what you posted to work with a gui front end and to put all images in an images/ folder beside extracted mobi html (made it easier to fine their names and extensions to create the links in the standard html version.
Next I have heavily modified Calibre / Mobi/ reader.py to not need or use non python standard libraries and C code (i.e. removed lxml, BeautifulSoup, etc plus all of the code that unpacked things since your version does that). It now converts the xhtml quite nicely except that I have no title or author information (or any other metadata).
I assume that is stored someplace in the original prc file.
Do you have any plans to dump the title, author, isbn, publisher, etc metadata to a file in MobiUnpack or is this something I should try doing?
Thanks!
February 7, 2010 at 10:13 pm
I don’t have any plans to write metadata extractiuon routines, but I can tell you where to find the metadata. It’s all in the first record, in the MOBI header, and (if the file has them) the EXTH records.
As much info as has been reverse engineered about the MOBI header and the EXTH records can be found at
http://wiki.mobileread.com/wiki/MOBI
It might be possible to work out more through careful use of Mobipocket Creator.
It /would/ be rather splendid to extract enough metadata into an OPF file that the original Mobipocket file could be re-created using Mobipocket Creator or Kindlegen.
This would provide a simple way to do minor textual changes without messing up any formatting.
Hmm… I don’t think I’ll have any time to do anything myself over the next week, but I’d be interested to see what you come up with, and maybe I’ll have some time next weekend.
Feel free to contact me off-blog if you wish (see website).
February 9, 2010 at 1:01 am
Hi Paul,
Okay I added in support to mobiunpack for getting the title and codec, and things like that as well as parsing the exth (if it exists) to get the basic metadata, as well as made many small changes to mobiupack.py to make it play well with a gui front-end if people so desire.
I have also extracted Calibre’s mobi reader.py and removed the need for lxml tree parsing and the like as well as removing all of the code for unpacking a mobi (since mobiunpack now does that part) and created a still quite rough mobiml2html.py.
I have placed them along with their GUI front-ends into Kindle_Mobi_Tools_v1.0.zip which can bde downloaded from here:
http://www.mediafire.com/?ynme2mmmigl
Unzip it and look in the lib subdirectory for mobiunpack_v0.15.py and mobiml2html.py
The files in the main directory are the prototype gui versions in case you want to play with those at all.
I am not sure what you mean by enough info to recreate the opf file. I grabbed everything I thought might be useful from the exth metadata. Is the Mobi opf file the exact same format as an epub content.opf file? If there is some spec, I would be happy to take a shot at creating that as well if you don’t have time.
Also, using .html as the file extension for what we get from mobiunpack does not seem right to me since it is not even correct html 3.2 as it stands. So I converted the extension to .rawml but would be happy to rename it to something like .mobiml or whatever you might want.
Thanks again for all of the help and pointing me at the Wiki. It worked out quite well.
Also fwiw, for the Wiki, I found that for my book in exth in id = 208 seems to be a thumbnail image (the wiki had that as unknown), probably of the book cover itself.
Hope this helps,
February 9, 2010 at 7:17 am
That all sounds very good. I agree that using some different extension for the raw text is a good idea. I believe the Mobipocket opf file is very nearly the same format as an ePub opf file, both being closely derived from an earlier version of the Oen Packaging format.
Would you mind any or all of your additions being made available at Mobileread? Since the code contains no de-drm code there’s no reason it couldn’t be hosted there.
February 9, 2010 at 4:10 pm
Hi Paul,
Yes, everything I write or anything I add to is free to be used in any way by anybody and posted anyplace as far as I am concerned.
I will go look at the original oeb opf spec and see what I can find as a format. Right now, for ease of integration into the html, I simply write out what I found as tags.
Take care,
February 9, 2010 at 8:08 pm
I’ve had a go, having dug out the specs at http://openebook.org/oebps/oebps1.0.1/download/oeb101-xhtml.htm and also by comparison to some .opf files I have from compiling dome Mobipocket files.
I’ve uploaded to Mobileread. http://www.mobileread.com/forums/showthread.php?p=780915#post780915
(see message #5)
I had fun trying to convert from Mobipocket’s weird language and sublanguage IDs back to a standard format. I found a table in one of the MobiPerl files that gave the data, and a bit of GREP massaging, and research into Python dictionaries did the trick.
To be able to usefull use the generated opf, however, a lot of work needs to be done on the ‘html’ file. Your code culled from Calibre might well be able to do most if not all that’s needed, with a bit of tweaking.
February 9, 2010 at 8:36 pm
Hi Paul,
Wow that was fast! Nice work. I have grabbed mobiunpack_v0.16.py and will incorporate it into the next version of Kindle_Mobi_Tools (gui stuff).
Yes, please feel free to hack up mobiml2html.py to create any conversions you need.
I added a parser very similar to the parser used in ereader2html.py (and therefore in xpml2xhtml.py) but even simpler because mobi rawml tags are clearly beginning or ending tags (or single tags like ) so a lot of the in_tag stuff needed to convert pml markup to html was not needed.
All that is really going on is that the parser returns all text before any tag, then the tag itself each time it is called.
I added a routine to parse the beginning tags for attributes and returned a tag type (begin, end, single), tag name, and a hashtable of name content pairs for the attributes.
Most of what is done in Mobi reader.py is simply remapping the tag attributes to css style classes and cleaning up a few unsupported attributes and tags.
It should be readily adaptable to convert .rawml to whatever input format is needed by KindleGen and the like.
Let me know if I can help.
BTW: I have not written much applescript. How hard would it be to take the python gui tools I have and convert them to an .app (to prevent the damn terminal from opening a window when it is a pyw file on a Mac). Can I use your other post – the PythonWrapper to do that? Are there any Mac specifc python tools I should be using?
Thanks
February 9, 2010 at 10:32 pm
Yes, I can’t post a long message. But I get the error wrong – it’s “Discarded”. I’ll comment on one of the other posts.
February 9, 2010 at 6:32 pm
Just to let you know that I will compile the scripts in the next few days and then I will have to stop for the foreseable future as I am deploying to Afghanistan again, so will be out of touch for a few months.
I am having some problems compiling unswindle. I can compile ok, but it always errors out when I run the exe file. Are you willing to look at my error and try to help me?
B.
February 9, 2010 at 7:32 pm
Hi Brutusbum,
Sure,
I don’t have access to py2exe since I am on a Mac but I can read the source code of py2exe (and have already studied a bit of it) and I can study unswindle source code as well so I should be able to come up with something if you can post a full error message.
Take care,
February 9, 2010 at 8:29 pm
One of my old soldiers told me today he is also leaving Friday to Afghanistan. Good luck brutusbum. I might be seeing another rotation to the same place in June.
Stew
February 5, 2010 at 6:09 pm
Are there any scripts that can take a DRM-free mobi file and add the DRM? Since mobi’s purchase by Amazon they closed the door on new publishers and authors wishing to remain anon and self-publish for Kindle are also now closed.
February 5, 2010 at 7:17 pm
Umm… why on earth would you want to add DRM to a Mobipocket file? I would have thought that this blog made it obvious that Mobipocket DRM doesn’t work.
Adding DRM to an ebook only irritates your paying customers, and discourages some people from becoming your customer.
February 6, 2010 at 5:15 pm
Its primarily to serve authors and provide new entrants some emotional comfort (however mistaken they may be).
As a user of these scripts (and I can’t thank members here enough) I’m certainly in this camp but atm e-book users in this and similar forums are outliers. Publishers have fought hard to maintain and even expand their legal privileges. I would not be surprised if e-books have a deja vu experience to the DRM battles fought over digital music downloads. In the end DRM is a dead end but you can’t convince many publishers and authors.
February 6, 2010 at 2:35 am
hmm, Pro-DRM gives a new light to this thread. Something to think about anyway. It seems castell wants Self-Publishing with Amazons’s DRM without Amazon. (or mobi-DRM without mobipocket) To what end, I wonder.
Forgive me if I am wrong but castell probably thinks of DRM as some kind of general “copy-protection-only” method, in a material way that was with game CDs and Diskettes. I guess that’s as far as average-non-technical-authors-or-authors-to-be understand.
February 5, 2010 at 10:59 pm
Hi Paul,
That’s great. Thanks for doing that.
Have a nice weekend.
February 6, 2010 at 3:00 am
Hey all,
Before I get to begging for help, I wanted to say awesome job on all of this. When I first read that topaz books were glyphs I was convinced there was no way to convert them to a different format; imagine my surprise when I realized it’d already been done!
Unfortunately, I have so far been unsuccessful in converting my own Kindle4PC Topaz book. When I run the following command (with 1.71 of brutusbum’s package):
python cmbtc_dump_nonK4PC.py -v -d -p K72X7IW6 -p VZaw3Dgo -o outdir ~/My\ Kindle\ Content/B000W90YK0_EBOK.prc
I get the error:
Book key could not be found. Maybe this book is not registered with this device.
The first PID is from running wine cmbtc_dump.exe -i — it is listed as the device PID. The second PID is from running wine cmbtc_dump.exe -v -d ~/My\ Kindle\ Content/B000W90YK0_EBOK.prc — I get the same error, but it first lists the book PID.
I’ve tried several combinations of the -p options, including none (that doesn’t work, obviously), the first one alone, and the second one alone. As far as I know, the book is downloaded on my computer. At this point I’m somewhat at a loss for where to go next. If anyone has any ideas or suggestions, I would greatly appreciate it.
Thanks again!
February 6, 2010 at 4:30 am
If your have a Kindle4PC Topaz book then you can not use the cmbtc_dump_nonK4PC.py to unencrypt it since a book downloaded to Kindle4PC will not just use the PID but will use a book specific PID (unlike standalone Kindles and iPhones).
So you must run cmbtc_dump.exe on the machine you run Kindle4PC on and you will not need to add any PID. Instead it will look up the correct Book PID and take care of it for you.
That is why we named the cmbtc_dump.py and cmbtc_dump_nonK4PC.py.
The second script only works for books purchased and downloaded to standalone Kindles and iPhones/iPods which use only one PID.
So try cmbtc_dump.py and it should do the trick.
February 6, 2010 at 5:23 am
Ah, I misunderstood the intention of cmbtc_dump_nonK4PC – I thought it was just the Mac/Linux version. Of course what you said makes a lot more sense. It looks like K4PC conversion isn’t currently possible on Linux, since the cmbtc_dump.exe gave the same error and the cbmtc_dump.py script gave an error about needing Windows dlls. This isn’t such a big deal really, since the latest version of K4PC doesn’t even begin to work in Wine…
Anyway, I installed WinXP in VirtualBox and used cmbtc_dump.exe and it worked! I’m so happy to have an HTML format. There’s a tiny bit of garbage and formatting issues, but it’s so much better than the tables being shrunk down to an unreadable size, with no search capabilities, etc.
As a side note, it looks like the documentation (at least the HOW-TO file) is a bit behind as far as the process goes, or perhaps just a little confusing. For example, it indicates that you can give genxml an output directory, but no matter what the second argument is it always generates the files into the input dir/xml. Tripped me up for a couple minutes, but not a huge deal. Again, huge thank-you for everyone’s hard work and some_updates, thank you for your help!
February 6, 2010 at 3:24 pm
Hi Kitsuta,
Glad to hear you got it working. Yes the “output directory” of cmbtc_dump* becomes the “input directory” for all of the other Topaz programs and they create other files and subdirectories inside of the original output directory.
So in my thinking they are the same thing but it is confusing. I will fix the readme.txt to make this clearer.
February 6, 2010 at 5:34 pm
You are correct. K4PC conversion cannot take place on linux, since the script must run on the machine that has K4PC installed.
And obviously, there is no version of K4PC for Linux….. yet!
February 6, 2010 at 3:38 pm
I downloaded the new v1.71 but when I double click one of the scripts (exe file) a small window pops up for a second then nothin. I have python 2.6 installed on my win xp machine and unzipped the library into the same dir as the scripts in a folder called library. what am I doing wrong?
February 6, 2010 at 4:22 pm
Hi Amber,
Those programs have been compiled to run on Windows without need for python but they are still command line tools. You must actually pass arguments on the command line.
There are experimental gui tools (that still need python which you have) that all you have to do is double-click.
You can grab a set of the experimental tools from here:
http://www.mediafire.com/?5ngybiamztw
Download that zip and unpack it. Inside is a bunch of tools each in their own folder. Pick the tool you need based on the type of ebook you have.
Open that folder and you should be able to double click on a program and have it do what is it supposed to.
If you let us know exactly what you are trying, I can lead you through it.
I would appreciate hearing any issues or problems you run into if you do try the new gui tools.
February 9, 2010 at 3:15 pm
Hi some_updates
Just downloaded the tools. I thought the script exe files were already “gui-ready” 🙂
I’ll give the gui a try when I am off from work and lyk if there are issues. was trying to convert a book to read in my sony reader is all.
Thanks for your response
Cheers
Amber
February 6, 2010 at 5:30 pm
Anyone willing to host a new group blog / website like this one?
Unfortunately this site has grown in size so much it takes measurable time to load and posts no longer seem to be in order which makes it hard to find your post and nay replies even with the RSS feed.
So is anyone willing to host a new group blog / website like this one? Preferably one not hosted in the US given its draconian laws.
A thought anyway …
February 6, 2010 at 5:41 pm
I agree that a new site with actual Forum software would be of a great deal of help. But I caution all that choosing a site that will be safe against this country’s (USA) laws and sovereignty issues with other nations be of paramount concern. While I love the United States recent decisions of our Supreme Court will enable large corporations (such as publishers) to absolutely control the election of Federal officials such as Congressmen and Senators. This will, if not stopped in some manner, result in far tougher and repressive laws placing corporate wealth above public good as well as a general leaning toward right-wing Fascism. I would hate to see a Forum begin and then, in a year, be wiped out and all information and data erased. So, just be careful.
February 7, 2010 at 6:03 am
How about hosting on a hidden Tor server, for example the free and open http://xqz3u5drneuzhaeo.onion Can also be resolved for those without Tor installed or a Tor browser, http://www.torproject.org/torbrowser/, via a gateway (http://xqz3u5drneuzhaeo.tor2web.com/). I know Tor can be slow but for forum use it should be more than adequate.
February 7, 2010 at 1:56 am
Hi folks,
Let me start off by saying that I am not a huge computer guy, I know the basics of using the command prompt to run mobidedrm and ereader2html, but if something doesn’t work right I have a hard time figuring out why.
I have been having issues with removing the DRM off of a B&N pdb book and was hoping that you might be able to help. I have used ereader2html successfully on two other books, but with this newest book I go through the steps like usual and it begins to process, but then I get an error which starts off with a traceback and then a string of files and ends with an error which reads: IndexError: String index out of range does anyone have any ideas as to what might cause this?
February 7, 2010 at 3:03 am
Hi,
Please present the exact error so that we have some idea of what is wrong.
Also, try erdr2pml.py which converts the file to pml as step 1 and then try xpml2xhtml.py which converts the pml to html.
They have a few bug fixes and improvements in them that have not made it back into the original ereader2html.py which they are derived from.
If you do not have those, you can grab Brutusbum’s 1.71 archive of precompiled exe versions to get them or you can grab the gui versions which can be found here.
http://www.mediafire.com/?5ngybiamztw
f those work fine, if they fail too, then please post the detailed error message so that we can try and figure out what is happening.
February 7, 2010 at 3:57 am
I can only get one version of mobidedrm to work with my files and it’s an old 02 version. However, I’d like to get it to work as a plugin for Calibre. Any help?
February 7, 2010 at 4:04 am
Using the newer versions of mobidedrm I get the “Error: No key found. Maybe the PID is incorrect”. However, using an older version of it, it works just fine. Any reason why?
February 7, 2010 at 5:08 am
Hi R,
Have you tried with mobidedrm version 09 ? As far as I know it is the very latest and it should be backwards compatible with the earlier versions. Some of the versions broke things for others as it went along, but I thought version 09 did the trick.
If you have tried it and it doesn’t work while version of 02 does, then something is messed up and people here might be able to figure out what and add it to the newer versions.
Please let us know more – someone here may be able to reproduce the problem if they new which book, etc.
February 7, 2010 at 6:13 pm
brutusbum,
For genhtml, if there is only a low number of glyphs (e.g. the only topaz book I’ve looked at so far only got 59 in total) wouldn’t it be possible to supply a list of glyphs that are italic/bold (e.g. by generating an svg that shows each glyph with it’s number separately and having the user create an italic.info and bold.info file in the glyphs folder that lists the number of the ones that are italic or bold?
That would certainly beat going over the html later and manually restoring the italic and bold sections.
February 7, 2010 at 9:30 pm
Hi,
It might be doable in your case. But it certainly is not in mine. My book’s glyphs.svg is over 1.3meg long and has some 2,810 different paths.
If you are interested in writing some code to show each glyph one by one of the screen and allow users to record, the font size, font weight, and italics for each one, I would be willing to include it in the next release of the tools.
Even then, code would still have to be written to take this information and code it up in some reasonable way so that the html conversion code can figure out the right tags to use on a character by character basis.
Another approach is to try to automate the process somehow in gensvg.py where you would use software/code (and not a person) to somehow determine ?mathematically? if the letters are slanted (italic) or thicker than normal, what their and size might be, etc.
If anyone knows of any open source code that can take glyph paths and do that, I would be more than happy to try and incorporate it into some future release.
Not easy.
February 7, 2010 at 10:21 pm
I’m trying to convert a Topaz book so I can read it on Mobipocket, using the v1.71 tools it seems I can dump the book and convert to xml using cmbtc_dump.exe and genxml.exe just fine. But when I run genhtml.exe I get the following error:
Processing …
metadata0000.dat
other0000.dat
Using font size: 111
Using page height: 0
Using page width: 0
Traceback (most recent call last):
File “genhtml.py”, line 157, in
File “genhtml.py”, line 138, in main
File “stylexml2css.pyc”, line 236, in convert2CSS
File “stylexml2css.pyc”, line 149, in process
ZeroDivisionError: float division
Can anyone advise?
February 7, 2010 at 11:11 pm
Hi,
Yes, the page height and width are all wrong. They should not be zero.
The code in getpagedim.py tries to extract this for the first TEXT page.
Hmm, very strange. Please look in metadata.txt that should have been created.
Look to see what the firstTextPage value is.
Then go look at the xml for that page (look in the xml directory inside the book directory). Look at pageNNNN.xml (where NNNN is the number of the firstTextPage. (these are all text files so notepad should open them).
Then look in the page for the page height and page width values. Then compare them to the same thing on another page just for comparison.
Do they really say 0?
If all of the other ones have typical values (height of near 11000, and width of 8500) then for some reason the firstTextPage is NOT the first page of Text for you book.
If you have python installed, I can tell you how to workaround this youself (with just notepad). If not and you are using the .exe files, then I will have to code something up for you to try and Brutusbum would have to compile it to exe.
If you do have a text editor,
after the following lines in genhtml.py
if (ph == ‘-1’) : ph = 11000
if (pw == ‘-1’) : pw = 8500
add the following 2 lines (with the same indentation)
ph = 11000
pw = 8500
and then run genhtml.py again and it should work.
February 8, 2010 at 12:48 am
Thanks for the help!
It was as you said firstTextPage was 2, checked that and it was really 0 for h & w. Made the changes to genhtml.py and ran again, book came out just fine.
February 9, 2010 at 10:36 am
I’ve made an improved version of MobideDRM.
This version, 0.10, now also decodes Mobipocket files that aren’t encrypted with a specific PID, but that can only be read with genuine Mobipocket software.
In addition, if run on an unencrypted file, it does not give an error but just outputs the file using the output file name. This means that when used as a Calibre import filter there’s no problem importing unencrypted Mobipocket files to Calibre when the filter is installed.
http://www.mediafire.com/?2ootaz2wood
regards,
Alf.
February 9, 2010 at 12:28 pm
and mirrored here:
http://ifile.it/d8optsf
and here:
http://www.mediafire.com/?1qimdjmzdg0
may god bless every hair on your lovable head! amen.
February 9, 2010 at 10:31 pm
I’m getting “rejected” messages. Perhaps this thread has an upperlimit on comments/size of comments?
I suggest moving to one of the other posts
February 9, 2010 at 11:43 pm
I also posted this on ilovecabbages blog. I’m having trouble with unswindle. It worked when I first d/l it but today it won’t work. I get a brief glimpse of it, then away it goes. Does anyone have any idea what I can do? I tried d/l a new version but nothing comes up when I use that.
February 10, 2010 at 1:30 am
I noticed you called it unskindle over at Cabbages. Unswindle is for removing DRM from Kindle For PC ebooks. Unswindle paired with a copy of mobidedrm should open Kindle For PC when you double click it.
February 10, 2010 at 1:35 am
I meant unswindle over there but I couldn’t edit the post. I have both unswindle & mobidedrm in the same python folder. When I double click unswindle, the python box opens for a second or two, then disappears and nothing happens.
February 10, 2010 at 1:39 am
That sounds odd. Unswindle should open Kindle For PC and nothing else. Usually when you double click a py extension file, a command window will open and then close. But when you double clicka pyw extension, a GUI will open, or in this case Kindle For PC. I would grab another copy of K4PC v6 at Cabbages and try again.
February 11, 2010 at 12:09 am
I just tried the topaz tools and rediscovered a Windows problem.
I was running them from a folder within My Documents and getting errors.
I moved the tools to a different, standard, folder, and everything worked.
February 11, 2010 at 5:36 pm
I’ve started a blog.
http://apprenticealf.wordpress.com/
I probably won’t blog much, but I will try to keep track of the latest updates.
regards,
Alf.
February 12, 2010 at 7:15 am
Hello all,
Have to admit I am new to this whole python scripting for Topaz. I can’t get the script’s properly executed. One of the most important questions is how to discover or generate the PID(s) (I use K4PC) any help? Thanks!
February 12, 2010 at 12:44 pm
If you have K4PC and the Topaz book was downloaded to K4PC on that machine, you should use cmbtc_dump.py on the same machine as K4PC and it will find the correct PID for that book. K4PC does not just use 1 PID, it has a book specific PID. There is a readme.txt that specifies the arguments for cmbtc_dump.py. There are a set of gui tools if you would prefer here:
http://www.mediafire.com/?5ngybiamztw
February 12, 2010 at 11:13 pm
I have been reading this blog for over a month now and even with the GUIs I am having trouble getting even one of my Topaz books to convert. I have Python 2.6.4 installed and when I start TopazExtract_K4PC.pyw and choose my input file and my output directory, the program just hangs. I have even left it for hours and come back and it has done anything except create the output directory itself (no files in it though). My question is: How do I (and I think others who may be having this same issue) get it to work? FYI, I even brought out another Win XP running box and tried it on it and it does the same. The book was downloaded to each box seperately so I wouldn’t create any more headaches that way.
I hope you can help me and then I can document what you say for other users in the future.
Thank you.
February 13, 2010 at 2:28 am
What do you mean by hangs? What exactly does it say in the Conversion Log window? Does it say anything? It should be trying to invoke cmbtc_dump.py. Do you see the command information in the “Conversion Log” info?
Did you by any chance move the TopazExtract_K4PC.pyw away form where it was unpacked? It needs to be right beside its “lib”subdirectory where it finds all of the other pieces of code it needs.
Also, please double check you can actually open and read that book in K4PC?
Let us know what is in the “Conversion Log window in detail (you should be able to copy and paste info from the Conversion Log to here) and we might be able to help. Many people have been successful wit the conversion. We just have to figure out what is different about your case.
February 13, 2010 at 2:33 am
One other thing. Please make sure that these Topaz books were downloaded to your Kindle For PC application on the PC and that you can actually read them. If these Topaz books were actually downloaded to a standalone Kindle or iPod / iPhone, then you will need the PID and to use the TopazExtract_Kindle_iPhone.pyw program instead.
February 13, 2010 at 5:09 am
I think that I have exactly the same problem as wyo_sony_user. However Topaz File Extraction says “Please Wait…” and nothing happens. This is what I’ve done so far.
Extracted Topaz_Tools to root of “C” with lib subfolder in it.
Made sure Topaz book is in the Kindle4PC and that I can read it. The book is in .prc format (unswindle tells me that this is Topaz file). Went back to Topaz_Tools directory and double-clicked TopazExtract_Kindle4PC
When TopazExtract_K4PC.pyw opened I’ve chose my input file and my output directory and this is what I got in Conversion Log:
Command = “python cmbtc_dump.py
Topaz Path Path = “C:\Users\Owner\Documents\My Kindle
Content\Nameoffile.prc”
Output Directory = “C:\temp\Topaz_Tools”
First 8 chars of PID = “”
Please Wait…
And nothing else happens
Running the above on Windows 7 Ultimate with Python 2.6.4 installed
Any thoughts?
February 13, 2010 at 2:52 pm
Yes, you should have seen the first message from cmbtc_dump.py after the “Please Wait” but you did not.
So the gui could not find and invoke cmbtc_dump.py for some reason. We just need to figure out why.
Are you sure you extracted the full lib/ subdirectory? I know some unzip programs will only extract one specific file unless you explicitly tell it to extract everything?
I will fire up my Win box under Parallels desktop on my Mac and see if I can recreate what is happening and get back to you today.
February 13, 2010 at 2:56 pm
One other thing to check.
The cmbtc_dump program does not actually run K4PC and does not use it (just the info it saves after reading the book).
From your description, it sounds like you have K4PC open and running and reading the book at the same time you are trying to run TopazExtract. I am not a Windows person, but one application may in fact “lock” a file while running and not allow others programs to read it. So please exit out of K4PC before running TopazExtract. This may not help but it can certainly not hurt.
February 13, 2010 at 3:03 pm
Hang on. I had the same thing happen to me so I can recreate the problem.
I will post a fix today.
February 13, 2010 at 3:19 pm
It was a typo that randomly could cause the problem. It did not in my initial tests but did for me now (and obviously you two as well).
So here is a new version of tools.zip (with a new version of TopazExtract_Kindle4PC.pyw in it.
Please grab this version of the experimental gui tools and let me know if this fixed the hanging issue or not.
http://www.mediafire.com/?vxd2dudgykk
Thanks for testing it and helping me find the problems.
February 13, 2010 at 5:48 pm
YESSSS! Now it is working. Your new version of Topaz tools decrypted the book. Now I have to go back to reading and find out how the rest of it works. I will keep you posted. Thank sooooo much for your hard work.
February 13, 2010 at 6:35 pm
Glad to hear it works.
Next step is:
1. Doubleclick TopazFiles2SVG.pyw and select the directory you created with TopazExtract_Kindle4PC.pyw
2. Doubleclick TopazFiles2HTML.pyw and select the same directory as above.
You will see a bunch of file names go by as they are processed.
After all is done, in that directory, you should look in the svg/ directory and you should have perfect graphical images of the pages that you can open in your browser. (i.e pictures of what each page look likes) similar to what the Kindle shows you.
And in the main directory, you should also have a book.html file that will be an imperfect version of your book in html – how good or how bad it ends up being depends on how good or how bad the optical character recognition was in the original topaz file.
You will need to fix any ocr mistakes in book.html as well as add back any italics, and bolds that were in the original. The svg version (pictures) of the pages can help you do that.
Good luck.
February 13, 2010 at 7:01 pm
So, if I understand instructions correctly, now I have to run TopazFiles2SVG. However when I run that file I get the following error:
Conversion Log
Command = “python gensvg.py”
Book Directory = “C:\temp\Decripted book”
Please Wait …
Traceback (most recent call last):
File “C:\temp\tools\Topaz_Tools\lib\gensvg.py”, line 388,
In
sys.exit (main (‘ ‘))
File “C:\temp\tools\Topaz_Tools\lib\gensvg.py”, line 289
In main
Flat_xml = convert2xml.main (‘convert2xml.py –flat-xml ‘ + dictFile + ‘ ‘ + fname)
File “C:\temp\tools\Topaz_Tools\lib\convert2xml.py”, line 774, in main
dict = Dictionary (dictFile)
File “C:\temp\tools\Topaz_Tools\lib\convert2xml.py”, line 121, in _init_
Self.fo = file(dictFile, ‘rb’)
IOError: [Errno 2] No such file or directory:
‘C:\\temp\\Decripted’
Processing Meta Data …
metadata0000.dat
Processing Glyphs …
glyphs0000.dat
Error: SVG conversion Failed
Thanks again for all your help
February 13, 2010 at 7:15 pm
Hi,
Yes, the space in the name of the output directory is causing the problem.
Please simply rename
“C:\temp\Decripted book”
to
C:\temp\Decripted_book
(notice the underscore used to replace the space) and then rerun TopazFiles2SVG.pyw and all should go well.
I will try to work around this issue in the code so that spaces in directory names do not cause problems in the future.
Thanks,
February 13, 2010 at 7:47 pm
Yessss again. I have made the changes to directory name and everything is working as per your instructions. Now I have graphical pages in svg directory and a file called book.html
OCR is almost perfect but it needs few changes. As I’m not computer wizard as you, I’m not sure what other programs I can use to make changes in book.html Can you suggest any, please? Also, after corrections, Can I use program like Calibre or maybe Mobipocket Creator to convert the html to mobi. Sorry again for my ignorance in this as I’m new to it. Thanks again, from the bottom of my heart which is deep inside my body, for all your superior advice. (this is small book with only 200 pages but next one is about 1500)
February 13, 2010 at 8:19 pm
Yes there are programs that can make it easier to edit html files. The easiest one that comes to mind is Sigil which you can find links to and a support forum on http://www.mobileread.com.
The only problem with Sigil right now is a bug that exists in Sigil 0.1.9 (the latest and all earlier version) that messes up the images since as it happens the image names that topaz extraction uses and the image names used inside Sigil clash and this causes problems. This is scheduled to be fixed in Sigil version 0.20 which is coming out in a few weeks time. So I would recommend Sigil is you can wait 2 or 3 weeks.
Until then, I have seen people import the code into OpenOffice.org Writer (open source, cross-platform and free) and edit it until their heart is content and then save it as html. Some people also use Microsoft Word for this but the html you get back is not pretty but it does work and the un-pretty part can be cleaned up by Calibre upon import.
Either way, you should be able to use Calibre to convert the edited html to mobi quite easily.
February 13, 2010 at 9:45 pm
Thank you again for all your help. I will wait for Sigil 0.20 before I’ll start using it. In the mean time I will post couple suggesting at Sigil wiki which might help in editing Topaz books. You are my hero. Thanks again for your support.
February 12, 2010 at 5:48 pm
Thank you thank you!! The mobidedrm plugin has been great–I appreciate being able to use my ebooks in whatever reader I happen to have, and I have been slowly going through my library. However, I find that some of my files, apparently some prc files from fictionwise, are not dedrmed by your excellent program. They don’t seem to use a pid; instead, they require me to enter my password to fictionwise before I can read them, even in the computer mobireader program. I was hoping that version 10 would deal with this but it doesn’t seem to do anything. Any ideas?
February 12, 2010 at 6:05 pm
The files you have aren’t fully encrypted. After connecting to fictionwise, I think Mobipocket Reader wil create a copy of the file that is encrypted with the Mobipocket Reader’s PID, and store it in somewhere like My Documents/My eBooks.
If you run MobiDeDRM on the copy that’s made in there, using the Mobipocket Reader PID, I think it’ll work.
February 12, 2010 at 9:32 pm
Hmmm. Currently, I’m running mobidedrm as a plugin for Calibre, using a PID…..Wait. I looked in fictionwise and there are a *list* of different PIDs(who knew?) …I put all of them in and it seems to be working! Hah!! Thanks; I had not thought of multiple PIDs.
February 13, 2010 at 8:31 pm
The latest version of topazscripts_2.2.zip is now included as part of a full set of GUI tools and python based conversion scripts.
tools_v1.0.zip
which can be downloaded from:
http://www.mediafire.com/?zji0nyzwo5t
Changes include fixes for topaz extraction and spaces in paths, updated mobidedrm to v10, new programs to unpack (mobiunpack.py) non-drm mobi files and convert them to html (mobiml2html.py), a new beta version of ineptkey.pyw (version 4) that will is supposed to work with Adobe Digital Editions 1.72 (downloaded and included but not personally tested yet).
February 14, 2010 at 4:49 pm
I’m using v1.71 of the tools to extract a topaz file. Everything works without error until I get to the GENHTML stage. Then I get the following messages…
Processing …
metadata0000.dat
other0000.dat
Using font size: 125
Using page height: 12782
Using page width: 8467
page0000.dat
page0001.dat
page0002.dat
page0003.dat
page0004.dat
page0005.dat
page0006.dat
page0007.dat
Error: – incorrect word_semantic ordering inside paragraph
Traceback (most recent call last):
File “genhtml.py”, line 157, in
File “genhtml.py”, line 147, in main
File “flatxml2html.pyc”, line 675, in convert2HTML
File “flatxml2html.pyc”, line 612, in process
File “flatxml2html.pyc”, line 395, in getParaDescription
ValueError: invalid literal for int() with base 10: ‘LOCATION_INFORMATION’
I have successfully converted topaz files before, so I’m wondering what’s the problems here. Any ideas?
February 14, 2010 at 6:11 pm
Hi,
It means the xml inside of your topaz script is really messed up and seems to be putting the end before the beginning in a sequence (which no book until now has done).
So if you would please run genxml and then post for me the xml/page0007.xml file someplace so that I can figure out what they are doing on that page that is so confusing the xml parser.
Once I know what they are doing, I should be able to figure out a way around it and post a revised version for you to try.
February 14, 2010 at 7:07 pm
Thanks. I have tried this with both the Kindle topaz file and the K4PC downloaded file and get the same error.
Here is the xml/page007.xml file
http://www4.zippyshare.com/v/99918576/file.html
February 14, 2010 at 9:06 pm
Hi Sue,
Yes, the xml used in page0007.xml is quite strange and out of order – it provides lastWord information before it provides firstWord information which confused the hell out of my parse code (and I have never seen this before).
I have changed the code to deal with these out of order conditions.
I have posted the new topaz scripts as part of the package:
tools_v1.1.zip
which can be grabbed from here:
http://www.mediafire.com/?ju3qzwmmjgy
The only problem is Brutusbum has not used pyexe to compile these to the exe format. So in order to make this work, you will need to install Python 2.6. You can find links on how to do that for Windows at the very top of the original Dark Reverser’s blog.
The good part is that you can use the gui and simply Double-Click to launch the tools.
Since you have already run gensvg.py and genxml.py all you have to do is double-click on TopazFiles2HTML.pyw and select the directory you dumped your book into, and it should handle the rest.
Please let me know if you run into trouble and I can try to help.
February 14, 2010 at 10:36 pm
I already had Python installed, so it was a breeze. Your fixed script worked like a charm. Now I have an HTML file I can use.
Thanks a bunch.
February 15, 2010 at 12:55 am
This IS WORKING!! The previous version did not… Absolute genius!!! Thanks a lot!!
February 15, 2010 at 5:24 pm
I’m trying to understand the .mobi/azw file formats. Is there any more detailed source than http://wiki.mobileread.com/wiki/MOBI ? In particular, I’m trying to understand the settings associated with expiry.
February 15, 2010 at 5:41 pm
The binary file format is not documented anywhere. And in particular, the DRM and expiry system aren’t documented anywhere. If you look at the source for MobiDeDRM, I think you’ll find a couple of variables named expiry and expiry2. I’ve no idea if these are related to the expiry date for certain DRM systems, but they might be.
Why do you need to understand the expiry settings?
February 15, 2010 at 6:05 pm
I want to loan some of my Kindle books to online people I have met. These people appear to be noobs and not likely to use deDRM tools and I’d rather not remove the DRM and easily enable the books to be file-shared. Setting the Expiry fields seems like a possible alternative.
February 15, 2010 at 6:48 pm
*IF* the expiry fields mentioned in MobiDeDRM are related to the expiry time, you’d need to work out how the time is encoded in these two 32-bit values, and then you’d need to work out how to reverse what MobiDeDRM does with the DRM entries, so that you could encode a new PID and expiry time in one of the fields. I suppose that it might be possible to encode the ‘default’ PID (whatever that is), so that the files wouldn’t need a particular PID, but would work on any genuine Mobipocket reader.
Possible, I think, but it seems a lot of work compared to giving them a de-drmed sample file of (say) half the book.
February 15, 2010 at 7:53 pm
How does one de-DRM just 1/2 a book or did u mean to somehow edit the de-DRM’s version and delete 1/2 the content?
I still prefer to use the Expiry fields. Here is what I think I know:
-expiry1 is typically set to 0xFFFFFFFF, expiry2 to 0x00000000
-first one seems to be the expiration date, 2nd the validfrom date
-the PRC/PDB format typically uses dates in the format unix_epoch – 2082844800L
-however there are 2 variants apparently, signed and unsigned
Perhaps a structured approach with automated sweep generations of the bitfields in order to catch both trigger points between valid/expired and notactiveyet/valid. If I get that narrow enough, I can find the way that the time is encoded.
Any other ideas?
It seems my coding skills are a bit weak for this. Perhaps you, some_updates or another member might find this interesting/important enough to work on. Might a $50 bounty, paid in Liberty Reserve or some other digital currency help?
February 15, 2010 at 8:22 pm
If I was doing it, one way would be to de-drm the file, convert to ePub using Calibre, delete half in Sigil, convert back to Mobi using Calibre.
Since the expiry fields are encoded in the same field as the PID, you would have to work the generation of this field backwards from the code in MobiDeDRM that decodes it.
This would enable you to change the PIDs and expiry dates stored in encrypted mobipocket files. But it’s not something I’m interested in working on.
February 15, 2010 at 11:40 pm
Hi Tedd,
There is one thing you could try. Please edit TopazExtract_Kindle4PC.pyw and right **before** this line:
p2 = Process(cmdline, shell=True, bufsize=1, stdin=None, stdout=PIPE, stderr=PIPE, close_fds=False)
You could try adding the following line (with the same level of indentation).
cmdline = cmdline.encode(sys.getfilesystemencoding())
If that works, you can add the same thing to all of the other pyw scripts in the Topaz_Tools directory and they should work too.
This is of course assuming the problem is in subprocess and not in Tk itself.
If that works please let me know and I will add that to all of the tools and create a tools_v1.2.zip.
February 16, 2010 at 7:31 pm
Newest version of B&N reader doesn’t have the bd file anymore… So the ignoble stuff doesn’t work. Anybody got a newer version?
February 16, 2010 at 10:38 pm
Hi Davo,
Can you be more precise. Are you saying the keygen script by IHeartCabbages does not work anymore?
http://i-u2665-cabbages.blogspot.com/2009/12/circumventing-barnes-noble-drm-for-epub.html
What do you mean by the “doesn’t have the bd file anymore”?
February 17, 2010 at 11:19 am
Try using the KeyGen instead of the other one. Just enter you’re details. I think this is the best format, you keep ePub, and strip DRM.
KeyGen has always worked for me. Retrieve never has.
February 17, 2010 at 11:27 am
To all, thanks for the wonderful work. Tools v1.1 ereader seems to work flawless. I’ll try others later!?!
Anyone set up a new board for us on this topic? Something that can be more easily followed?!
February 17, 2010 at 2:51 pm
There seems to be something at
http://apprenticealf.wordpress.com/
February 17, 2010 at 7:06 am
I made some additions to erdr2pml.py version 0.13 that makes it more useful for me. Straight after the step:
convertEreaderToPml(infile, name, cc, outdir)
I added the following:
#——————————————————————-
#Addition to make a file more suited for Calibre
import zipfile
import shutil
print “Creating PMLZ file for Calibre…”
list = os.listdir(outdir)
for file in list:
filePath = os.path.join(outdir,file)
if os.path.isfile(filePath):
myZipFile.write(filePath)
elif os.path.isdir(filePath):
imageList = os.listdir(filePath)
for image in imageList:
imagePath = os.path.join(filePath,image)
if os.path.isfile(imagePath):
myZipFile.write(imagePath)
myZipFile.close()
print “Removing output directory…”
shutil.rmtree(outdir)
#End of addition
#——————————————————————-
February 17, 2010 at 3:32 pm
Hi,
Thanks for sharing your code additions. I am a little worried about including them in the next release of tools_v1.x.zip, for two reasons:
1. your code assumes you have chosen an outdir that is **only** used for that book’s output and nothing else, and that you actually want that outdir to be deleted. In the general case this need not be true at all. So somehow, we would need to test that outdir is empty except for the *.pml file and image directory before doing this.
2. on many platforms, simply typing zip -r mybook.pmlz outdir/
will generate the same thing with no changes needed at all.
So the solution can be:
A. fork to create an erdr2pmlz.py script that has your additions and remove the outdir option completely and instead simply create a temporary directory to hold the results until the zipping can be done.
or
B. add a command line options to erdr2pml.py to produce a pmlz
(–make-pmlz) file that always ignores the outdir info if provided and instead creates a temporary directory to hold the output in before zipping.
or
C. modify erdr2pml to always create the pmlz archive using the outdir but instead of deleting the outdir, simply store the final pmlz zip archive in the outdir along with the pml and images so that users can have the choice of which pieces they want to use
Comments welcome on which would be the best way and I can code up the changes and include them in the next tools_v1.x release.
February 18, 2010 at 8:57 am
Very valid concerns, and:
zip -r mybook.pmlz outdir/
is probably the “better” option, but I do like option B, a command line option –make-pmlz.
February 18, 2010 at 9:08 am
Just to add, Calibre already has a plugin which searches for images and converts a PML file to PMLZ. I made the additions because I work on two different systems, and for me this simplified the sending of books to be loaded in calibre.
February 17, 2010 at 4:19 pm
I have a Topaz formatted book and was succefully able to generate xhtml pages (800), however using the genhtml.py gives an error, list index out of range. I have already tried the adapted script that was posted here somewhere above by some_updates. Suggestions? should I post error codes as described above?
February 17, 2010 at 4:28 pm
Hi Felix,
Please double-check that you are using the version of the Topaz tools that comes with tools_1.1.zip posted earlier. It has fixes for a similar problem although yours may be unique.
1. If you are using the very latest (which is available only in that tools zip archive), then please post a full description of the error. Please note what the last page number to be processed was. It failed while trying to process that page so noting the number NNNN is important. If you run TopazFiles2HTML.pyw you can copy the exact error message from the Conversion Log screen and paste it which would be a big help.
2. run genxml.py (or if you are using the GUI scripts TopazFiles2XML.pyw) and post for me the xml/pageNNNN.xml file where NNNN is the number of the page it was processing when it failed so that I can look at what the xml is actually doing that is confusing the html generation code and figure out how to work around it.
Thanks
February 18, 2010 at 3:01 am
The ineptkey is upgraded to v4.2 so it also works with 1.7.2 and the old scripts. you might upgrade to ineptpdf 6.2 anyway …
February 18, 2010 at 3:32 am
Isn’t it 4.3 now?
February 18, 2010 at 3:49 am
yeah you’re right – 4.3 is the newest version
February 18, 2010 at 5:32 am
Sorry, where can I find ineptpdf 6.2 (and ineptkey)? Thanks.
February 18, 2010 at 1:17 pm
@Terisa
I sent you something at MR.
February 18, 2010 at 1:40 pm
Just do a google search like
ineptkey 4.3
February 18, 2010 at 1:49 pm
Check my blog.
http://apprenticealf.wordpress.com/
I’m trying to keep each post up-to-date on the various drm removal scripts for the different ebook formats.
regards,
Alf.
February 18, 2010 at 4:36 pm
I’m using ereader2html09.py on some B&N ereader .pdb files so that I can then use Calibre to convert to .epub for my Sony Reader. Works great!! Thanks.
My question is how can I add to the script to call Calibre command line process after the html is produced and have it convert all in one Python script? I’ve spent a couple of days and gotten to nothing but dead ends. Any ideas?
Thanks.
February 19, 2010 at 2:18 pm
Here is the latest update to scripts and gui tools:
tools_v1.3.zip
http://www.mediafire.com/?yjumy0dn0hy
changes:
– update to mobidedrm version 12
– fixes for all of the gui tools to work on asian based machines
– fixes for more out of order xml in the Topaz Tools
– now includes ineptkey_v4.3.pyw for use on Windows for ADE 1.71,
and 1.72
– now includes mac version of ineptkey
– new versions of mobiunpack.py and mobiml2html.py that work with normal (unencrypted) mobi tools to unpack them so that small changes can be made easily and then use MobiCreator to recreate them, plus a tool that converts them to xhtml for archival purposes
February 19, 2010 at 4:47 pm
@some_updates
Thanks for all your hard work.
February 20, 2010 at 11:45 am
Some_updates,
Thanks to you and everyone else for all of YOUR HARD WORK! We all appreciate it!
I am going to post you’re reply at the very bottom of this list to make it easier for people to see.
February 21, 2010 at 2:01 pm
Thank you John. At least RSS feed spots comments all right.
February 20, 2010 at 11:48 am
Some_updates posted this above, and with the weird wordpress system, it got lost in the mess, so, posted from above (all credits to some_updates).
—————-
Here is the latest update to scripts and gui tools:
tools_v1.3.zip
http://www.mediafire.com/?yjumy0dn0hy
changes:
– update to mobidedrm version 12
– fixes for all of the gui tools to work on asian based machines
– fixes for more out of order xml in the Topaz Tools
– now includes ineptkey_v4.3.pyw for use on Windows for ADE 1.71,
and 1.72
– now includes mac version of ineptkey
– new versions of mobiunpack.py and mobiml2html.py that work with normal (unencrypted) mobi tools to unpack them so that small changes can be made easily and then use MobiCreator to recreate them, plus a tool that converts them to xhtml for archival purposes
—————-
Thanks some_updates and everyone else for your hardwork!!!
February 21, 2010 at 1:43 am
Hi,
I am running the latest unswindle for a Kindle for PC book. I run unswindle, pick the book which is in my library, and close kindle for pc. The save file dialog opens and I pick a name for the unDRM file.
MobiDeDRM then pops up asking for the output file and my PID. I have no PID or serial number. I am using Kindle for PC. Any ideas on what is going on and how to get this to work?
February 21, 2010 at 1:46 am
Hi,
I am running the latest unswindle for a Kindle for PC book. I run unswindle, pick the book which is in my library, and close kindle for pc. The save file dialog opens and I pick a name for the unDRM file.
MobiDeDRM then pops up asking for the output file and my PI
February 21, 2010 at 4:04 am
Go to the link below. Follow the instructions and let us know if you still have the problem.
http://nyquil.org/archives/1235-Converting-Amazon-Kindle-For-PC-Books-Now-Possible.html
February 21, 2010 at 11:35 pm
Hi Quicksilver24,
Make sure you have the right mobidedrm.py file. If you have downloaded the tools_v*.zip archive, make sure you are not mistakenly using the MobiDeDRM.pyw program (notice the extension is pyw and not py) which is the front end gui for the real program which is called mobidedrm.py and is stored in the lib/ directory. That is the one you want not the front end gui program.
February 22, 2010 at 2:47 am
There are 2 main problems with topaz to html conversion right now. First, it loses any text style formatting (italic / bold text). Second, there are still a lot of errors in the OCR text.
For a one off conversion of a single book I’ve done some experiments how to improve that situation and the result has been quite successful, but the code I wrote to do that is specific to that book (and not in python or C, it was just running against the files generated with genxml and gensvg) and would be quite useless for anyone else if I released it.
But the ideas I used do apply to all topaz books and I think someone might be able to write general purpose programs for it.
First step is to identify as many glyphs as possible. Going through the XML file, found all cases where the length of the OCRed word and the number of glyphs is equal (large majority of the cases) and assuming a 1:1 mapping between characters and glyphs, counted how often each glyph represents a specific character (generating an html file that showed each glyph and what character(s) it mapped to). Except for punctuation marks, all cases where a glyph mapped to more then one character represented OCR errors and looking over that html file it was possible to very quickly identify these cases and build up a list of definite character mappings for ambiguous glyphs. This information can then later be used to fix some of the errors in OCRed text.
Next step, armed with the information from the previous step, it is now possible to look at all words where the number of characters does not match up with the number of glyphs. It is possible to generate a string based on the glyphs (with some place holder character for each unknown glyph) and match this string up with the OCRed text. This allows the identification of glyphs that represent multiple characters as long as they are flanked on both sides with know glyphs. What follows is the same as in the first step, display of these glyphs and their possible mappings via a html file, and creation of a definite glyphid to character (sequence) list. It might be necessary to repeat this step a number of times as every identified glyph might enable the identification of more glyphs.
At the end of this step the majority of all glyphs should be clearly identified and that information can be used to fix the majority of OCR errors automatically.
That leaves a handful of glyph sequences that simply can not be mapped to useful text on a glyph by glyph basis (usually because a single character comes out as multiple glyphs), it is possible to to generate an html file that shows how these glyph sequences look and enter a proper translation for them.
e.g. In the book I was working on there was the glyph sequence 13568,10754,12288,1029,5146 which got translated in most cases into |An| but the correct mapping for that sequence should be |Ari.|
Armed with this last list of glyph mappings, it should now be possible to fix up almost all errors in the OCRed text.
If I hadn’t had to write the program to do all this the whole process would have only taken 10 minutes, much quicker then later trying to fix up all the errors in the final HTML.
Now on to text styles. The user has to specify the glyphid of one (or more) glyphs that are italic. That, in combination with the information generated above and the assumption that if one character in a word is italic, all are italic, makes it possible to automatically identify all italic glyphs. The process isn’t perfect because that last assumption isn’t absolutely always true. So I generate an html file that shows, in order in which they were found, all the glyphs that are assumed to be italic. Going through that list it’s easy to visually spot the first glyph where the identification went wrong. add that glyphid to “definitely not italic” list and rung the program again until all glyphs are correctly identified as either italic or not italic (I ended up with 2 glyphs that I had to manually identify as definitely italic, and 7 as definitely not italic to automatically find all two hundred something italic glyphs in that book).
Again, if I hadn’t had to write the program, the above process would have only taken a couple of minutes.
If the book also uses bold characters, the above process can be repeated for them.
Now we should have the following information:
– glyph to character mapping information for correcting OCR erros
– flags for each glyph specifying if it’s italic or bold (or both).
By feeding that information into the html creation process it becomes possible to fix a lot of the OCR errors and to generate the right tags for bold and italic text styles, all in a fraction of the time it would take to manually fix up the html as it’s currently generated.
February 22, 2010 at 2:52 am
Repost because this stupid blog somehow mixed in my comment quite some way up and not at the end (at least that’s how it’s displayed to me):
There are 2 main problems with topaz to html conversion right now. First, it loses any text style formatting (italic / bold text). Second, there are still a lot of errors in the OCR text.
For a one off conversion of a single book I’ve done some experiments how to improve that situation and the result has been quite successful, but the code I wrote to do that is specific to that book (and not in python or C, it was just running against the files generated with genxml and gensvg) and would be quite useless for anyone else if I released it.
But the ideas I used do apply to all topaz books and I think someone might be able to write general purpose programs for it.
First step is to identify as many glyphs as possible. Going through the XML file, found all cases where the length of the OCRed word and the number of glyphs is equal (large majority of the cases) and assuming a 1:1 mapping between characters and glyphs, counted how often each glyph represents a specific character (generating an html file that showed each glyph and what character(s) it mapped to). Except for punctuation marks, all cases where a glyph mapped to more then one character represented OCR errors and looking over that html file it was possible to very quickly identify these cases and build up a list of definite character mappings for ambiguous glyphs. This information can then later be used to fix some of the errors in OCRed text.
Next step, armed with the information from the previous step, it is now possible to look at all words where the number of characters does not match up with the number of glyphs. It is possible to generate a string based on the glyphs (with some place holder character for each unknown glyph) and match this string up with the OCRed text. This allows the identification of glyphs that represent multiple characters as long as they are flanked on both sides with know glyphs. What follows is the same as in the first step, display of these glyphs and their possible mappings via a html file, and creation of a definite glyphid to character (sequence) list. It might be necessary to repeat this step a number of times as every identified glyph might enable the identification of more glyphs.
At the end of this step the majority of all glyphs should be clearly identified and that information can be used to fix the majority of OCR errors automatically.
That leaves a handful of glyph sequences that simply can not be mapped to useful text on a glyph by glyph basis (usually because a single character comes out as multiple glyphs), it is possible to to generate an html file that shows how these glyph sequences look and enter a proper translation for them.
e.g. In the book I was working on there was the glyph sequence 13568,10754,12288,1029,5146 which got translated in most cases into |An| but the correct mapping for that sequence should be |Ari.|
Armed with this last list of glyph mappings, it should now be possible to fix up almost all errors in the OCRed text.
If I hadn’t had to write the program to do all this the whole process would have only taken 10 minutes, much quicker then later trying to fix up all the errors in the final HTML.
Now on to text styles. The user has to specify the glyphid of one (or more) glyphs that are italic. That, in combination with the information generated above and the assumption that if one character in a word is italic, all are italic, makes it possible to automatically identify all italic glyphs. The process isn’t perfect because that last assumption isn’t absolutely always true. So I generate an html file that shows, in order in which they were found, all the glyphs that are assumed to be italic. Going through that list it’s easy to visually spot the first glyph where the identification went wrong. add that glyphid to “definitely not italic” list and rung the program again until all glyphs are correctly identified as either italic or not italic (I ended up with 2 glyphs that I had to manually identify as definitely italic, and 7 as definitely not italic to automatically find all two hundred something italic glyphs in that book).
Again, if I hadn’t had to write the program, the above process would have only taken a couple of minutes.
If the book also uses bold characters, the above process can be repeated for them.
Now we should have the following information:
– glyph to character mapping information for correcting OCR erros
– flags for each glyph specifying if it’s italic or bold (or both).
By feeding that information into the html creation process it becomes possible to fix a lot of the OCR errors and to generate the right tags for bold and italic text styles, all in a fraction of the time it would take to manually fix up the html as it’s currently generated.
February 22, 2010 at 3:09 am
Hi Elminster,
Sounds interesting. But how can the software you describe be specific to your book … especially if it is run against the xml and svg files. It simply can not be useless for anyone else.
Please post what you have written (it does not matter the language) and people can work from there to try to generalize it.
Thanks,
February 22, 2010 at 7:34 am
Because I didn’t really keep individual programs for individual steps. I just started with the first step, put the data I acquired through each step into the source and kept modifying the source for additional steps.
Anyway, here it is:
genpages.dpr: http://pastebin.com/fbee6571 – turns the page0*.xml file into a pages.xml and removes the double use of the “word” tag
pages.pas: http://pastebin.com/f7f4a79c1 – wrapper classes for access to pages.xml
processpages.dpr: http://pastebin.com/f7e96a74 – processes pages.xml
The code can make spaghetti jealous and it really is in no shape to even modify it into something useful in my opinion. But it DID help me to figure out what exactly it takes to turn a topaz file into usable html, and that’s what I described above.
Especially the final html creation part of this code is much more primitive then the code in genhtml, e.g. I doesn’t create a style.css and so on. But it was sufficient for my needs.
February 22, 2010 at 2:35 pm
Hi Elminster,
Thank you! Anything is a help because it will serve as an example to make your ideas more concrete.
I have a way to parse the xml as a sequence of assignment statements (use of –flat-xml) which makes it really easy to parse which should help simplify things a bit, and I can certainly collect glyph number to character match statistics to see how effective this would be for larger numbers of glyphs.
I will try to put something python based to look at this in more detail.
Thanks again for the ideas and the code.
February 22, 2010 at 3:29 pm
some_updates,
the book I was working on had 737 glyphs (59 glyph dat files).
As far as single character glyphs go, the automatic detection works very well (with the exception of punctuation marks, e.g. it happens often that both , and ‘ share the same glyph, just their positioning is different).
It gets more difficult where a single glyph contains multiple characters. especially fi, fl, ff and are running into each other quite often and end up as a single glyph.
And then there are the rare cases where a single character gets split up into multiple glyphs, that really messes things up.
The one thing to be aware of is that you need to allow some give and take, you can’t expect to get a perfect glyph to character match for all glyphs.
But getting good matches for the majority of them is enough to produce markedly better results then the current genhtml does.
February 23, 2010 at 9:59 pm
Hi Elminster,
I have as step 1, createed a use count for every glyphid in every word where the number of glyphids is equal to the number of the letters in the word (your first pass)
Statistically, I have run into a few problems.
I have 2810 different glyph paths:
Of those I found at least 1 character match for 2584 after just your first pass. This seems to be quite good.
Unfortunately, statistically speaking there really is only reasonable power in about 10% of the glyph cases to be reasonably assured that character to glyph mapping is correct:
2 and under uses: 1334 glyphs
from 2 to 5 uses: 326 glyphs
from 5 to 10 uses: 297 glyphs
from 10 to 100 use cases: 469 glyphs
greater than 100 use cases: 158 glyphs
In fact, for my particular book, most of my ocr issues were with glyphs only used once or twice in the book.
(confusing “l”, “1”, “I”, “\”, “/”, and “|”)
Even some of the highest use count glyphs were technically wrong – mapping u umlaut repeatedly to just u, similar problems with o umlaut, and with accent symbols.
These were again all caught by my spellchecker (you can get a spellchecking plugin for Firefox) with nice recommendations (in context) for how to fix them.
The good news is it is quite easy to create a glyph to character mapped html file showing the glyph and how many different characters it is mapped to. You were quite right that it does not take long to check visually even 2000 some glyphs when they are written right beside that character they are supposed to represent.
The problem is we really need to make a very very easy to use user interface to show the glyph, characters it maps to use (default ot one with the highest count),counts, and checkboxes for italics, bold, and maybe even relative font size that can be used.
I am no client-side web-developer, but if someone was (using javascript or java) able to create an interface like that, I would certainly include it in future releases.
If we had that and then preloaded it with the output of your step 1, we could use that type of program to collect all of the corrections and have it output a “fixup” table that could be imported into genhtml.py for processing.
I will keep playing, but I am beginning to think your book must have had quite a few more ocr typos than my book did. I was able to load the html in Firefox and spellcheck the html and have it point out the fix in most cases without much effort.
That said, your approach is truly the only way to handle italics versus non-italics issue and you were dead right about how fast it can be to visually check even a large number of glyphs.
If anyone else wants to play around, I can post my glstats.py and glcount.py python scripts so we can see how use of character to glyph mapping can be used.
February 24, 2010 at 1:55 am
some_updates,
My findings in regards to statistical distribution of glyph usage were relatively similar (with about a hundred glyphs used over 10000 times each and most other glyphs used quite rarely). But with under 800 glyphs in total that naturally resulted in a much smaller number (and percentage) being in the rarely used category.
Also, the book I was working on was in English, so no umlauts and accents and also as far as I can remember no use of / \ |.
Outside of punctuation marks (where a proper glyph to character mapping is simply impossible), the most common errors were in regards to confusion between i / l and f / t. But these were by far not the only ones.
Before I started writing that program, I did start with the normal genhtml output and a spellchecker, but decided to see if I can’t come up with a better solution after it took me more then 30 minutes (with hundreds of different misspelled words) to get less then 25% through.
I would like to point out though that the ability to fix up most (not all) OCR errors automatically was only a side-effect, my primary goal was to automate the creation of proper italic tags because the book I was working on had over a thousand separate sections of italic text and correcting all of these by hand would have taken a very long time.
Once I got to working on identifying italic glyphs I found that visually identifying all the italic glyphs can get very tedious and it’s sometimes very hard to distinguish an italic and non-italic glyph when just looking at a singly glyph. That’s why I wrote the code to try and automate finding all italic glyphs after specifying just a handful manually.
February 22, 2010 at 9:55 pm
to some_updates:
Thank you for your great work.
I have a Topaz book and while trying to convert it, I’m getting some errors. I use tools_v1.3.zip, but I’ve been getting the same error with all the previous versions. I start TopazExtract_Kindle4PC.pyw and extract the book. Everything goes well. Then I start TopazFiles2SVG.pyw
Here is what I get:
Conversion Log
Command = “python gensvg.py”
Book Directory = “C:\tools\Topaz_Tools\lib\New”
Please Wait …
Processing Meta Data …
metadata0000.dat
Processing Glyphs …
glyphs0000.dat
glyphs0001.dat
…….
(Everything goes well)
…….
glyphs0595.dat
glyphs0596.dat
Processing Pages …
page0000.dat
page0001.dat
…….
(Everything goes well)
…….
page0332.dat
page0333.dat
Traceback (most recent call last):
File “C:\tools\Topaz_Tools\lib\gensvg.py”, line 396, in
sys.exit(main(”))
File “C:\tools\Topaz_Tools\lib\gensvg.py”, line 320, in main
flat_xml = convert2xml.main(pargv)
File “C:\tools\Topaz_Tools\lib\convert2xml.py”, line 778, in main
xmlpage = pp.process()
File “C:\tools\Topaz_Tools\lib\convert2xml.py”, line 703, in process
snippet = self.injectSnippets(self.snippetList[0])
File “C:\tools\Topaz_Tools\lib\convert2xml.py”, line 559, in injectSnippets
aso, atag = self.injectSnippets(asnip)
File “C:\tools\Topaz_Tools\lib\convert2xml.py”, line 559, in injectSnippets
aso, atag = self.injectSnippets(asnip)
File “C:\tools\Topaz_Tools\lib\convert2xml.py”, line 551, in injectSnippets
name = tag[0]
IndexError: list index out of range
Error: SVG conversion Failed
I tried to delete page0332.dat and page0333.dat. As a result, it goes to page0352.dat and spews out the same error.
February 22, 2010 at 11:29 pm
Hi Mars,
It sounds like something on page0333.dat is messing things up.
Please try running TopazFiles2XML.pyw (or genxml.py) and post the file xml/page0333.xml and send me a link. Once I see what is on that page, I can try to figure out what is wrong and find a fix.
If you get an error running the xml generation, code, then I will need a copy of page0333.dat and dict0000.dat to see what type of xml you are running into that is confusing the parser.
Once I see the problem, the fix should be straightforward. It may be the blank page issue reported earlier. If it is, there has already been a contributed fix which will be in the next release.
Thanks
February 23, 2010 at 12:59 am
To some_updates,
First, many thanks to you for your tireless effort!
I cannot get either TopazFiles2XML.pyw or TopazFiles2HTML.pyw to open. (even if placing them within the Python directory) All my other .pyw scripts have always run flawlessly. Am I missing something? Do I need to switch to command line scripts?
Feeling rather stupid here…
February 23, 2010 at 2:06 am
Hi Patricia,
So you can double-click other *.pyw files and they work but if you double-click the TopazFiles2XML.pyw (or any of them) they do not work. Is that correct?
If so, then it could be one of 2 things:
1. you have moved the Topaz*.pyw files away from the lib/ library they need.
If you simply unpack tools_v1.3.zip, and open Topaz_Tools, you should be able to double-click them and have them launch. Inside that Topaz_Tools directory you should see the .pyw files and the lib directory. That lib directory has the other python scripts the .pyw files need to work.
2. Some machines have things set up do not allow running things from the root of the C drive or in certain directories. I am not a Windows user so I can not tell you how to test for that. I would simply move the entire Topaz_Tools directory to be in some place you know has worked in the past and try again and see if that helps.
Please let me know if you still can’t get them to work.
February 23, 2010 at 6:45 am
It seems to go wrong after page0326.dat or page0327.dat, so here is a zip archive with page0326.dat – page0333.dat from page directory, page0326.xml – page0332.xml (there wasn’t page0333.xml) from xml directory and dict0000.dat.
http://www.mediafire.com/?jt5dojdny2m
Conversion Log
Command = “python genxml.py”
Book Directory = “C:\tools\Topaz_Tools\lib\New”
Please Wait …
Processing …
metadata0000.dat
other0000.dat
page0000.dat
page0001.dat
…….
(Everything goes well)
…….
page0326.dat
page0327.dat
Traceback (most recent call last):
File “C:\tools\Topaz_Tools\lib\genxml.py”, line 145, in
sys.exit(main(”))
File “C:\tools\Topaz_Tools\lib\genxml.py”, line 122, in main
xmlstr = convert2xml.main(pargv)
File “C:\tools\Topaz_Tools\lib\convert2xml.py”, line 778, in main
xmlpage = pp.process()
File “C:\tools\Topaz_Tools\lib\convert2xml.py”, line 703, in process
snippet = self.injectSnippets(self.snippetList[0])
File “C:\tools\Topaz_Tools\lib\convert2xml.py”, line 559, in injectSnippets
aso, atag = self.injectSnippets(asnip)
File “C:\tools\Topaz_Tools\lib\convert2xml.py”, line 559, in injectSnippets
aso, atag = self.injectSnippets(asnip)
File “C:\tools\Topaz_Tools\lib\convert2xml.py”, line 551, in injectSnippets
name = tag[0]
IndexError: list index out of range
page0328.dat
page0329.dat
page0330.dat
page0331.dat
page0332.dat
page0333.dat
Error: XML conversion Failed
February 23, 2010 at 5:54 pm
Hi,
The bug was actually in page0333.dat, the error message appeared out of order since it is printed before and stdout information.
You actually have a new tag! First one in a while.
“empty_text_region”
They created a whole new tag just to say there is nothing in that region!
I have posted a potential fix. Please grab:
http://pastebin.com/w5Vy0WFn
and rename it to
convert2xml.py
and store it in the Topaz_Tools/lib/ directory (it will replace the older version that is there).
Then try rerunning things.
Please let me know if it fixes the problem or not. If it does, I will include it in the next official release.
Hope this does the trick
February 23, 2010 at 7:01 pm
some_updates, my friend, you are totally made of win. Everything worked perfectly and now I have a clean and good looking html file of my book. Can’t thank you enough.
February 23, 2010 at 2:07 am
Disregard my above post. My problem was file corruption – so I re-downloaded and scripts are working correctly.
Thanks again to all of you involved – awesome work!
February 23, 2010 at 2:56 am
some_updates,
It appears our posts crossed. All works fine with a re-download of the zip file. 🙂 I’m running all scripts on an XP Pro machine, BTW. In regard to your answer # 2 – all my python files reside together in the same directory.
I am one of many who cannot thank you enough for your work (along with all the others) and also for taking time to respond to our posts.
No more ugly Topaz formatting – yahoo!!!
February 24, 2010 at 7:23 pm
New tools release
tools_v1.4.zip
http://www.mediafire.com/?nwzjmmkyzdy
Changes in version 1.4 include:
– bug fix by “Chaos Incarnate” in Topaz Tools for gensvg.py
– add support for new “empty_text_region” tag to Topaz Tools
– eRdr2Pml.py new version 0.14 now supports –make-pmlz option
code contributed by calibreuser
– new gui tool: eReaderPDB2PMLZ.pyw to use the new eReader option
February 25, 2010 at 9:04 am
Can anyone tell me the difference of “python.exe” vs “pythonw.exe”?
I never noticed that I had the “w” file.
Pythonw.exe seems to run when I double click the pyw file?
Is one for GUI and one for command console?
February 25, 2010 at 6:16 pm
Hi John,
The executables python.exe and pythonw.exe are very similar. Both will run gui program and non-gui programs. The real difference is that pythonw.exe will not automatically open a terminal window (console) so it is especially useful for programs with a GUI. You can always run a .pyw gui style tool with python.exe but you will get an extra terminal window you may not need or want.
check out: http://docs.python.org/using/windows.html
for more info.
February 26, 2010 at 8:01 am
Hello John,
v1.4 doesn’t seem to work for a free book sample called Strengths-based Leadership at Amazon. After successfully extracted, they can’t be converted at all.
Conversion Log
Command = “python genhtml.py”
Book Directory = “C:\Users\Supermann\Desktop\tools_v1.4\tools\Topaz_Tools\strengths”
Please Wait …
Traceback (most recent call last):
File “C:\Users\Supermann\Desktop\tools_v1.4\tools\Topaz_Tools\lib\genhtml.py”, line 183, in
sys.exit(main(”))
File “C:\Users\Supermann\Desktop\tools_v1.4\tools\Topaz_Tools\lib\genhtml.py”, line 172, in main
flat_xml = convert2xml.main(pargv)
File “C:\Users\Supermann\Desktop\tools_v1.4\tools\Topaz_Tools\lib\convert2xml.py”, line 780, in main
xmlpage = pp.process()
File “C:\Users\Supermann\Desktop\tools_v1.4\tools\Topaz_Tools\lib\convert2xml.py”, line 692, in process
self.doLoop72(‘number’)
File “C:\Users\Supermann\Desktop\tools_v1.4\tools\Topaz_Tools\lib\convert2xml.py”, line 484, in doLoop72
snippet.append(self.procToken(self.dict.lookup(val)))
File “C:\Users\Supermann\Desktop\tools_v1.4\tools\Topaz_Tools\lib\convert2xml.py”, line 433, in procToken
subtagres.append(self.procToken(self.dict.lookup(val)))
File “C:\Users\Supermann\Desktop\tools_v1.4\tools\Topaz_Tools\lib\convert2xml.py”, line 140, in lookup
print “Error – %d outside of string table limits” % val
TypeError: %d format: a number is required, not NoneType
Processing …
metadata0000.dat
other0000.dat
Using font size: 116
Using page height: 12240
Using page width: 7920
page0000.dat
page0001.dat
page0002.dat
page0003.dat
page0004.dat
page0005.dat
page0006.dat
page0007.dat
page0008.dat
page0009.dat
page0010.dat
page0011.dat
page0012.dat
page0013.dat
page0014.dat
page0015.dat
page0016.dat
page0017.dat
page0018.dat
page0019.dat
page0020.dat
page0021.dat
Error: HTML conversion Failed
February 26, 2010 at 2:43 pm
Hi Superman,
It seems something on page0021.dat is confusing the xml parser. (errors are printed first but the last page number actually printed is the culprit)
Please try running TopazFiles2XML.pyw and see if it generates the same error.
If not, please look in your book directory in xml/ for a file called page0021.xml and post it someplace and provide a link.
If it fails, then please create a zip or rar archive which has the following files in it and provide a link:
page/page0020.dat
page/page0021.dat
page/page0022.dat
dict0000.dat
so that I can recreate the problem on my end, and hopefully get you a version that works.
February 26, 2010 at 4:06 pm
Hi Supermann,
This could also be the result that this is a “sample” and the data in the final page itself might be truncated. If you have the full book, would you please try converting that and let us know if it has the same error or not.
February 26, 2010 at 5:39 pm
Thank you so much for the response. TopazFiles2XML.pyw has the same result. But after you said it’s sth about page 21 and i read somewhere here that as long as there is a blank page at the end, it would always give problems. So I just deleted that page 21 and TopazFiles2html then WORKS!!!
Hooray, thank you so much for this great tool!!!!
February 27, 2010 at 1:13 am
Hi Superman,
Actually, I went and looked at the sample book and it actually introduces a whole new type of page*.dat file called a ‘marker’ and it is the first thing I have seen with external links, and spaces in style names. This last page uses images with external links to allow you to purchase the book (Buy now) or get more details about the book directly from Amazon. They add this type of page to the end of the samples only.
So the next release of the tools will have support for:
– these new type of marker page*.dat files
– fixes for spaces in style class names
– fixes to handle external links
– fixes to handle inline images with links
– and better guessing for new/unknown region types
So a very useful bug report all in all! Thanks!
February 27, 2010 at 7:18 pm
I’m using mobirdedrm and have been having mostly good success, except that every few pages I get a corruption that presents itself as a couple garbage characters and then a repeat of some prior characters. For example:
as anything but friendly.
becomes
anything but fr saything iendly.
Perhaps I am using an outdated version of mobidedrm (hard to tell in this very long thread. 🙂 ).
I would be happy to send a sample of the source azw file and the PID if you think there is still a lingering issue with the stripper.
Piling on the thanks for all the work here. It’s going to be very nice to know that my Kindle investment is not going to go poof when (if) I switch readers in the future.
February 27, 2010 at 7:46 pm
Latest MobideDRM is 0.12, I think – it’s in the tools_v1.4.zip collection. Are you sure the corruption is being introduced by the stripping, or is it just a poorly-formatted file?
February 27, 2010 at 8:31 pm
As Pardoz says, get MobiDeDRM 0.12. This has many advantages over earlier versions, including bug fixes that should eliminate the corrupt characters (which are caused by earlier versions not understanding about various extra bits of data that can appear at the end of text records).
http://apprenticealf.wordpress.com/ is a good place to check for a summary of the latest tools available, along with download links.
February 28, 2010 at 3:56 am
I was using and older version (0.09), but the file created with 0.14 produced the same result (diff binary equivalent). The book was published in the last few months and displays correctly on the Kindle. I can’t tell for sure, but it looks like there is a periodicity to the corruption. In other words, it looks like after a certain number of correctly decrypted characters I get a burp, then the same number of good characters, then a burp, and so on. Not all the time, of course, because that would make it easy.
How can I tell if the source file is corrupt? I simply pull the files off of the Kindle via a USB cable, and since it works on the Kindle, I’m assuming the file made it over the wire okay.
Is there a utility that will simply stream out the unencrypted mobi data in its unpacked form? In other words, whatever Stanza or the like is using to read the mobi file. I should be able to use that to get an accurate byte count between hiccups. Or do you have other suggestions on how to proceed?
February 28, 2010 at 7:45 pm
Hi Tank,
You wrote:
“I was using and older version (0.09), but the file created with 0.14 produced the same result (diff binary equivalent).”
Was this a typo? Did you mean to write 0.12 as the version number for mobidedrm.py instead of 0.14? The reason I ask is that I have seen a number of different mobidedrm.py scripts with different patches by different people, but the one that is the most advanced when it comes to extra data flags is the 0.12 version that Paul mentioned.
Please double check that the version you have (0.14?) has the following it its change log:
# Changelog
# 0.01 – Initial version
# 0.02 – Huffdic compressed books were not properly decrypted
# 0.03 – Wasn’t checking MOBI header length
# 0.04 – Wasn’t sanity checking size of data record
# 0.05 – It seems that the extra data flags take two bytes not four
# 0.06 – And that low bit does mean something after all 🙂
# 0.07 – The extra data flags aren’t present in MOBI header < 0xE8 in size
# 0.08 – …and also not in Mobi header version < 6
# 0.09 – …but they are there with Mobi header version 6, header size 0xE4!
# 0.10 – Outputs unencrypted files as-is, so that when run as a Calibre
# import filter it works when importing unencrypted files.
# Also now handles encrypted files that don't need a specific PID.
# 0.11 – use autoflushed stdout and proper return values
# 0.12 – Fix for problems with metadata import as Calibre plugin, report errors
Just to be safe.
February 28, 2010 at 7:09 am
It sounds like you’ve found a file where the rules used by MObiDeDRM to guess whether there are trailing bytes fail.
There’s MobiUnpack that will extract the raw text from an unencrypted Mobipocket file. http://www.mobileread.com/forums/showthread.php?p=774836#post774836
Which book is it? I’d be interested in getting a copy to check out the problem.
February 28, 2010 at 8:49 am
The book is “Kitty Raises Hell” by Carrie Vaughn, available in paperback in March 2009.
The Kindle file name is Kitty Raises Hell-asin_B001RTC0MY-type_EBOK-v_0.azw
If you tell me how, I will LOAN you my copy of the book.
February 28, 2010 at 10:08 am
Thanks – I’d hoped to be able to test using the sample, but I’d forgotten that the samples aren’t DRMed. Could you email me (see website) the book file and PID? Then I can try to work out what’s going wrong in the decoding, and whether it is the trailing bytes or not.
And then, hopefully, it’ll be possible to make a new version of MobiDeDRM that works on this file too, without breaking support for other files.
February 28, 2010 at 9:14 pm
My bad. I used v0.12 of mobidedrm but had 14 on the brain because of tools_1_4.
March 2, 2010 at 1:53 pm
Hi,
I’ve been using the scripts from some_updates and I’m able to succesfully generate the separate Xhtml and SVG pages.
However, when trying to generate a HTML file I get the “list index out of range” error.
I have zipped the decrypted book and uploaded it, maybe somebody is in the mood to fix it.
Ultimately I’d like to make a PDF of this book, since Its out of print it.
Thank you all for the hard work!
http://rapidshare.com/files/357867810/E-book.zip.html
March 2, 2010 at 5:22 pm
Hi Felix,
It seems they are using newer software to generate your book, and this new software has new version info stored in the other0000.dat file.
These new tags include:
version.bad_text
version.glyph_mismatches
version.margins
version.paragraph_continuation
version.staggered_lines
version.toc
I will release a new set of tools (v1.5) that will handle this case as well as fix a number of other issues sometime this afternoon, after I test the changes.
Also, if you want to make an image based PDF you may be better off using the command line version of gensvg.py and passing an option that will make pure svg images of each page that you can more easily convert into a pure image based pdf.
March 2, 2010 at 6:15 pm
Hi Felix,
Grab tools_v1.5.zip from here.
http://www.mediafire.com/?njoxq4lnzqn
That should do the trick. I looked at the generated html and it looks very very good on my machine except for some headings that simply ocr’d into garbage characters. So the new topaz code may help improve things.
March 2, 2010 at 6:12 pm
A new version of tools (v1.5) is available.
tools_v1.5.zip
http://www.mediafire.com/?njoxq4lnzqn
Changes include:
– updating Topaz_Tools to version 2.6
– fix for many additional version tags
– fixes to generate better links
– fixes to handle external links
– now handles new “marker” page .dat files
– improved special region handling
– properly handle class names with spaces
– handle default alignment for synthetic regions
– now includes ApprenticeAlf’s high ascii pml cleanup from his v0.15 into erdr2pml.py
March 2, 2010 at 8:43 pm
Hey some_updates,
Thank you for your effort, greatly appreciated. Im trying to make a HTML file as we speak.
One question: Can I use an older version of the script that I still have that is command line to force SVG images. I remember trying that before and ended up with 800 svg figures that I could only open with Inkscape.
What would be the best route for generating a PDF through the svgs?
THANK YOU AGAIN! wonderful work!
March 2, 2010 at 9:39 pm
Hi Felix,
I have never made a pdf but ClarkNova has and if you follow this link and start reading around post 349, there is a discussion about how to make a pdf from the .svg files and how to streamline its size.
Check out this link:
http://www.mobileread.com/forums/showthread.php?t=63413&page=24&highlight=clarknova+pdf+svg
Also, if you just grabbed tools_v1.5.zip simply look in tools/Topaz_Tools/lib/ and you will see gensvg.py . If you open up a terminal and go there, you should be able to run the command line as follows:
gensvg.py -r PATH_TO_YOUR_BOOK_DIRECTORY
All the *.pyw files are simply front-ends. They still use the same code (gensvg.py, genhtml.py, etc) but now these are hidden so as not to confuse people in the lib subdirectory.
Have fun!
March 4, 2010 at 4:26 pm
To some_updates -and others still reading this-,
I have managed to produce AWESOME LOOKING PDF’S from the generated HTML files with the Topaz tools.
The downside is, this method is for LaTeX typesetting system -in my case Protext-. Dont use it if youre a n00b.
This is how it works:
-I downloaded a nice little JAVA app that converts HTML to Latex document. Its called HTML2latex
This worked reasonably well, and with some minor tweaking I managed to produce better looking PDF’s than the original hardcopy book -really-.
Those unfamiliar with latex: you type your documents like you program a website. LaTeX compiles to PDF, hence my choice for this. It is an often used program in science, due to its easy handling of references and mathscript.
Anyway, those unfamiliar with it I would highly discourage installing it, instead mail your stuff to me and I will make you a PDF.
I would only recommend this for folks who want to print their documents.
If you have text only documents it is really, really easy and for me just 5 clicks.
I will upload the resulted PDF tomorrow or this weekend, so the users here can compare the result.
I will likely put the book up as a torrent aswell since it is out of print and as a FU to Amazon for encrypting the stuff and giving me a lot of work.
THANK YOU FOR READING!
March 4, 2010 at 4:55 pm
> will likely put the book up as a torrent aswell since it is out of print
> and as a FU to Amazon for encrypting the stuff and giving me a lot of work.
Please remember, that by doing so you are hurting the author(s) of the book by taking money out of their pockets – something they do not deserve IMHO.
March 4, 2010 at 5:32 pm
Not cool Felix. Not cool at all.
March 4, 2010 at 6:26 pm
Yeah, what has the author done to you, Felix?
If you want to hurt Amazon, buy a Nook.
March 4, 2010 at 7:02 pm
Felix — I have been really proud to be a member of this blog for some time now. I hate DRM. I think it is awful and I do not understand why, for example, Random House and Penguin can publish a white paper saying that DRM actually leads to piracy and does not protect ebooks and yet still use DRM. One of the things I have enjoyed about this community is to watch people — who program better than I can ever dream I will — help others to defeat this horrible problem so that, for example, when I buy a new ebook reader next week I will be able to transfer all my ebooks. Because I have purchased over 100 ebooks from Amazon, from BoB and other sites and would be sh_t out of luck without Dark Reverser. OK, that makes me happy. What makes me proud is that this site has always been about defeating DRM, NOT defeating authors or publishers or sellers out of what they JUSTLY deserve. People who research these areas just to make a point will be liable to take your comment out of context to prove that “These people are all lousy pirates!” Well, we’re not. We should make certain that the point is always to be against DRM here and DRM only. Just IMHO.
March 4, 2010 at 9:08 pm
Everybody,
I agree and I take back what I said. You guys are right.
Anyhow, anyone interested into generating a PDF can email me the HTML file and I can see what I can do:
felix AT arthecom DOT nl
Apologies for upsetting everybody!
March 4, 2010 at 10:15 pm
Better yet, just post the instructions on how to do it. There’s some pretty sharp people here. Volunteering to do it for them sounds nice and all, but you could be just trolling for free ebooks for all we know.
March 4, 2010 at 9:10 pm
Oh one more thing, I can’t buy the book on paper cause its out of print and that is the very reason I had to buy the Kindle edition: I even contacted the publisher 😉
March 4, 2010 at 9:40 pm
I think even purchased books from amazon are still encrypted. I tried the latest v1.5 on this book called “Are you a stock or a bond” and got the following errors:
Conversion Log
Command = “python cmbtc_dump.py”
Topaz Path Path = “E:\Office Application\Amazon Topaz Conversion Tool v1.5\Topaz_Tools\stock or bond\B001FA0OGE_EBOK.prc”
Output Directory = “E:\Office Application\Amazon Topaz Conversion Tool v1.5\Topaz_Tools”
First 8 chars of PID = “”
Please Wait …
Traceback (most recent call last):
File “E:\Office Application\Amazon Topaz Conversion Tool v1.5\Topaz_Tools\lib\cmbtc_dump.py”, line 880, in
sys.exit(main())
File “E:\Office Application\Amazon Topaz Conversion Tool v1.5\Topaz_Tools\lib\cmbtc_dump.py”, line 819, in main
parseTopazHeader()
File “E:\Office Application\Amazon Topaz Conversion Tool v1.5\Topaz_Tools\lib\cmbtc_dump.py”, line 378, in parseTopazHeader
raise CMBDTCFatal(“Parse Error : Invalid Header, not a Topaz file”)
__main__.CMBDTCFatal: Parse Error : Invalid Header, not a Topaz file
Device PID: I6IBXUYJ
Error: File Extraction Failed
March 4, 2010 at 10:06 pm
Well, the reason it’s giving you an error when you try to decrypt it is that the book appears to be a Mobipocket book, not a Topaz book.
March 4, 2010 at 10:09 pm
Hi Superman,
The error message you received indicates that you have chosen the wrong software to try and remove the DRM from this book.
You are trying to use tools designed for Topaz books when the book in question is not in Topaz format.
__main__.CMBDTCFatal: Parse Error : Invalid Header, not a Topaz file
Your book is probably a normal Kindle azw style book which is basically a Mobi book.
You should try the MobiDeDRM.pyw program if you have a standalone Kindle and know the full PID (it can be found in Kindle_Mobi_Tools directory.
If you do not and are instead using Kindle For PC you need to use unswindle.pyw and copy the mobidedrm.py file (which can be found in Kindle_Mobi_Tools\lib\) to be located beside the unswindle.pyw program.
Another alternative is to use SKINDLE.
March 5, 2010 at 12:03 am
Yes, I used Kindle-for-PC. wow. This is pretty amazing. I followed exactly what the mighty some_updates told me. At first. I thought it did not work, because unswindle would open the kindle-for-pc program and then when I exit the program, it asked “select unecrypted mobipocket book to produce” when I was expecting a window of “choose the location where you want to save the unecrypted book.” So I had to manually copy the file out of the default location and put it somewhere else to be decrypted.
After the decryption took place, the file was still there with file size unchanged seemingly. I was entiredly confused at this point. I thought nothing happened. but then tried to use Calibre to open the book. It’s now openable in Calibre. Previously, it’s unopenable saying “DRM-protected.” Of course, then Calibre can’t really do a successful conversion, but I managed to just print to PDF which works successfully.
Again. Salute to the mighty some_updates. But I am curious how to use SKINDLE when windows7 said it just can’t open those C or H file. Thanks so much.
March 5, 2010 at 1:14 pm
supermann,
When unswindle opens K4PC, open the book that you want to remove DRM just like you were going to read it in K4PC. Then you can close K4PC and it will ask where to save it and file name.
When unswindle opens K4PC and you close it before opening a book, I am sure that is why it didn’t know which one to work on.
March 5, 2010 at 3:39 pm
The version of SKINDLE in tools v1.5 does not seem to have the proper skindle.exe program in it for some reason.
I will fix that in the next release of tools. In the meantime, if you go to ApprenticeAlf’s Blog, he has a direct link to skindle version 6, I think.
March 4, 2010 at 11:23 pm
DiapDealer, I am not trolling and I have uploaded the PDF I made in the link below so you can check it out -I think theres 10 downloads max-
Check it out and let me know if you like it. There are no easy and straightforward instructions for LaTeX, it is just relatively hard to use.
http://rapidshare.com/files/359038359/boekv2.pdf.html
heres the pdf generated from the HTML files. Only flaw are sometimes misalligned figure captions.
Last, I do not care about free ebooks, I have only 1 purchased and its a book on programming techniques, I have no interest in any book whatsoever. If I want I have access to thousands of books in my university library anyways -except this one of course :p –
March 5, 2010 at 3:18 am
Hey all,
Thanks so much for all you have done – I am an avid computer user with a small amount of programming knowledge (from my college courses many moons ago!). I have been able to use many of the python scripts to decrypt my purchased epubs for my Sony Reader (what an awesome help that has been!). I recently purchased an Amazon Kindle book to test the abilty to convert the Kindle book to something I could put on my Sony Reader (since the book is not available on the Sony store).
I am getting the following message when running the TopazExtract_Kindle4PC:
File “lib\cmbtc_dump.py”, line 774
except Exception as message:
^
SyntaxError: invalid syntax
Error: File Extraction Failed
I have searched this site and others and not found mention of it. It is extremely possible that I am missing a step. Any help that could be provided would be wonderful – your posts have been invaluable to me!
Thanks!!
Cheron
March 5, 2010 at 1:51 pm
Hi Cheron,
That error message means the program could not find the KindleInfo() file. This file only exists on machines that have Kindle For PC (as opposed to a standalone kindle or an iphone/ipod).
If you do not have Kindle for PC and are instead using a standalone Kindle or an ipod/iphone then you have run an incorrect program. For these devices you need to run TopazExtract_Kindle_iPhone.pyw which works for the Kindle and ipods/iphones.
If are using Kindle for PC (the program that allows you to read books on your PC without owning an ipod or a Kindle) please double check that you have bought and downloaded the book and can read it in that program.
March 6, 2010 at 12:16 am
some_updates,
Thanks for the info. I have purchased and downloaded the book and can read it in Kindle 4 PC. I have also uninstalled Kindle 4 PC, deleted the Kindle info file and re-installed and re-downloaded the book. I still get the same error. My Kindle 4PC software is loaded in the following path:
C:\Users\Cheron\AppData\Local\Amazon\Kindle For PC\{AMAwzsaPaaZAzmZzZQzgZCAkZ3AjA_AY}
Could this be causing the problem? Is the script looking for the file to be located in a “program file” directory?
Any help would be greatly appreciated! Thanks!!
March 6, 2010 at 2:29 am
Hi,
If you look in the script lib\cmbtc_dump.py it looks exactly for that funny path.
I looked again and the funny thing is that the error message is actually a “syntax error” and not an exception.
It is almost as if the cmbtc_dump.py program is corrupted (and whitespace is important in a python program).
Please try getting a fresh copy of tools_v1.5.zip and try again with that version. I will do the same from my end to make sure a bug or mistake has not crept into the code.
Please let me know if a fresh version of the tools helps or if you get the same error.
March 7, 2010 at 5:03 pm
Hi Cheron
I think you are right and the software can not find your kindel.info file.
Unfortunately, I am neither a windows user or the author of cmbtc_dump.py (cmbtc is!).
From what I can tell from looking at the code, it queries your windows registry entry
“Software\\Microsoft\\Windows\\Current Version\\Explorer\\Shell Folders\\
for for the value of ‘Local AppData’
On my machine this returns:
C:\Documents and DSettings\Administrator\Local Settings\Application Data\
It then appends to this path the string:
Amazon\Kindle For PC\{AMA..a bunch of chars here.}\kindle.info
In your case the path is:
C:\Users\Cheron\AppData\Local\Amazon\Kindle For PC\{AMAwzsaPaaZAzmZzZQzgZCAkZ3AjA_AY}\kindle.info
The question then becomes what does the query of the registry for ‘Local AppData’ return for you?
Does it return
C:\Users\Cheron\AppData
or
C:\Users\Cheron\AppData\Local
The best way to find out where it is looking is to edit the lib\cmbtc_dump.py script and add a print statement right after it builds that path.
Eid this file with any text editor (be careful to keep whitespace as it is in that file) and change this blob of code:
def openKindleInfo():
regkey = winreg.OpenKey(winreg.HKEY_CURRENT_USER, “Software\\Microsoft\\Windows\\CurrentVersion\\Explorer\\Shell Folders\\”)
path = winreg.QueryValueEx(regkey, ‘Local AppData’)[0]
return open(path+’\\Amazon\\Kindle For PC\\{AMAwzsaPaaZAzmZzZQzgZCAkZ3AjA_AY}\\kindle.info’,’r’)
to look like this
def openKindleInfo():
regkey = winreg.OpenKey(winreg.HKEY_CURRENT_USER, “Software\\Microsoft\\Windows\\CurrentVersion\\Explorer\\Shell Folders\\”)
path = winreg.QueryValueEx(regkey, ‘Local AppData’)[0]
print path
return open(path+’\\Amazon\\Kindle For PC\\{AMAwzsaPaaZAzmZzZQzgZCAkZ3AjA_AY}\\kindle.info’,’r’)
And then run the TopazExtract_Kindle4PC.pyw program and look in the Comversion Log window to see where it thinks your ‘Local AppData’ is stored.
If need be you can copy the kindle.info file to where-ever this is looking and everything should work for the dump.
Once dumped, none of the other programs are windows dependent and all should work fine.
Perhaps someone here who knows more about windows registries and things can then propose a better fix for the future.
Sorry I can’t be more help, but TopazExtract*.pyw runs just fine on my Windows XP box running under Parallels Virtual Machine, so I can’t actually recreate your error to help fix it.
March 5, 2010 at 10:00 am
Any chance of someone putting together a Calibre plugin for the scripts that handle Topaz files?
March 5, 2010 at 1:54 pm
Hi Gwen,
Unfortunately not. The generated html can have lots of errors in it if the ocr that was done on it is not very good. So the book.html file really should be cleaned up in other programs before being imported into Calibre for conversion.
I typically load the spellcheck module in Firefox and open and spellcheck and fix ocr mistakes there. Then I load up that fied file in Sigil (the epub editor) so that I can add in the proper italics, fix other mistakes, etc. Then and only then do I try to import it into calibre.
March 6, 2010 at 9:09 pm
Some_updates:
I grabbed a new copy of the tools (v1.5 from the March 3rd post above), unzipped/extracted the files – not changing any directories or paths at all, and I still get the same error.
Thanks!!
Cheron
March 7, 2010 at 6:28 pm
Ok guys. Have been trying to convert a mobipocket book using mobidedrm latest version but keeo getting syntax error and script does not run
Does the book have to be in a special folder?? Have python 2.6.4 installed and nothing works! I don’t have the pid either.
can someone help me convert this book?
March 7, 2010 at 7:38 pm
Amber, without the PID, the eBook cannot have the DRM stripped.
March 8, 2010 at 12:43 am
Amber,
Did you buy the book, and do you know how to get the PID?
You need to use the serial number of your Kindle to obtain the PID.
March 8, 2010 at 8:18 pm
I’ve posted a cleaned up and reformatted version of MobiDeDrm v0.13 here:
http://mobi.pastebin.com/pmrvHgwP
The changes are mostly cosmetic, and clean up some long-running issues with formatting, trailing whitespace, and *nix-unfriendly line endings. The code should continue to work fine in Windows, too.
The one change that may *possibly* impact someone is that I moved the code for unbuffered output to the __main__ block at the bottom. If it causes you grief (say, when using Calibre) then just move the line into the global context after the Unbuffered class is declared.
March 8, 2010 at 8:37 pm
Hi Omega Pythonista
The unbuffered stuff was only added to allow the code to be run from a gui using a spawned (sub) process and pipes.
It should not be needed for Calibre plugin operation at all but will not hurt anything either.
So it should be okay where you moved it as long as it is the first thing done before stdout is used when invoked via main.
It was just easier to make it the very first thing after the global level import of sys to be safe.
March 10, 2010 at 4:44 am
Has anyone worked out how to download ePub books using the URLLink.acsm files directly? It’s a pain having to boot a Windows VM just to run Adobe Digital Editions for the download.
March 10, 2010 at 12:26 pm
Hi “that guy”
Thanks for the patch I will include it in the next release.
The code was never originally meant to handle sample books because Topaz samples are truly “hacked up” when they truncate it to make the sample. I have tried to support them but …
Thanks
March 11, 2010 at 1:56 am
Hi — I bought a book that turned out to be Topaz and this was only the second time I have converted a Topaz file. It was the first time I had access to the new tools. So I used TopazFiles2SVH.pyw and then on the svg files I had to use the old genhtml.py script to get an actual book that I could add to Calibre. Am I not seeing how to avoid two steps? Is there a pyw script that does one-step to an actual non-0RM book file?
March 11, 2010 at 12:47 pm
Hi,
It is actually a 3 step process:
1. TopazExtract_*.pyw is used to remove the encryption and unpack the sections from the archive to make separate files (one for Kindle for PC, and the other for standalone Kindles and iPhones/iPod Touches)
2. TopazFiles2SVG.pyw then processes the glyphs*.dat files and creates paths for each glyph and then processes each of the page*.dat files that tell it how to assemble each page by giving x and y positions for each glyph
3. TopazFiles2HTML.pyw then uses the glyph information generated in the last step to create svg images for tables, dropcaps, and the like, as well as the xml page descriptions and ocrtext to to build an html version of the book.
The only step you can actually skip is:
TopazFiles2XML.pyw which creates human and machine readable descriptions of the xml that describes the page*.dat files and the xml description of the glyphs*.dat files.
That is there to show you exactly what is being processed and it can help track down bugs especially new tags and things.
So unfortunately, the processing does take 3 steps and putting it into one program would not gain you anything, It would still require 3 passes and the output would still take a long time to generate.
March 11, 2010 at 4:29 pm
OK, one thing that confused me is that somehow using TopazFiles2HTML did not generate a book.html file but instead a separate html file for every page in the book. That is why I fell back to the old genhtml.py script to massage the SVG files. Having just one script might still require three passes and take as long but from a user viewpoint (especially a user like this one who is pushing his own individual envelope of computing skills here) it would be much easier to use.
March 11, 2010 at 5:15 pm
Hi Thothamon,
Unless there is a serious bug, Topaz2HTML.pyw only creates 1 book.html file in the end. All Topaz2HTML.pyw does is provide a gui wrapper for genhtml.py, genthml.py does all of the actual work.
If your actually creates and writes many different html files, then that is a bug which we would have to debug. Are you sure you did not run TopazFiles2XML.pyw by mistake. It does generate an xml page description of every page.
Topaz2HTML.pyw invokes genthml.py which does loop through and creates and collects html pieces for each of the pages which it appends together to create the book.html in the end.
Given the amount of editing and cleanup needed afterwards, I don’t think the number of programs to be run has much impact on the overall useability of the conversion from Topaz to HTML. Luckily Topaz books are few and far between and hopefully going away for new books in the future when fully electronic documents will be required by the publisher.
March 11, 2010 at 5:30 pm
I probably did use the wrong program then. Oh, well, out of the 200+ books that I have bought only two were Topaz. Very happy for this site though as I am going to be switching from Kindle to iPad soon and wouldn’t have had that option otherwise.
March 12, 2010 at 3:54 am
Can anyone tell me where I can get the ereader2html09.exe? I had a copy, but somehow managed to corrupt it – shows as 0Kb size now and windows tells me it is not a valid file.
I manged to find a working link to the topaz tools which contained ereader2html.exe. this is working for the most part, but I have a couple of books where the chapter.dat file is empty. Am hoping the 09 version will remedy this.
March 12, 2010 at 12:58 pm
Hi Darren,
The chapter.dat file need not be used at all. If you want a .pdb simply take the pml output and use Dropbook (it does not use or need the chapter.dat file) If you want html, then the chapter.dat file is not used either. The chapter.dat file was normally printed as an FYI only thing.
The most up-to-date conversion is to grap tools v1.5 and then run the gui tool eReaderPDB2PML.pyw to convert the book to pml (from here you can use Dropbook or Calibre).
If you want the html, then run the gui tools PML2HTML.pyw to convert the pml to html.
The only problem is you do need to have python installed.
If you don’t have and want to install python, then your only option is to try and find ereader2html.exe (most recent versions are version 9). I think you can find a link to a pre-compiled rar file someplace (see this list near the end).
March 12, 2010 at 1:12 pm
thanks. I am actually converting my pdb to html and then using calibre to create an epub.
I did not know that about the chapter .dat. Thanks. Now I don’t have to worry about it.
So you are saying it is better to go to pdb to pml? Keeping in mind I will ultimately end up at pdb.
I see the eReaderODB2PML.py and .exe, but not pyw. I to have topaz 1.5. Also, you mention a gui – how do I get that to work.
March 12, 2010 at 1:13 pm
my one line there should read “I will ultimately end up at epub. “not pdb.
March 12, 2010 at 2:36 pm
Hi Darren,
Grab the latest version of the tools (version 1.5) from here:
http://www.mediafire.com/?njoxq4lnzqn
If you have python installed, then you should be able to use the gui tools inside.
To convert from an eReader .pdb file to an .epub (the conversion I do all of the time), I recommend the following:
1. Download tools_v1.5.zip
2. Extract the entire archive (not just one file)
3. open the eReader_Tools directory in tools
4. Double click on eReaderpDB2PML.pyw
This should start up a gui tool to extract the pml and images from the .pdb file
5. Double click on PML2HTML.pyw
That should convert the pml into html (and it will do a better job of keeping the original styles than a direct conversion using calibre at this point instead.
Now you should have a properly converted html file and a set of images. I recommend either of the following:
A. Import the html file directly into Calibre for the conversion to epub (most automatic)
or
B. Use Sigil if you need to edit/change the file in any way and have Sigil it to convert to epub (allows you to change the TOC specify metadata, etc).
Hope this helps
March 12, 2010 at 2:40 pm
Darren,
All of the pyw files in the v1.5 tools package are GUI. You don’t run them using the command line. Just double-click the GUI(pyw file), and fill in the blanks.
Thnaks to some_updates for these easy to use tools.
March 12, 2010 at 2:59 pm
Thanks a lot! I will give this a try when I get home.
One last question (for now). I have already converted a reasonable number of my pdb to html using eReader2html.exe.
Is it worth re-doing those using the 1.5 tools as you explain above?
I do remember on a number of the conversions, I was getting a non-critical error – something about non-standard fonts or such.
Does tools 1.5 resolve this error or is the error nothing to worry about?
March 12, 2010 at 3:33 pm
Hi Darren,
That font message is not part of the ereader to html conversion process.
So I would not worry about it.
As for reconverting older files again, that is completely up to you. It should not be needed.
The newest parts of the conversions are:
1. better handling of footnotes and sidebars
2. better at determining paragraph indentation and converting that to paragraph style.
3. better handling of metadata
4. more technically correct xhtml
My guess is only a very few small number of books would look noticeably different using this method compared to older versions of ereader2html. So it would probably be a lot of work without much difference for the bulk of the books.
March 12, 2010 at 4:01 pm
Alright. Thanks. I will give this a go tonight and decide from there if I want to redo what has already been converted.
March 12, 2010 at 11:11 pm
Well, I just extracted the entire tools 1.5 file. Tried double clicking eReaderpDB2PML.pyw and no gui appears.
I tried extracting both in its own directory and into the tools directory within my python directory. No go.
The .pyw file does have the python icon next to it so I know the association is there. Any other thoughts? Does the .pyw need to be placed somewhere else?
On another note, I did find another copy of eReader2html09.exe does this file address the 4 improvements you noted above that the gui fixes?
Assuming I can’t get the gui to work, am I better off using the 09 exe?
Thanks.
March 13, 2010 at 5:04 am
Hi Darren,
Well the gui tools have been tested and do work on all platforms (Windows, Mac, and Linux).
So there can only be 2 reasons why it is not working for you.
1. You did not properly extract all of the files inside the tools_v1.5zip or somehow moved one of the files after the extraction.
or
2. Your python installation is not correct
So let’s walk throug checking things …
If I download those tools to Win XP and double-click on the tools_v1.5.zip, it opens the zip archive and shows it as the tools directory but actually does not extract anything. You literally have to look at the left side of the window and under “Folder Tasks” you will see the “Extract all files”. You need to hit that to properly unzip all of the files in the zip archive. Otherwise you end up trying to run things form inside of the zip which does not work.
Once you hit that, it runs the “Extraction Wizard” which properly extracts all of the files into a tools_v1.5 directory.
Once extracted, do not move anything anyplace unless you move the entire tools_v1.5 directory. The programs need to find files inside their local lib subdirectory to work.
Then open the tools_1.5 directory that you just extracted. Then open the eReader_Tools directory, and then finally double click on the eReaderPDB2PML.pyw file.
If that does not launch the program, then something is wrong with your python installation.
To test your python installation, open a terminal (run cmd.exe) and then simply type the following on the command line and hit rreturn
python
You should see something that tells you the python version number 2.6.X. To exit this python program simply type quit()
Once you have python installed and properly extract all of the files in the archive, the gui programs do work simply by double-clicking.
Please try one more time and let me know if you can get it working.
March 13, 2010 at 6:01 am
Wow. Thanks for the detailed instructions. Turns out I had python 3.1 installed. Guess it didn’t like that . Installed Python 2.6 and all is good.
Question about the images that get extracted:
The cover gets extracted and put into an images folder. Upon loading the html book into caliber (Assuming I do not also import the pml file) and then converting to epub, Caliber inserts its own default cover on the first page and then the actual book cover (from the images directory shows up on the second page).
If I have Calibre fetch the cover before doing the conversion, then that cover appears on the first page of the epub book and the cover from the images directory appears on the second page. I assume this is expected behavior?
What is your work flow for adding a converted pdb to Calibre?
March 13, 2010 at 2:07 pm
Hi Darren,
In calibre you can simply set the cover image to whichever you want. It is one of the conversion options. So set that and all should be fine. Many ebooks do have both a cover image and a bokt title page that are both graphics. I am never sure what is more correct.
Personally, I don’t use Calibre much. If I do use calibre to make an epub, I then open the epub in Sigil (latest version is 0.20b2) and fix up anything I don’t really like (remove a created cover image), etc.
March 13, 2010 at 3:04 pm
Alright. Thanks for all your help, I really appreciate it. And thanks for the great tools.
March 13, 2010 at 5:47 pm
I am actually finding that for my work flow the command line is easier for me to use. Having to only go through the one process and only specify one input an output directory is easier for me.
So, that said, am I better off using eReader2hmtl09.exe vs eReader2html.exe?
If so, does 09.exe offer the same ‘quality’ output as the gui?
If not, is there an ‘all in one’ command line version that works as well as the gui?
Thanks.
March 13, 2010 at 11:36 pm
Actually, I have pretty good process down using the gui. thanks again for the tools.
March 20, 2010 at 3:03 pm
Hi,
I just bought and have removed the drm on 2 .pdb books. Both by the same author. Using the gui, I receive similar errors when converting the pml to html. The error is as follows:
Processing…
Converting pml to raw html
Error: Improperly Nested Link Tags: expected q found P
after processing #ch58″>Cover
elapsed time: 0.62 seconds
output is in file PelicanBriefNovel_106542.html
Finished Processing
File successfully converted
When I try to convert the outputted html to epub (Using Calibre) the font size is quite large, but more importantly, the entire book, looks like it is one big link – all the text is blue and underlined. You can see what I mean from viewing the linked html below.
I have also tried using the old command line ereader2html.exe. No errors are raised and the book converts to epub without the large font and underlined sentences, but there are no spaces between the paragraphs and it is very difficult to read.
I’ve tried modifying the html based on the error message, but I’m not a html guy and have not had any luck. Was hoping some one could take a look and see if they can tell what the issue is – and if it is a simple fix.
This is the html from the gui
http://dl.dropbox.com/u/3007540/book/PelicanBriefNovel_106542.html
March 20, 2010 at 3:07 pm
If it matters, the drm pdb looks fine when viewing in Stanza on the iphone.
March 20, 2010 at 11:41 pm
Hi Darren,
In the dedication change:
<h1 title=”Dedication”></h1><p><a href=”None”> <span id=”ch59″>
to this
<h1 title=”Dedication”></h1><p><a href=”None”> </a> <span id=”ch59″>
and you should be good to go.
March 21, 2010 at 2:10 am
Hi,
Thanks for taking a look.
But that didn’t seem to do it. Most of chapter 1 looks okay (it did before as well), but the last part of chapter 1 and a good chunk of everything forward is all underlined.
Also, the spacing between paragraphs isn’t there. I’m not quite sure what I’m looking at, but the html around the actual story seems a lot more complex than with my other books. My other books just have mostly tags, this book has all kinds of stuff going on.
I’m guessing eReaderPDB2PML.pyw is either having trouble with this pdb or the pdb itself is ‘weird’….
Not sure that there is an easy fix, but I’d be interested in hearing any comments.
March 21, 2010 at 2:11 am
hmmm should be ‘my other books have mostly BR tags’
March 21, 2010 at 3:27 am
Hi Darren,
The right way to fix this is to fix the xpml2html.py code that is run by the gui to handle the poorly formatted pml that seems to be in your book.
So can you please post the pml file (I don’t need the whole file just from the beginning of the book through the first couple of chapters) someplace for me to grab.
I would like to see just what kind of pml markup tags that are being used that confuses the conversion so much and hopefully make the code more robust to these type of errors:
> Converting pml to raw html
> Error: Improperly Nested Link Tags: expected q found P
> after processing #ch58″>Cover
I think the above means that an internal link actually includes a page break right in the middle of it – but I would need to see the exact pml to be sure.
March 21, 2010 at 3:22 pm
Here are the 2 books (pml) that were giving me the same sort of error.
http://dl.dropbox.com/u/3007540/books.rar
Let me know when you grab them and I will disable the link.
March 21, 2010 at 4:18 pm
Hi Larry
I have them. I will be posting a new version of xpml2xhtml.py for you to test hopefully by this evening.
Thanks,
March 21, 2010 at 4:36 pm
Hi Darren,
I have them (sorry wrote back to the wrong person before) so you can remove the rar file.
I also have found the problem and fixed it (at least I hope so)
The difference is that your book uses tabs inside of links which drove the converter crazy. Most books simply use the tab first and then start the link if they want an indented link.
In fact, both of these books have tabs (\T markers) flying around all over the place. Unfortunately html does not support tabs so everything is not perfect but it is the best I can do unless I simply ignore them completely.
Anyway, grab the file xpml2xhtml.py from here:
http://pastebin.com/ZPDX7DFz
rename it if need be. It should be exactly named:
xpml2xhtml.py
(in case it gets renamed during download from pastebin) and copy it to
eReader_Tools\lib\
where it should overwrite the file of the same name.
Then run the Pml2HTML.pyw gui and select your pml file and give it a try.
Please let me know if this works, and if so, I will include it in the next release.
March 21, 2010 at 4:46 pm
Hi Darren,
One other thing. There are so many tab stops not at the beginning of lines – after the first character of a new chapter, that leave funny spaces. If you need it I can add an option switch to xmpl2xhtml.py that can be used to ignore either all tabs or any tabs that do not begin at the start of line.
Either of those options would make these particular books look better, I think. Or, as an alternative, simply edit the resulting html using search and replace (or Sigil) to fix things up.
March 21, 2010 at 5:01 pm
Hi,
Thanks a lot. Things look great now. The table of contents has double or triple spaces between each chapter, but that is no big deal.
The actual story looks good now. there are still no spaces between paragraphs, but that is easily fixed in Calibre – before your changes, Calibre wouldn’t add spaces, now it adds them no problem.
Both books look fine. These are brand new releases, not sure why the formatting was so different, maybe a sign of things to come??
thanks again.
March 21, 2010 at 5:04 pm
Regarding the tab stops, yes I noticed how there is one after the first character at the start of each chapter.
If there is a way to remove them, that would be great. Like I said, it only looks to be an issues after the first character at the start of each chapter, no tab issues elsewhere, as far as I can tell.
March 12, 2010 at 6:49 pm
Hi.
The pastebins for the eReader and Mobi scripts are coming up empty. Am i doing something wrong, or are they deleted?
March 12, 2010 at 7:05 pm
Oops. I found them. Never mind.
Thanks
March 13, 2010 at 5:45 pm
[…] really not too hard. All you need is the book you want to read on your iPad, unswindle, mobiledrm, and Stanza, which is a program that reads ePub files. It runs on the iPhone and iPod […]
March 14, 2010 at 12:47 am
[…] really not too hard. All you need is the book you want to read on your iPad, unswindle, mobiledrm, and Stanza, which is a program that reads ePub files. […]
March 16, 2010 at 9:51 pm
I used latest version of mobidedrm (014), it didn’t work with my prc file though
I have installed python 2.6
I started the script but even after 15 (!) minutes nothin happened 😦
on the command line I get the following message:
D:\Downloads\MobiPocket-Ebooks>python MobiDeDRM014.py Duden_Deutsches_UniEE1290F
5.prc test.prc XXXXXXXXXX
MobiDeDrm v0.14. Copyright 2008-2010 The Dark Reverser.
MOBI header length = 228
MOBI header version = 7
Extra Data Flags = 7
Decrypting. Please wait…
Traceback (most recent call last):
File “MobiDeDRM014.py”, line 295, in
strippedFile = DrmStripper(data_file, pid)
File “MobiDeDRM014.py”, line 235, in __init__
self.patchSection(i, PC1(found_key, data[0:len(data) – extra_size]))
File “MobiDeDRM014.py”, line 149, in patchSection
self.patch(off + in_off, new)
File “MobiDeDRM014.py”, line 140, in patch
self.data_file = self.data_file[:off] + new + self.data_file[off+len(new):]
KeyboardInterrupt
I would send you my prc file and PID for debugging.
thanks in advance
Rusty
March 16, 2010 at 9:54 pm
I should add that anything beneath the line where it says
Decrypting. Please wait…
was produced after pressing strg-c
March 16, 2010 at 9:55 pm
strg (GER) = ctrl (US) 😉
March 16, 2010 at 10:56 pm
It looks like it was working fine. For very big files (dictionaries, etc) that are tens of megabytes, it can take tens of minutes.
But perhaps the latest script has a problem with your file. Feel free to send me email and I’ll take a look.
But first try letting it run for an hour. That should be long enough for even the biggest files.
March 17, 2010 at 9:08 am
alright, thanks
I’ll give it another try and let it run for an hour
I’ll let you know whether it worked or not…
March 17, 2010 at 2:27 pm
And by the looks of things, it might be worth your while picking up MobiDeDRM 0.15 from http://apprenticealf.wordpress.com/
March 17, 2010 at 6:22 pm
thanks again, patience sometimes pays off *g*
March 19, 2010 at 4:42 am
After a “file successfully decrypted” message using ineptpdf7.2, I cannot open the resulting file in Adobe Reader or ADE. I receive the following message in Adobe Reader: “There was an error opening the document. The root object is missing or invalid.”
I do not know what this means or how to resolve it. Any suggestions?
March 19, 2010 at 3:13 pm
I have downloaded all scripts, python2.6, unswindle, mobidedrm, K4PC and downloaded a book. It came as a .prc book. I first tried modidedrm and it couldn’t find the PID. Then I tried unswindle and it went all the way to entering name and destination but then said: failed to determine PID. Not sure where to go from here. So nice to be able to read books from anywhere. Thanks so much all of you for your work.
lar
March 19, 2010 at 3:39 pm
I’m having trouble with unswindle. I have all the tools and downloaded a bood from Amazon. Came as .prc. I tried mobidedrm but it said it was a wrong format. So I tried unswindle. Worked all the way to entering a name and destination. The result was : failed to determine PID. Not sure what to do next. Any help is appreciated. Thanks for much for all this great work. Life savers for sure.
lar
March 19, 2010 at 4:59 pm
Can you read the book in Kindle for Windows? Are you running the script on the same machine as Kindle for Windows?
If not, then it won’t work.
Also… a .prc from Amazon is probably not encrypted at all. Are you sure it’s an encrypted file?
March 19, 2010 at 6:23 pm
Paul, I can read the book on windows. I redownloaded the book and tried again and got a message that it was a topaz book. So, I tried the topaz tools and got a message that the syntax was bad. Not sure where to go from here. Thanks for the help.
March 19, 2010 at 7:21 pm
Hi Larry,
If the book is a topaz book, then please download the tools_v1.5.zip and extract the **entire folder**.
Then cd into tools_v1.5 (not the zip archive – but the extracted folder)
then cd into Topaz_Tools.
Double click on TopazExtract_Kindle4PC.pyw
If you get an error message, please copy the entire error message from the Conversion Log and post it here so that we can figure out what is up.
If that succeeds:
Then double-click on TopazFiles2SVG.pyw
Once that completes, double click on TopazFiles2HTML.pyw
Please note, unlike MobiDeDRM, Topaz books are not perfect. Some editing and cleaning up in an editor will be required.
March 19, 2010 at 8:44 pm
Couldn’t find a way to copy the whole thing. Thought I could just right click and choose copy but that didn’t work. Anyway, I will try to duplicate it here:
Conversion Log
Command = “python cmbtc_dump.py”
Topaz Path Path=”C\Documents and Settings\Larry\My Documents\My ebooks\tools\tools\Topaz_tools”
Output Directory = C:\(same as above)
First 8 chars of PID=””
Please wait
File “Lib\cmbtc_dump.py”, line7740
except Exception as message:0
^0
Syntax Error: invalid syntax0
Error: File Exctraction Failed
So thats what I see except I used O for little boxes.
Hope this helps. Thanks so much.
March 19, 2010 at 11:57 pm
Hi Larry,
Can you please double check the line number of the exception: I don’t think there are actually 7740 lines.
Second, your output directory should NOT be the same directory as the tools directory. It should be a new separate directory.
Think of this program as unencrypting and unpacking the the different pages of the book into its own set of files.
Also, please make sure you are using Pythin 2.6.X and not the 3.X version.
March 20, 2010 at 4:57 am
You are right, the number is 774 with a little box after the number. I’m not familiar with programing so don’t know what that little box is all about. I did change the output directory but still got the same syntax error. I am using python2.6. Any other ideas. Thanks again.
lar
March 20, 2010 at 1:45 pm
Hi Larry,
I really running out of ideas here. Here are a few things you might try:
1. some virus/protectins programs do not allow anything to execute with the folder “My Documents” (this is a security “feature” not a bug”). So please try unpacking the set of tools someplace that is not inside of the “My Documents” folder and running it again?
2. Could this be what ProTitan said in his response? Are you really sure this is a Topaz book. If you try and open the file with a editor you should see TPZ somewhere near the top if it is truly a Topaz Book.
3. Be sure you do not have 2 different versions of python installed in your path:
To do that, simply run cmd.exe to get a command line prompt and then type:
python
and hit return and see what version is found first. It should be 2.6.something.
Aside from these, I have no ideas why it works for others and myself but not for you.
What version of Windows are you running?
March 20, 2010 at 4:32 pm
Wow, I should give up on this but just too stubborn. I changed the file location away from D&S and ran it from there. I did look the book up on an editor and it is tpz. Since unswindle also said it was a topaz book, I was fairly sure it was. I checked the python and there is only one, 2.6. Since I’m never sure what I’m doing here, would it be correct to make the topaz ebook input file the book in the kindle file? And then the output file some other file that I created? So far, I still get the syntax error on line 774. Thanks so much for trying to help. Who knows what is going on. I will keep trying stuff.
March 20, 2010 at 11:45 pm
Hi Larry,
When you run the GUI you should most definitely select the topaz book file (ie. the *.prc or *.azw1 file that is the book), and make the output directory point to a new directory.
Hope this helps,
March 21, 2010 at 5:03 pm
Paul, when I run topas extract for k4pc, in the place calling for 8 chars of PID, I assume I should leave that blank since that is what I’m trying to find. Is that correct?
March 21, 2010 at 5:38 pm
Hi Larry,
Yes for, for a Topaz book when you are using Kindle4PC, the PID can be blank. It should find all it needs from the kindle.info file.
March 21, 2010 at 6:40 pm
I got it I think. After redownloading and redoing everything and it still didn’t work, I decided to do it all on a different computer. And it worked. However, now I’m not sure what I have. I have run three different .pyw’s and have a bunch of stuff in the file. What do I really have and what do I do next? Thanks again for the help.
March 21, 2010 at 8:45 pm
Hi Larry,
I am glad to hear using another computer worked. I just wish we knew what was different so that next time it can be simpler.
For a Topaz Book there are 3 steps:
1. TopazExteact_Kindle4PC.pyw to unencrypt the book and dump its pages and glyphs into separate files in their own directories inside the book directory
2. TopazFiles2SVG.pyw which reads in all of the glyphs and then processes “pictures” of each page in the directory svg/ inside the book directory. You can read these in a svg enabled web-browser to see picture images of what each page looks like.
3. TopazFiles2HTML.pyw whcih reads some of the glyph information from the last step and uses the optical character information stored in the page*.dat files to try to build an html version of the book called book.html which is inside the book directory.
So use a web-browser to open book.html. It will not be a perfect copy of the book because Amazon’s character recognition inside the files is not perfect nor is there information on italics and the like.
So I use Sigil or some other html editor to fix it to the way I like and then I have an html version of the book. You can use calibre to convert it to epub or whatever.
Hope this helps.
March 22, 2010 at 12:06 am
thankyou, thankyou, thankyou. Everything works and I am reading the book on my sony reader. Thanks so much for your help. I’m going to try to find out the problem with my computer. Wish me luck.
March 22, 2010 at 3:00 am
Hi Larry,
If you ever figure out why none of this worked on the old computer, please let us know here just in case anyone else runs into the same problem.
Thanks,
March 19, 2010 at 8:00 pm
Larry, the problem you are having comes from the way scripts are packaged. Unswindle.pyw needs mobiderm.py to operate. The way it is packaged now Unswindle.pyw is packaged together with Mobiderm.pyw so once when you click Unswindle.pyw, the file will take mobidedrm.pyw as mobidedrm.py. Solution? Copy Unswindle.pyw together with mobidedrm.py (found in lib directory) to separate directory and try running unswindle again.
March 21, 2010 at 12:35 am
Guys,
We just launched our Alpha site, http://firstsale.i2p.to, as a complement and homage to the fine work here and by others to free ebooks from DRM. Its aimed at newbies and those timid to download software from less publicly trusted sources (no slight intended).
Please drop by and take a look. Your constructive comments are welcome.
FS operator
March 22, 2010 at 6:14 am
Thanks for all the information here, I have read the entire list of comments. I have a sony reader, which I love, however the Sony store often doesn’t have the books I want so I bought one from Amazon. I tried unswindle and it converted it, but left off about 10 pages of the beginning and I don’t know how much of the ending, and different parts intermittently. I tried skindle and the program flashes and goes away. Any ideas? I have successfully converted the B&N files.
March 23, 2010 at 2:30 pm
Hi Elizabeth,
Sounds very strange.
What versions of unswindle.pyw and mobidedrm.py and skindle have you tried?
What versions of Python have you installed?
Are you using the same computer and Kindle4PC application tht you purchased the book with?
Can you read the full book under Kindle4PC?
Have you tried redownloading the book just in case it has become corrupted?
skindle-06.tgz should be able to find the kindle.info file on your machine and use that to decode the book. Are you sure you are logging in as the same userid on your machine?
Perhaps someone here with a Windows box can help walk you through the process.
March 24, 2010 at 12:37 am
When it comes to unswindle, please go to the website below for the comprehensive instructions
http://nyquil.org/archives/1235-Converting-Amazon-Kindle-For-PC-Books-Now-Possible.html
If you still have a problem with conversion please let us know.
March 24, 2010 at 5:17 am
Thanks for the response! What ProTitan told me to do worked, but I thought I would answer your questions in case anyone else comes here with the same problem.
What versions of unswindle.pyw and mobidedrm.py and skindle have you tried? ~The versions on I ❤ cabbages site and the one in tools v1.5
What versions of Python have you installed? 2.6
Are you using the same computer and Kindle4PC application tht you purchased the book with? Yes
Can you read the full book under Kindle4PC? Yes
Have you tried redownloading the book just in case it has become corrupted? No, I didn't because it seemed to be perfect on Kindle4PC
skindle-06.tgz should be able to find the kindle.info file on your machine and use that to decode the book. Are you sure you are logging in as the same userid on your machine? Yes, I'm the only one that uses this computer. Because I was curious I ran skindle on 'cmd' and it stated it found the kindle info. I don't know why it wouldn't stay open after I clicked on it
March 24, 2010 at 5:21 am
ProTitan, Thank you! This totally worked.
March 24, 2010 at 8:54 pm
Hi Elizabeth,
What did you change to make unswindle work after reading Pro-Titan’s site, when it did not work previously?
Did you have to change to 32 bit python? I just wondered what changed to make it work.
Thanks,
March 25, 2010 at 3:27 am
Thank you for the vote of confidence some_updates but I have to make something clear. The site that I have listed above is not mine. I direct many people there just because the owner of the site has clear instructions. I must also confess that I had problems converting books with unswindle from your package v. 1.5. Because I’m not a programmer I cannot really tell what is wrong with it. (Maybe different version of the program or maybe it is unwindle and mobidedrm combination in your package? Again I have no clue) But what I know for sure, unswindle problems disappeared for those people who I have directed to the above website. Thanks again for all your hard work.
March 25, 2010 at 4:21 am
Hi again. I started with the unswindle from I ❤ cabbages site. I don't know if I didn't paste it right or what, but that's the one where the first few pages were missing and pages intermittently were gone. On the unswindle from tools 1.5, after I save the name for the mobi book a screen pops up that says "Mobi eBook Encryption Removal" that asks for a 10 digit pid. I don't have the pid so it won't work. I already had python 2.6 because I used it to convert the B&N books and I have 32-bit windows on my computer, my atom processor does not come in a 64-bit extension. The only thing I did was download the unswindle from the other site, unzip it in a new folder, and click it , I didn't change anything on my computer. Thanks again for your help! And I hope my questions and answers can help someone else 🙂
March 25, 2010 at 2:21 pm
Hi ProTitan and Elizabeth,
Okay, I ran a diff (a program to compare two files character by character) to on the unswindle.pyw from unswindle.zip compared to the one in tools_v1.5. They are identical.
So that is not the issue.
Next I compared the mobidedrm.py from the unswindle.zip with the very latest one just released on Apprentice Alf’s blog site and the one from unswindle.zip is very very old (version 0.5) which is known not to work for some files. So unless there is some serious problem with the newer mobidedrm.py (which I don’t believe since there have been many many success stories), we really should be using the very latest version of mobidedrm.py and not an older version like 0.5.
Assuming that the newer version is not the issue, then why do people have trouble with the identical version in tools_v1.5?
The reason is that the mobidedrm.py file is not sitting right beside the unswindle.pyw program in the tools_v1.5 (the MobiDeDRM.pyw file does not count) – and the unswindle.pyw program does not know to look in the lib subdirectory to find it.
So I will create a separate folder in tools that houses both the unswindle.pyw and the latest mobidedrm.py in its own folder to make using it more simple.
March 25, 2010 at 7:37 pm
Thank you for the new update however I have encountered an error when I was converting kindle book with the unswindle in your package.
Error: problem running MobiDeDRM
MobiDeDrm v0.15. Copyright 2008-2010 The Dark Reverser.
MOBI header version = 5, lenght = 232
Extra Data Flags = 1
Decrypting. Please wait ……done.
But there is no converted file anywhere in my computer.
Running:
Windows 7
Python 2.6.5
tools v1.6
I should also mention that I’m getting converted file with tools from above website. Any suggestions?
March 25, 2010 at 7:47 pm
Hi Protitan,
The version of unswindle.pyw is exactly (byte for byte) identical to the version in unswindle.zip on the site you pointed out to us.
So if you are having trouble, then it must be because of the differences from the old MobiDeDRM.py used there (version 5) and the latest version used in tools_v1.6 – MobiDeDRM.py (version 15).
I will boot up my windows box under a virtual machine and try to see what changed with mobidedrm.py that is causing the problems.
March 25, 2010 at 8:19 pm
Hi Protitan,
I was able to recreate your error. It seems there is something about version 0.15 of mobidedrm.py that makes it not work properly with unswindle.
I tried the mobidedrm.py from tools_v1.5/Kindle_Mobi_Tools/lib/ and that worked fine (v0.13).
I next tried mobidedrm.py version 0.14 (downloaded from Apprentice Alf’s site) and that worked fine as well.
But when I upgraded to the latest version (0.15) from Apprentice Alf’s site, then it errored out.
So please grab mobidedrm.py (version 0.14) from Apprentice Alf’s site and use it to replace the one in tools_v1.6 by replacing the file in Kindle_Mobi_Tools\unswindle\ and it will not hurt to replace the one in Kindle_Mobi_Tools\lib\ as well.
Then give unswindle another try and let me know what you find out.
Thanks
March 25, 2010 at 8:53 pm
Hi ProTitan,
Found it!
unswindle looks for the exact string “done\n” to know that mobidedrm finished properly and if not that, it prints and error and removes the file it converted (thinking the mobidedrm had failed).
In all versions up to and including 0.14, mobidedrm.py prints “done\n”.
But somehow that one line in mobidedrm.py version 0.15 has been changed to “done.\n” by the addition of a period!
So that extra period is enough for unswindle.pyw to think that mobidedrm.py did not finish properly and so it prints an error message and removes the properly converted file.
So much for proper grammar!
So the solution is to change mobidedrm.py version 0.15 to remove the period that it prints at the end of done:
change
print “done.”
to
print “done”
to make it consistent with all earlier versions and then it should work.
I will pass this along to Apprentice Alf so that the v0.16 comes out eventually without the extra period.
March 25, 2010 at 9:37 pm
Perfect! After replacing mobidedrm from v.15 to v.14 the unswindle is working perfectly. I will try v.16a latter on today but I’m sure it will work fine. (on the side note: I’m still working on my grammar hahaha) Thank you again for your work. You are my hero.
March 26, 2010 at 5:09 am
Hi some-updates
I have downloaded the mobidedrm v0.16 from apprentice alf’s blog (as per your note above) and I’ve tested with the unswindle. I must say that unswindle with this new mobidederm is working fine for me. I’ve converted about 5 books without any problems. The conversion process was very fast if you compare it to previous versions of the mobededrm. Thank you again for all your work.
March 25, 2010 at 3:44 pm
New version of tools now available:
tools_v1.6.zip
http://www.mediafire.com/?jaqhdgwmzmu
Changes
———-
– update mobidedrm.py to version 0.15 from Apprentice Alf
– bug fixes and improvements to mobiunpack.py by Paul Durrant
– bug fix for Topaz Tools when used on sample books by ‘that-guy’
– bug fix for eReader_Tools to handle tabs (\T) inside links \q
– update xpml2xhtml.py to use SGF Markers for sigil >= 0.20B3
– repackaging unswindle.pyw into its own folder with latest mobidedrm.py
March 25, 2010 at 4:08 pm
The SKINDLE folder is removed?
March 25, 2010 at 5:02 pm
Hi Supermann,
No … just moved. It is now inside the Kindle_Mobi_Tools folder where I think it fits better since it is specific to Kindle4PC.
March 25, 2010 at 8:38 pm
Just in case:
An amended version of tools – version 1.6a
– downgrades to mobidedrm.py version 0.14 since something in the change from 0.14 to 0.15 seems to make mobidedrm.py stop working with unswindle.pyw (the heartbeat? – or some other change?).
tools_v1.6a.zip
http://www.mediafire.com/?jjyvnumnjzy
March 26, 2010 at 12:48 pm
Hi Victory1,
It works fine for me.
tools_v1.6a.zip
http://www.mediafire.com/?jjyvnumnjzy
March 27, 2010 at 11:09 am
[…] protected with your username and credit card number. If push comes to shove, there's also the "ereader2html" tool. It's written in Python, so you need a python runtime. With it, you can check whether the file's […]
March 27, 2010 at 1:32 pm
Man, thse tool evaporate faster than piss on a hot rock in Texas! But thanks for the last link (v1.6a lives). Help me understand a little-I successfully decrypted a .pdb book that I had purchased from somewhere-the eReaderPDB2PML script ran fine in active python (once i told it to stop opening unitron). The result was a folder with the jpgs and the text. But there was also another file generated named book.pml. what is this file all about?
I also have some books I bought a few years ago (WOW time flies!) But in the meantime, I forgot what cred card I used to buy them. Hence I am stuck. Is there a way to get at these books?
I struggled with ineptkeymac.py and this generates a file which I suppose is a key? I am stuck at this point. ineptepub.pyw doesnt work for me. Or am I barking up the wrong branch?
In any event, thank you for your contributions. I rue the day when we are born into debt and the hospital stamps our babies’ feet onto a license agreement that says we dont own our own bodies.
March 27, 2010 at 3:52 pm
Thank you for your contributions but I need a little help. I did successfully use the ereader script to decrypt a pdb book I had bought. However, I have a couple of books that I have lost the cc info used to purchase them.
In an attempt to get these titles to stanza on my iphone, I need to convert them to epub format but they are encrypted. I have tried to use ineptkeymac.py and this generates a 4kb file which I assume is the key. I then tried to use ineptepub.pyw but was unsuccessful. Any pointers?
I also tried eReaderPDB2PML.pyw and that generates a pml file. What do I do with that?
March 27, 2010 at 4:20 pm
Without the name and CC name you can’t decrypt eReader (.pdb) files. ineptkeymac is to recover the key that is used for encoded ePub files. It won’t help you with pdb at all.
PML is the source file use to create eReader files. I suspect the one you have there is empty, since you don’t have the required info (name & CC no.) to decode the file.
March 28, 2010 at 5:21 pm
Hi,
There are ways to track down old credit card numbers:
1. look in your old receipts drawer for old credit card slips
2. look for old credit card bills (I keep mine with my tax files)
3. buy a credit history record check (or ask for copies of an old one from your bank, or where-ever you last took out a loan). In many states, if a bank did a credit history check on you, they are obligated to provide a copy to you at no cost (ie. you already paid for it!). They often-times show complete credit card history.
4. Finally if you truly know the exact name on your credit card and can’t make any of the above work, you could write a python script to brute force try all of the possible last 8 numbers. This would take some programming but would work as long as you know your exact name from your credit card and have not changed things (i.e. started using a middle initial,, stopped using a middle initial, started using a first initial, etc). All the program would do is try all of the numbers from ‘00000000’ to ‘99999999’ (simply increment a count and convert the count to a string keeping leading zeros).
The last step might take a very long time to run, but once you find it, you can use it for all of your old books.
March 27, 2010 at 5:01 pm
Hi,
I’ve long been stripping DRM from my .azw files using MobiDeDRM so that I can read my Kindle-purchased books on my Windows Mobile Phone and, more recently, in Stanza on my iPhone, and am very grateful for all the folks who have contributed to the learning on this.
I am now – notwithstanding the success others seem to be having — struggling to successfully deDRM my few .tpz files.
I have read the comments on this Blog — in particular the March exchange between Darren and Some_Updates — and do not seem to be able to get the *,pyw files to execute. I’m running Windows 7 on a 64 bit Dell, and, when I run cmd to check my python version, I have only one installed and it is ActivePython 2.6.4.8 (and it has always worked for using MobideDRM from the command line on my .azw files).
I have downloaded (several times) Tools_v1.6a and extracted ALL FILES and gone to the extracted Topaz Tools folder and double-clicked (actually, single-clicked, as I have my options set up that way) on the TopazExtract_Kindle_iPhone.pyw file — NADA. Nothing happens.
I must be doing something basic wrong — but I cannot figure it out. If anyone can offer elucidation, I would be most grateful. Alternatively, if someone can step me through how to use the cmd syntax with cmbtc_dump_nonK4PC.py, I will try that instead — when I run it (using cmbtc_dump_nonK4PC.py – d -o TARGETDIR -P MYPID (first 8 only) NAMEOFTPZFILE) it runs, generates pages of numbers, but does not appear to create a TARGETDIR anywhere. Again, something basic I’m missing?
Thanks in advance for anyone who can help.
March 27, 2010 at 11:00 pm
Hi Rogerinnyc,
Can you please double-check / verify that you have installed the 32 bit version of Active Python and not the 64 bit version. If you have the 64 bit version, please completely uninstall it and then install the 32 bit version.
Then give things a try (double click on any of the *.pyw files just to see if they properly launch).
If not, then please try the following:
1. open a console (run cmd.exe).
2. once in the console change directory to where you extracted the tools
ie. something along the lines of …
cd tools_v1.6a
cd Topaz_Tools
Then try running the *.pyw by invoking python first as follows:
python TopazExtract_Kindle_iPhone.pyw
If that does not work, then try running it with pythonw as follows:
pythonw TopazExtract_Kindle_iPhone.pyw
I am beginning to think that the 32 bit version of Python is required to make the windows GUI programs work correctly.
Something here should give you at least an error message that we can use to figure out what is going wrong.
March 28, 2010 at 12:45 pm
Hi Some_Updates,
Thank you for the quick and thoughtful reply. And also helpful. I did indeed have 64 bit python installed. I uninstalled, rebooted, and then installed 32 bit (version 2.6.5.12).
The Windows gui now opens as expected when I click on a *.pyw file. I haven’t done the actual deDRM of my Topaz file yet (I want a little more time to study the three-part process) but I’ll let you know how that goes. In the meantime, however, I wanted to let you know that your diagnosis was spot on.
Thanks again.
March 28, 2010 at 4:39 pm
Last week I successfully converted a kindle book using tool 1.5. It was a .prc topaz book. Then Yesterday I downloaded tools 1.6a and tried another book. But it wouldn’t work. I really don’t know anything about programming so you would have seen the problem right away but it took me several hours and some experimenting to find out that the file cmbtc_dump.py was missing from the library. I copied that file from tools 1.5 to 1.6a and it works fine now. So this morning I downloaded tools 1.6a again and opened it and that file was missing again. I don’t know if its a problem with my downlaod or extraction. So I do have a correct set of files and they work but thought you would like to know that this happened to me and it might help someone else. Again, thanks so much for your work.
March 28, 2010 at 5:03 pm
Hi Larry,
My bad!
I took it out while trying to figure out what was going on with the issues people had in running it and simply did not put it back into the archive.
I think the issue was that cmbtc_dump.py uses crypt32 dll and uses ctypes to access it. With 64 bit python, this library does not seem to be interfaced with properly. I am sure there is a crypt 64 dll someplace, but not being a windows person or the author of the cmbtc_*.py code, I have not played around with it enough to know if that is what is causing the dependecy on the 32 bit version of Python. The same issue may happen with PyCrypto
A tools v1.6b will be posted momentarily.
Thanks for finding that! It would have caused no end of grief.
March 28, 2010 at 5:08 pm
I hope the 3rd time is a charm!
Here is the latest version of tools:
tools_v1.6b.zip.
http://www.mediafire.com/?mn3vmttbwrt
It should now have all of the needed files to actually work for Topaz books.
March 28, 2010 at 9:41 pm
I just got Kindle for PC and the unswindle.pyw script.
Unfortuanately, I have no way to de-drm the topaz ebooks.
March 28, 2010 at 9:51 pm
Hi Drak Bibliophile,
I think skindle-0.6 will remove the drm from Topaz books.
If you want to actually convert a Topaz book to a sequence of svg images or html, then you need the Topaz_Tools.
They are both part of a collection of useful tools form many sources:
tools_v1.6b.zip.
http://www.mediafire.com/?mn3vmttbwrt
Unzip the entire archive and find the tools you need for specific types of books.
You will need to have Python 2.6.X installed on your machine (32 bit version only – the 64 bit version will not work for some tools) to get things to work properly.
March 28, 2010 at 11:27 pm
Okay, what version of Tools_X.x.zip and Topazscripts_X.x.zip are we up to and where can I get them? Mediashare seems to have timed out.
March 28, 2010 at 11:50 pm
Hi DelphiNikonian,
The best way to read this list (since it no longer keeps good time sequence) is to use the RSS 2.0 feed available at:
feed://darkreverser.wordpress.com/2008/02/13/new-blog/feed/
From there you can see things in reverse chronological order and find the link to the latest things including tools_v1.6b.zip
Also you can check out Apprenctice Alf’s Blog to find links to the most up-to-date pieces.
March 29, 2010 at 4:36 am
Where are the latest files and tools, please? All of the mediafile links above return Invalid or Deleted file
Regards, Alex
March 29, 2010 at 11:23 am
Hi Alex,
The latest link is still valid. You just have to find the latest link which is hard given this forum seems to sometimes post out of order.
So, the best way to read this list is to use the RSS 2.0 feed available at:
feed://darkreverser.wordpress.com/2008/02/13/new-blog/feed/
From there you can see things in reverse chronological order and find the link to the latest things including tools_v1.6b.zip
Also you can check out Apprenctice Alf’s Blog and its comments to find links to the most up-to-date pieces.
tools_v1.6b.zip.
http://www.mediafire.com/?mn3vmttbwrt
March 29, 2010 at 1:31 pm
Thank you some_updates!
March 31, 2010 at 4:32 am
I’m having a problem with erdrPDB2pml.pyw. After entering everything I get this:
Conversion Log
Command = “python erdr2pml.py”
PDB Path = “C:\Users\Me\Documents\Downloads\eReader_Tools”
Output Directory = “C:\Users\Me\Documents\Downloads\eReader_Tools”
Name = “Me”
Last 8 of CC = “12345678”
Please Wait …
Traceback (most recent call last):
File “lib\erdr2pml.py”, line 692, in
sys.exit(main())
File “lib\erdr2pml.py”, line 648, in main
convertEreaderToPml(infile, name, cc, outdir)
File “lib\erdr2pml.py”, line 570, in convertEreaderToPml
sect = Sectionizer(infile, ‘PNRdPPrs’)
File “lib\erdr2pml.py”, line 323, in __init__
self.contents = file(filename, ‘rb’).read()
IOError: [Errno 13] Permission denied: ‘C:\\Users\\Marcy\\Documents\\Downloads\\eReader_Tools’
eRdr2Pml v0.15. Copyright (c) 2009 The Dark Reverser
Processing…
Decoding File
Error: Conversion Failed
Any suggestions would be appreciated!
March 31, 2010 at 4:35 am
Never mind. I tried again and it worked. Don’t know why.
April 4, 2010 at 9:10 pm
Hi,
Yes, you did not properly select the file to be converted (the *.pdb file) the first time and so it tried to read from the eReader_Tools directory itself and failed with an I/O error.
The hint was here in the log …
PDB Path = “C:\Users\Me\Documents\Downloads\eReader_Tools”
This should be pointing at the *.pdb book to be converted not at the eReader_Tools directory.
Which in turn caused this error:
IOError: [Errno 13] Permission denied: ‘C:\\Users\\Marcy\\Documents\\Downloads\\eReader_Tools’
The second time you must have entered in the right path or hit the … button and properly selected the book to be converted.
March 31, 2010 at 6:43 am
Dark Reverser, You’re a f*&^%$ing Genius! Thanx a bunch for your ereader2html.py. You, sir, are a Godsend….
April 3, 2010 at 2:51 pm
Is there any way to decrypt Adobe Digital V. l.7.2 epubs with ineptepub. V2. I keep getting “no such file or directory” in the Out File section of the decrypter. Any help greatly appreciated.
April 4, 2010 at 4:21 pm
You need ineptepub version 4.1
April 3, 2010 at 3:39 pm
Hi,
I’m wondering if anyone can explain/fix an error message I’m getting with Topaz Tools. I’m running it on a new Kindle-purchased book — Matterhorn — that runs about 600 pages in book form.
When I run “KindleExtract_Kindle_iPhone.pyw (on Windows 7), the extraction process runs smoothly through all the page xxxx.dat and then ran smoothly through the glyphs xxxx.dat until it got to glyphs 0078.dat. It then stopped and returned “Error – -10 outside of string table limits”
“Error – XML conversion failed”
I nonetheless tried the SVG script, and that too failed. However the HTML script did produce a book.html file that, with the usual conversion errors, looked pretty much OK.
Any thoughts about what’s going on here? Is it just the length of the book?
Thanks to anyone who can enlighten me here.
April 4, 2010 at 5:49 pm
Hi,
There is something in glyphs0078.dat that is not currently “known” xml so that when it try to decode it, it freaks out then program.
If you could please post someplace the dict00.dat and the glyphs0078.dat and send me a link, I should be able to figure out what is there that is confusing the xml parser so much and provide a fix for it.
April 5, 2010 at 3:15 pm
Thanks, some_updates! Here’s a mediafire link (first time I’ve tried, so I hope I did it right). http://www.mediafire.com/?sharekey=40bab2d51fa32e6ad8f14848abf485dd21147eff6f858391a4648785df63f216
April 5, 2010 at 6:24 pm
Hi Rogerinnyc,
I looked at the files you posted and the problem is that the glyphs0078.dat file is not based on the dict0000.dat file. In other words they seem to be from two different books!
Could it be that the glyphs0078.dat file is a leftover from an earlier conversion effort? Or perhaps you got the wrong dict0000.dat file from someplace.
That would explain why the html conversion worked (there was no page that needed glyphs0078.dat) but the svg, and xml conversion failed, since they will attempt to process all files in the glyphs/ directory.
Please try again using a completely clean directory to extract your book files into and then look in glyphs/ and see if there actually is a glyphs0078.dat at all. If not, that explains things. If so, then perhaps it would not let you overwrite the file, so try running the svg conversion again to see if it still fails.
April 4, 2010 at 5:00 pm
So how do we manage iPad ebooks from the new Apple bookstore? They are epubs but I am not sure what type or even how to proceed. I am coming from a Kindle so have been using the various tools on mobi and azw and tpz files. Which scripts mightbwork on the Apple drm? right now prices in the Kindle store are so much better I would tend to buy there and mobidedrm the files convert to pub and move them over. But sooner or later I will want to purchase something from the iPad bookstore….
April 4, 2010 at 5:35 pm
iPad epubs will be using Appl’es own encryption system, that they’ll be calling Fairplay. I doubt that any existing tools wil work with it, and reverse engineering it will be tricky, as I don’t think Apple are producing software to decode the encryption except on the iPad.
Someone might be able to work it out, but I don’t know how long it’ll take.
April 4, 2010 at 9:20 pm
Wow that’s too bad. Well, I’ll keep buying books from Amazon then to transfer to the iPad! Here is an absolutely perfect example of a business (Apple) that is going to lose money because of unbreakable DRM and one that will make money because their DRM is cracked. It is incredible.
April 5, 2010 at 2:18 am
I’m running into the same d-None error that one person above was having with a B&N PDB file. Sadly, it doesn’t appear that Barnes and Noble is still carrying the book in ebook format that I purchased, so it doesn’t look like it’s going to be made available as an epub (I just triggered a download of a new copy on the Windows client and it still pulled the same PDB from months ago). I’m having a KeyError: ‘x’ problem as well and I can make the PML file available if that will help fix this error.
Let me know.
April 5, 2010 at 3:34 am
Hi Aaron,
The d-None error is meaningless and can safely be ignored. It is a placeholder for “This book belongs to …”. It is not part of the official pml spec.
As for KeyError “x”, that means that somehow the \x commands are not being properly used (at least according to the pml specification.
Please post the pml file for me someplace and send me a link and I should be able to see how messed up the pml markup is and see if there is a way to make the xpml2xhtml.py work around it.
If there is a viable workaround, I will post a new xpml2xhtml.py program for you to test with.
April 5, 2010 at 7:42 pm
Thank you some_updates. You nailed it again. Indeed, using a clean directory, there was no glyphs 0078.dat — so lesson learned — use a different extract directory for each Topaz conversion.
Having said that, I did a full re-conversion of my book and it appears not to be any different than the conversion I did that generated the error codes — which kinda makes sense — I guess the conversions (both XML and SVG) just ignored the glyphs left over from the prior book conversion
April 6, 2010 at 2:34 pm
Hi there….I’ve been using the scripts for a couple years now. But, I’ve only recently been turned on to the Kindle scripts for Kindle for PC. Great stuff…
Recently, I got a Topaz book…and used the Topaz scripts from Alf’s blog.. Worked like a charm…
Then, I wanted to read some samples of books…so I downloaded a couple Topaz samples…and converted them to mobi so I could read on my handheld.,.. Again…no problem.
Then my laptop crashed…and it’s beyond recoverable…
So, I am now using my desktop and have ported over all the python scripts and the Tools v1.6b… Since I liked those samples, I went ahead an ordered the full versions..
Well…I tried Topaz…and it would not give me a conversion at all! And, the ebook that I had earlier converted with no problem…I can’t convert THAT one either any longer… I checked Unswindle…in the same Tools folder….and it works perfectly. No problem.
So, it’s either the Topaz script…or the copy of the book from Amazon. I’m confused as to which it is. I believe Alf might already be looking into it so I hate to sound like I’m all over the place bugging everyone.
I just wanted to see if anyone else has issues with Topaz recently…like in the last day or two, maximum. Because, 3 days ago it was working just fine…
I’m writing here because I wanted to know if it matters what version of Python I’m using because I cannot recall what I had on my laptop before it crashed. It was definitely a 2.6 version…but does it matter if it’s 2.6.4 or 2.6.5….or even earlier?
Anyway…I think I’ve gotten everything right on my desktop…but I cannot be sure. Is there a written log of what to install…and then how to proceed step-by-step in order to create a Topaz into HTML? I’m not sure if that will help because I get stuck at the Topaz Extract part… the PID is what’s screwing it up… It keeps showing that it starts with ” ” instead of numbers….Weird, huh?
I apologize if I’m bothering anyone…or Alf.. I just am so confused as to how this could happen? Lol..
I even changed to a new user id…and everything…so I could download a new Topaz file to play with…but no luck. Still can’t get past step one…
BTW…one quick question (ha ha…yeah right, huh?) Out of curiosity, I DO change the extension to AZW1, right? Because, that’s what I did before and it worked just fine…now it doesn’t. Maybe the new script needs it to be left in the PRC extension? Just curious. BTW…thank you ALL for such great info here on this blog. It’s amazing all the great and hard work you’ve done just to let those of us not be bullied into purchasing devices to read our favorite books when they should be readable on ALL devices. Many thanks….
April 6, 2010 at 6:40 pm
Hi Alex,
Please see my response to your comment number 4 on Apprentice Alf’s blog.
Other answers:
If you are using Kindle for PC and you can open the book in Kindle for PC in that same account, then you do not need the PID, the program will figure it out for you.
Python versions do matter. Almost any 32 bit version in the 2.6.X series will work. Python 3.X will not work. And some scripts are not happy with the 64 bit version of Python at all (so I strongly recommend only using the 32 bit version)
Once you have the right version of Python installed (and you have fully removed the 64 bit version and installed the 32 bit version) of Python 2.6.x please try again (following my response in your post on Apprentice Alf’s blog)
The steps are:
Double click on the following in this order (being care to select the right things in the gui).
1. TopazExtract_Kindle4PC.pyw
2. TopazFiles2SVG.pyw
3. TopazFiles2HTML.pyw
April 6, 2010 at 6:41 pm
Also,
No need to rename anything from .prc to .azw1
April 9, 2010 at 12:18 am
Do you also have the scripts from the sub directory? I think the .pyw scripts just pass the info to those scripts.
April 10, 2010 at 5:16 am
Anyone interested in tackling the so-called SPDF format? More details at
http://landofthefreeish.com/security/sealed-pdf/
It seems the solution posted is to print the purchased ebook on a LINUX system into a ordinary PDF using some so-called cups-pdf program that normal folks won’t know how in the first place. Can our mighty windows simply do the job? Planning to buy a eChapter at Cengage the publisher who use this kind of DRM as well. Thank you so much.
April 11, 2010 at 6:48 pm
I have a problem with one of the programs: MobiUnpack, so I don’t know if I’m using it correctly. After running the program, somewhere from the middle of the book, some images are substituted with images from different parts of the book. Any ideas?
April 11, 2010 at 7:14 pm
Which version? The latest on 0.21, which fixes up some problems with trailing byte handling.
http://code.google.com/p/ebook-conversion-tools/source/browse/lib/mobiunpack.py
If you still have the problem with 0.21, I’d love to see the original Mobipocket file. Thanks.
Paul
April 11, 2010 at 7:47 pm
The one I’m using is the one from tools_v_1.6b.zip which I believe is v 0.21
Here is the file I’ve the problem with:
http://www.mediafire.com/?ntkztuhdztm
Thank you for your help.
April 11, 2010 at 7:55 pm
OK, I’ve got the file and reproduced the problem, so you can take it down from mediafire now.
I don’t have time tonight to debug it, but I’ll do so tomorrow, I hope. The image substitution code is mine, and it looks like I’ve messed up counting about 9! Probably my inexperience with Python, so hopefully it will be an easy fix.
April 11, 2010 at 8:12 pm
The file is off the mediafire. When it comes to the fix take your time, I’m in no hurry at this time. Thank you again for your help.
April 12, 2010 at 8:24 am
Yes, a silly bug in the regular expression code that updated the image links. Now fixed in version 0.22.
http://www.mobileread.com/forums/showthread.php?p=774836#post774836
April 12, 2010 at 2:48 pm
Again, Thank you very much for your help.
April 20, 2010 at 1:29 am
Hey all! First I want to say thanks a MILLION for the hundreds (if not thousands) of man-hours that have gone into this.
I’ve got the tools_v1.6b package, and it has been working fine up until tonight. Just bought a new book from the Amazon site, (using Kindle4PC) and I went to use the TopazExtract_Kindle4PC, and after setting where the book was, and setting where to extract it to, I got this message:
Conversion Log
Command = “python cmbtc_dump.py”
Topaz Path Path = “C:\Users\Darksun\Documents\My Kindle Content\B0031RS6PU_EBOK.prc”
Output Directory = “C:\Users\Darksun\Desktop\Utilities\Kindle\tools\Topaz_Tools\Temp”
First 8 chars of PID = “”
Please Wait …
Traceback (most recent call last):
File “C:\Users\Darksun\Desktop\Utilities\Kindle\tools\Topaz_Tools\lib\cmbtc_dump.py”, line 880, in
sys.exit(main())
File “C:\Users\Darksun\Desktop\Utilities\Kindle\tools\Topaz_Tools\lib\cmbtc_dump.py”, line 819, in main
parseTopazHeader()
File “C:\Users\Darksun\Desktop\Utilities\Kindle\tools\Topaz_Tools\lib\cmbtc_dump.py”, line 378, in parseTopazHeader
raise CMBDTCFatal(“Parse Error : Invalid Header, not a Topaz file”)
__main__.CMBDTCFatal: Parse Error : Invalid Header, not a Topaz file
Device PID: X77F1MIE
Error: File Extraction Failed
The thing is though: it IS a Topaz file! And just to make sure the script was still working, I re-converted some older books that I bought last month, and it worked just fine. It’s just this one book that’s not working, and it’s the newest. Did amazon change their format, or is there something I’m missing?
April 20, 2010 at 1:49 pm
Hi,
How do you know it is a Topaz file?
Simply open the file with any text or hex editor and look for the ‘TPZ0’ as the first 4 bytes of the file and see if it is there. That is what the cmbtc_dump.py program is looking for.
If not, what string does make up the first 4 bytes of the file?
Could your download be corrupted in some way?
April 23, 2010 at 4:04 pm
I have similar issues. Files I just downloaded are giving me the same error message although older ones seem to be working fine.
April 23, 2010 at 5:51 pm
Never mind, stupid user error. Some of the files were not topaz and I either forgot how to use the script or something subtle has changed. I don’t recall having to open the book when running unswindle, just select it in K4PC which would result in an error message. If I actually open the book after running unswindle everything is fine.
April 27, 2010 at 5:13 am
Thanks to everybody that has made this process possible…especially for computer illiterate old fogies. I read this blog all the way through and only understood about 10%. What I did understand allowed me to put the right files in the right place…and then I just followed directions and it did its thing. I have no idea how it worked…but it did. Thanks.
I used unswindle and mobidredrm to allow me to “clean” up my ebooks. I kept reading about mobiunpack too…and have no idea what it’s for or how to use it. I put it in the same directory as unswindle and mobidredrm…but I’m fairly certain this isn’t how it’s used. Any help on this would be appreciated.
I’m also confused about books in Adobe Digital Editions. Is there a way to “clean” these up?
Thanks again. DRM is just another freedom taken from us.
April 27, 2010 at 2:43 pm
I have a silly question. I have a Mac. It has Python installed in it. Can I run unswindle and mobidedrm with K4Mac somehow? Are there any tweaks that would allow this to work. I don’t mind going to Bootcamp…but would prefer staying in OS X if possible.
April 27, 2010 at 3:32 pm
To the best of my knowledge, no-one has yet worked out how Kindle for Mac manipulates the PIDs for each book. And so it’s not possible to strip the DRM from books downloaded using Kindle for Mac, except by brute force.
If you have an iPod touch, just download Kindle for iPod, and you’ll be able to get a PID from the iPod Touch serial number that will work for any books downloaded for the iPod just using MobiDeDRM.
April 27, 2010 at 3:58 pm
Thanks. As I mentioned earlier…as far as programming goes I am completely clueless. I follow directions fairly well and am fairly stubborn, so can usually work out how to do things. I’m a little lost with a few things though.
I’ve determined what mobiunpack does. Now you’ve given me a hint on how to run mobidedrm on my mac. Would you be willing to explain to me:
1. How to run the scripts from terminal (I’m assuming mobiunpack and mobidedrm will work with the same syntax…just different script names)?
2. Where and exactly how I determine my iPod Touch PID?
3. Once I have the PID, how is it used in mobidedrm?
I apologize for my ignorance. I’ve searched the forums and googled quite a bit and can’t seem to find relevant information.
Thanks for everything
April 27, 2010 at 4:09 pm
You might find it easier asking questions at
http://apprenticealf.wordpress.com/
where there aren’t 1250 comments on one post. 🙂
As for running this stuff on a Mac. I just use Applescript wrappers. You’ll find some links to them at Alf’s blog. It makes doing se-srm as simple as drag&drop.
For Mobiunpack help, you can ask at Mobileread.com – an excellent ebook forum that will be happy to help out, as Mobiunpack only works on non-drmed Mobipocket files, and so there are no awkward legal problems.
iPod Touch PID – again, you’ll need to have your iPod Touch serial number, and then run the kindlepid script to get the equivalent PID. Once you have that, and have registered your iPod touch at amazin through downloading the kindle app, you should have no difficulty in de-drming Kindle ebooks (so long as they’re no Topaz format).
MobiDeDRM – easiest to use the Applescript wrapper mentioned above. Then you just enter your PID once, and it stores it for use in calling the MobiDeDRM script.
April 27, 2010 at 4:20 pm
Thanks…I’ll use those resources. You’ve been very helpful and gracious.
April 29, 2010 at 2:04 am
I have successfully worked with a topaz.prc book. I just downloaded a book from amazon with a .prc. When I tried kindle4pc with topaz tools, it says it is not a topaz book. I tried unswindle but got an error mobidedrm error saying:Mobi v0.15
Mobi header version=6, length =232
Extra Data Flags =2
Decrypting. Please wait… done.
But nothing is done and the book does not show up so I am assuming it wasn’t decrypted
Thanks again for all your help.
larry
April 29, 2010 at 7:00 pm
When doing swindle, the kindle4pc comes up, a book is selected and kindle4pc is closed. Next comes up a page asking for the undecrypted mobi book. So I choose a place to save the book, a folder I made, and then askes for the file name. I guess I’m not sure what to put in for the file name. I have tried many things but always get an error message. Error running mobidedrm. This book is not a topaz book so thats not the problem. I also downloaded tha book again so thats not the problem. Any ideas would be appreciated. Thanks again.
April 30, 2010 at 2:07 am
Since I was having trouble with mobidedrm using unswindle, I replaced mobidedrm from v1.6b with mobidedrm from v 1.5. Still won’t work but get different errors. With 1.6, just says problem running mobibderm. With v 1.5, it says “no module named subasyncio. ” Wow. Anyway, I can do topaz books just fine, but can’t seem to make unswindle work. Help!!
larry
April 30, 2010 at 4:07 am
Finally it worked. I downloaded tools v1.6b again. The diffrence was in the swindle file which did not contain the file .DS_store before. With that file incl, the program worked just fine. Just gotta keep trying different things. thanks again for all your work.
larry
May 2, 2010 at 4:17 pm
I have a PID which is only 10 characters long (taken from the mobireader software), and the program says “invalid PID checksum”
Any help greatly appreciated!
May 2, 2010 at 5:30 pm
You’ve probably got I /1 or O/0 mixed up in the PID. Check again.
May 2, 2010 at 5:44 pm
Thanks Paul, but I copied and pasted – I surrounded the PID with single quotes and it worked, at least so far.
I have run the py in the terminal, as for some reason Calibre doesn’t use the plugin. I converted from .prc to ,prc, then using Calibre converted to mobi, and then to fb2. It doesn’t seem to like PDF conversion. It’s probably long-winded, but it seems to work OK.
Thanks again!
May 3, 2010 at 9:57 pm
Anyone having trouble with Unswindle.pyw recently? I have ver6, and K4PC updated, and now whenever I try to run unswindle, it comes up with the error message “unsupported version of K4PC”.
May 3, 2010 at 11:07 pm
Hi,
Yes, there is a new binary for Kindle for PC (and it can do an autoupdate) and since unswindle has to patch the binary it includes a check to make sure it is a binary that the program knows where to patch. The new version of Kindle For PC has a new binary and so unswindle does not yet know where to patch it to make things work.
So you have 2 options:
1. Go to the I Heart Cabbages website and wait for a new version of unswindle that has been updated to work with the latest binary. Or instead, use skindle (see the tools_v1.6b.zip archive) for a binary version that will work regardless of the binary being used since it simply decodes the proper PID and then knows how to modify it to calculate the book specific PID.
If someone is good with python, they can read the skindle source code and modify unswindle to do the same thing if needed.
May 4, 2010 at 1:57 pm
some_updates, outside of the unswindle thing, the tools could use an update with the new mobinpack mentioned earlier – what’s the outlook on getting compiled windows versions by the way?
May 4, 2010 at 6:36 pm
Hi,
There are two new things to waiting for the next update:
mobiunpack.py (new version)
mobidedrm.py (new version)
None of these are critical as the latest mobidedrm.py can be grabbed from Apprentice Alf’s Blog site (the current version works perfectly, it is just not as fast as Alf’s new one), and the latest mobiunpack.py is available from Paul’s mobileread.com/forum/ link since it has nothing to do with DRM removal it can be publicly posted and is readily available.
So I was planning on waiting for an update to unswindle before putting out a tools v 1.7.
As for pre-compiled tools for Windows (pyexe), unfortunately, I can not produce those and the person who supported them previously has been shipped out to Iraq or Afghanistan I believe.
So we will need to wait for his return or for some other kind soul to build things with pyexe and release them.
May 5, 2010 at 12:31 am
When I heard about the new kindle4pc, I quickly downloaded a book to see if unswindle would work but of course it didn’t. So I tried skindle. I tried double clicking on it from tools 1.6b and dos flashes but nothing happens. I tried a command prompt but only get Kindle info file found. Nothing more. So I guess I don’t know how to use skindle. Could someone help me out here. I have done most other types, swindle and topaz but this one has me confused. Thanks for any help.
larry
May 5, 2010 at 12:41 am
Hi Larry,
Skindle is a command line tool. Run it from a terminal
Here is the usage info from the kindle.c program.
./skindle [-d] [-v] -i -o [-k kindle.info file] [-p pid]
-d optional, for topaz files only, produce a decompressed output file
-i required name of the input mobi or topaz file
-o required name of the output file to generate
-k optional kindle.info path
-v dump the contents of kindle.info
-p additional PID values to attempt (can specifiy multiple times)
May 5, 2010 at 1:03 am
Wow. Not sure about all the letters. Do I actually write[-v] and do I write -i and then the output mobi file or just the mobi file. Do I actually write [-p]? Just not sure what to write or are the letters just place holders for input files and output files. I must have done something wrong because I still just get the answer, kindle.info file found. Sorry to keep asking. So much to know. thanks so much.
larry
May 11, 2010 at 9:07 pm
I just used skindle to remove the DRM from a mobi ebook, and that worked fine. I was hoping I could use the PID it displays as input to mobidedrm used as a plugin in Calibre, but that kind of failed (“couldn’t decode blah bleh”).
Am I doing something wrong, or do I have to use unswindle in order to figure out the PID for K4PC?
May 12, 2010 at 2:32 pm
Hi,
There is no one PID anymore for Kindle For PC. Each book has its own specific PID which is why you need to use skindle to figure things out.
May 5, 2010 at 2:59 am
Larry,
I have placed all the skindle files into one directory \skindle.
Make sure that skindle.exe is included.
I copy the book I plan to convert into this directory for example B000RG1ONE_EBOK.prc
then in the DOS window CD C:\skindle to get into the right directory.
then type the following: (make sure no spaces in book name)
skindle -i B000RG1ONE_EBOK.prc -0 book_name.prc
this works for non-topaz I have not converted a topaz book using skindle
hope this helps
Mark
May 5, 2010 at 3:31 am
Mark and Some_updates, thankyou very much for your help. I’m ready to read the book. Wasn’t nearly as difficult as I was making it. You all deserve a medal for your work. Thanks again.
Lar
May 5, 2010 at 1:31 pm
I’m running into my first problematic conversion of a .pdb file. I’ve not seen this sort of error before when converting/decrypting all of my other files and I was wondering if there was a way to correct it.
When using eReader2html, I get the following messages:
eReader2Html v0.07. Copyright (c) 2008 The Dark Reverser
Processing…
Traceback (most recent call last):
File “ereader2html.py”, line 813, in
sys.exit(main())
File “ereader2html.py”, line 799, in main
convertEreaderToHtml(infile, name, cc, outdir)
File “ereader2html.py”, line 761, in convertEreaderToHtml
file(os.path.join(outdir, htmlfilename),’wb’).write(pml.process())
File “ereader2html.py”, line 687, in process
r = self.next()
File “ereader2html.py”, line 561, in next
c = self.s[p+1]
IndexError: string index out of range
When using the PML/HTML python scripts I run into:
Converting pml to raw html
WARNING:root:Unknown tag: d-None
Traceback (most recent call last):
File “/xpml2xhtml.py”, line 858, in
sys.exit(main())
File “/xpml2xhtml.py”, line 833, in main
html_src = pml.process()
File “/xpml2xhtml.py”, line 492, in process
r = self.next()
File “/xpml2xhtml.py”, line 195, in next
c = self.s[p+1]
IndexError: string index out of range
Any tips or help is greatly appreciated.
May 5, 2010 at 1:48 pm
TJovian,
It appears your pml file is truncated in some strange way that makes conversion to html not work. The code is expected more pml from the file, but the file is already at its end.
You could try any of the following.
1. Take the pml output from erdr2pml.py and give it to Calibre to see if it works. If it converts to your liking, you are done!
or
2. Try using Dropbook to create a new (non-DRM) pdb file using the pml you get from erdr2pml.py. If that works, you can read the .pdb on your device or use Calibre to convert it.
or
3. post the pml file someplace and post a link and I will debug what is different from the typical pml and will then modify the xpml2xhtml.py program to handle that case.
May 5, 2010 at 2:09 pm
Thanks! DropBook had been erroring telling me it was unable to read the PML file, but Calibre seemed to do the trick. I must’ve had a brainless moment where I didn’t think to try the PML file in Calibre since I generally use it for all of my eBook organization and decrypted conversions.
The Table of Contents seems to end up on the last page of the book (perhaps it was designed to be at the end?), but my CyBook Gen3 doesn’t seem to mind and recognises it regardless of its location. The rest of the book and formatting seems unaffected. Thank you SO much for such a quick reply and so much useful information!
May 7, 2010 at 7:00 pm
I can’t tell you enough how much I appreciate the work you’ve done!!!! I own nearly 500 eReader formatted books. Recently, I bought a couple of books from B&N. I opened them with my Palm TX just fine but discovered you can’t adjust the font. An email to customer support resulted in “sorry, can’t help.” I’m not a programmer and it wasn’t until I stumbled over your Tools package that I was finally able to make heads or tails of any of this, but it worked. Now my books are in a nice, readable font size. May the force be with you always.
May 7, 2010 at 8:14 pm
@nellybly: That’s odd. You might ask Barnes and Noble technical support for a copy of eReader 3.0.0fp, if you can’t download it from their site. It does permit font adjustments.
May 7, 2010 at 9:34 pm
I thought it was extremely odd, too. I also have noticed that eReader.com no longer offers most of the popular best sellers, just romance, fantasy, sci-fi and little known authors as of recently. And the BN website no longer offers the eReader download for Palm OS, just iPhone, Blackberry, Windows and Mac. My sense is that BN is dumping the Palm OS format and the eReader.com website for new book sales. With all that in mind, I’ve downloaded as much of my library as is still available. Another example of DRM tyranny.
May 7, 2010 at 9:49 pm
BTW. I own both font packs from eReader, and while they do adjust for books downloaded from eReader.com, they won’t adjust for books downloaded from BN.
May 12, 2010 at 10:17 am
Hi, all!
I have Kindle for PC v 1.1.0 (30135) on XP Pro and I get “Wrong version” on using the tool pack. Any suggestion?
Thanks,
P.
May 12, 2010 at 2:30 pm
Hi Peter,
What where you trying to do? What software were you using? unswindle, skindle, etc?
If you are using unswindle and if you have the latest updated Kindle for PC then unswindle will no longer work. You will have to use skindle instead. There are instructions on how to use skindle posted quite recently in this blog so look near the very end of this blog to find that information if needed.
May 13, 2010 at 5:21 am
Look, I’m very tech-unsavvy, so please bear with me here…
I have determined that I need skindle because I have the latest K4PC and unswindle doesn’t work, however I don’t understand how to use skindle.
With unswindle, I just clicked a link in the blog and it took me to the download page, I see no link like that for skindle.
Is there a quick link you could post for me to access skindle?
Or can you tell a dummy like me how to do this step-by step?
I have K4PC downloaded, I have Python 2.6.4 downloaded, I just don’t know what to do next to de-drm my kindle ebook.
Thanks a bunch for making this stuff possible, it’ll be such a life-saver. 🙂
May 13, 2010 at 1:57 pm
Hi,
Please go to the Apprentice Alf’s Blog.
http://apprenticealf.wordpress.com/
At the bottom of that page you will find a link to a single archive that is a collection of tools that people can use:
http://www.mediafire.com/?mn3vmttbwrt
It has skindle in it as one of the tools.
Earlier on that same blog page, they have a direct links to skindle as well.
Skindle is a command line program written in C, and therefore does not actually need python.
So you open a terminal in Windows:
The format for the command line is given earlier in this blog, and I will repeat it here:
skindle [-d] [-v] -i -o [-k kindle.info file] [-p pid]
-d optional, for topaz files only, produce a decompressed output file
-i required name of the input mobi or topaz file
-o required name of the output file to generate
-k optional kindle.info path
-v dump the contents of kindle.info
-p additional PID values to attempt (can specifiy multiple times)
So to decode a non-Topaz file you should do the following:
1. In Windows, create a new folder and copy in the skindle.exe program, and copy in the book you want to remove the drm from
2. open a terminal (run command.exe) and use the cd command to change directories to your new folder (the one with skindle.exe and the book in it).
3. type in the following command line
skindle -i NAME_OF_INPUT_FILE -o NEW_NAME_FOR_OUTPUT_FILE
replacing NAME_OF_INPUT_FILE and NEW_NAME_FOR_OUTPUT_FILE as appropriate and it should do the rest.
May 15, 2010 at 6:55 pm
I’m new to posting here, just want to know two things (unrelated)
1 – Did anyone discover a way to get the PID from the Blackberry kindle app?
2 – While removing the DRM from my eReader books, I have found some that the script couldn’t decrypt, mainly the dictionaries (merriam-websters dictionary and the thesaurus) and the book “Band of Brothers”. I don’t remember the error on the dictionaries, but Band of Brothers had an error 11, which I had never seen before.
May 16, 2010 at 4:23 am
I don’t know where to find my kindle book files. I looked in:
\Kindle For PC\application
\Kindle For PC\Cache
\Kindle For PC\legal
\Kindle For PC\style
\Kindle For PC\updates
The only thing I found that might have been a book was this file: B000SW16L2. Skindle gave me a “Bad Open” error when I tried to run it with the following command: skindle -i B000SW16L2 -o
I’m not sure if this is the book file or not.
Any help would be greatly appreciated. Thanks for all of the hard work in the fight for freedom.
AN
May 16, 2010 at 6:01 am
look in “my documents” folder for “my kindle content”
May 16, 2010 at 8:09 pm
Thanks Mark
May 17, 2010 at 8:06 am
On last thing. I ran a book through Skindle and it said “Success! Enjoy!”
However, I can’t seem to open the file in any ebook reader that claims to read prc files. I also tried to reformat the prc to a txt file, and that didn’t work.
Any help would be appreciated.
May 17, 2010 at 5:12 pm
I keep getting the “bad open” message when using skindle. This is true for both older and newer prc books. The older books I had used unswindle which won’t work with newer Kindle for PC.
Thanks for all of your help and info!
May 17, 2010 at 7:15 pm
Lawrence, I was getting that message while attempting to strip the DRM from the wrong file. Make sure that the file you are referring to is actually the PRC book file from My Documents > My Kindle Content (assuming you’re using windows).
Hopefully that helps.
May 17, 2010 at 7:36 pm
Thanks, Ian. Actually, I finally got it to work, by putting skindle and the book in the root folder, and spelling out the whole path to everything, thus:
C:\skindle\skindle.exe -i C:\book.prc -o C:\newbook.prc
It wasn’t working for me with the book in the My Kindle Content folder.
I also renamed the azw to prc. I’m not sure what did it, but it finally worked perfectly!
Thanks.
May 18, 2010 at 11:00 pm
This is absolutely inredible. Such a combined effort of a lot of people. It is really amazing. I bought yesterday my first e-book on amazon just to try it in my new device, and I still don´t know many things, but with the entrance of some_updates on 13th of may, I could manage finally to change the format of the document.
I don´t have words.
Thank you to make lifes much much more easier, really.
Oscar
May 19, 2010 at 6:49 pm
Hi — Just got an iPad so am working with Adobe ePubs after having used the tools for a long time with the Kindle and mobi files. Anyway, I can’t get the ePub to work. I have a Mac and first ran ineptkeymac.pyw which did give me a file named adeptkey.der which I put into the same folder. Then I saw I needed PyCrypto. I found a link to a Mac insaller and that gave me a folder called PyCryto.pmdoc with three files inside 01pycrpto-contents.xml, 01pycrypto.xml and index.xml. I moved these three files to the folder with the ineptpub.pyw. But whenever I now run ineptpub.pyw it tells me to install PyCrypto. When I search on the disk for PyCrypto is just returns the stuff above. But the installer did say that the install was succeful. Any ideas where I could go from here?? THanks!
May 19, 2010 at 8:52 pm
Look here: http://dearauthor.com/wordpress/2010/02/21/how-to-install-python-and-pycrypto/
That PyCryto.pmdoc [sic] is probably just some documentation. It does not install PyCrypto.
May 20, 2010 at 12:42 am
Well, I think that almsot worked. I did the setup.py build and it spit out a whole bunch of stuff but at the end I get an error having to do with something called GMP and then when I try the setup.py install it immediately fails. Here’s a dump (just changed my name to “username” in a few spots ):
gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -fno-common -dynamic -O3 -fomit-frame-pointer -Isrc/ -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c src/MD2.c -o build/temp.macosx-10.3-i386-2.6/src/MD2.o
unable to execute gcc: No such file or directory
error: command ‘gcc’ failed with exit status 1
Macintosh:pycrypto-2.1.0 username$ python setup.py install
running install
running build
running build_py
running build_ext
warning: GMP library not found; Not building Crypto.PublicKey._fastmath.
building ‘Crypto.Hash.MD2’ extension
gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -fno-common -dynamic -O3 -fomit-frame-pointer -Isrc/ -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c src/MD2.c -o build/temp.macosx-10.3-i386-2.6/src/MD2.o
unable to execute gcc: No such file or directory
error: command ‘gcc’ failed with exit status 1
Macintosh:pycrypto-2.1.0 username$
Do I need something called GMP???
May 20, 2010 at 1:05 am
Hi,
Your error was:
unable to execute gcc: No such file or directory
error: command ‘gcc’ failed with exit status 1
So you do not have the gcc compiler installed on your machine.
Typically on a Mac, you need to install the XCode package to get the compilers (like gcc).
The Developer XCode package can typically be found as an additional install on your Mac OSX install disc or can be downloaded (it is big) from Apple.
It installs is in /Developer on your main disk drive unless you specify someplace else and then creates symlinks in all of the proper places.
On my Mac (Snow Leopard), gcc can be found here:
/usr/bin/gcc
So grab the XCode Developer package off your latest Mac OS X Install disc and install it.
Then try building pycrypto again.
That should do the trick.
May 21, 2010 at 6:46 am
Nope, no go. I installed the Xcode off the install disk and it did put in Developer/SDKS the MacOSX10.4u sdk. But I still get a gcc error when I run setup.py build. Here is the dump. I appreciate any help. And, I apologize for taking up this much vertical space here.
running build
running build_py
creating build
creating build/lib.macosx-10.3-i386-2.6
creating build/lib.macosx-10.3-i386-2.6/Crypto
copying lib/Crypto/__init__.py -> build/lib.macosx-10.3-i386-2.6/Crypto
copying lib/Crypto/pct_warnings.py -> build/lib.macosx-10.3-i386-2.6/Crypto
creating build/lib.macosx-10.3-i386-2.6/Crypto/Hash
copying lib/Crypto/Hash/__init__.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Hash
copying lib/Crypto/Hash/HMAC.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Hash
copying lib/Crypto/Hash/MD5.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Hash
copying lib/Crypto/Hash/RIPEMD.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Hash
copying lib/Crypto/Hash/SHA.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Hash
creating build/lib.macosx-10.3-i386-2.6/Crypto/Cipher
copying lib/Crypto/Cipher/__init__.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Cipher
creating build/lib.macosx-10.3-i386-2.6/Crypto/Util
copying lib/Crypto/Util/__init__.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Util
copying lib/Crypto/Util/_number_new.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Util
copying lib/Crypto/Util/Counter.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Util
copying lib/Crypto/Util/number.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Util
copying lib/Crypto/Util/python_compat.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Util
copying lib/Crypto/Util/randpool.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Util
copying lib/Crypto/Util/RFC1751.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Util
copying lib/Crypto/Util/winrandom.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Util
creating build/lib.macosx-10.3-i386-2.6/Crypto/Random
copying lib/Crypto/Random/__init__.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Random
copying lib/Crypto/Random/_UserFriendlyRNG.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Random
copying lib/Crypto/Random/random.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Random
creating build/lib.macosx-10.3-i386-2.6/Crypto/Random/Fortuna
copying lib/Crypto/Random/Fortuna/__init__.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Random/Fortuna
copying lib/Crypto/Random/Fortuna/FortunaAccumulator.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Random/Fortuna
copying lib/Crypto/Random/Fortuna/FortunaGenerator.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Random/Fortuna
copying lib/Crypto/Random/Fortuna/SHAd256.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Random/Fortuna
creating build/lib.macosx-10.3-i386-2.6/Crypto/Random/OSRNG
copying lib/Crypto/Random/OSRNG/__init__.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Random/OSRNG
copying lib/Crypto/Random/OSRNG/fallback.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Random/OSRNG
copying lib/Crypto/Random/OSRNG/nt.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Random/OSRNG
copying lib/Crypto/Random/OSRNG/posix.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Random/OSRNG
copying lib/Crypto/Random/OSRNG/rng_base.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Random/OSRNG
creating build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest
copying lib/Crypto/SelfTest/__init__.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest
copying lib/Crypto/SelfTest/st_common.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest
creating build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Cipher
copying lib/Crypto/SelfTest/Cipher/__init__.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Cipher
copying lib/Crypto/SelfTest/Cipher/common.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Cipher
copying lib/Crypto/SelfTest/Cipher/test_AES.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Cipher
copying lib/Crypto/SelfTest/Cipher/test_ARC2.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Cipher
copying lib/Crypto/SelfTest/Cipher/test_ARC4.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Cipher
copying lib/Crypto/SelfTest/Cipher/test_Blowfish.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Cipher
copying lib/Crypto/SelfTest/Cipher/test_CAST.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Cipher
copying lib/Crypto/SelfTest/Cipher/test_DES.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Cipher
copying lib/Crypto/SelfTest/Cipher/test_DES3.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Cipher
copying lib/Crypto/SelfTest/Cipher/test_XOR.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Cipher
creating build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Hash
copying lib/Crypto/SelfTest/Hash/__init__.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Hash
copying lib/Crypto/SelfTest/Hash/common.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Hash
copying lib/Crypto/SelfTest/Hash/test_HMAC.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Hash
copying lib/Crypto/SelfTest/Hash/test_MD2.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Hash
copying lib/Crypto/SelfTest/Hash/test_MD4.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Hash
copying lib/Crypto/SelfTest/Hash/test_MD5.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Hash
copying lib/Crypto/SelfTest/Hash/test_RIPEMD.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Hash
copying lib/Crypto/SelfTest/Hash/test_SHA.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Hash
copying lib/Crypto/SelfTest/Hash/test_SHA256.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Hash
creating build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Protocol
copying lib/Crypto/SelfTest/Protocol/__init__.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Protocol
copying lib/Crypto/SelfTest/Protocol/test_chaffing.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Protocol
copying lib/Crypto/SelfTest/Protocol/test_rfc1751.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Protocol
creating build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/PublicKey
copying lib/Crypto/SelfTest/PublicKey/__init__.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/PublicKey
copying lib/Crypto/SelfTest/PublicKey/test_DSA.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/PublicKey
copying lib/Crypto/SelfTest/PublicKey/test_RSA.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/PublicKey
creating build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Random
copying lib/Crypto/SelfTest/Random/__init__.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Random
copying lib/Crypto/SelfTest/Random/test_random.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Random
copying lib/Crypto/SelfTest/Random/test_rpoolcompat.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Random
creating build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Random/Fortuna
copying lib/Crypto/SelfTest/Random/Fortuna/__init__.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Random/Fortuna
copying lib/Crypto/SelfTest/Random/Fortuna/test_FortunaAccumulator.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Random/Fortuna
copying lib/Crypto/SelfTest/Random/Fortuna/test_FortunaGenerator.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Random/Fortuna
copying lib/Crypto/SelfTest/Random/Fortuna/test_SHAd256.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Random/Fortuna
creating build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Random/OSRNG
copying lib/Crypto/SelfTest/Random/OSRNG/__init__.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Random/OSRNG
copying lib/Crypto/SelfTest/Random/OSRNG/test_fallback.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Random/OSRNG
copying lib/Crypto/SelfTest/Random/OSRNG/test_generic.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Random/OSRNG
copying lib/Crypto/SelfTest/Random/OSRNG/test_nt.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Random/OSRNG
copying lib/Crypto/SelfTest/Random/OSRNG/test_posix.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Random/OSRNG
copying lib/Crypto/SelfTest/Random/OSRNG/test_winrandom.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Random/OSRNG
creating build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Util
copying lib/Crypto/SelfTest/Util/__init__.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Util
copying lib/Crypto/SelfTest/Util/test_Counter.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Util
copying lib/Crypto/SelfTest/Util/test_number.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Util
copying lib/Crypto/SelfTest/Util/test_winrandom.py -> build/lib.macosx-10.3-i386-2.6/Crypto/SelfTest/Util
creating build/lib.macosx-10.3-i386-2.6/Crypto/Protocol
copying lib/Crypto/Protocol/__init__.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Protocol
copying lib/Crypto/Protocol/AllOrNothing.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Protocol
copying lib/Crypto/Protocol/Chaffing.py -> build/lib.macosx-10.3-i386-2.6/Crypto/Protocol
creating build/lib.macosx-10.3-i386-2.6/Crypto/PublicKey
copying lib/Crypto/PublicKey/__init__.py -> build/lib.macosx-10.3-i386-2.6/Crypto/PublicKey
copying lib/Crypto/PublicKey/_DSA.py -> build/lib.macosx-10.3-i386-2.6/Crypto/PublicKey
copying lib/Crypto/PublicKey/_RSA.py -> build/lib.macosx-10.3-i386-2.6/Crypto/PublicKey
copying lib/Crypto/PublicKey/_slowmath.py -> build/lib.macosx-10.3-i386-2.6/Crypto/PublicKey
copying lib/Crypto/PublicKey/DSA.py -> build/lib.macosx-10.3-i386-2.6/Crypto/PublicKey
copying lib/Crypto/PublicKey/ElGamal.py -> build/lib.macosx-10.3-i386-2.6/Crypto/PublicKey
copying lib/Crypto/PublicKey/pubkey.py -> build/lib.macosx-10.3-i386-2.6/Crypto/PublicKey
copying lib/Crypto/PublicKey/qNEW.py -> build/lib.macosx-10.3-i386-2.6/Crypto/PublicKey
copying lib/Crypto/PublicKey/RSA.py -> build/lib.macosx-10.3-i386-2.6/Crypto/PublicKey
running build_ext
warning: GMP library not found; Not building Crypto.PublicKey._fastmath.
building ‘Crypto.Hash.MD2’ extension
creating build/temp.macosx-10.3-i386-2.6
creating build/temp.macosx-10.3-i386-2.6/src
gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -fno-common -dynamic -O3 -fomit-frame-pointer -Isrc/ -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c src/MD2.c -o build/temp.macosx-10.3-i386-2.6/src/MD2.o
In file included from /Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/unicodeobject.h:4,
from /Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/Python.h:85,
from src/MD2.c:31:
/Developer/SDKs/MacOSX10.4u.sdk/usr/include/stdarg.h:4:25: error: stdarg.h: No such file or directory
In file included from /Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/unicodeobject.h:4,
from /Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/Python.h:85,
from src/MD2.c:31:
/Developer/SDKs/MacOSX10.4u.sdk/usr/include/stdarg.h:4:25: error: stdarg.h: No such file or directory
lipo: can’t figure out the architecture type of: /var/folders/3D/3DvqDPYQGJq4AEDIiWVFb++++TI/-Tmp-//ccCUH9Ax.out
error: command ‘gcc’ failed with exit status 1
Macintosh:pycrypto-2.1.0
May 21, 2010 at 6:53 am
Alsp I looked on my disk and in that path and there is a stdarg.h file there. But I don’t know what the :4:25: means. Is it possibly a date (April 25) as in that case my stdatg.h file is dated May 4 (2007). This is driving me nuts.
May 21, 2010 at 12:47 pm
From http://bugs.python.org/issue6957:
/Developer/SDKs/MacOSX10.4u.sdk/usr/include/stdarg.h:4:25: error:
stdarg.h: No such file or directory
This can be especially baffling to those unfamiliar with nested include
files because the base file stdarg.h does exist in that directory.
The root cause here is that Snow Leopard now uses gcc-4.2 by default but
the 10.4 SDK does not include headers for 4.2. The solution is to force
use of gcc-4.0, the default on 10.4 and 10.5 and included in 10.6, by
defining the CC environment variable:
$ export CC=/usr/bin/gcc-4.0
$ python setup.py install
May 21, 2010 at 7:19 pm
Thank you, thank you, thank you! That was the problem OK. Once I defined that CC environment variable (I only suspect what that means!) verything went very smoothly. I installed Crypto fine and the inept scripts saw it fine and I was able to easily strip the DRM off the couple Adobe Digital Edition that I have purchased. This site does a true service to all — and I want to point out to by all I mean the publishers and authors and sales agents as well as readers. Had this site not been here, or less helpful, I would never, never, never again buy digital editions. Now I can add places that sell such to the places I will shop at and I buy a LOT of ebooks. Thanks again!
May 20, 2010 at 6:28 pm
Hi,
Did you have any success with fileopen plugin DRM?
June 17, 2010 at 6:15 am
Use the ineptpdf 8.3 script (with Fileopen support)
May 20, 2010 at 6:38 pm
Anybody have advice concerning un-DRMing B&N epubs? tools_v1.6b doesn’t seem to recognise the newest B&N Desktop Reader. I use a PC and want to be able to use the B&N ePubs on my EZReader.
May 20, 2010 at 10:13 pm
Does anybody know of a program which will take an unencrypted epub file and turn all the chunks into one unified HTML file? Thanks to all for all of these tools.
May 20, 2010 at 10:37 pm
Hi Paul Howard,
The nest eReader EPUB files do work with the tools from tools_v1.6b
Simply use ignoblekeygen.pyw (works on Macs and Windows) to enter the information to generate the proper key.
Then use ignobleepub.pyw and the key file you generated above, and it does work (at least for me it does).
If you still have errors, please post the exact error message that appears on the screen.
May 20, 2010 at 10:46 pm
I’m embarrassed to say that I must not have been “holding my tongue right”.
I got what I had to work.
May 21, 2010 at 1:18 am
Okay, I’m having an immensely difficult time trying to figure out how I can download books via Kindle’s store and extract the file to be able to convert it and use it with my Sony reader.
Here’s what I have:
Windows 7 64 bit (with UAC disabled)
Kindle 4 PC Version 1.01 Beta
Python V2.6 32 bit
Ereader-Topaz-Mobi-ePub-Kindle-Scripts_1.71
I run the unswindle.pyw file and the Kindle4PC comes up. I open a book, close the Kindle4PC and am prompted to save the book somewhere. I select my desktop, but when I click save, I get the following error:
“Error: problem running MobiDeDRM:
MobiDeDRM v0.09. Copyright (c) 2008 The Dark Reverser
MOBI header length = 232
MOBI header version = 6
Extra Data Flags = 2
Error: no key found, maybe the PID is incorrect.”
I’ve tried upgrading/downgrading my MobiDeDRM.py (or .pwy) between all of the versions I could find (.13 I think, and the ones before .09)
If anyone can be of any help, I’d GREATLY appreciate it! Thanks!!!!
May 21, 2010 at 3:27 pm
Is it possible the ebook isn’t encrypted?
I’ve downloaded a few from Amazon and fought with them for a long time before I realized they just weren’t encrypted at all.
May 23, 2010 at 9:56 am
Nah, I’m using one for testing purposes, and I’m sure it has DRM. I tried popping it into Calibre and converting it, but it said I needed to remove the DRM first.
May 21, 2010 at 4:31 am
I just wanted to thank everyone. I looked through these comments and thanks to all of your work was able to convert some evil Topaz ebooks to epub and my Sony! 🙂
May 21, 2010 at 12:27 pm
Jeff, go to Apprentice Alf’s blog and grab the tools there. The mobidedrm isn’t current (latest is 0.16), but it should have fairly updated files otherwise. They also have skindle, which should work if unswindle doesn’t.
May 23, 2010 at 10:18 am
Hey,
I tried the newest mobidedrm and it came up with the same error.
I also tried downloading Swindle, but whenever I try running the .exe, a cmd window flashes for a brief second than disappears.
I tried typing the full path in a new cmd window, and it came up with this:
“Attempting to locate the kindle.info
Found kindle.info location”
But nothing else after that…
May 24, 2010 at 1:19 am
1. If you are using Windows create a separate directory for skindle such as C:\skindle
2. Copy the book you want to work with into this same directory. Looks like this: B001PGMAV3C_EBOK.prc
3. Open a dos window. Start – All Programs – Accessories – Command Prompt
4. go to the skindle directory. CD C:\skindle
5. To convert type as follows:
For Non-Topaz type this command:
skindle -i inputfile.prc -o outputfile.prc
For Topaz file type this command:
skindle -d -i inputfile.prc -o outputfile.prc
Output file name must be different from input name and no spaces are allowed in the file names.
May 23, 2010 at 4:45 am
Thank you, thank you. I don’t work with commands at all so I did not get it the first time I tried (or the second time for that matter) but after mulling the problem over for a while something clicked and it all fell together. Now I feel safe, knowing that whatever I buy, I can use where I want and keep it as long as I want without having to jump through hoops every time I update my computer etc.
May 23, 2010 at 11:54 pm
1. If you are using Windows create a separate directory for skindle such as C:\skindle
2. Copy the book you want to work with into this same directory. Looks like this: B001PGMAV3C_EBOK.prc
3. Open a dos window. Start – All Programs – Accessories – Command Prompt
4. go to the skindle directory. CD C:\skindle
5. To convert type as follows:
skindle -i inputfile.prc -o outputfile.prc For non-Topaz
skindle -d -i inputfile.prc -o outputfile.prc For Topaz
Output file name must be different from input name and no spaces are allowed in the file names.
May 23, 2010 at 11:58 pm
The example did not show the input samples properly.
the “For non-Topaz” and For Topaz are not part of the typed command.
Command for Non-Topaz:
skindle -i inputfile.prc -o outputfile.prc
Command for Topaz:
skindle -d -i inputfile.prc -o outputfile.prc
May 29, 2010 at 2:18 pm
I have converted topaz books with the topaz tools and mobi books with both swindle and skindle but I was curious to see how skindle handles topaz books. So I got 4 sets of numbers. Volume serial # – 9digits. Device PID -8 digits. Book PID 8 digits. DRM key 16 digits. Says no magic pid numbers. Did leave a book in the file but Calibre couldn’t convert it. So I’m not sure what to do with these sets of numbers since the 10 digit PID wasn’t found. Just curious what to do next. Any ideas. Thanks for the help.
May 30, 2010 at 2:14 am
Larry,
As far as I can tell skindle removes the DRM from a Topaz file so that it can be read on another Kindle. However, it is still a Topaz file which is a very strange file type. It is a collection of scanned images, where each letter and or picture is an image with a specific location on the page. As such it has to be converted into a text before it can be converted into any other file format.
The Topaz tools are the only way I know to make this conversion. You can run the Topaz to HTML then the HTML file can be converted by Calibre.
First you have to set up a directory with the following files:
1. TopazExtract_Kindle4PC.pyw (I do not know if this will work with the new version of the Kindle for PC. )
2. TopazFiles2SVG.pyw
3. TopazFiles2HTML.pyw
4. Subdirectory called lib with all the topaz library files.
Second, run the files in this order:
TopazExtract_Kindle4PC.pyw
TopazeFiles2SVG.pyw
TopazeFiles2HTML.pyw
Make sure you specify and output directory! I use output.
When you have finished the book will be in the output directory as a HTML file. However, there will be other files and subdirectorys which were generated by the conversion process.
Finally, move the HTML file to Calibre (It will show up as a zip file in Calibre) You can then convert it to another file format.
It does a pretty good conversion, however, be aware that it is doing a OCR process to convert the images to text. I had several books where the Chapter titles were in a unique font which it had problems converting. I had to use a html editor to go through them an edit the Chapter headings.
Hope this information is helpful for you.
May 30, 2010 at 3:42 am
Thanks Mark. Glad we have the topaz tools to work with and don’t have to rely on skindle for the conversion . I was curious because the information out there in internet land seemed to suggest skindle would work with topaz if you just included -d in the command line. Obviously thats not quite correct. Not that it matters because most of the online book stores seem to have agreed to all price the books almost the same now. Think there is a name for that. Thanks again for all your help.
May 31, 2010 at 4:24 am
So. I downloaded a B&N ePUB book, successfully converted it, but can’t figure out what the resulting file is. It won’t open in a text or html editor and still appears to be encrypted. I want to format it for Palm eReader, but MakeBook and Calibre don’t recognize it. What now?
May 31, 2010 at 11:24 am
First you need two files to do the job: ignoblekeygen.pyw and ignobleepub.pyw. You must have python installed for these to work. Run ignoblekeygen.pyw first filling in your information to generate your key. Then run ignobleepub.pyw. That has one line for you to find the original epub and you fill in another name for the new epub. Be sure to put .epub on the end of the new name. Click for it to run and it should indicate it is done pretty quickly or else give an error. Then find your new epub and it should run with an epub reader or put into calibre and read there. Is this what you did?
May 31, 2010 at 2:32 pm
Yes. I generated the key and then ran the script. It returned with a message that said conversion successful. I then tried to open the file in a text editor and it still appears to be encrypted.
May 31, 2010 at 4:18 pm
You cannot read the file in a text editor. It is a compressed file which can be opened with an archive operner, at least I can do that with the one I have in Ubuntu. But that is not the way you read them. You can put the file on an ebook reader that does epub. Or you can put it in calibre and that comes with a reader. I’ve used that program in Windows and it works there. On the computer in Linux I use FBReader as well, though I usually read these on my BeBook. FBReader comes in a Windows version if that is what you are using: http://www.fbreader.org/win32/
June 1, 2010 at 1:27 am
Silly me. I finally figured out that I had to give it an .EPUB file extension. The I used Calibre to convert it to .pdb for my Palm eReader. Worked like a charm. Beautiful!!
June 4, 2010 at 1:04 am
I’m clicking on your link for mobidedrm to use with Cabbages unswindle, but it takes me to a page that reads this: ”
– Unknown paste ID, it may have expired or been deleted!”
June 4, 2010 at 6:02 am
http://apprenticealf.wordpress.com/
June 4, 2010 at 6:36 am
Hello Mark,
I was wondering if you can help me. I used your instructions to use Skindle to get rid the DRM from a Topax ebook I bought from Amazon. Running your prompts it said Success! Enjoy! and a similar looking book popped up in my C:\ skindle folder that looks like the first store bought one I placed in there but is the renamed one so I am assuming that part worked. I then went and installed Active Python 2.6 and then downloaded the tools for the HTML conversion. I then tried running Topaz Extract Kindle 4 PC. For input I used the same skindle folder it had generated into and for output I put a folder I made called C:\topazhtmloutput. This is what happened and I am lost at this point:
Command= “python cmbtc dump.py”
Topaz Path Path= “C:\skindle\sixpresidents.prc”
Output Directory= “C:\topazhtmloutput”
First 8 Chars of PID= “”
And then a whole bunch of numbers with this at the end:
Book PID: JKqSuROd
Error: File Extraction Failed
I hope the book did strip the DRM properly, as it said it did. Maybe I am missing something in this whole enterprise? Can someone help a newbie on what to do? I feel embarassed because normally I am tech savvy but this has me over the moon. Thank you everyone!
June 4, 2010 at 7:10 am
Rob
Skindle and Topaz tools are different processes to remove the DRM and are not used together.
TopazExtract_Kindle4PC.pyw is the first step use it on the original topaz file which has drm. It will create a number of files in your output directory.
TopazFiles2SVG.pyw uses the files created by the first process and creates an image of each page of the book.
Finally TopazFiles to HTML.pyw will scan each page image and create an OCR output in a html file called book.html in the output directory.
The book.html file can be copied into the Calibre program where it will show up as a zip file. Then it can be converted into whatever format your ereader can use.
Don’t try to use skindle and Topaz tools together as far as I know they are not compatible.
Hope this clears things up for you.
June 4, 2010 at 7:57 am
It worked. Ok, just got a bit screwed up. So, how what do I do with Calibre to get it formatted? I just need the last final steps.
Also, I can use Skindle for non-topaz DRM amazon ebooks and then just use Calibre right after, right?
June 4, 2010 at 9:22 am
Yes for non-Topaz only use skindle.
June 4, 2010 at 8:09 am
I loaded up Calibre….searched for the folder, and it was not in an archive like people said. This is how it split it up.
Three folders: glyphs, img, and page….img are the pics from the book and the other two are tons of DAT files
Then there are three other things beside the folders: dict0000.dat, metadata0000.dat, and other0000.dat. Just wondering how to go from here.
June 4, 2010 at 8:29 am
Nevermind, it worked! I just went through all the steps again. So, if I have a non-Topaz Amazon book I can just run it through the same process or is skindle better? If I run it through skindle will Calibre convert it right after or do I have to do the whole HTML thing again?
June 4, 2010 at 9:30 am
As the name implies these are Topaz tools. They do not work with non-Topaz files.
Use Skindle or Swindle for non-Topaz files. Swindle does not work with the new updated Kyndle for PC.
Just for the record I did not write or develop any of these scripts. I have only played around with them to figure out how they work with some of the books I have.
I think some people saw the DRM formats as an intellectual challenge to figure out. Once they figured it out and released the tools as they now exist they have moved on to other challenges. So there is little support for the programs outside of the postings here and on a few other similar sites.
June 4, 2010 at 4:12 pm
> think some people saw the DRM formats as an
> intellectual challenge to figure out. Once they
> figured it out and released the tools as they
> now exist they have moved on to other
> challenges. So there is little support for the
> programs outside of the postings here and on
> a few other similar sites.
In general I agree with most of what you say but I think “need” is also a factor. Given skindle exists and works, there is little need for an update to unswindle so their is less interest in supporting it given a solution already exists.
You will find lots of support for ereader to pml and html, mobidedrm, and the topaz tools on this list, simply ask. There is also active support for the inept tools by others who have picked up that banner (see Apprentice Alf’s blog for links to all of these).
And the B&N epub (ignoble tools) work on all platforms (use keygen now for Mac, Linux, and Windows and you can find people willing to help with those tools on this list as well.
It is just that unswindle is kind of superceded by skindle in that unswindle needed to patch the K4PC binary and skindle does not, which makes it more robust in the end. The full source code for skindle exists and it would be almost trival for some good python programmer to convert unswindle.py to use the same approach to get the book pid as is done in skindle if someone wanted to volunteer.
That said, a more important contribution would be to work on either of these projects:
1. convert skindle to work with Kindle4Mac for Mac users
2. work on the FairPlay DRM being used by Apple/iBooks
Neither of these have existing solutions.
Any volunteers?
June 5, 2010 at 12:07 am
My comments were not meant as criticism just stating my theory as to why there is not a lot of effort going into fixing unswindle.
I am playing around with writing a shell program for skindle to make it a little more friendly for people not familiar with working at the DOS prompt. If I get it working I will post a link to it here.
I am just working with it as an intellectual exercise. I am trying to decide if Visual C or Python will be my route. I am leaning toward Visual C since C is what skindle was written in.
June 4, 2010 at 11:42 am
OK, gotcha. Last question….how can I tell if an ebook I just bought, or am going to buy, is Topaz or not? I am referring to the books bought from Amazon. I live overseas in Taiwan and I find the Sony ebook store is just carrying less and less product, so being able to buy from Amaon is amazing. Thanks for helping out. It’s actually quite easy when you start to figure it out but can be tricky from the get go.
June 4, 2010 at 12:12 pm
Some say that Topaz books give the number of pages instead of the file size. I am not sure how to tell before buying. I do know that Topaz books have much bigger file sizes than not-Topaz. Also, the first few bytes of the file identify it as a Topaz book.
We appear to be in the same time zone.
June 4, 2010 at 2:45 pm
Not working anymore 😦
Kindle 2.5 update seems to have changed the drm.
A quiick hex diff between an old (downloaded with 2.3 fw) and new (dl with 2.5) ebooks shows two changes where the keys seems to be.
June 4, 2010 at 3:58 pm
Hi CyranoB,
Use skindle instead of unswindle. It works.
June 6, 2010 at 10:24 am
Hi,
Well this is actually for books coming from Kindle (no problem with K4PC)
Some books are only available for Kindle hardware. It seems the DRM scheme has been changed a little bit, and MobileDeDRM, which was working until 2.3 FW doesn’t anymore with books downloaded from a kindle with 2.5 FW.
I suspect the master key has been changed.
June 6, 2010 at 8:33 pm
Could you post the 2.3 and 2.5 versions of the same ebook somewhere?
(My Kindle is still at 2.3)
Thanks.
June 5, 2010 at 10:08 am
I have written a small program which simplifies the use of skindle for less experienced people. Download at this location and give it a test if you like. I wrote it for Kindle for PC and I don’t know if it will work on non-windows platforms. I have included the source code if anyone wants to play with it on other platforms.
http://rapidshare.com/files/395505897/ezskindle4pc.zip.html
This is only good for 10 downloads but if anyone wants they are free to upload with the tools.
June 5, 2010 at 12:31 pm
I guess I should give a little explanation. the ezskindle4pc program makes skindle very simple. Put it in the same directory as skindle, copy the file you want to remove drm from in the same directory.
Click on the ezskindle4pc file in explorer and it will ask you to enter the file name of the file you wish to remove drm from. Enter the file name press enter and it will do the rest. It will determine if it is Topaz or not and will generate the command line code and execute skindle for you.
The only thing the user needs to type is the input file name. Please read the readme.txt file for full instructions.
June 5, 2010 at 3:07 pm
Hi Mark. Thanks for doing the ezskindle. I did try it, twice, and each time I got the error: Application failed to start because MSVCP100D.dll cannot be found. I unzipped it twice and downloaded it again but the same thing happened. I think this is a great idea. Hope this helps.
larry
June 5, 2010 at 3:33 pm
Hi Mark,
Download limit reached at that site. Once the missing dll issue has been worked out, would someone please anonymously post it to the mediafire site and place a link here and I will see it gets added to the next release of the tools (v1.7).
Thanks!
June 5, 2010 at 10:43 pm
Sorry about not having the dll included. This is the first time I have shared a file generated with this compiler. I have now added the dll to the zip file.
http://rapidshare.com/files/395706560/ezskindle4PC.zip.html
Please give me feedback if there are any more issues.
I think keeping it to a small download limit is good until it is working good.
June 6, 2010 at 12:49 am
I am working on a fix for this problem.
June 5, 2010 at 11:10 am
We are indeed in the same time zone. I am from Alberta, Canada. I also wonder, now that Apple is gearing up their large ebook store, is if people will try to crack that next.
June 6, 2010 at 8:42 am
Ok the situation with EZskindle4PC is that I originally wrote in using Visual C++ express. Which is a free compiler from Microsoft. Because it is a free compiler it does not have the ability to create a version that runs on a computer that does not have Visual C++ express. To create a program that can run on computers with out Visual C++ express Microsoft wants you to pay lots of money for the full version.
So I have backed up and rewrote the program in a very old Turbo C++ compiler. This resulted is a few problems and issues. Since it was from the days when DOS only allowed 8 character file names I had to do some trickery in order to open the files to see if they are Topaz. The limitation is that the program will only accept prc files as input files (As far as I know all files downloaded to Kindle for PC have prc file extensions). And if you have multiple prc files in the directory and the first 6 characters of the files are the same it will have problems. As long as the first 6 characters are different there should be no problems.
If anyone needs to use different file extensions please let me know and I will see what I can do to make the change.
http://rapidshare.com/files/395830418/EZskindle4PCv1_0.zip.html
Please give me feedback on how this works for you. Also, if anyone knows of a nice free C++ Compiler let me know what it is.
June 6, 2010 at 11:17 am
“Also, if anyone knows of a nice free C++ Compiler let me know what it is.”
gcc/g++
June 6, 2010 at 3:49 pm
Mark, I tried your ezskindle again. I’m assuming the book should be in the same directory as skindle. So I got this message: An installable virtual device driver failed DII initialization. Choose close to terminate the application. So I chose ignore instead and the process worked. I used Calibre to convet the book to .lrf and it worked fine also. Hope this helps.
larry
June 7, 2010 at 12:49 am
I’m assuming the book should be in the same directory as skindle. Yes you assumed correctly.
Not sure what the message is about. I only have 2 computers to test on both running XP. If you are not running xp I think the message may be because the program is now compiled as a 16 bit program not 32 bit. I am still looking for a free compiler I like – or a way to export from VC express. I wish I had remembered to bring my disks for the full featured Visual C++ when I retired to the Philippines.
How many books did you try?
What do you think of its functionality? Do you think it makes skindle easier?
New Link:
http://rapidshare.com/files/396102488/EZskindle4PCv1_0.zip
I would appreciate feedback from other people
June 7, 2010 at 3:14 am
Hi Mark,
Some thoughts …
If you are all all interested in python, it should be easy to take any of the front-end gui programs from the tools v1.6b and modify them to invoke skindle instead of the Topaz Scripts, eReader scripts, etc. All they are is a basic widget set to gather info to launch command line programs via a subprocess and pipe. It is the command line programs that do all of the work.
Also, if you are interested you can get cygwin (same compiler used by skindle) to run on Windows (and it is free) that includes gcc/g++. That way you could compile both packages with the same Windows compiler.
June 7, 2010 at 3:33 am
Hi Mark. I tried three books and got the same message each time. I run xp also. Regardless, when I choose “ignore”, the program works very well. It certainly is much easier to use. Thankyou for your effort.
larry
June 7, 2010 at 3:58 am
Larry
Thanks so much for providing additional testing. Unfortunately, nearly all programs show some bugs when transferred to end users computers.
Some_updates
I have never used Python so that would require a steep learning curve. But I may try later just to learn Python. Thanks for suggesting cygwin but a little late I found Dev-C++ which I have used for the latest version 1.1. I think this will get rid of the error message Larry was seeing and it fixes the handicap that was required to run on the antique compiler.
Latest version is available at:
http://rapidshare.com/files/396137127/ezskindle4pcv1_1.zip
Larry do you think you can run a few more tests for me? Also, anyone else who tries please post if you have any error messages.
I have added information that shows when my program calls skindle and when it resumes so that we can see if the error is mine or skindle’s.
June 7, 2010 at 6:18 am
Hi Mark. I tried the new ezskindle and it works great. There were no error messages and the process was quick and easy. I believe you’ve done it. Yeaaa!!Thanks again for doing this. It will really help a lot of people.
lar
June 7, 2010 at 6:29 am
Thanks for doing testing for me Larry. Without your feedback I would not have been able to put out a working version for public use.
My next phase will be to see if I can get it to convert all the books in a folder without any input from the user that will be ultimate in easy conversion.
June 6, 2010 at 10:23 am
Hi,
Well this is actually for books coming from Kindle (no problem with K4PC)
Some books are only available for Kindle hardware. It seems the DRM scheme has been changed a little bit, and MobileDeDRM, which was working until 2.3 FW doesn’t anymore with books downloaded from a kindle with 2.5 FW.
I suspect the master key has been changed.
June 6, 2010 at 7:57 pm
Hi Cyrano,
So you think the update to kindle (standalone) has broken the simple use of a single PID?
This is the first I have heard of that. If so, I expect they have moved to a book speciific PID similar to how K4PC now works.
I think if you look at how the device PID is used along with data from the book itself to create the book specific PID, it might work. See the skindle source code to see how that is being done.
Sorry, I can’t be much help with this as I do not have a standalone Kindle.
Please let us know what you find out.
June 7, 2010 at 9:52 pm
My Kindle updated to 2.5.2 today. Comparing books downloaded for 2.3 with books downloaded now I have 2.5 (the same book), it does appear that the PID for the book has changed. Whether this is just a change in the PID used for each Kindle (which would need an update to kindlepid) or whether this is a change to a per-book PID, rather like for Kindle for Windows/Kindle for Mac, I don’t know.
Apprentice Alf has posted a note about it at his blog. Hopefully someone will work out a fix soon.
Until then, downloading to Kindle for Windows and using skindle seems to be working.
June 7, 2010 at 11:35 pm
I pulled a couple of books off of my Kindle while I still have 2.3. When I get the 2.5 update, I’ll re-download them and hopefully have some time to play around with the differences.
I’m thinking I should be able to download books from my Amazon account page directly to my computer and still use MobiDeDRM, right? So at least we have a couple of options until this gets resolved.
June 8, 2010 at 12:31 am
If it’s a change used in the pid for each Kindle, that will probably break the tools used for topaz conversion as well. Can anybody that has 2.5 get a hold of a topaz book and verify this?
June 8, 2010 at 5:25 am
I have taken down the link for EZskindle4PCv1_1 I am now up to ver 1_1_1 fixed minor typo in text output.
I have now created what I call Lazy skindle ( LZskindle4PCv1_0.exe ) this version will convert multiple books and requires no typing. Great for the typing impaired. Just a double mouse click to run.
LZskindle4PCv1_0 The Lazy skindle program for those who are typing impared
To setup:
1. Create a new folder: example C:\skindle
2. Place LZskindle4PCv1_0.exe and skindle.exe in this folder.
3. Create 2 subfolders: C:\skindle\input
and C:\skindle\output
To run:
1. Copy the book(s) you wish to remove DRM from into the input directory (leave the originals in my kindle folder).
2. Double click on LZskindle4PCv1_0.exe
3. A DOS window will open and will show skindle being called for
each book in the input directory.
4. The books with the DRM removed will now be in the output directory.
If the program is run with no files in the input directory you will get a
“File not Found” error and the program will terminate.
PLEASE USE ONLY FOR YOUR PERSONAL USE – ON BOOKS YOU PAID FOR!!
This program is provided to allow you to archive your library in a format that will outlast the kindle. Technology moves on and you should not have to reinvest in an entire new library when the Kindle is obsolete. Please do not use this program for piracy.
The EZskindle4PCv1_1_1 is also included in the download for those who just have to type for themselves.
http://rapidshare.com/files/396544256/LZskindle4PCv1_0.zip
June 8, 2010 at 10:26 pm
Hi Mark. I tried LZSkindle but got an error: file not found. I made the two subdirectories and copied the book to the input directory. Not sure what the problem is. I did try EZSkindle1_1_1 and it worked great with the book in the same directory. I’ll be glad to try anything you suggest.
lar
June 9, 2010 at 1:03 am
Larry,
Thanks for the feedback. It is hard to fix errors when I don’t know about them.
When I wrote my program I assumed that all Kindle for PC application books used the prc file extension. This may have been a bad assumption on my part. Did the book you tried have a different file extension? If so then maybe I need to change my assumption.
I did write the program for use with kindle for PC application books.
If that is not the case I will maybe need more information. Make sure that the input and output directory are sub-directories of the directory where LZskindle4PC is located. They also need to be called input and output. (Not trying to say you made a mistake but need to know exactly how you have it on your computer)
My program makes a file called books.txt with list of all the books in the input directory. This list is what it uses to send command lines to skindle program. I only make a list of books with the prc extension so maybe that was a bad assumption on my part. I made this assumption because all of the books in my kindle for PC library have the prc extension. Let me know if you have a different file extension.
The other problem may be that the program is not been able to create the books.txt file on your computer.
Try this for me. Make sure you have some books with .prc extensions in the input folder. Open a DOS window and go to the skindle directory. Then type the following and press enter. dir /b .\input\*.prc > books.txt
This should create a text file with a list of your books in it. Let me know if that works.
I really appreciate you willingness to test this program.
June 9, 2010 at 2:53 am
Hi Mark. You are right about the extentions. The boos that I downloaded from Amazon were of two types. The topaz books were .prc and the non-topaz books were .azw. I was trying the non topaz books and they didn’t work because of the extention. I tried a topaz book with .prc and it worked as far as your program works, showing a list of books in the output file. The problem, of course, is that skindle doesn’t really strip the drm fom the topaz books. The topaz tools seems to be the way to go with them. If you can include .azw in your code then I think it will be perfect. I don’t know if there are other extentions that come from Amazon for K4pc. Hope this helps. Good luck.
lar
June 9, 2010 at 4:06 am
Larry,
Thanks for the feedback. I will work on a solution. The problem being I now need to write code to reject file extensions that are not usable with skindle. I now know that it will take prc and azw inputs but do not know of others that might work. So I need to test to see if skindle has problems with other file extensions.
In the mean time try to rename the file with a prc extension this should work.
June 9, 2010 at 5:26 am
Larry,
Actually skindle does strip the drm from Topaz files. This allows them to used on any kindle not just the one it was purchased for. However, that being said Topaz is only readable on kindle software and it is not convertible to other formats by Calibre which is why the Topaz tools.
I did some testing and skindle will take any file extension you put on a valid file. It looks at the file content to determine if it is a valid file type. If for example you try to run a BN book it it will give and invalid file message and quit even if you gave the file a kindle extension. So I don’t really need to do a lot of checking of the file extensions so now I am just taking whatever you put in the input directory. I am letting skindle sort out if it is a valid file.
Here is the link for the new version 1_1
http://rapidshare.com/files/396930650/LZskindle4PCv1_1.zip
June 9, 2010 at 3:49 pm
Hi Mark. Works great !!! I just assumed that because the topaz file wouldn’t be converted by Calibre that it didn’t work with skindle. My mistake. Thanks for your hard work. You have helped a lot of people with this.
lar
June 9, 2010 at 1:17 am
So is anyone working on freeing up Apple iPad books yet? IPad has two million units sold now so could help a lot of people!
June 9, 2010 at 5:32 am
I do not know of any work on the ipad drm. Try a google search on Ipad DRM maybe you will get a hit.
June 9, 2010 at 11:50 pm
I had an interesting problem with TopazFiles2SVG (from Topaz_Tools in tools_v1.6b).
The Topaz_Tools had worked trouble free for several Topaz files until I attempted to convert Mike Resnick’s Flagship (Starship 5).
When running the TopazFiles2SVG, I got the error message below.
Error Message
Traceback (most recent call last): File “lib\gensvg.py”, line 405, in
sys.exit(main(”)) File “lib\gensvg.py”, line 329, in main flat_xml =
convert2xml.main(pargv) File “C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 789, in
main xmlpage = pp.process() File “C:\Documents and Settings\Drak
Bibliophile\My Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”,
line 703, in process tag = self.procToken(self.dict.lookup(v)) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in
procToken subtagres.append(self.procToken(self.dict.lookup(val))) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in
procToken subtagres.append(self.procToken(self.dict.lookup(val))) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in
procToken subtagres.append(self.procToken(self.dict.lookup(val))) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in
procToken subtagres.append(self.procToken(self.dict.lookup(val))) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in
procToken subtagres.append(self.procToken(self.dict.lookup(val))) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in
procToken subtagres.append(self.procToken(self.dict.lookup(val))) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in
procToken subtagres.append(self.procToken(self.dict.lookup(val))) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in
procToken subtagres.append(self.procToken(self.dict.lookup(val))) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in
procToken subtagres.append(self.procToken(self.dict.lookup(val))) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in
procToken subtagres.append(self.procToken(self.dict.lookup(val))) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in
procToken subtagres.append(self.procToken(self.dict.lookup(val))) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in
procToken subtagres.append(self.procToken(self.dict.lookup(val))) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in
procToken subtagres.append(self.procToken(self.dict.lookup(val))) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in
procToken subtagres.append(self.procToken(self.dict.lookup(val))) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in
procToken subtagres.append(self.procToken(self.dict.lookup(val))) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in
procToken subtagres.append(self.procToken(self.dict.lookup(val))) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in
procToken subtagres.append(self.procToken(self.dict.lookup(val))) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in
procToken subtagres.append(self.procToken(self.dict.lookup(val))) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in
procToken subtagres.append(self.procToken(self.dict.lookup(val))) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in
procToken subtagres.append(self.procToken(self.dict.lookup(val))) File
“C:\Documents and Settings\Drak Bibliophile\My
Documents\tools_v1.6b\tools\Topaz_Tools\lib\convert2xml.py”, line 140, in
lookup print “Error – %d outside of string table limits” % valTypeError: %d
format: a number is required, not NoneType
Error: SVG conversion Failed
June 10, 2010 at 12:30 am
Hi Paul,
That error typically means that your book has an xml token or keyword that we simply have not seen before.
if you look at the conversion log you can find the very last file that was being converted that caused the problem. Either a page****.dat or a glyphs***.dat file.
If you can post an archive that inclues the dict0000.dat and that file, I should be able to identify the new token and modify the code to recognize and process it. Just post a link in reply.
June 10, 2010 at 12:57 am
Ouch! I don’t have a website or know anyplace where I could put the files (I know what page caused the problem).
June 10, 2010 at 1:59 am
I hope this works: http://www.mediafire.com/drakbibliophile
June 10, 2010 at 3:15 am
By the way some_updates, drakbibliophile (me) is the Paul S Howard who had the problem with TopazFiles2SVG. Please see the mediafire ‘site’ for the files.
June 10, 2010 at 1:45 am
I have not received any more bug reports so here is the final upload for EZskindle4PC and LZskindle4PC
http://www.mediafire.com/file/jbcmjmjjzdg/LZskindle4PCv1_1.zip
June 10, 2010 at 11:52 am
Hi Paul,
Grabbed them and I will look at them this evening after work and get back to you.
Thanks,
Kevin
June 10, 2010 at 1:49 pm
Hi Paul,
No new token. The problem is that the dict000.dat file and the page0012.dat file do not come form the same book (or one or both are very corrupt)
So either the dict0000.dat file was left over from earlier conversion effort or the page0012.dat file was left over from a previous effort (perhaps the new book only has up to page0011.dat?).
So please try reconverting this book but starting with a completely new output directory so that there are no older files left lying around anyplace.
If you still run into trouble, then either something has corrupted those files (you did not use skindle on them I hope! – skindle and the topaz tools are not compatible) or something is really strange and I will need to see more of the book to tell exactly what is going on.
Please let me know what you find out.
June 10, 2010 at 2:02 pm
The “fun” is that I created a seperate Output folder for each Topaz file.
After the first time it happened, I deleted everything in that Output folder and tried again with the same results.
I did run it just now with a completely new Output folder and it happened again.
Now, I noticed that Flagship didn’t download automatically when I opened K4PC.
Do you want me to upload the Flagship-Out folder to Mediafire?
June 10, 2010 at 2:44 pm
Wow! So they do come from the same book. Very strange. That only leaves corruptions.
So please regrab and download a new copy of the book on your K4PC. Open it under K4PC and make sure you can read through and past the first 14 or 15 pagges just to be safe and to prove there is no corruption in the book.
Then run the TopazFiles on it and zip or rar up the output folder and post it for me. You can pm “KevinH” on Mobileread with the link if you do not want to post it openly on this blog.
I will try to see if I can figure out what has changed.
June 10, 2010 at 3:06 pm
Hi Paul,
Ignore my earlier reply. I went and looked more closely and the damn file has a new “magic” string that starts it. Instead of __PAGE___ it has _PAGE__ so it was not recognized as a “page” and that confused the hell out of the parser and so it started sending out bunk which convinced me the dict or page were bad.
I have posted a trial convert2xml.py file to see if this fixes your issue:
Please grab it from here:
http://pastebin.com/zDWTA8pU
name it convert2xml.py and then copy it into Topaz_Tools/lib/ to replace the file of the same name.
Then try rerunning the conversion to svg and html and let me know if this does not fix the problem.
If it does fix it, I will release a new version of the tools with this fix and a few others soon.
Thanks for reporting the problem and working with me to help fix it.
June 10, 2010 at 4:18 pm
some_updates, that did the trick!!!! Thanks for the help!!
Paul (Drakbibliophile) Howard
June 11, 2010 at 1:42 pm
There is a new version of the tools available:
tools_v1.7.zip
The changes include:
1. updated to version 16 of MobiDeDRM which speeds up conversions (Thanks Alf)
2. updated to the latest version of mobiunpack.py (Thanks Paul)
3. added latest version of LZSkindle tools (Thanks Mark)
4. bug fixes for Topaz conversions with strange “magic” file type “indicators”
The old version has been removed.
The URL is:
http://www.mediafire.com/?yywynwqoqtm
June 14, 2010 at 3:33 pm
This morning I downloaded a book from Amazon and went right to the new tools 1.7 and LZSkindle4pc. Didn’t work. I knew that LZSkindle works so I tried it in one of my older files with LZ and it worked fine. So after trying all the usual redownloads and copying stuff, I remembered Mark saying that skindle.exe must be in the same directory. So I checked and my LZ file didn’t have Skindle.exe so I copied it and it works fine. Don’t know if my download messed up or it was missing from the new tools 1.7. Anyway, now I’m reading the book.
lar
June 14, 2010 at 4:21 pm
Hi Larry,
That’s not a bug but a feature 😉
Not everyone uses LZSkindle (yet!) so I did not package skindle.exe inside of LZSkindle, instead I placed it in its own folder so that people could find it. You still have to read the follow the LZSkindle instructions to get things to work.
Hope that explains things.
June 12, 2010 at 9:54 pm
I have some ebooks I bought from B&N(pdb) and I’m trying to remove the DRM.
so far……
Step 1
At the command line type ereader2html.py –Enter
Step 2
ereader2html input_filename.pdb outdir “your name” credit_card_number –Enter.
Error: ‘ ereader2html.py:8: Deprecation: the sha module is deprecated; use the hashlib module instead
import struct, binascii, zlib, os, sha, sys, os.path
eReader2Html v0.3. Copyright 2008 The Dark Reverser
Converts eReader books to HTML
Usage:
ereader2html infile.pdb outdir “your name” credit_card_ number
Note: It’s enough to enter the last 8 digits of the credit card number
June 13, 2010 at 1:56 pm
Hi Kevin,
Two things:
1. make sure you are using python 2.X and NOT python 3.0. None of the scripts work with the upcoming Python 3.X series yet as far as I know.
2. ereader2html.py is now old and no one seems to be keeping it up actively. I would use erdr2pml.py to convert it to pml and then xpml2xhtml.py to convert the pml to proper xhtml.
Both of these tools exist in the latet tools_v1.7.zip archive
http://www.mediafire.com/?yywynwqoqtm
and both have nice GUI front-ends so all you need to do is double-click and go.
But everything here relies on 32 bit python 2.X being installed and not the 64 bit version or the upcoming python 3.x series
June 13, 2010 at 1:54 am
I have found one Kindle for PC book which skindle does not correctly remove DRM. Unswindle did correctly remove the DRM.
The book is a free book called: “The Static of the Spheres”
If the author of skindle wants to take a look at it.
June 13, 2010 at 1:52 pm
Mark,
Are you sure that book is even DRM encoded?
June 13, 2010 at 2:04 pm
Calibre says it is DRM’d. It is a very small book only 64kb. Skindle gets about 1/4 through then fills the rest with garbage. Unswindle is able to remove the DRM.
Ran into it while testing LZskindle. Get the same result when trying from the command line.
June 13, 2010 at 6:05 pm
Hi Mark,
Maybe it is an issue with how they converted MobiDeDRM.py to “C”. Please try one thing, use skindle with that book and just have it print out the book specific PID, Then try that PID with mobidedrm.py and see if it properly removes the drm from the book.
If so, then the problem is most probably in the C equivalent of MobideDRM inside the skindle code and that should help narrow things down.
June 14, 2010 at 1:22 am
I tried to run MibiDeDRM but was not able to get it to work. I have never used MobiDeDrm and I am not sure if it was me or the software or the book. I was only getting a 8 char pid and the software was asking for 10 so maybe that was the problem.
It also may be just a problem with the book having some unique character or symbol that is causing the problem.
June 14, 2010 at 10:16 pm
some_updates
Is anyone maintaining the skindle code? Or has it become an orphan program?
If someone else is maintaining it I do not want to muck around in their code.
June 15, 2010 at 1:20 pm
Hi Mark,
I am not sure. I think “BartSimpson” does maintain the skindle code (he is the author) but I do not think he reads or watches this website or even Apprentice Alf’s site. He posted his original code on OpenRCE .
see http://www.openrce.org/forums/posts/1199
I think Clarknova found and reported a bug to him on that site (see last post) but I am not sure if it was ever incorporated into any release.
So there might be an active maintainer for skindle, but someone would need to track him down and send him the bug report.
So if anyone knows someplace a post can reach “BartSimpson” then please let this list know.
June 14, 2010 at 10:46 pm
For anyone looking into the Kindle 2.5 update/mobidedrm issue:
Against my better judgement, I updated my Kindle to 2.5.2. Before updating, I downloaded one of my books from Amazon to my computer and verified that mobidedrm v0.16 worked. I proceeded with the update and 10 minutes later, downloaded the same book from Amazon to my computer and verified that mobidedrm v0.16 does NOT work (as I expected).
I don’t claim to know anything about encryption, but I’m getting some differences in debug output (between the two files) before mobidedrm even uses the pid. The drm_ptr variable has a different value before it and the pid (and others params) are passed to the parseDRM function. The portion of the code referred to can be seen at http://pastebin.com/hw0fikXt
Hopefully, someone smarter than me can use this information toward a solution… because skindle is not an option for me right now.
June 15, 2010 at 1:41 am
Just out of curiosity DiapDealer, did you allow the 2.5 upgrade to occur “naturally”over the Whispernet or did you download the installation binary and install it manually? I ask because I suspect that the Whispernet procedure allows Amazon to know that you can handle the new form of DRM that they have developed. If you downloaded the installation file and did the installation manually with Whispernet disabled then I suspect that Amazon would be unaware of your upgrade and continue to provide you DRMed files in the old format. If I am correct in my suspicions then once you enable Whispernet it would just be a matter of time before your upgrade status would be passed back to Amazon and the new DRM would be in effect for you.
By the way, can anybody confirm that older ebooks obtained before the 2.5 upgrade are still readable under the new regime with their older DRM still intact?
June 15, 2010 at 3:20 am
I downloaded the file and updated manually. You could be right about maintaining the status quo by leaving Whispernet disabled during and after a manual update. Unfortunately, someone else will have to test your theory as I had my wireless turned on throughout the update.
As to your question about ebooks obtained before the 2.5 upgrade… the answer is yes. I pulled several drmed books off of my kindle while I still had 2.3. After the 2.5 upgrade (and with Whispernet off) I transferred those drmed books back onto the device and had no trouble reading them.
June 15, 2010 at 12:00 am
FWIW, I just “upgraded” to 2.5.2 and books I download now will not convert using a Mobidedrm plug-in in Calibre. This is ridiculous, Amazon. All you have accomplished is now I will buy ebooks from B&N or Fictionwise or other sites we can still de-DRM.
June 15, 2010 at 2:06 pm
Well, I succeeded in tracking down the old 5.2Mb beta of K4PC and got it working with Wine 1.1.42 on Ubuntu. I was able to register it with Amazon and send ebooks to it and read them just fine.
Skindle also seems to work in Wine — at least it finds the kindle.info file and parses it ok. It finds a device pid and a book pid, but still can’t find any valid keys for the DRM.
Since no other versions of K4PC seem to work with Wine, this was my last hope for a workaround to the 2.5 update issue. For now, this Linux user is well and truly screwed. Won’t be buying any new Kindle books for me for a while.
June 16, 2010 at 12:28 am
Any book downloaded after 2.5.2, even those bought before and downloaded from Manage Your Kindle, will not work with Mobidedrm. Have to use skindle.
June 16, 2010 at 1:10 am
I think you you need to clarify your remarks. Did you update manually? Did you have Whispernet on at any time before or after the update? If one leaves WN off and it remains off, then the device will not communicate with AZ and AZ will not know you have updated manually and, presumably, subsequent purchases will be based on the older firmware (2.3.).
June 18, 2010 at 2:53 am
I did update manually, but had my WN on. I was just verifying that even a book purchased before 2.52 carries the new encryption if it is downloaded after 2.52. This must be why I had to redownload my Topaz books, which gave me a “licensed to a different device” error.
June 16, 2010 at 1:26 am
Trying to use EZ or LZ skindle and get the following error. Any idea what I am doing wrong? Thanks for your help.
Attempting to locate kindle.info
Found kindle.info location
Using UserName = “xxx”
Using SystemDrive = “C:\”
Using VolumeSerialNumber = “XXXXXXXXX”
Device PID: NBHJ2XXX
PID for Cry_Sanctuar-ck_Pass_series is: W7mAkXXX
No valid pids available, failed to find DRM key
An error occurred, unable to process input file!
The command line used was:
skindle -i test.azw -o test.azw
the XXXs are my doing not the generated error code. I am not sure how much those PIDs and the volume serial number share about me.
June 16, 2010 at 1:44 am
> skindle -i test.azw -o test.azw
Did you really use the same name for the input file and the output file. That is not a good idea. If so, you may have corrupted the original file since you tried to write to it.
I would download a new copy of the file and try skindle again, but this time use a new name for the output file.
June 16, 2010 at 1:45 am
well i found out what i was doing wrong. I had downloaded the book to be transferred by computer not to Kindle4PC. Brain in neutral today.
June 16, 2010 at 2:33 pm
I’ve done a hexdump of an ebook delivered to the Kindle via Whispernet while still on version 2.3. Then I did a hexdump of the same ebook (delivered via Whispernet) after the upgrade to 2.5.2. I don’t know if this will help anyone working towards a solution, but I thought it would at least allow people who don’t have a Kindle to offer ideas. I uploaded this info to MediaShare if anyone wants to take a look. I also included a diff file that shows differences between old and new. http://www.mediafire.com/?2nmzjnjummi.
My skills are limited but I’m willing to share my feeble attempts at a solution. 😉
June 16, 2010 at 5:02 pm
The differences between the two files are very subtle. I was able to open the two files in vbindiff and edit the new file to match the old one within two minutes. MobiDeDRM v0.16 then worked fine on it obviously. There’s a couple of bytes that are different early on… then the rest seems to be a case of the data being shifted.
I don’t know exactly what it all means, but it gives me hope that a new solution won’t involve completely overhauling the existing process. But I admit I’m in way over my head here.
June 16, 2010 at 7:03 pm
Hi Diap Dealer,
That is good news actually. If the encryption key was changed then the bulk of the content of the two files would be very very different, not just a few bytes changed and/or shifted. My guess is they have simply wrappered the file in some way (perhaps to store information on collections?) and that the all we must do is figure out how the data has been shifted or wrappered and then write a program that unwraps/unshifts it before passing it through the standard mobidedrm with the original kindle PID.
Do you have a second pair of books (ie. two different version os the same book) that you can binary diff again so that we can see which bytes are being used and if the offset into the files of those changed bytes are identical or not.
If the offsets are consistent we could write a quick and dirty unwrapper for people to test with.
June 16, 2010 at 8:09 pm
I have other books, but I chose the smallest for that first compare. This one is slightly bigger. I apologize.
I really don’t know what I’m looking at in the data, but it seems to me that the differences – again – are limited to one small section. But I don’t know if they are consistent. That’s for greater minds!
Anyway… http://www.mediafire.com/?kjiunqbmcin
June 16, 2010 at 8:40 pm
I’m uploading a third set of diffs for good measure… and because even I can begin to see some consistency in where the data changes.
Especially with these last two that I’ve looked at. I think the first one had some slightly different metadata that should be no issue. These last two definitely look like consistent changes to a specific record.
http://www.mediafire.com/?my1mydlhdtc
June 16, 2010 at 10:55 pm
Hi DiapDealer,
It seems all of the serious changes are things added to the end of the EXTH (extended header).
From looking at your diffs, it appears that most of the differences seem to be related to a cover or thumbnail image. Only the last change seems to be something meaningful (as far as I can tell, but this is a guess on my part).
From your take2 file, can you try only changing this very last diff and see if mobidedrm will successfully work with only this one change:
246,248c246,248
< 00000f50 39 00 00 00 da b8 d7 34 a3 b0 8c 0a d6 13 ec d4 |9……4……..|
< 00000f60 a7 90 fa d6 28 15 cd f6 d3 49 06 ce 63 38 05 bb |….(….I..c8..|
00000f50 f8 00 00 00 d2 86 0d 01 5a 11 5f 28 30 66 12 66 |……..Z._(0f.f|
> 00000f60 87 09 17 1c fb 35 52 46 a1 2f 1a 9b db dc e3 84 |…..5RF./……|
> 00000f70 22 78 84 82 00 00 00 00 00 00 00 00 00 00 00 00 |”x…………..|
Please let me know if that one change is enough. I can write some code for you to run to decode the EXTH changes that are going on and we can see what if anything is confusing the mobidedrm script.
June 16, 2010 at 11:36 pm
some_updates,
I think copying and pasting might have messed up the changes you posted. So, for the take2 file, I took the very last diff (three lines different between the files) and made the 2.5.2 version match the 2.3 version. The good news is… mobidedrm decrypted the 2.5.2 version just fine after that small change! Now, all we have to do is determine whether I did what you wanted or not. 😉
My last diff for take2 looked like this (crossing my fingers that the copy paste works):
246,248c246,248
< 00000f50 39 00 00 00 da b8 d7 34 a3 b0 8c 0a d6 13 ec d4 |9……4……..|
< 00000f60 a7 90 fa d6 28 15 cd f6 d3 49 06 ce 63 38 05 bb |….(….I..c8..|
00000f50 f8 00 00 00 d2 86 0d 01 5a 11 5f 28 30 66 12 66 |……..Z._(0f.f|
> 00000f60 87 09 17 1c fb 35 52 46 a1 2f 1a 9b db dc e3 84 |…..5RF./……|
> 00000f70 22 78 84 82 00 00 00 00 00 00 00 00 00 00 00 00 |”x…………..|
2.3 file was first… 2.5.2 second. I changed 2.5.2 to match (just those three lines) the 2.3. Did I do what you wanted?
Thanks for your help! You can PM me on MobileRead (DiapDealer) if you need anything emailed… or here works too. Whatever’s easiest.
June 16, 2010 at 11:39 pm
Yeah, the copy/paste isn’t working so hot for that diff data, but I think we’re on the same page.
June 17, 2010 at 12:56 am
Hi DiapDealer,
Yes, I tried to copy and paste the last pair of changes from your diff set but it got all messed up. But it sounds like you did the exactly what I asked you to do and from the sounds of things it worked!
So something in that small change is what we have to figure out.
We have some python code that Paul and I wrote that unpacks Mobi files that can be tweaked to examine if that changed data is really in the EXTH section and what if anything it might represent.
The link to the python code to unpack a Mobi file is:
http://code.google.com/p/ebook-conversion-tools/source/browse/lib/mobiunpack.py
We can use it as a model to look at the EXTH header info and see what is anything changed. This mobiunpack.py program only works for no-drm files but for the time being, we can ignore that and have it dump the EXTH header and Mobi Header info.
If Paul Durrant does not beat me to it (I believe he knows this code and the Mobi file format much better than I do – and he has a standalone Kindle), when I get some time on Friday, I will try to hack it to do what we want.
Sorry but I am tied up all day tomorrow (Thursday). But it sounds like we are close and we should be able to figure out what is different and a way to work around it. The nice thing is the encryption does appear to use the exact same old PID as before so this should be doable.
June 17, 2010 at 2:09 am
Good deal.
Let me know if you don’t get my PM with the requested info on mobileread. My sent PM’s aren’t showing up.
June 17, 2010 at 7:23 am
It’s not a surprise that most of the file is identical. The actual encryption of the text is done with what is (I beleive) essentially a random encryption key. The PID is just a key to decode the actual encryption key. That’s how a Mobipocket file can be encrypted for use with more than one device – up to four different PIDs can be used to encrypt four separate copies of the actual encryption key.
So – there’s about 48 bytes after the EXTH headers that contain the PID-encrypted encryption key. It’s this small area of change that will copy the PID from one file to another. But it will, of course, only work on the same book – the underlying encryption key has to be the same. (The offset to the encryption bytes is given in the MOBI header. See the Moobileread wiki for details.)
It seems certain that the new Kindle firmware is allowing Amazon to change the PID used in books downloaded for Kindles with the new firmware. I suspect that Amazon have moved to some kind of book-specific PID, as for Kindle for Windows and Kindle for Mac. It’s just possible that they’ve only changed the way the PID is generated from the serial number, but I doubt it.
Theoretically, it would be possible to disassemble the code in the 2.5.2 update, and work out exactly how this new per-book PID is being generated, and what info from the Kindle is being used. I’m not able to do this.
It might also be possible to work it out by knowing what the new PIDs are for various newly downloaded books. This is hard, since we don’t know the PIDs.
Hopefully someone with an interest in hacking the Kindle is looking at this. I haven’t heard anything though.
June 17, 2010 at 10:19 pm
Hi Paul,
Thanks for explaining that. So the diffs simply will not help us then at all.
I guess I will take a long look at the skindle source code to see if I can figure out the algorithm for creating the new PID. I looked at it a bit and it seems to use data from exth item 208 or item 209 to generate the book specifc pieces (using sha1 sums to create the key value).
We really will have to wait until someone cracks the new kindle firmware and searches for a kindle info file and then disassembles the source to see how things are done.
June 18, 2010 at 1:05 pm
I think I just died a little on the inside.
Oh, well… gotta appreciate a challenge.
June 19, 2010 at 6:46 pm
DiapDealer
Have you tried to tst out any topaz files with a before and after test or do you have just awz files? I was just wondering.
I also was thinking about you reverting to the previous 2.3 vesion of the Kindle software and then applying one of the kindle hack mocifications that will give you an alternative font. I believe that Amazon’s automatic upgrade feature will not attempt to overwrite one of the hacked installations. I that is truly the case then once you connect up through WhisperNet Amazon should detect you altered status and reverse their changes to your ebook DRM style.
Do you have your previous binary installation file?
June 20, 2010 at 3:01 pm
Yes, the topaz tools (at least for linux users) do not work for ebooks downloaded after the 2.5 upgrade (as expected). I’ve not compared the binaries because it seems rather fruitless at this point.
Somebody else will have to test whether back-revving to 2.3 would actually reverse the changes on ebooks downloaded. I don’t mess with the Kindle itself – through hacks, jailbreaks or back-revving. I use it too much. I couldn’t afford to replace it if I bricked it.
Plus, I don’t like to go backwards… “never straight, but forward” has always been my policy 😉
June 21, 2010 at 6:35 pm
I downloaded the tools_v1.7 programs mentioned in Apprentice Alf’s blog. Thank you everyone for creating and maintaining these programs. I’ve done one test run with eReaderPDB2PML on a DRMed PDB file and the book decrypted perfectly. I lost all the metadata, however. Since I have 70-80 PDB books to decrypt and just spent most of yesterday collecting all of the metadata for them that’s a bit frustrating. So 2 questions. First, is there a way to do this that will keep the metadata? Second, is there a batch conversion method?
I notice there’s another program called eReaderPDB2PMLZ. Should I use that instead?
June 21, 2010 at 10:35 pm
Hi,
The official pdb format metadata is not lost. It is typically encoded in a comment as one of the top lines in the pml file. If you convert this to xhtml using the Pml2HTML.pyw program, the metadata is converted to meta tags in the output html. Sigil will recognize these meta tags and the official metadata should also make the leap to epub via Sigil.
If you have hand collected additional metadata that is stored in Calibre (or whatever program you have to manage your books) then that will probably not make the conversion unless you use Calibre to export the metadata and then re-import it.
Simply use any text editor and open the pml file and you should see the line with the official metadata in it. If you drop this pml file on DropBook, it should preserve the official metadata.
As for batch conversion, I would not use the gui tool instead I would create a .bat file (or .sh file under MacOSX/Linux) to invoke the python scripts directly for every file in a given directory. If you need hints, I am sure someone here can help you do that.
June 22, 2010 at 5:58 am
Hi,
Thanks for your reply. You’re right — the official metadata is still attached. It is the hand-collected information that disappeared. I checked Calibre’s manual and see that by checking one box I can have metadata inserted as the first page in the book. I’ll try that and see what the result is.
As for the batch conversion, my husband will be able to figure that out, now that I have a direction to point him in.
Thanks again for your help.
Debra
June 22, 2010 at 9:53 pm
I have successfully removed the drm from .pdb book and then converted to epub. Seeing as I have over 200 books to convert how do create a .bat file to batch remove the drm.
June 23, 2010 at 5:44 pm
Hi Greg,
The idea is to write a .bat file that invokes the individual python scripts for every file in a folder.
The command to invoke the python script is:
This assumes you have zipped the tools_v1.7.zip to create a directory called tools.
python tools\eReader_Tools\lib\erdr2pml.py –make-pmlz “NAME_OF_INPUT_FILE” “NAME_ON_CREDIT_CARD” CREDITCARD_LAST_8_DIGITS
So your .bat file would need to invoke this once for each file in the input directory.
I don’t use Windows so I can’t really help with the required .bat file so perhaps someone here who uses windows a lot can help with the exact .bat file contents.
If no one does, I can probably write a python script that might work on a Windows box.
June 23, 2010 at 5:56 pm
i think i get the concept but what would i use for input file so it would go from one to the next. And i am using 1.7 tools for ereader books. ive never had to create a .bat file for anything before now but it is pretty slow going one by one. I will need to run one batch to strip drm then another batch to convert to html. thanks for the quick reply.
June 23, 2010 at 6:52 pm
Hi Greg,
The .bat file would look in a directory to find a list of all of the input files and then invoke the python script once for each file passing in the proper name.
This is a common automation task so there should be example .bat files that will run a command on every file in a directory to be found someplace here on the internet.
You can modify a script to both remove the drm and convert it to html by having it run two different python scripts back to back.
Perhaps someone on this list that knows windows can post a sample .bat file?
Let’s give them a day and if not, I will scrounge up something in python to do the trick.
June 23, 2010 at 7:03 pm
Hi Greg,
I quick google search gave me the following as the contents of a .bat file that will run a command on each file in a specific directory
for /f %i in (‘dir /b’) DO command %i
So you would replace the /b with the path to directory with your pdb files (make sure you copy all of them first to back them up for safety).
Then assuming you have the tools in the path I gave you earlier, you would replace the “command” with the python command line I gave you earlier and use the %i in place of the input file name.
Then you would invoke the specific .bat file from the command line and it should handle things.
Again, someone with more info on Windows should be able to set this up to test and make it work. I can’t really help with that.
Hope this helps.
June 24, 2010 at 8:57 pm
Hi Greg,
I fired up parallels desktop and was able to create a doit.bat file that converted everything automatically.
Here are the steps:
1. create a new folder on your Desktop
and change its name to “autoconvert”
2. inside of autoconvert, create a new folder
and change its name to “books”
3. copy all of your DRM pdb books to this directory autoconvert\books\
4. inside of autoconvert, create another new directory
and call it tools
5. copy the entire eReader_Tools\ directory out of the tools_v1.7 folder and copy it into autoconvert\tools\
6. paste the following line into a text file inside of autoconvert and name it doit.bat (use Notepad)
FOR %%i IN (books\*.pdb) DO python.exe tools\eReader_Tools –make-pmlz “%%i” “YOUR_NAME_ON_CC” LAST_8_DIGITS_OF_CC
7. edit doit.bat and replace YOUR_NAME_ON_CC and LAST_8_DIGITS_OF_CC with the correct info (do not remove the quotes that should be around your name).
Now all should be good to go.
On your Desktop you should now have a folder called autoconvert and inside of it you should have a folder books with all of your pdb books in it and a folder called tools that has the eReader_Tools folder in it, and a doit.bat file
Simply double-click on doit.bat and it should remove the DRM from every one of the .pdb files in books and create a .pmlz file (a zip archive of the pml and any images) for each book (and be ready to wait a long time as stripping the drm from each book is a slow process)
This works for me on Win XP
June 24, 2010 at 8:58 pm
whoops, slight tyo,
The command line should have two dashes infront of make-pmlz so it should look like this:
–make-pmlz
and not
-make-pmlz
June 24, 2010 at 8:59 pm
Argh!
The damn blog is stripping out the two dashes and replacing it with one.
When you are typing it in just make sure there are two dashes in front of make-pmlz
June 26, 2010 at 5:48 pm
On that, is anyone updating the ereader PDB2PML? I have some books I haven’t been able to decrypt, like Merriam-Webster’s Thesaurus and Dictionaries, and Band of Brothers which gives a previously unseen error 11.
June 26, 2010 at 6:18 pm
Hi Rafael,
Yes, it is supported here. If you have books that will not convert then we will need some help tracking down why. Can you please post the entire contents of the conversion log window (replace your name and CC number with filler info) so that we can see exactly what the problem is and have some info on what actually be causing it.
Also, please double-check that the >PDB books in question are actually DRM encoded. Try importing them in Calibre and see what it says for example.
July 2, 2010 at 12:45 am
I will retry to decrypt it again, just to get the error codes. I remember something about Error 11, but am not sure. Since I have a newborn son now, I’ll take a bit (I guess) to post back here.
July 4, 2010 at 1:14 am
Bingo! Working like a charm now. Thanks for all the work and help!
July 2, 2010 at 2:50 pm
some_updates, I tried your batch conversion script and get the following error: can’t find ‘main_.py’ in ‘tools\\eReader_Tools’. I’ve had no trouble running the eReader tools on individual books. The python26 folder is in my C drive directory.
July 2, 2010 at 7:56 pm
Hi Nellibly,
That;s because my paste of the line that makes up doit.bat somehow missed the entire name of the program to actually run. The line wrapped at that point and somehow part of the line was dropped in the paste. My bad!
So please edit doit.bat with Notepad (or some other text editor) and replace this line:
FOR %%i IN (books\*.pdb) DO python.exe tools\eReader_Tools –make-pmlz “%%i” “YOUR_NAME_ON_CC” LAST_8_DIGITS_OF_CC
with
FOR %%i IN (books\*.pdb) DO python.exe tools\eReader_Tools\lib\erdr2pml.py –make-pmlz “%%i” “YOUR_NAME_ON_CC” LAST_8_DIGITS_OF_CC
And remember to put two hyphens in front of make-pmlz instead of the one longer one shown by this blog.
Sorry about my paste error.
Try this one and it should work.
July 3, 2010 at 3:51 pm
Hi some_updates,
I made the change (and double checked the double hyphen) and this time got this error message:
C:\Documents and Settings\tamole\Desktop\autoconvert>python.exe tools\eReader_To
ols\lib\erdr2pml.py –make-pmlz ôbooks\DiffusionofInnovation_16370.pdbö ôMy Nameö 12345678
eRdr2Pml v0.15. Copyright (c) 2009 The Dark Reverser
Processing…
Decoding File
Traceback (most recent call last):
File “tools\eReader_Tools\lib\erdr2pml.py”, line 692, in
sys.exit(main())
File “tools\eReader_Tools\lib\erdr2pml.py”, line 648, in main
convertEreaderToPml(infile, name, cc, outdir)
File “tools\eReader_Tools\lib\erdr2pml.py”, line 570, in convertEreaderToPml
sect = Sectionizer(infile, ‘PNRdPPrs’)
File “tools\eReader_Tools\lib\erdr2pml.py”, line 323, in __init__
self.contents = file(filename, ‘rb’).read()
IOError: [Errno 2] No such file or directory: ‘\xf4books\\DiffusionofInnovation_
16370.pdb\xf6’
What did I miss?
July 3, 2010 at 3:54 pm
Hi,
The problem is your doit.bat file seems to have curly quotes in it and not normal old-fashioned straight quotes.
The best solution is to turn off any auto-replacement of old quotes with smart or curly quotes and then make sure all quotes inside doit.bat are the old normal straight quote.
July 3, 2010 at 5:58 pm
BTW:
My original doit.bat used only old fashioned straight quotes but this damn blog replaced them with curly quotes at the same time it replaced the double dash with one longer dash.
So much for technology!
A Blog should leave thing exactly like they were typed but this one seems to think it knows better.
June 26, 2010 at 5:26 pm
No surprise, but the mobidedrm plug in for Calibre works just fine with mobi library books. Such an elegant solution – really hope someone can build something similar for the new Amazon DRM scheme (hint, hint)
June 26, 2010 at 6:14 pm
Hi DLR,
The new Amazon DRM scheme uses a book specific PID (combination of book specific info, drive volume info, username, and literally pseudo random data) so a Calibre style plugin would not be easy to do, you will always need to provide the ever changing PID to the plugin (or code that calculates it the same from from kindle info file – and that would work only for Kindle 4 PC).
June 29, 2010 at 11:39 pm
Thanks some_updates. I have been using kindle4pc with no problem (so much better if you want to convert, oh, your last 30 books). I won’t look for a plug in. I find ineptepub and ineptpdf to be very easy to use, and skindle just takes a couple clicks. I don’t mean to sound ungrateful. All your work has made me a very happy Kindle owner. I would have gone to another reader long ago, if not for you.
June 28, 2010 at 8:45 pm
Hi Rafael,
If you search this very blog for PDct PPrs you will see a posting that says this works with only minor changes to the ereader code. So it should be easy to support dictionaries.
July 1, 2010 at 7:36 pm
Hi — I am just hoping that someone will please take the time to post here an update about the Kindle giving us all some idea as to what we should or could be doing at this point. I believe I have read here that SKINDLE for the PC is now working? Will it also work on the Macintiosh? I have Kindle for the Mac so what I would do to run SKINDLE? I realize it is probably on the board but everything is all so run together and a search turns up so many old hits that are of no use or help.
I have been working before with MobideDRM and using it as a Calibre plug-in — but I see that will never probably work again? I am now buying my ebooks from Barnes and Noble and thus far have deDRMd them pretty easily with the ePub scripts. But I miss buying from the Kindle store which often has the best prices.
I truly appreciate everything that everyone has done here and this is not to be taken as a criticism. But, every so often, a message capsulizing what is currently happening would be very handy. Thanks.
July 1, 2010 at 8:15 pm
Can anyone verify if the TTS flag can still be changed using mobileDRM?
Please someone with the new version give a try delete the TTS flag and tell me if it enables the TTS.
Thanks.
(thats the only reason right now I care about DRM issues.) Some day I hope to convert my books to take them with me. But for now they work.
July 2, 2010 at 3:30 pm
Got an error of the “Error – -1 outside of string table limits” while trying to convert a Topaz file.
http://www.mediafire.com/?tgztmm4mjum
http://www.mediafire.com/?nwxzmy32hmx
are the page and dict files.
Any ideas?
thanks
July 2, 2010 at 8:07 pm
Hi Terry,
Please update to the latest tools version 1.7 (links available at this site near the end of the blog or on Apprentice Alf’s Blog in the comments). The bug that you are seeing exists in version 1.6b but is fixed in version 1.7.
July 2, 2010 at 8:30 pm
Hi,
I can’t find the link to the 1.7 tools. Could you reply with a link?
thx
July 2, 2010 at 9:42 pm
Hi,
Simply doing a “Find” on the Dark Reverser’s New Blog page for tools_v1.7 reveals this:
There is a new version of the tools available:
tools_v1.7.zip
The changes include:
1. updated to version 16 of MobiDeDRM which speeds up conversions (Thanks Alf)
2. updated to the latest version of mobiunpack.py (Thanks Paul)
3. added latest version of LZSkindle tools (Thanks Mark)
4. bug fixes for Topaz conversions with strange “magic” file type “indicators”
The old version has been removed.
The URL is:
http://www.mediafire.com/?yywynwqoqtm
July 2, 2010 at 11:23 pm
thanks!
July 2, 2010 at 10:20 pm
My apologies! I thought I was using the new version but I reloaded it and started over. Works just like we want it to.
Thanks again!
Tery
July 3, 2010 at 10:04 pm
Hi — Need some help with Skindle and Topaz ebooks. I am NOT a PC expert as I always use my Mac but with the new DRM techniques from Amazon have had to switch to Swindle on my PC laptop. That has worked well for me with azw books. But today I managed to download a Topaz one (one that was just uploaded to Amazon yesterday (Married to the Mouse — a book about Disney World and Orlando). I cannot get Skindle to work with it. The primary problem is that it fails trying to read the DRM’d Topaz book file. I use the command line of skindle -i -o but it will not go past the -i parameter and says it has quit as it can’t open the book. Secondarily, I am unsure also what to name the output — do I use the azw extension or what in the command line? But not opening the book in the -i paramater is the first problem. Any ideas? I thought Skindle could do Topaz as well? No?
July 3, 2010 at 11:47 pm
Hi,
skindle can remove the DRM from Topaz books but they are still Topaz books and they can not be read in anything other than a Kindle (Kindle 4 PC, Kindle 4 Mac, etc).
The command line to use skindle to remove the DRM from a Topaz book says to use the -d command line option before the -i input file option.
If you want to actually read the Topaz file on something other than a Kindle, you will need to convert the book from its original Topaz with DRM state to xhtml or svg format using the Topaz_Tools which are part of tools_v1.7.
July 4, 2010 at 2:21 am
Hi some_updates — Well, I got the ebook file from the PC and put it on a flash card to move to the Mac where I am more comfortable in Python. I tried to run TopazExtract_Kindle4PC.pyw on the file and got the following error:
Conversion Log
Command = “python cmbtc_dump.py”
Topaz Path Path = “/Volumes/8GB FLASH/B0016ODGAU_EBOK.azw”
Output Directory = “/tools 1.7/Topaz_Tools”
First 8 chars of PID = “”
Please Wait …
Traceback (most recent call last):
File “./lib/cmbtc_dump.py”, line 37, in
from ctypes import windll, c_char_p, c_wchar_p, c_uint, POINTER, byref, \
ImportError: cannot import name windll
Error: File Extraction Failed
I then tried the supposedly unDRM’d version done with Skindle and that failed in the same manner.
What should I try? Thanks!
July 4, 2010 at 2:51 am
Oooops, never mind I’ve done it! My mistake was to move the book file to the Mac. I guess I did not understand what the scripts are looking for. Anyway, I used the Topaz tools on my PC laptop and they worked fine. Congrats to the script authors on designing a GUI that even a Mac Lover can comprehend!
July 7, 2010 at 12:23 am
what is the latest version of topaztools? the latest i have found is v1.8. there was mention of v2.0, but it is available nowhere.
July 7, 2010 at 6:16 am
Hi TWiT,
I stopped giving new version numbers to the Topaz Tools long ago. There were simply too many files in the lib/ directory to keep up with.
The latest version of Topaz Tools can only be found in the latest collection of tools which is now tools_v1.7.zip
Simply grab tools_v1.7.zip and unpack it to get the very latest version of Topaz Tools and ignore any internal version numbering in the Topaz Tools files themselves.
July 7, 2010 at 8:39 am
ah. ok. i learned how to get rid of adept drm from my sony ebooks some time ago, and now i shall try this one. i am primarily interested in this to preserve my purchases in an unencumbered format.
by the way, does this work on a mac, or has it been compiled to be windows-specific.
thanks for all the work.
July 7, 2010 at 11:49 am
Hi TWiT Com,
There are scripts that do the unpacking for Kindle4PC that are windows specific. There are also a Kindle_iphone script that will work on books purchaseod for Kindle for ipod , iphone, or ipad which are crossplatform and do work on a Mac. There is now no solution for books purchased on updated standalone Kindles or for Kindle4Mac. If you have any of those you are out of luck until someone reverse engineers a script for those.
Hope this helps.
July 7, 2010 at 12:22 pm
right now i have a mac running the kindle software. what i will probably end up doing is installing windows 7 in parallels (all legit software by the way; i believe in paying fair compensation for other people’s work; i know it’s old-fashioned). i also have an older k4pc installer from http://nyquil.org/archives/1235-Converting-Amazon-Kindle-For-PC-Books-Now-Possible.html. this one supposedly works with unswindle.
the blogpost linked above does say that the method outlined does not work with topaz, but that was in february IIRC. i am hoping that the scripts linked on this post will be able to at least strip the drm from topaz.
once that’s done, i’ll mess around with this http://stream-recorder.com/forum/convert-topaz-ebooks-html-remove-drm-tpz-t5426.html?s=073053d54bf6bf4f1991640ae51a24df to get the book into html format. from there, i want to see how hard can it be to get a working epub file.
like i said before, my main motivation is to archive my digital purchases which i have bought with my money. i am not interested in stealing. stealing is wrong. as trivial as it was to circumvent adept, unfortunately for me, most of the technical/engineering books i am interested in are only available from amazon, perhaps in topaz format as it is more advanced than mobi if i read correctly. i have searched high and low, but these titles are not available in epub format anywhere.
i’ll do this once the weekend is here. i’ll probably post my progress here.
July 7, 2010 at 1:39 pm
Hi TWiT Com,
That streamrecorder link just points to very old versions of Topaz_Tools. The latest version is in tools_v1.7.zip.
There is now a nice GUI that handles the bulk of the conversion process.
Removing the DRM from Topaz files will not allow them to be converted to anying else (just allows being read on another Kindle wiout the pid).
If you want an epub you should simply use the Topaz Tools to convert the file to xhtml and then manully clean it up to remove any OCR related qdefects, add in italics, bolding, etc. The topaz format is imply an x,y glyph poistion description of each page and so eithermust be converted to xhtmlor view each page as an svg image.
Have fun!
July 7, 2010 at 2:10 pm
i noticed from what other people have written about the peculiarities of topaz. if i understood correctly, it’s a scan of the original material with individual glyphs stored in some kind of a table. the reflowable layout is reconstructed by stringing the glyphs together, kind of like how individual letters in the alphabet make up whole words. i may be wrong regarding this as you say that it is like images with x,y positions, but i’ll examine this further when i get a topaz file decrypted.
on stream-recorder, they have a procedure to convert topaz to xhtml. how is that different from your method? my ultimate aim is to convert topaz into epub. having said that, i can also archive the decrypted topaz files since they can be read in k4pc or kindle.
right now, i am travelling (work-related). i should be getting my kindle 2 b y the end of the month when i return home. in the mean time, i’ll see what can be done with k4pc and the scripts.
by the way, i did notice that the two topaztools were similar when i compared the script names, the difference being the version numbers.
July 7, 2010 at 2:41 pm
by the way, the decrypted topaz file should open in stanza. am i right? stanza can open unencrypted azw files (which may be mobi or topaz inside if i am not mistaken). from there, the sky is the limit: epub, mobi, pdf, html, rtf, txt, etc.
i may be getting too optimistic.
July 7, 2010 at 4:34 pm
never mind about stanza. the only image it can see is the cover image. any other which may be in the middle of the file, like charts or figures disappear in the converted file.
by the way, i was able to get an 8-digit pid for kindle pc software and decrypt one of my prc/mobi files.
one of the gui was asking for a 10-digit pid. am i missing something?
July 7, 2010 at 4:44 pm
the more i dig into this, the more stuff i discover:
1. sample files are not encrypted.
2. $0 books are encrypted.
3. if you get a book in the old version of k4pc, you get a prc/mobi. when you download the same book in latest version of k4pc, you get an azw/topaz.
4. calibre’s rendering of azw and mobi is buggy when you encounter bullets, numbered lists and tabs – lots of accented characters.
July 6, 2010 at 12:52 pm
I work with unswindle version 7 – but something doesn’t work. I always get another book “unswindled” – one that is taken frommy calibre library.
What canI do?
July 7, 2010 at 1:42 am
Help… Can anyone verify if the TTS flag can still be changed using mobileDRM?
Please someone with the new version give a try delete the TTS flag and tell me if it enables the TTS.
Thanks again.
(thats the only reason right now I care about DRM issues.) Some day I hope to convert my books to take them with me. But for now they work.
July 10, 2010 at 6:45 pm
I recently ran into an issue stripping DRM from a .PDB file purchased from B&N. The book converted fine without any noted errors.
The title is “Moloka’i” and it seems during the conversion using erdr2pml.py it all of the accented characters (é, í, etc…) used in the text are rendered as a “?” symbol.
Is there a way to prevent this from happening?
July 11, 2010 at 2:11 pm
I know this has most likely been done somewhere, but I am about 24 hours into converting ebooks (i.e. pretty new) and have some DRM Topaz books that need conversion for my nook. I have the Topaz_Tools from tools_v1.7.zip, and can not seem to find a comprehensive instruction set for non-python experts on how to use it. I have active python and mobidedrm installed inside calibre, but this won’t work for topaz, so I need to convert these topaz books first. Thank you for your patience.
July 13, 2010 at 1:04 am
Nvm, figured it all out by trial and error with the extractor, the xml, svg, and http converters.
July 13, 2010 at 12:38 pm
Hi,
Yes that is the way. If someone would like to write up a readme.txt or howto.txt that is readable on Windows that explains the steps, I would be happy to include them in the next release of the tools zip archive.
I would love to have something for each tool that explains how to launch each gui tool for each type of conversion (not just for the Topaz Tools).
The last readme.txt I wrote on a Mac and people on windows complained that it was unreadable since it all ran together into one line.
All contibutions welcome.
July 13, 2010 at 1:34 am
wow really not many people answering questions on here is there?
July 13, 2010 at 12:44 pm
Hi,
Many of these questions are already answered here many times. The is another blog Apprentice Alf’s blog ( a Google search term) that has summarized many answers and it has many fewer comments to read through (he prunes them).
If after reading that site you still have problems, post here and someone will generally respond within a few days.
July 13, 2010 at 11:14 am
My girlfriend’s got a kindle, and I have a sony 505. I’d like to read kindle ebooks without having to register her device on my computer. Where can I find her kindle.info file, because that’s apparently what I need to make skindle work on my machine?
July 13, 2010 at 9:23 pm
Problem with Topaz Tools in tools_v1.7
TopazExtract_Kindle4PC.pyw ran OK, but TopazeFiles2SVG.pyw gave me the following error messages.
Processing Pages …
page0000.dat
Traceback (most recent call last):
File “lib\gensvg.py”, line 405, in
sys.exit(main(”))
File “lib\gensvg.py”, line 329, in main
flat_xml = convert2xml.main(pargv)
File “C:\Documents and Settings\Drak Bibliophile\My Documents\tools_v1.7\tools\Topaz_Tools\lib\convert2xml.py”, line 791, in main
xmlpage = pp.process()
File “C:\Documents and Settings\Drak Bibliophile\My Documents\tools_v1.7\tools\Topaz_Tools\lib\convert2xml.py”, line 716, in process
snippet = self.injectSnippets(self.snippetList[0])
File “C:\Documents and Settings\Drak Bibliophile\My Documents\tools_v1.7\tools\Topaz_Tools\lib\convert2xml.py”, line 567, in injectSnippets
aso, atag = self.injectSnippets(asnip)
File “C:\Documents and Settings\Drak Bibliophile\My Documents\tools_v1.7\tools\Topaz_Tools\lib\convert2xml.py”, line 567, in injectSnippets
aso, atag = self.injectSnippets(asnip)
File “C:\Documents and Settings\Drak Bibliophile\My Documents\tools_v1.7\tools\Topaz_Tools\lib\convert2xml.py”, line 566, in injectSnippets
asnip = self.snippetList[j]
IndexError: list index out of range
Error: SVG conversion Failed
July 14, 2010 at 10:09 am
Hi,
This message is typically the result of the parser getting confused by an xml tag that it does not yet understand.
Please post the LAST page****.dat file shown at the very end of the conversion log message and the dict0000.dat file someplace and I will grab it and enable debugging to see exactly what confused the parser and fix it and then post a test program for you to confirm the fix.
July 15, 2010 at 8:08 pm
Sorry to take so long with this. The files are at the following link (if I did everything correctly [Wink])
http://www.mediafire.com/?sharekey=685ffe78617e74ca00b0c48982c309a23bb7fa34e3aa397b7e985b5830f975cb5c1b0bdb8f904aafbb1db29dca7ac1a2
July 16, 2010 at 7:06 am
Hi Paul,
Your code included a new tag which I have not seen in a long time (img.color_src).
I have modified the code to understand that new tag and have a test program for you to try.
So please go to
http://pastebin.com/RDERReyk
and grab the file and name it convert2xml.py and use it to replace the file of the same name in tools\Topaz_Tools\lib\ and then try processing the files again and let me know if it works for you.
July 16, 2010 at 12:24 pm
Hi Some_Updates, I tested your test version of convert2xml.py and it worked on both the Topaz file I told you about and another one that had the same problem.
Thanks for the help.
July 14, 2010 at 2:25 am
Hi,
Just tried converting a Topaz file using the topaz_tools. The SVG conversion failed with ‘maximum recursion depth exceeded’.
Any ideas?
Thanks
July 14, 2010 at 2:26 am
Oops – sorry, it was the Topazfilestohtml stage that failed.
Thanks
July 14, 2010 at 10:16 am
Hi Glenn,
Please copy and paste the entire Conversion log window contents so that we can see where and how the program is failing. It could be an unknown xml tag that confuses the parser into reparsing somthing which causes the recursion limit to be reached. I need the conversion log to see the context of the error and the last page****.dat file name at the very end of the log to see which file is the problem case.
July 14, 2010 at 3:02 am
I have just discovered that Kindle 4 PC was recently upgraded, making version 6 of Unswindle found in some_updates tools package 1.7 obsolete. However I have also discovered that there is now a Version 7 of Unswindle available that recognozes the newest Kindle 4 PC. It’s a moving target out there. Kindle 4 PC has an option to automatically update itseld without asking. I would suggest turning this option OFF to avaid surprises.
July 14, 2010 at 4:49 am
Link to version 7 of Unswindle??
July 14, 2010 at 10:19 am
Hi,
Yes I Heart Cabbages site has links to the new unswindle and new versions of the inept tools that we can include into the next version of tools v 1.8 when it is released in about 2 weeks (I am traveling in europe and do not have access to my home computer) right now.
July 14, 2010 at 10:38 pm
Here is an extract from the log.
Conversion Log
Command = “python genhtml.py”
Book Directory = “C:\Users\Rose\Documents\hood”
Please Wait …
Processing …
metadata0000.dat
other0000.dat
Traceback (most recent call last):
File “C:\tools\tools\Topaz_Tools\lib\genhtml.py”, line 183, in
sys.exit(main(”))
File “C:\tools\tools\Topaz_Tools\lib\genhtml.py”, line 158, in main
xmlstr = convert2xml.main(pargv)
File “C:\tools\tools\Topaz_Tools\lib\convert2xml.py”, line 791, in main
xmlpage = pp.process()
File “C:\tools\tools\Topaz_Tools\lib\convert2xml.py”, line 705, in process
tag = self.procToken(self.dict.lookup(v))
File “C:\tools\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in procToken
subtagres.append(self.procToken(self.dict.lookup(val)))
File “C:\tools\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in procToken
subtagres.append(self.procToken(self.dict.lookup(val)))
>>>>>> This call then repeats hundreds of times
File “C:\tools\tools\Topaz_Tools\lib\convert2xml.py”, line 439, in procToken
subtagres.append(self.procToken(self.dict.lookup(val)))
File “C:\tools\tools\Topaz_Tools\lib\convert2xml.py”, line 413, in procToken
tkn = self.get_tagpath(j)
File “C:\tools\tools\Topaz_Tools\lib\convert2xml.py”, line 360, in get_tagpath
for j in xrange(i+1, cnt) :
RuntimeError: maximum recursion depth exceeded while calling a Python object
Error: HTML conversion Failed
Thanks
July 15, 2010 at 6:49 am
Hi Glenn,
This does look like the parser got confused. Was the last file name processed other0000.dat or was there a later pageNNNN.dat file in the log?
If other0000.dat is actually the latest then the TopazFiles2SVG.pyw should have run completely through to the end successfully. Is that right?
If so, please post for me the dict0000.dat and other0000.dat and I will see if I can track down the problem and get it fixepd.
July 15, 2010 at 10:39 am
Yes TopazFiles2svg did run through to the end. The failure was in the conversion to html. The dict0000.dat file is quite large (166Kb) and full of binary. The other0000.dat is smaller but still full of binary. Can I attach the files or do you just want me to cut and paste them in a reply?
Thanks
July 15, 2010 at 2:41 pm
Hi Glenn,
Please zip them up and post them to mediafire or some other file sharing site and post a link here and I will grab them.
July 15, 2010 at 10:16 pm
OK the files are here
http://www.mediafire.com/file/yntd4azmgwdjou2/gvfiles.zip
Thanks for all your help!
July 16, 2010 at 7:11 am
Hi Glenn,
I grabbed them. The problem is that an xml “token” called “book” is mapped to exactly the same thing as the the command code to do a loop (0x76). That confuses the parser and it keeps looping and looping and finally the recursion limit is reached which results in failures.
So somehow I have to add code to will understand when the 0x76 in the binary file means a loop and when it instead means the xml token “book”.
That is going to take some thinking. So it may be a few days until I can figure out something that works.
Sorry, but I will get you something to test with but it may take a while.
July 16, 2010 at 12:59 pm
Hi Glenn,
I think I have a workaround that should solve the issue.
Please go to:
http://pastebin.com/1XiL1eS2
and download the file there and rename it convert2xml.py and use it to replace the file with the same name in tools\Topaz_tools\lib\ and then rerun things.
Please let me know if this does the trick.
July 16, 2010 at 11:15 pm
Well done! I can’t believe how quickly you resolved the problem. The conversion looks to be almost perfect! I have read that topaz book conversions sometimes need a fair amount of editing after conversion. As far as I can see the only issue with this one is that it uses a fancy font for the first character of each chapter which hasn’t converted properly sometimes. Will only take a few minutes to fix.
Thanks for your help!
July 17, 2010 at 2:39 am
I have a Topaz book that fails in the gensvg step with the following error.
Processing Meta Data …
metadata0000.dat
Processing Glyphs …
glyphs0000.dat
Traceback (most recent call last):
File “gensvg.py”, line 376, in
sys.exit(main(”))
File “gensvg.py”, line 277, in main
flat_xml = convert2xml.main(‘convert2xml.py –flat-xml ‘ + dictFile + ‘ ‘ +
fname)
File “C:\DeTopaz\convert2xml.py”, line 789, in main
dict = Dictionary(dictFile)
File “C:\DeTopaz\convert2xml.py”, line 121, in __init__
self.fo = file(dictFile,’rb’)
IOError: [Errno 2] No such file or directory: ‘o’
I’ve pasted the book’s directory at
http://www.mediafire.com/?d9eijk8gtockq6y
July 17, 2010 at 7:05 am
Hi,
This error means the program can not find the dict0000.dat file.
That means you either did not properly unpack the drm topaz file or you did not properly pass the name of the directory that you did unpack them into to to the TopazFiles2SVG.pyw program.
I can’t tell which because you did not show the full conversion log.
So simply doublecheck that the directory where you unpacked the topaz book files into is the same as the directory that you give to the TopazFiles2SVG.pyw and TopazFiles2Html.pyw programs.
July 17, 2010 at 1:09 pm
Hmmm, I use a batch file that uses %~n1 to keep the directory name the same as the book file name minus the extension. The book name doesn’t have any spaces or DOS metacharacters, so it should be getting the same directory name throughout. I’m not using the GUI tools; I’m using the topazscripts 2.0. Let me try the Tools scripts.
July 17, 2010 at 1:16 pm
I get a little farther with the tools. Here’s the error when converting to html.
Conversion Log
Command = “python genhtml.py”
Book Directory = “C:\tools\Topaz_Tools\testbook”
Please Wait …
Processing …
metadata0000.dat
other0000.dat
Error – -404 outside of string table limits
Error: HTML conversion Failed
July 17, 2010 at 1:24 pm
Hi Davo,
The Topazscripts are very old and now unsupported. The only current source is in the tools archive.
The error you are getting seems to be very similar to Glenn’s. So TopazFiles2SVG.pyw should have successfully completed.
So please see my last response to Glenn and go to pastebin.com and grab the test version of convert2xml.py and use it to replace the file of the same name in Topaz_Tools\lib\ and try again.
July 17, 2010 at 2:20 pm
Hi Davo,
With tools v1.7.zip and with
http://pastebin.com/1XiL1eS2
downloaded and renamed to convert2xml.py and used to replace the file with the same name in tools\Topaz_tools\lib\
your book converts successfully.
Your problem was the same one experienced by Glenn
Upon my return from my travels I will post a new tools_v1.8.zip with these updates as well as updates to the latest unswindle and the latest inept tools from I Heart Cabbages.
July 24, 2010 at 3:20 pm
Hi,
The latest version of the tools collection has been updated:
tools_v1.8.zip
http://www.mediafire.com/?3ro5pqsjbetc6bs
Changes from the previous version:
– added support for img.color_src tag and more robust parsing of stylesheet for Topaz Tools
– update to latest version of unswinde (v7) from IHeartCabbages
– update to latest versions of Inept Tools for epubs from IHeartCabbages
July 24, 2010 at 3:46 pm
The Adobe_EPUB_Tools folder in 1.8 does not contain the ineptkeymac.pyw script as did 1.7. Should Mac users stay with 1.7? Could the 1.7-generated key just be dropped into the 1.8 folder? (I have not used Adobe ePub in a long while so don’t have a test file).
July 24, 2010 at 5:13 pm
Hi Thothamon,
The new version by IHeartCabbages works on both Mac OSX and Windows so their is no longer a need to keep up a ineptkeymac.pyw scripts.
One key script does it all.
See the IHeartCabbges website for more info.
July 25, 2010 at 1:25 pm
Does anyone have a Calibre plugin with the latest MobiDeDRM? My script is a tad bit old and I didnt notice one in the lates tool archive
July 25, 2010 at 1:50 pm
Hi SR1,
The best place to look for links related to ebook tools is Apprentice Alf’s Blog.
http://apprenticealf.wordpress.com/
Search that page for Calibre plugin and you will see the link you need.
July 25, 2010 at 6:53 pm
I have looked there and the link to mediafile does not work…
July 25, 2010 at 8:01 pm
Hi SR1,
I just tried the link from the Apprentice Alf page:
http://www.mediafire.com/?z1nrd1jyttz
and it works fine.
I tried it on both Windows (FireFox), and MacOSX (Safari) and it works fine. Perhaps you need to enable javascript for that site?
August 24, 2010 at 6:36 pm
Went to the site to get the plugin, how do you make it work once you have it?
August 24, 2010 at 6:37 pm
Once you have this plugin, how do you make it work?
August 24, 2010 at 9:32 pm
Sent you a PM on MobileRead, Chilady.
July 25, 2010 at 5:50 pm
Well, with the latest release of unswindle, I finally have a linux-only solution for decrypting my ebooks after the 2.5 Kindle update. It’s not pretty… I now have two python installations (linux native plus ActiveState python under Wine), but it does work. I’m not nuts about the duplicate python installs (not to mention K4PC under Wine), but it does save me from having to borrow a Windows machine.
Running things in Wine still makes me feel all dirty and tainted, but there you have it. 😉
July 31, 2010 at 12:37 am
You really have to laugh….why do they bother with DRM?
I thought I had lost the ability to read Kindle on my Bebook as unswindle stopped working after the 1.1 update. But the new V7 plus Darkreverser have made it work again…so thanks.
I paid for the book at Amazon so why shouldn’t I be able to read it on my favourite ebook reader?
July 31, 2010 at 6:22 pm
Preaching to the choir man.
August 1, 2010 at 5:19 pm
Question about Kindle Mobi Tools in tools_v1.8
I don’t have a Kindle reader so I use Kindle For PC (an older version) for my Kindle ebooks.
I currently use unswindle to de-DRMed the Kindle ebooks but I’m want to know what order to use the scripts in Kindle Mobi Tools as well as any ‘special’ instructions.
By the way, thanks for these tools. They’re very helpful.
August 7, 2010 at 7:52 pm
Hi Paul,
Here is s short synopsis of the other tools there:
– Kindleizer can take any normal Mobi ebook and make it work/be recognized on a Kindle. Early kindles did not actually work/read regular mobi books.
– KindlePID can take an older standalone Kindle and convert its serial number to the normal PID. Unfortunately, standalone Kindles now use book specific PID’s so this program no longer works for standalone Kindles. It does continue to work for iPad’s, iPod Touch’s, and iPhones where is converts a UDID to the single PID still being used on those devices.
– LZskindle4PC is a front end for running skindle that makes it easier for some people who do not like command lines to use it.
– skindle has completely reverse engineered how the book specific PID is generated for Kindle4PC so it can be very useful when new version of Kindle4PC are released and unswindle has not yet been updated.
– unswindle can be used to find the book specific PID but it needs to be updated for each version of Kindle4PC that Amazon releases (and therefore is also useful for Linux users who have Wine). This program “patches” the Kindle4PC executable and therefore is very release specific.
– MobiDeDRM.pyw is a simple graphical front end to the mobidedrm.py progam that actually does the DRM removal as long as you know the correct PID to use with it (which depends on the versions of Kindle software used). It is needed by unswndle to do the final drm removal after unswindle finds the right book specific PID
– MobiUnpack.pyw is a graphical front end to mobiunpack.py that will actually unpack all of the original pieces from a Mobi ebook (assuming the DRM has already been removed) so that you can see the actual mobi specific html and other things. If you have a mobi ebook and would like to recreate it after a few changes, you can use this program to get to the pieces and then repack it to create a modified Mobi ebook.
– MobiML2HTML.pyw is a graphical front end that can be run on the output of MobiUnpack.pyw to convert the Mobi-specific html to normal xhtml for archival purposes or for later conversion to epub or any other ebook format that uses xhtml. (It is a highly modified version of what is done by calibre – in fact the calibre code was the original basis for much of the python code in this program). I use this one a lot to actually “archive” my ebooks by storing the xhtml versions of each book to a data DVD for long-term storage in a format that almost any browser today can read.
That is pretty much it. Simply play around with these tools. Most are graphical or have graphical front-ends and are easy to use.
August 2, 2010 at 7:54 pm
Hi — I downloaded a free ebook from the Univ. of Chicago site and it came down as an Adobe Digital Edition but not an ePUB but, rather, a DRM’s pdf file of some sort. When I try to open it in Mac Preview I get gibberish. When I try to open it in Adobe Reader it says that page is not available. It’s not a placemarker or link file as it is 33 MB. Works, of course, in Adobe Digital Edition itself. Is there any way to unDRM this pdf?
August 2, 2010 at 8:24 pm
The ineptpdf tools handle Adobed pdf files – you can get them from Apprentice Alf’s or I Heart Cabbage’s blog, I don’t think they’re included in the Tools distribution.
August 3, 2010 at 12:45 am
Thank you! Those worked great for me.
August 3, 2010 at 3:11 pm
Seeing as the adeptkey.der produced by both ineptepub and ineptpdf is the same then I am wondering why ineptpdf is not included in the Tools releases? Just would have to rename the folder from Adobe_EPUB_Tools to Adobe_EPUB/PDF_Tools and it would work. Yep, it was easy to find but isn’t it important to keep the Tools folder complete for all uses? Just a thought.
August 7, 2010 at 7:34 pm
Hi Thothamon,
The reason is that Adobe seems to be more “converned” about removal of DRM from pdf files than they are about epub files. They threatened IHeartCabbages when he first posted it and he was forced to remove the ineptpdf program from his site (see his blog for more info). I did not want to have a repeat of that happen for the “tools” set. In addition, I personally do not think of pdf as an “ebook” format since is not really reflowable, needs to have the screen size specified and therefore is not truly device independent, etc), so I chose not to include it.
There are links to it easily found on the web.
Feel free to repackage any of the tools and post them along with ineptpdf if you so desire, I simply did not feel it was worth the risk.
August 7, 2010 at 10:38 pm
Hi Some_Updates and thank you for the response. I agree with your reasoning 100%. PDF is really not a good ebook format and is becoming less and less used. Of the 150 or so eBooks I have removed the DRM from (all of which I bought — unless they were free PD — and none of which I WOULD have bought did unDRM Tools not exist) only this one was a DRM’d pdf. I certainly hope the day comes soon when DRM is looked back on as some sort of deviant law like Prohibition. Until then, hey, this is the best speakeasy in town and thanks for all your efforts!
August 15, 2010 at 6:00 pm
PDF has the attributes for an e-book format. It doesn’t matter if it is reflowable or not. You can’t reflow paperback books (or would you say these aren’t books either?).
It’s a nice feature to allow reflowing an e-books but it’s not a deciding attribute.
August 4, 2010 at 8:16 pm
Could anybody tell me which version of Kindle for PC works with unswindle v7?
I’ve tried 1.1.0 (30136) and 1.2.0 (30413) and swinlde keeps saying “unsupported version”
I am on Windows7 64bit
August 6, 2010 at 7:48 am
I’m successfully using 1.1.1 (30156). I did have to completely uninstall the program and reinstall directly from Amazon.
August 6, 2010 at 9:57 am
Thanks. I guess it’s 1.1.1 that’s is supported for now.
Amazon currently serves 1.2.0, I may have to find 1.1.1 elsewhere. (They don’t have an archive of K4PC, do they?)
Does anyone have any luck with 1.2.0?
August 7, 2010 at 7:56 pm
Hi,
If you have tried the latest unswindle and it does not work, then the only solution is skindle which does not need to patch the Kindle4PC application and therefore will work as long as it can find your .kindle_info file on your PC.
August 13, 2010 at 4:29 pm
Modidedrm2 is not working on newly purchased kindle books. is there an updated one out yet?
August 13, 2010 at 4:37 pm
You should use MobiDeDRM 0.16, which can be found at http://apprenticealf.wordpress.com/
However, that’s unlikely to solve your problem, which is probably due to an upgrade to firmware 2.5 or later on your Kindle.
The only way at the moment to remove DRM from Kindle books is with Kindle for Windows and either unswindle or skindle.
Unswindle requires the MobiDeDRM script. I’d advise using MobiDeDRM even with skindle, as skindle’s decoding is not perfect, although it does an excellent job od working out the per-book PID.
August 29, 2010 at 7:18 am
I guess I am lucky. My Kindle 2 is stuck at 2.3. The official Amazon firmware checks that files on the Kindle haven’t been modified.
Hacks such as the screensaver or the UFH hack modify some files; and as such the official firmware update will fail until they are removed. So, if your Kindle 2 has not updated to 2.5 and you want to prevent such updates just install one of the hacks, for example the screensaver
August 14, 2010 at 2:22 pm
skindle worked a treat for me (with LZskindle4PCv1_1 ), on a Kindle4PC book purchased today 15 Aug 2010.
many thanks to the people who put the hard work in to achieve this (it’s way beyond me!)
AMAZON – For what it’s worth, I wouldn’t have had to do this if you didn’t make it so damn hard for Australian Kindle users to buy certain books. But instead I had to create a Kindle UK account, buy the book using Kindle4PC, the use skindle to break the DRM then move the book over to my Australian registered Kindle.
August 16, 2010 at 12:16 am
I need help with topaz files.
If someone could explain to me how to de/drm’ing/converting to (any) other format on a Mac, that would be great.
I’ve downloaded the tools zip, and read through the Readme, but I got as far as cmbtc_dump_nonK4PC.py -d -o TARGETDIR -p 12345678 YOURTOPAZBOOKNAMEHERE before getting lost.
I will forever be in your debt.
August 16, 2010 at 8:07 pm
Hi Anarel,
On a Mac, for conversion of a Topaz book to work, you will need either Kindle for iPod Touch, iPhone, or iPad (and iPhone Backup Extractor.app and the UDID for the device) — or — run Parallels Desktop (or any other virtual machine – assuming your Mac is Intel based) and install Windows XP – or some other version of Windows and get Kindle4PC up and running (try for an older version of Kindle4PC that is better supported).
These are the only devices that the correct PID can be determined for right now (Kindle4Mac will not work since no one has reverse engineered it yet) and the update to actual Kindles (to version 2.5 or later) has broken the approach that worked on them.
If you have one of those set-ups and properly downloaded the book to it, we can help lead you through how to convert it to html and svg by running the GUI applications in tools_v1.8/Topaz_Tools/ on your machine.
August 16, 2010 at 2:03 am
Anarel,
Was a TARGETDIR created?
August 16, 2010 at 2:50 pm
Anyone else getting corrupt books when decoding topaz books using skindle? I’ve had two that appear to be okay until I get a ways into the book, then my Kindle says there is a problem with the book and requires me to download it again. I also tried using the topaz tools, and it looks like there is some OCR flaws being introduced in the svg –> html step, garbled words etc.
August 16, 2010 at 7:48 pm
Hi,
I would not use skindle to remove the DRM from any Topaz book. skindle’s code that does the equivalent of MobiDeDRM.py has some bugs and it will not work reliably.
As for ocr problems in the Topaz conversion using the tools, yes that can happen. But please understand the topaz tools do NO OCR. The OCR is actually stored inside the Topaz Book and if flawed, it is because Amazon did a very poor job on the OCR. So converting flawed OCR to html will in fact result in flaws.
The the Topaz Tools conversion to a set of svg files does not use the OCR info from Amazon and instead simply arranges a bunch of graphic images (glyphs) to create an image of the page. It should be a perfect replica of the page. However it is an image and can not be searched for words, reflowed, etc.
Sorry, but Amazon’s Topaz format is actually a very poor man’s PDF format with lots of bad OCR to provide input for any search done on the text and images of words (not! made with fonts) just independent glyphs.
August 16, 2010 at 8:00 pm
That explains why searching for terms does not always yield the expected result. Thank you very much for the explanation.
August 19, 2010 at 12:56 am
Just downloaded and installed Kindle4PC, and attempted to strip the DRM on an Amazon book (azw). I have tools version 1.8, and have tried both unswindle and skindle. Unswindle reports that it is an unsupported version of Kindle4PC. Skindle reports a device PID as well as a PID for the book itself, then reports that it can find no valid PIDs available and that it can find no DRM key. Then it terminates.
Does anyone have any ideas what to do now???
August 19, 2010 at 8:20 am
It sounds like the book doesn’t actually have any DRM applied.
If it did, the best course is to use skindle to get the book PID, and then use that with MobiDeDRM 0.16 to remove the DRM.
Unfrotunately, the skindle de-drm code gets things wrongs with some Mobipocket files, leading to intermitant corruption to the text of the book.
August 19, 2010 at 4:52 pm
Unfortunately it does have DRM, which I verified with MobiPocket Reader.
I tried using both the device PID and the book PID with unwindle, and both times I got a checksum error on the PID.
Anybody got any more ideas? They would be most appreciated.
August 19, 2010 at 6:41 pm
Mike-
You did not mention what version of K4PC you have installed. I believe you must have an earlier version (i.e. v1.1.1) of K4PC. The current v1.2 will not work. (unless something has changed and I missed it)
August 19, 2010 at 7:16 pm
Hi Mike,
The only pid you want to try with is the Book PID that skindle provides.
Then grab mobidedrm.py (the latest version) and slightly modify it to accept only 8 digit PIDs instead of 10 digits.
You can do that by simply editing mobidedrm.py and making the following changes:
change:
def __init__(self, data_file, pid):
if checksumPid(pid[0:-2]) != pid:
raise DrmException(“invalid PID checksum”)
pid = pid[0:-2]
to
def __init__(self, data_file, pid):
# if checksumPid(pid[0:-2]) != pid:
# raise DrmException(“invalid PID checksum”)
# pid = pid[0:-2]
Then try running mobidedrm.py passing in the 8 digit book PID provided by skindle on that same book put use the original copy of the book and not what is produced by skindle.
Then let us know what happens …
Basically we are letting skindle figure out the book specific PID that is needed but we are ignoring its attempt to remove the drm itself since skindle’s code that does the same as mobidedrm.py has some flaws that can come back to bite you.
We then modify mobidedrm.py to only need 8 digits of PID (ignore the checksum stuff) and then let the modified mobidedrm.py which now only needs the 8 digit PID to try to remove the drm from the original copy of the book (NOT! the output from skindle) and let it do the removal.
I know this is messy but please give this a try and let us know if it works for you or not.
August 23, 2010 at 7:20 pm
Some_UPdates,
Sorry it took so long to get back – I was out of town.
I tried modifying mobiDeDRM0.16.py as suggested. Unfortunately it did not work. The result that I got from running mobiDeDRM was as follows:
MobiDeDrm v0.16. Copyright 2008-2010 The Dark Reverser
MOBI header version = 6, length = 232
Extra Data Flags = 2
Error: no key found. maybe the PID is incorrect
Anything else I can try???
August 23, 2010 at 7:58 pm
Hi,
If skindle provides the book specific PID and you have tried that with a modified mobidedrm.py and it thinks the PID is wrong, then I would guess the copy of the file you have was downloaded to some other Kindle version than the version of Kindle4PC you have right now.
Please try again after using your current Kindle4PC to download the book again from Amazon (see the Archived items) and use this freshly encoded copy of the book.
Then try DiapDealers new K4PCDeDRM script on the same machine and in the same account you run Kindle4PC on.
If that fails , then you have me stumped unless the book is actually a Topaz book in disguise or one of the really old TEXtREAd style books. If it fails, then please try using any text editor to simply open the fresh copy of the book to look at the text information embedded near the front of the book to make sure it has the right Mobi headers and not TEXtRead or Topaz.
This is strange indeed.
August 24, 2010 at 1:47 am
some_updates:
I am not sure what was going on. I went through my computer and deleted all versions of the book. Downloaded a fresh version, then tried DiapDealers new tool which worked just fine. I will continue to fool with it a bit to see if I can figure out what happened, but in essence I think DiapDealer’s solution has resolved my problems.
Many thanks for all your help…
August 21, 2010 at 2:26 pm
I tried this and it worked. The DRM books decryped with skindle and with mobidedrm016 are the same (same checksum).
August 21, 2010 at 2:52 pm
For some books skindle and Mobidedrm will produce the same output. For some, the skindle output will be corrupt. When that happens, MobiDeDRM 0.16 should produce a good copy.
August 23, 2010 at 4:52 am
Help! I tried using the unswindle thing, followed all the steps to a T and I get a message saying “unsupported version of Kindle for PC”. I’m not exactly a super tech, I had a hard time with the unswidle process, thank God for youTube, and the guys at TooSmartGuys.com (the only reason I figured it out). Anyone have any suggestion, I need my freakin’ textbook!
August 23, 2010 at 5:24 pm
I slapped together a Windows python script that combines Skindle’s K4PC pid-finding technique with MobiDeDRM’s superior decrypting routines. The result is K4PCDeDRM. The script works on files downloaded to K4PC only and must be run on the machine that has K4PC and Python installed on it. It MAY work in virtual Windows environments that have K4PC and python installed, I don’t know. Just feed it two parameters – the input file and the output file – and it will do the rest.
The good news for Windows users is that if you have K4PC, Calibre and python installed on the same machine, you can also install this script as a Calibre plugin… just like MobiDeDRM before the Kindle 2.5 update. That’s the level of functionality I was looking to restore.
It works on all of my books, but I expect errors – so give it a workout 😉
Download the script from: http://www.mediafire.com/?fw5ahj31kzbwgxm. It’s ready to be installed as a plugin, but if you want to use it as a stand-alone script… unzip it and change the name to something handy.
August 23, 2010 at 5:49 pm
It’s possible that I’m too stupid to figure this out (it’s happened before with scripts) but all I get is a flash of a python terminal and that’s it.
I know it’s a pain in the butt, but if someone figured this out, could they please tell me how they did it…in detail?
August 23, 2010 at 6:32 pm
You can’t double-click on .py scripts and run them in that manner. You have to open a command prompt (Start, Run, type “cmd” click OK) then switch to the directory where the script is saved. Then you have to type a command to run it. For example:
python K4PCDeDRM-v.0.0.1_plugin.py encrypted-book.prc name-for-unencrypted-book.mobi
August 24, 2010 at 4:17 am
DiapDealer you are my hero! This worked perfectly and is so much easier than messing about with the older scripts. Thank you!
September 12, 2010 at 6:10 pm
DiapDealer, you are made of pure awesomeness! Thanks so much!
August 23, 2010 at 6:31 pm
Hi DiapDealer,
Nicely done! This will make things easier for many Windows/K4PC users.
Thanks!
August 23, 2010 at 6:34 pm
Hi DiapDealer,
One thought …. If you look at MobiDeDRM.pyw in tools_v1.8.zip, I bet it would be easy to modify it to make a K4PCDeDRM.pyw for people who do not like command lines.
Hint hint!
Thanks
August 23, 2010 at 6:49 pm
I’m on it. 😉
August 23, 2010 at 8:48 pm
I created the gui version of K4PCDeDRM for those who don’t like the command line stuff. Download it and unzip it somewhere. Double-click the the K4PCDeDRM.pyw file and away you go!
The previous link is still viable for a Calibre plugin.
**Note** Extract ALL the files from the zipfile and leave the directory structure intact. All of the files are necessary for it to work properly.
Note to Some_Updates… The K4PCDeDRM.pyw file is expecting to find the k4pcdedrm.py in the lib subdirectory. This way, it will be easier to incorporate it into the tools-vX.X archive… after some testing proves that it’s somewhat stable, of course.
http://www.mediafire.com/?1p6c45fdzdcm95d
September 6, 2010 at 8:49 pm
DiapDealer:
Many thanks for the GUI version of K4PCDeDRM, it is most appreciated.
If you are ever feeling bored, I have a couple of suggestions for the GUI version, 😉
– Add the ability to process multiple files.
– If possible, get the internal (actual) name of the book and use it as the output filename.
– Remember the K4PC book directory.
– Remember the output directory.
– If there is no DRM, then simply copy the file to the output directory with the correct file name.
Again, many thanks for the application…
August 23, 2010 at 6:43 pm
Yes… I know… DUHHHH.. Sorry, that’s what I get for trying this BEFORE I have any caffeine. My apologies….it’s been too long since I’ve tried this. Now I feel like a COMPLETE idiot…
August 23, 2010 at 10:01 pm
Hi DiapDealer,
The GUI version works like a charm on Parallels (virtuual Machine) running Windows XP on my Mac!
Very nice work! I never knew that the cmbtc python code implemented the same pid calculation process that was implemented in skindle. I just assumed that the process for Topaz PID determination would be completely different and so never really looked at it.
With this tool, no one will need skindle or unswindle as it does both jobs without having to patch any code and without any known bugs.
I would be happy to release a new tools collection to include this as soon as you say it is ready.
Great!
August 23, 2010 at 11:04 pm
Ok, I’m not sure I’m doing this right. Do I just drag and drop the K4PCDeDRM into python? Also, I have Calibre installed on the comp as well, how do I make it a plug for that? If I do that, will I be able to DeDRM while converting to an EPub at the same time? Cause that would be awsome.
August 23, 2010 at 11:35 pm
Heather,
You just unzip the K4PCDeDRM-Gui file, go to the folder where you extracted the files and double-click on the K4PCDeDRM.pyw file.
To use it as a Calibre plugin, just download the earlier file (http://www.mediafire.com/?fw5ahj31kzbwgxm) and install it as a plugin from Calibre’s Preferences page.
The plugin works automatically when you import a book into Calibre – it strips the DRM (behind the scenes) as it adds it to the library, so it happens before you would do any converting to EPUB.
August 23, 2010 at 11:24 pm
some_updates,
So glad to hear it works on Parallels! I can’t get it to work under Wine on Linux yet, but I couldn’t get skindle to work under Wine either, so it’s no real surprise. I’ll figure it out.
Yeah, I do believe that skindle borrowed the cmbtc_dump routines and just ported them to C. Looking at the skindle source, the only difference between topaz and mobi PID determination was where it looked for two particular tokens from the ebook itself. Everything else was the same.
Using K4PCDeDRM as a Calibre plugin should really please a few people too. Many have been missing that functionality.
Go ahead and release a new version of the Tools collection (with K4PCDeDRM included) anytime you’re ready… the more people that use it, the quicker the bugs can be found and eliminated 😉
Thanks!
August 24, 2010 at 12:22 am
OMG! Thank you sooooooo MUCH!! It worked! I can use my textbook on my Nook! 😀 Do you happen to have any plugins to De DRM Barnes and Noble Ebooks? That would be really helpful to.
August 24, 2010 at 3:13 am
Wow! K4PCDeDRM plugin works perfectly! I did get the “unable to convert DRM” message when importing my azw book into Calibre, but I ignored that, imported the book, then was able to convert to epub.
Well done! And thanks very much to all responsible for the different tools. I have used most of them at some point.
August 24, 2010 at 3:47 am
Anyone know how to remove the DRM from B&N epubs?
August 24, 2010 at 4:28 am
I don’t know how to use them, but the latest Tools collection has scripts that work on B&N Epubs.
http://www.mediafire.com/?3ro5pqsjbetc6bs
August 24, 2010 at 5:44 am
I clicked the download link and it took me to yahoo hotjobs, do you have another link for the download. I’d like to try to figure it out, see if making it a plugin will work for that to.
August 24, 2010 at 1:32 pm
Heather, to do B&N epubs, use the 3 files in the Barnes_and_Noble folder in the tools found at the link DiapDealer gave. You must have python and the crypto installed, but I think you need that for all of these tools; so you will have that already.
Run ignoblekeygen.pyw to generate the key. You only need to run that once unless you buy your books with a new credit card. Then run ignobleepub.pyw to do the job on the book. It will use the key made from the keygen in the first line. The second line is the book original, and the third line you put the decrypted book title which I give a slightly new name so as not to overwrite the original.
ignoblekey.pyw can be used to get a key from the B&N Desktop reader if you use that. I’ve never tried it.
August 24, 2010 at 4:43 pm
A new version of the tools collection has been posted:
tools_v1.9.zip
http://www.mediafire.com/?r8qj7eqorj4hrkp
The changes include:
– Reorganizing folder structure to put all Kindle For PC applications in one place
– Adding PIDCheck.py to convert 8 digit PIDs to 10 digit PIDs (thanks Paul)
– Adding in DiapDealer’s just released K4PCDeDRM gui and plugin that make properly removing DRM from Kindle4PC books a snap. Thank you DiapDealer!
K4PCDeDRM should remove the need for skindle which has bugs when decoding some books that can result in corruption (you can still use skindle to get the book PID and then mobidedrm to remove the drm if you so desire).
unswindle will still be needed for Linux installations since Kindle4PCDeDRM will not work under Linux/Wine directly.
August 24, 2010 at 4:49 pm
A Call for Help!
It is obvious from the questions here that people could really use some help/readme information to show how each tools is used, describe their purpose, talk about getting python installed under Windows, etc.
Would some kind souls please write up some html pages (with screen-shots if they know how to do that) that describes their favorite tool, etc.
You can then post them anonymously on mediafire or pastebin or where-ever and they could be collected and added to the next release of the tools archive to make things easier for people.
I think html would be best since almost any user could double-click the file to read things in their favorite browser.
Any help doing this would be greatly appreciated.
August 26, 2010 at 7:04 pm
Diap_Dealer: If this resounding success is what you get when you “slap together” something, I’d love to see what you could do if you were actually trying! Thank you for an absolutely painless solution for liberating K4PC files!
However, now you have me wondering….would it be possible to turn some of the other scripts into plugins for Calibre? Or would the different systems they use make that impossible?
August 26, 2010 at 11:42 pm
Thanks! But, all the code but was really already written. I just had to merge the code from three different existing scripts and add about ten lines of my own code to make it work. 😉
I’m sure some of the other scripts could be converted to Calibre plugins (with the exception of Topaz Tools), but I don’t really use formats other than azw/mobi, so I don’t really have any way of testing/playing.
August 26, 2010 at 11:46 pm
I’m going to be buying a bunch of epubs on the 7th (new release Tuesday for mysteries). I’ll volunteer to use those to test if you’ll see if the scripts can be made into plugins for Calibre. Some of those titles will be available as B&N pdb too, I’m betting, if you’d want to try that one.
August 27, 2010 at 4:30 pm
@ zanf
Thanks for the offer, but you’d have to give me some pretty sensitive data about what you used to purchase those epubs for them to do me any good. 😉
I’ll look into purchasing a few myself sometime and check into it.
August 30, 2010 at 3:03 am
If you were thoughtful enough to buy your Nook for cash at the B&N store and your e-books with a pre-paid card registered to some nym no one knows ur meat-space ID and you are free to share, cc information once you have spent all its balance.
August 27, 2010 at 1:43 pm
What about Drmed .PRC from Fictionwise is there a calibre plugin?
August 27, 2010 at 2:13 pm
The standard MobiDeDRM (which is available as a calibre plugin) will handle .prc from FW.
August 27, 2010 at 6:11 pm
Diap_Dealer, no, actually I was figuring it’d be like most DRM schemes, it’d have to be decrypted on the machine they were purchased on. I was thinking more along the lines of you try to figure out what coding that needed to be used, send me the script, I test it and give you the output.
Of course that would be more complicated than you buying the files and testing them on your own machine where you can make instant changes to the scripts as needed. But it would save you some money, LOL.
August 31, 2010 at 9:12 pm
Help! Skindle, LZ Skindle, K4PCDeDRM all suddenly stopped working for me. I have changed nothing.
If I run LZSkindle the error I get is:
The skindle program is called here.
Any errors reported between here and “The command line used was:”
Are errors from the skindle program. Not EZskindle4PC.
Attempting to locate kindle.info
Found kindle.info location
Using UserName = “Marcy”
Using SystemDrive = “C:\”
Using VolumeSerialNumber = “519629645”
Device PID: 1BFFN4SW
PID for Bake_Sale_Murder is: CySvmmhK
Found a DRM key!
Success! Enjoy!
Attempting to locate kindle.info
Found kindle.info location
.\input\It bad open, quitting
The command line used was:
skindle -i .\input\It Takes a Genome_ How a Clash Between O – Greg G -o .\outpu
t\\
August 31, 2010 at 11:48 pm
The output looks like it worked for at least one book. Are you trying to do a lot of books at once? If so, remove that problem one (It Takes a Genome_) from the list and try it again.
For that particular book:
It looks like LZSkindle isn’t building the input file path properly. It’s bailing after the first space in the filename. Change the name of that book to something simple (with no spaces) and try it again.
If K4PCDeDRM is failing, try selecting the ebook directly from the “My Kindle Content” folder with the original name that K4PC gave it.
Are the scripts not working for ANY book you try, or just that one?
September 1, 2010 at 9:16 pm
The scripts aren’t working for any book I try. I first tried using Kindle4PCDeDRM directly with the ebook in My Kindle Content folder. I only went to LZSkindle after Kindle4pCDeDRM failed on 4-5 books.
BTW, the Genome book *is* the original name. I didn’t change it.
The aggravating thing is this all worked perfectly when I first installed it.
September 2, 2010 at 3:42 am
Hi Marcy,
DiapDealer is right. That is a very strange name for a book downloaded from Kindle. It almost looks like a book that has been converted with calibre.
Either way, using skindle the way you did is not correct:
skindle -i .\input\It Takes a Genome_ How a Clash Between O – Greg G -o .\output\\
One problem is that the file name has spaces in it but no quotes were used to help the command line version of skindle figure out where the file name begins and ends. What you have above makes no sense to skindle.
Whenever you try to pass a file name or path that has spaces or any other special characters in it, you should put quotes around the the path/file name and if necessary ‘escape” any special characters.
This should be done automatically for you by K4PCDeDRM.pyw as long as you actually select the correct book file,
The kindle book file names typically end with .prc or .azw and look much like what DiapDealer has shown in his response to you. (ie. like B001T5WLMA_EBOK.prc).
So please make sure you are actually pointing the program at the actual file that is the book and not something else.
September 2, 2010 at 4:12 am
I will re-download and install K4PCDeDRM as soon as I finish typing this.
The Genome book is a red herring. I was using LZSkindle and forgot that book was in the input folder. I was doing the actual trial on a book I downloaded today which also was in the input folder.
If the skindle use is incorrect it is an error within LZSkindle, as that is what I used to generate that message.
September 1, 2010 at 9:46 pm
Hi Marcy,
Please grab a fresh copy of the latest tools archive (tools_v1.9.zip) and “extract” the entire zip archive just to make sure nothing has moved or renamed anything – the directory structure must be intact for the files to find their lib\ archive.
Then navigate to tools_v1.9\Kindle_4_PC\
and then simply double-click on K4PCDeDRM.pyw
Use the top … button to select the input file (the book you want to convert) and the second … button to select a new name and location for the output file.
Then hit start.
If an error happens, please copy everything in the Conversion Log window and post it here so that we can see the exact error message.
September 2, 2010 at 12:53 am
Marcy,
The only reason I questioned the name of that ebook is because I found that particular book on Amazon’s site. Since it is being offered for free, I downloaded it to my K4PC and it was delivered as ‘B001T5WLMA_EBOK.prc’… which is pretty standard for a K4PC filename.
Not sure why yours was different, but I would definitely try what some_updates has suggested first.
September 2, 2010 at 4:37 am
Re-downloading everything worked. All fixed! Thanks for the help.
I wonder if it was because my copy of Kindle for PC got “de-registered.” I had to re-register it and some time after that I noted the DeDRM’ing wasn’t working.
September 2, 2010 at 11:42 am
Marcy,
Glad everything is working again!
And yes, trying to decrypt books that were downloaded with a previously registered version of K4PC is going to give you trouble. The tokens from the kindle.info file is not going to match up with the tokens pulled out of the ebooks themselves.
September 2, 2010 at 2:00 pm
DiapDealer, I knew that I couldn’t decrypt the old books, so I deleted everything and re-downloaded them. But K4PCDeDRM still couldn’t decrypt the newly downloaded books for some reason. I even tried to “buy” some new freebies to try a book that definitely hadn’t been previously downloaded, but no go.
I don’t know exactly what the problem was, but I’m grateful it was solved so simply!
September 2, 2010 at 2:30 am
Trying to strip drm from a couple of amazon books. When I use any of the tools I get the same error everytime:
Parse Error: Failed payload read of record color:1 offset 0x31e3b0:0x0
It worked on a couple of other books but it didn’t work for me on these two. Any help?
September 2, 2010 at 3:29 am
Hi Prime Time,
Perhaps they are Topaz books?
The error you are seeing comes only from skindle. It appears in the source code in tpz.c
.//Kindle_4_PC_Tools/skindle-06/tpz.c: fprintf(stderr, “Parse Error : Failed payload read of record %s:%d offset 0x%x:0x%x\n”, name, index, fileOffset, fileSize);
Which other tools have you tried and what exact error message are you seeing in those tools? They can’t all be the same?
October 3, 2010 at 6:57 am
I’ve got the same error in skindle. In cmbtc_dump.py it looks like it works well but that isn’t true because, although you get all the page.dat, you only get the first glyphs, not all.
October 3, 2010 at 12:23 pm
Forget it, I suppose it was a random error. skindle gets the same error but Topaz Tools worked at last (not at first time) and I had to run them a couple of times, but at the end they worked.
September 2, 2010 at 4:01 am
Yes, I believe, even though they end with .azw, they are topaz books. Is there a different tool I am supposed to use to strip drm from that format? Thanks.
September 2, 2010 at 7:33 am
Cannot decrypt topaz format book
Command = “python k4pcdedrm.py”
K4PC Path = “C:\skindle\input\B001P80G0A_EBOK.prc”
Output File = “C:\skindle\output\B001P80G0A_EBOK_nodrm.mobi”
Please Wait …
K4PCDeDrm v0.01 provided DiapDealer.
Error: invalid file format
Error: Encryption Removal Failed
September 2, 2010 at 11:34 am
Yes, K4PCDeDRM does not decrypt topaz files. I saw no point in having the script produce a decrypted topaz file that is unusable by anything other than the Kindle itself.
If you need that functionality, skindle will work.
If you need to convert the topaz file to another format…the Topaz tools scripts are the proper way to decrypt and convert topaz files.
September 2, 2010 at 12:08 pm
Since updating B&N Reader, ignoblekey.pyw can not longer find the Reader installation. I get Error: Could not locate BN Reader location.
On the other hand, ingnoblekeygen works just fine, as does ingnoblepub, because I still have my bnepubkey in the folder.
I have tried replacing ClientAPi.dl in my Appdata folder, and I have tried changing a few details in ignoblekey.py, to insure that ClientAPI is properly named, but to no avail.
I’m just curious about what’s up. I can still convert my books, for which I am very grateful, but what changed that makes ignoblekey unable to find BN REader installation?
September 2, 2010 at 1:29 pm
In reference to my previous post, I tried uninstalling BNREader, and installing an older version, and I still got the Could not locate BNREader error message.
But, I’m not complaining. ignoblekeygen still works brilliantly.
Thanks, saviors of the digital book world.
September 2, 2010 at 2:08 pm
Hi Lawrence,
ignoblekey.pyw is very old and is no longer needed or wanted. It only worked on windows and only worked on some versions of the software. IHeartCabbages has replaced it with ignoblekeygen.pyw which generates the proper key based on the information you give it.
ignoblekeygen.pyw is the correct program to use (you only have to run it once unless you change the name or creditcard number used to buy the books) and it works on all platforms. You can simply throw out ignoblekey.pyw.
See IHeartCabbages blog for more information.
September 2, 2010 at 11:13 pm
So how do we decrypt topaz files? Which of the tools do we use?
September 2, 2010 at 11:18 pm
Okay, I used the Topaz ebook file extration but I have no idea what to do with all the .dat files that it produces? How do I get this into some usable format?
September 2, 2010 at 11:36 pm
Prime Time, there were instructions given earlier. I’ve posted them below. You will need the free calibre programs.
The Topaz tools are the only way I know to make this conversion. You can run the Topaz to HTML then the HTML file can be converted by Calibre.
First you have to set up a directory with the following files:
1. TopazExtract_Kindle4PC.pyw (I do not know if this will work with the new version of the Kindle for PC. )
2. TopazFiles2SVG.pyw
3. TopazFiles2HTML.pyw
4. Subdirectory called lib with all the topaz library files.
Second, run the files in this order:
TopazExtract_Kindle4PC.pyw
TopazeFiles2SVG.pyw
TopazeFiles2HTML.pyw
Make sure you specify and output directory! I use output.
When you have finished the book will be in the output directory as a HTML file. However, there will be other files and subdirectorys which were generated by the conversion process.
Finally, move the HTML file to Calibre (It will show up as a zip file in Calibre) You can then convert it to another file format.
It does a pretty good conversion, however, be aware that it is doing a OCR process to convert the images to text. I had several books where the Chapter titles were in a unique font which it had problems converting. I had to use a html editor to go through them an edit the Chapter headings.
September 3, 2010 at 2:51 am
Hi Drak,
Those instructions are correct. I just want to clarify something you wrote …
> It does a pretty good conversion, however, be aware that it is doing a OCR process to convert the images to text.
Actually, there is **NO** OCR being done by the Topaz Tools. The “text” for the book comes from OCR done by Amazon when it created the book and which is stored inside the Topaz book (so that the images that make up the book pages can be searchable by the Kindle end user).
All the Topaz Tools html conversion does is use the text stored inside the Topaz book to try to recreate an html version of the book. Any errors in the text are do to errors in the OCR that was done by Amazon when they created the Topaz book.
Also, when you use calibre to convert the book.html to something else don’t forget the image subdirectory and the css style sheet that are created and stored in the output directory. Calibre should be smart enough to grab all of the images and style sheet but, if you plan to archive the book itself, don’t forget these pieces as well.
You might also want to archive the svg directory as it has images of each page of the book in case you want to proof it or keep it for posterity.
September 3, 2010 at 3:00 am
“It does a pretty good conversion, however, be aware that it is doing a OCR process to convert the images to text.”
Not to be a stickler or anything, but the topaz tools are not doing any OCR. The original OCR data from when the ebook was created by Amazon is included in the topaz format. It’s there so you can do text searches from your Kindle or K4PC. But it contains mistakes (from Amazon not the scripts)… that’s why text searches on a topaz book don’t always produce the expected results.
In short, the OCR errors are Amazon’s… and the topaz tools recreate those errors accurately and diligently. 😉
Oh… and you don’t really have to go to all the trouble of setting up the correct directory structure if you just download the Tools_v1.9 archive. When you extract all the files, it puts everything where it needs to be.
September 3, 2010 at 3:03 am
Those are amazingly similar posts!
He, he 🙂
September 3, 2010 at 3:30 am
Okay, 2 books down 1 to go. It has a .tpz extension if that means anything. When I try to run it in topazextract, I get this error ” File “D:\tools\Topaz_Tools\lib\cmbtc_dump.py”, line 849, in main
dkey = getBookPayloadRecord(‘dkey’, 0)
File “D:\tools\Topaz_Tools\lib\cmbtc_dump.py”, line 405, in getBookPayloadRecord
raise CMBDTCFatal(“Parse Error : Invalid Record, record not found”)
__main__.CMBDTCFatal: Parse Error : Invalid Record, record not found” earlier in the log and “Error: File Extraction Failed” at the end. Any suggestions?
September 3, 2010 at 4:05 am
Hi,
My guess is there is something wrong with the book you downloaded. I don’t know of any .tpz extension. Either that or the book has no DRM key (ie. is not encrypted) – but I have never seen that either for a Topaz book.
Please download a new copy of the book from Amazon. And try things again.
If it still fails in the same way, open it in any text editor to see if the TPZ string is near the top of the file.
Please let us know what you find …
September 3, 2010 at 7:10 am
hmmm, TPZ is one of the extensions that LZSkindle adds to it’s output, no?
September 3, 2010 at 12:27 pm
Topaz files downloaded to the Kindle can have a tpz or azw1 extension.
I’ve not tried the Topaz tools since the Kindle 2.5 update, but I assume that the new DRM scheme would prevent you decrypting topaz books sent to the kindle–same as mobi(azw) files.
Make sure you are using the book that was downloaded to K4PC (should have a prc or azw extension.
Also make sure you are not running the Topaz tools on a book you already decrypted with Skindle (as soalla suggested). The scripts must be run on the original encrypted ebook.
September 8, 2010 at 1:22 am
i have tried all of the posted mediafire links that have been put up in the last 2 or so weeks and am unable to open any of the pages. Am i doing something wrong or do i need to sign up for something? I’ve been trying for days to get of the drm on a book i bought for k4pc but want to read on my pandigital novel. it has a prc extension but gives the error unable to decrypt topaz file when i unswindle. I’m hoping either your calibre plug-in or gui file will fix it up for me but i can’t open the links.
September 8, 2010 at 1:38 pm
Hi,
That means it is a Topaz book. The only want to convert its format is to start with a fresh copy (NOT the output of unswindle!) and use the Topaz Tools to convert it to html. Be warned … the conversion is not perfect (Amazon has many OCR errors in its Topaz books) and is a lot more involved then using a single program or plugin, although there are GUI tools to help. You can find full instructions in posts over the last month or so on this blog.
September 8, 2010 at 1:51 am
never mind, duh….pop up blocker!
September 8, 2010 at 7:54 am
I haven’t been on here in a long while!
What’s the best way to de-drm the BN Epub’s? Is it still I love cabbages one?
Also, what would be best for de-drming an Apple ebook, if possible!
Thanks!
September 8, 2010 at 1:42 pm
Hi John,
Yes, IHeartCabbages scripts are the only way to go (you can get them from the tools collection talked about on this site or directly from the IHeartCabbages blog.
There is no tool that will remove DRM from an Apple (Fairplay DRM) book. I recommend simply not ever buying ever from the Apple bookstore so that their format for DRM dies a quick death.
This is not much of a hardship outside of the US as the iBooks store for Canada is a complete waste of time. I would guess it is similar in other countries.
You can always buy other places and once the DRM is removed it can easily be read from within iBooks if you want to. So you can still read it with iBooks if you buy anyplace esle. If you but via the Apple bookstore, you will be tied to Apple hardware for a very long time and perhaps forever.
September 11, 2010 at 2:45 am
I have a topaz format book that I can’t get encryption removed on. Here is the output from the conversion log but it doesn’t look very helpful:
Conversion Log
Command = “python k4pcdedrm.py”
K4PC Path = “C:\Users\Mark.Mark-PC\Documents\My Kindle Content\B001GMAPCY_EBOK.prc”
Output File = “C:\Users\Mark.Mark-PC\Downloads\tools_v1.9\tools\Kindle_4_PC_Tools\B001GMAPCY_EBOK_nodrm.mobi”
Please Wait …
K4PCDeDrm v0.01 provided DiapDealer.
Error: invalid file format
Error: Encryption Removal Failed
September 11, 2010 at 3:40 am
Hi post_reply,
Wrong tool. A Topaz format book is very very different from a normal Amazon (Mobi) style book. The only way to convert it is to use the Topaz_Tools which are part of tools_v1.9.zip.
If you only want to remove the DRM and not convert it, then you can try using skindle but be aware you still will not be able to read it or even see it on anything other than a Kindle and that skindle has bugs in it which might come back to bite you.
September 13, 2010 at 12:31 pm
Is there any other tool that removes the DRM from topaz? I’ve been having problems with skindle, and the process of converting them to another format is too involved. Since I won’t buy another e-reader, just stripping the DRM would be enough for me…
September 13, 2010 at 2:59 pm
Hi Rafael,
If you won’t buy any other e-reader then you don’t ever need to strip the DRM from the Topaz files. There would be no point to it. The only reason to remove the DRM from a Topaz file but not convert it is if you want to give away copies of the book to someone else who also has a Kindle reader. For your own Kindle readers, you can simply download a new DRM copy to any KIndle you have.
So you can see why many people don’t think simply removing DRM from a Topaz book but leaving it a Topaz book is a good idea to support.
Perhaps someone else here can help you, but for me it makes no sense.
Sorry
September 13, 2010 at 3:37 pm
Everything you said is true. And I should swallow my words, because I already had a bad experience with eReader (formerly Peanut Press) books, which I converted with no greater problems to read in my Kindle. What I meant is that until Kindle dies as a platform, I won’t be using anything else, so I would be wise to keep my options. Problem is I abhor DRM per se, it is an outrageous concept. Also, while Topaz isn’t my first choice by far, some books are only available in it. And lots of freebies come as Topaz as well, so I have about 50 books which I am too lazy to convert to other formats then check the formatting, etc.
But I guess I’ll end up doing it…
September 13, 2010 at 3:55 pm
“If you won’t buy any other e-reader then you don’t ever need to strip the DRM from the Topaz files. There would be no point to it…. So you can see why many people don’t think simply removing DRM from a Topaz book but leaving it a Topaz book is a good idea to support.”
Uh, no offense and don’t consider this any sort of flame but I do think you are way wrong on that from every this standpoint. One of the main reasons that people remove DRM from their ebooks is device independency. And if you want to be sure that your Topaz ebooks will someday work on your Sony 5000 or your Nook Vesrion 10 then you better remove that DRM now while the tools we have work on the ebooks we have. We havbe already seen in the mobidedrm world how Amazon changed their DRM scheme on ebooks we already have and made it impossible to redownload and strip the DRM using the same techniques as before. If Amazon changes their Topaz formatting in some respect and breaks the Topaz Tools it may well leave you regretting that you did not unDRM AND convert those files while you had the chance!
September 13, 2010 at 5:28 pm
Hi Thothamon,
You wrote …
> If Amazon changes their Topaz formatting in some respect and breaks the Topaz Tools it may well leave you regretting that you did not unDRM AND convert those files while you had the chance!
That is my entire point. I am obviously not against removing the DRM to convert formats. But if you remove the DRM using skindle from a Topaz book, you are in no better position for archiving it or converting it. Nothing else will read a Kindle Topaz file but a Kindle. By running the Topaz tools, it removes the DRM and then unpacks the binary encoded xml files and glyph files that describe each page 1 by 1 to make them useable for something else (conversion, storage, archiving). Just removing the DRM will not do that. The program TopazFiles2XML.pyw will even output the XML in human readable form that can be parsed and even converted to another format if someone so desires to write one. All the programs TopazFiles2SVG.pyw and TopazFiles2HTML.pyw do is read that XML (in a machine readable form) and convert them to SVG pages and xhtml, respectively.
So unlike Mobi books where simply removing the DRM makes them convertable and archivable, removing the DRM from Topaz books does not allow anything else to read them, does not make them archivable, and does not allow anything else to convert them.
Even the person who wrote the DRM removal code (CMBTC) disabled its ability to do anything to a whole book because until someone figured out how to reverse engineer the binary coding to get back the original xml and convert it (even imperfectly) to some other format, its only purpose was for piracy.
My 2 cents …
BTW: Any c or python programmer can actually easily modify the cmbtc code, or skindle code to do what you want (only to remove the drm) so please ask and someone here might help you, I just am not willing to do that (speaking for myself only) as I can’t see a legitimate reason for using it since it will not allow conversion and won’t therefore allow for useful archiving.
All of that said, FWIW, if I am remembering correctly, someone did post a program to do that here to this list someplace so if you look around in the history (searching for cmbtc) you should be able to find something.
September 13, 2010 at 5:56 pm
You said:
>Even the person who wrote the DRM removal code (CMBTC) disabled its ability to do anything to a whole book because until someone figured out how to reverse engineer the binary coding to get back the original xml and convert it (even imperfectly) to some other format, its only purpose was for piracy.
I disagree to that, and as OP to the request, I think I should comment on that.
First, by saying that all the help I get here is *extremely* valuable, and I don’t want to start a flame war or anything else, so my say in this ends with this post.
As I mentioned on a previous post, the prospect of decrypting, converting and checking for errors (which I obsessively tend to do) fifty-plus Topaz e-books does not appeal to me. Perhaps I’ll do that to the couple I *had* to buy, only because they aren’t available in any other format. The others are Kindle freebies, mostly religious fiction, which I keep only because I got them, not that they are particularly appealing to me.
Someone said Topaz is a poor man’s PDF, and I agree to that, it was a bad choice Amazon made, which shows up when you compare the volume of Topaz books to the volume of Mobi ones.
*But* for all its faults, OCR errors and shortcomings, Topaz *can* produce better typesetting results than Mobi. An example is a book I purchased, Groundswell, which I tried to convert previously with Topaz Tools, but didn’t get glowing results.
Which, coming to think of it, is probably the cause of my willingness to put up with Topaz, not DRM. And certainly not piracy.
But, perhaps I was inept with the tools that were provided, and if I try again I may get better results.
Anyway this has turned into a rant, and it’ll stop. I won’t ask again for such tools, but please don’t assume I want to pirate stuff. I have legally bought lots of book, and will continue to do so, because I believe in the e-Book format. I think it’s the greatest thing since sliced bread, as I can carry around the four or five books I read simultaneously.
Anyway, sorry if I offended anyone, and please keep up the good work.
/rant off
September 13, 2010 at 6:42 pm
Hi Rafael,
You said,
> Anyway this has turned into a rant, and it’ll stop. I won’t ask again for such tools, but please don’t assume I want to pirate stuff.
I apologize. I never meant to imply you were a pirate. I was just trying to explain why that option does not exist or get much support (at least from me).
My purpose with these posts was trying to explain that even if you do remove the DRM from Topaz style Kindle books (all 50 or only 1), it does not *help* you in any way. Unlike Mobi, with or without DRM – nothing else but a Kindle can read it.
So the only way right now to archive a Topaz book (or convert it), is to run the Topaz Tools and include the XML options so that if in the future someone else writes a tool that can better handle the xml files for each page, you can use it regardless of whether the Kindle changes its format or dies.
So I do strongly recommend running the Topaz Tools (a script can be written to automate this process) and include the SVG, HTML, and XML outputs, copy in your kindle.info file and a clean copy of the topaz .prc or .azw book and then archive everything as a rar or zip , or tar.gz archive. I would only ever bother to clean-up the html for books that I really really wanted to convert to another format. For most, I would just leave them as is and not clean them up.
That way if Kindle dies, or changes its format or drm, you have everything needed (the binary files, the kindle.info and the xml, etc) to do with what you want. Just keeping the non-DRM version of a Topaz file, will not help with this given the current Topaz Tools will not even work on a non-DRM topaz books.
Again, sorry for any confusion my responses to Thothamon generated.
Hope this explains things more clearly.
September 13, 2010 at 6:52 pm
OK, I am obviopusly not understanding something here. You said that:
“My purpose with these posts was trying to explain that even if you do remove the DRM from Topaz style Kindle books (all 50 or only 1), it does not *help* you in any way. Unlike Mobi, with or without DRM – nothing else but a Kindle can read it.”
But I buy Topaz books from Amazon all the time and convert them to non-DRM’d ePUBs for use on my iPad. I have done this at least three or four times. Basically I use TopazFiles2XML.pyw. Then I use Calibre to read in the XML files by choosing the All Files option in Add Books as it will not read the files otherwise. However, it does read them in fine and from there just convert to ePub using Calibre.
September 13, 2010 at 7:02 pm
I spent two miserable obsessed crazed days trying to do what you just described, Thothamon…but apparently I missed a step with calibre (the all files option in add books). Off to try again…
September 13, 2010 at 7:31 pm
Hi Thothamon,
You are confusing two things:
1. removing the DRM from a Topaz book (as can be done via skindle) but NOT extracting the XML or HTML or SVG – just leaving it a binary encoded topaz book
2. removing the DRM and then extracting the contents, dumping the XML and converting it to SVG, XML, HTML so it can be converted to some other format (with Calibre) as is done *not* by skindle but by the Topaz_Tools
I am saying the just doing number 1 above will not help since nothing else but a Kindle can read it (making any archive of it worthless).
Instead I am recommending number 2 as the right approach for archiving or converting Topaz books.
Hope this makes things clearer.
September 13, 2010 at 7:44 pm
It does clarify and I apologize because I was leaving my how-to message below as, I guess, you were leaving this one.
September 13, 2010 at 7:58 pm
On a side note, does skindle damage mobi files as it does topaz?
September 13, 2010 at 8:16 pm
IHi Rafael,
It can but the bugs in skindle do not hit every file. They only impact some books depending on the size and how they handle the trailing data bits.
There are two bugs in skindle I know about:
1. the first is that skindle does not properly encodeNumbers for some number ranges as reported by clarknova and this corrupts some binary files:
See clarknova’s post at the end of this url for the details and a proposed fix.
http://www.openrce.org/forums/posts/1199
2. The second is that MobiDeDrm.py has had a number of fixes to handle trailing data bits properly that have never been incorporated into skindle’s C version of what Mobidedrm does.
Both bugs would have to be fixed before you could guarantee that skindle will not corrupt anything.
The skindle archive does have the c – code and both clarknova’s proposed fix and an updated version of the c that does what mobidedrm does could be added by someone.
September 13, 2010 at 7:10 pm
Zanf — I think the problem is that you have to tell Calibre to look for All Files and then it will load the xml book. But, apparently, this is luck because it is not a “supported” format. I just found it in desperation myself one day.
September 13, 2010 at 7:43 pm
Sigh. OK, I apologize to all as I was not properly recalling how I did this but just went and retried everything and here is the procedure I use to get Kindle Topaz books onto my ePUB-lovin’ iPad.
First I have the ebook sent to my Kindle for the PC from the Manage Your Kindle page. I then open the TopazExtract_Kindle4PC.pyw script. From there I choose the input file from the Kindle Content directory in the documents of my user directory. I usually extract the dat files of the unTopaz’d book to a new folder on the desktop just to make it all easy to find.
When that is done I run TopazFiles2SVG.pyw to get the svg directory and files the next step needs.
Then I run the TopazFiles2HTML.pyw script which results in a book.html file. Then I can just use calibre to read in that html book file. (Some confusion, I think in older scripts I needed the 2XML step but apparently not any loinger).
Once the book is in Calibre I just convert it from the html (it will say it is a ZIP file) to ePub or any other fomat. Calibre then transfers it to my iPad without problem.
The above is a step-by-step I just did on a book (“Cruise Confidential”) that is a topaz book I ourchased at Amazon. I apologize if this repeats other information here but I was surpised others were not able to buy Topaz books and convert them to other formats. Hope this helps and if I have made some stupid error of assumptrion I do apologize as, frankly, I am usually in over my head here. LOL
September 13, 2010 at 7:46 pm
Ok, I got that far, but how do you get it to convert after they load? All I get is “461 of 462 books unable to be converted, unsupported format”.
September 13, 2010 at 7:46 pm
Never mind, we cross posted…. LOL! Thank you Thothamon.
September 13, 2010 at 7:56 pm
It’s not that we were unable to convert the HTML to ePub (or at least that isn’t my particular problem…). The problem I was trying to get at is that when you ‘explode’ the decrypted topaz into HTML, the HTML leaves out bold and italics. So, what I was trying to do was find an easy way to work with the XML or SVG output, which still preserves those, (or at least the SVG output does). That’s why I got so excited when you said, “read in the XML files”. However, on further research, I’m not sure that the XML output does preserve bold and italics, so I’m no better off in any case.
September 13, 2010 at 8:25 pm
Hi Zanf,
No it does not save bold or italic info. That is simply inherent in the glyphs that are used to create the images of each page.
I do think that someone smart could take the output of the svg/ directory and create high resolution graphical images (jpegs) of each page from the svg images of each page (bulk convert the svg to jpegs by imagemajik or something like that) and then pass those jpeg images into a really good OCR program (one that supports text and image areas with some layout suppor that **should** be able to do a much better job of creating a cleaner more complete version of the book with fewer OCR problems and better mapping to actual fonts (as opposed to glyphs) with their associated font characteristics kept (italics, bold, etc).
You might also be able to create a full image-only pdf from those same images as well (but it would be huge).
September 13, 2010 at 8:26 pm
My answer to the topaz issue is simple: don’t buy topaz books. Yes, I know some ebooks are only available as topaz, and you may REALLY want the book… but the sooner people stop buying buying them (or returning them for a refund if they accidentally buy one), the sooner the format will die the horrible death that it deserves.
Other than that, my feelings on producing a decrypted topaz book from an encrypted topaz book are the same as Some_Updates… the result is a book that can only be read on the device that it could already be read on.
Having said that, the existing scripts (Python and C) could be fairly easily modified to achieve what you want, but it’s probably going to have to be a self-service kind of thing. Which is how I developed any scripting that I may have. 😉
So use what’s available or make your own. I don’t say that to be mean or rude… it’s just how this sort of community works. 🙂
September 13, 2010 at 8:40 pm
I don’t recall asking for anyone to write any scripts to make this simpler, I was only hoping that Thothamon had stumbled on something I’d missed. Not that that would be hard to do!
September 13, 2010 at 8:56 pm
Sorry zanf,
You and thothamon posted a lot of stuff in the time it took me to compose my post… which was addressing the original topic of decrypting vs converting topaz. The one about 10 or 12 posts previous.
Nothing to see here… move along folks. 😉
September 13, 2010 at 8:49 pm
Man, this is a bad post to reply from the iPhone 🙂
I have by now agreed that leaving books in Topaz is bad. I’m gonna try my luck with the scripts, but it’s not something that comes easy to me.
And from the last reply by some_updates, it is clear that it isn’t a 100% solution as well, since, as a picky graphic designer I believe typesetting an formatting are integral to books, even e-books.
That said, does the mapping of glyphs change from book to book or is it constant?
Could we remap the glyphs to formatting (bold/italic) and if so, how?
I tried to read the SVG with photoshop but couldn’t, is there any specific program to do so?
September 13, 2010 at 10:51 pm
Hi Rafael,
> I tried to read the SVG with photoshop but couldn’t, is there any specific program to do so?
The files in the svg directory are all svg images embedded inside .xhtml files.
You should be able to open any of the .xhtml files with a good browser (Safari, Firefox, etc) and see an exact graphic image of each page and use the arrow triangles to move from page to page of these images.
If instead you want the svg images for each page – not embedded in an xhtml file, there is a way to create those by adding a optional switch to the conversion program. This is not possible with the gui tool yet, but doable from the command line version.
I can modify the GUI tool to do that for you if you want to work directly with the svg images. Just let me know and I can post a version of TopazFiles2SVG.html that will create pure svg files versus svg files embedded in xhtml files.
Anything that understands svg images should be able to then open the pure .svg files (such as Scibus or Inkskape, or Imagemajik or …).
September 14, 2010 at 12:50 am
Hi some_updates,
No need for a special version of the tools, knowing How it works helped me a lot. Don’t wanna take up your time just for that. I was reading elsewhere and it seems the general consensus is Amazon blotched it by choosing to implement both Topaz and Mobi, when ePub would be a lot better for formatting purposes.
And it looks like Topaz was very poorly put together, as it could be better structured. Anyway, thanks for the offer.
September 13, 2010 at 9:03 pm
Ah, gotcha, DiapDealer 🙂 No worries, hon.
September 13, 2010 at 9:38 pm
“That said, does the mapping of glyphs change from book to book or is it constant?
Could we remap the glyphs to formatting (bold/italic) and if so, how?”
This is exactly where the problem lies. The map just points to the data for the particular glyph to display. There is nothing stored in the map about formatting whatsoever. The topaz reader just says; “I need a character here!” and the map points to the vector data that will draw an image (of a character) in a particular place on the Kindle’s screen.
So unless someone can come up with a highly intelligent algorithm to detect that the vector data will produce more slanted lines (or slightly thicker lines), there is just no way to know what was originally in italics (or bold)… except with a side by side comparison.
OCR software could work on the SVG files, but frankly, any program that would do a decent job is going to be pretty costly. There’s some free open-source stuff out there (Ocropus, Gocr, Tesseract and the like), but I’ve found that I could probably retype the book myself quicker and easier than re-training OCR software for each book’s particular typeface.
September 13, 2010 at 10:08 pm
I see the problem. If the table mapping was constant, like ASCII or such, then converting to italics or bold would be a lot easier.
That is why I disliked the converted results from the books I tried before.
I agree with you, Topaz should die a fiery death, but sometimes there are no other options.
September 14, 2010 at 1:29 am
Ok, I’m probably doing something wrong here, but please bear with me. I’m getting this kind of output from the tools:
Conversion Log
Command = “python cmbtc_dump.py”
Topaz Path Path = “D:\Tools\TOPAZ\B001C4TEIM_TOPAZ.prc”
Output Directory = “D:\Tools\TOPAZ\Decrypt”
First 8 chars of PID = “”
Please Wait …
File “lib\cmbtc_dump.py”, line 774
except Exception as message:
^
SyntaxError: invalid syntax
Error: File Extraction Failed
The books are in Topaz format for sure, and they have been downloaded from may through september.
I am using TopazExtract_Kindle4PC script, and my K4PC is old, version 1.0.1 Beta 1, from april
September 14, 2010 at 6:04 am
“So unless someone can come up with a highly intelligent algorithm to detect that the vector data will produce more slanted lines (or slightly thicker lines), there is just no way to know what was originally in italics (or bold)… except with a side by side comparison.”
Please search for my other posts here (under the name “Elminster”) where I describe a workable approach that helped me to produce a high quality html from a topaz book with 1000’s of italic and/or bold words.
In short, it is possible to:
– generate a mapping between glyphs and the character they encode by comparing used glyphIDs with the used character in the OCRed text
– for the glyphs where that mapping is not consistent, it’s possible to just show them to the user and have the user make a decision. Same for glyphs that encode multiple characters (e.g. italic ff, fl, lf, tl, lt things like that get very often lumped into a single glyph). Once there is a confirmed glyphID to character (-sequence) mapping, the glyphIDs can be directly used to generate the output text instead of the OCRed text, producing much better results.
– now the trick for identifying all italic and/or bold glyphs: have the user identify a single one (neither bold no italic; italic; bold and bold/italic) manually, then search for all glyphIDs that occur together with this one in a single word. Perform this search recursively for all newly identified glyphs until all glyphs are clearly identified (if necessary have the user again manually identify glyphs if some are “unreachable” from the ones originally manually identified).
I only implemented this as a one-off hardcoded solution for a single book (with hardcoded glyphIDs and such in the source, modifying the source as I successively got closer to the wanted output).
But it should be possible to wrap this up into a more general implementation which, with some user interaction, should be able to convert topaz to as-close-as-possible-to perfect html in a fraction of the time it currently takes.
It all comes down to not trying to identify individual glyphs, instead you make use of the fact that italic and/or bold glyphs occur in clusters, commonly know as “words”.
September 14, 2010 at 3:08 pm
Hi Elminster,
Yes, I remember and although it did not work for my strange book – too many unrelated single char errors on my book which had lots of foreign words, it did work well for identifying clusters of bold and italics chars.
So if you or anyone else if interested in coding up a generic version of your approach in python, I would love to include it with the tools.
Thanks
September 14, 2010 at 11:57 pm
Hi some_updates,
“So if you or anyone else if interested in coding up a generic version of your approach in python, I would love to include it with the tools.”
Unfortunately, Python is not a language I usually work with. Also, the one Topaz book that I already converted was the only one I ever bought (was before I figured out that books that don’t specify a filesize on amazon are topaz books) so I have neither a need for a generic version nor further material to test it on.
I fear someone else will have to take this up. When someone does, I’ll be happy to provide feedback and guidance about how to best perform the requires steps.
September 15, 2010 at 12:46 am
Elminster, thank you for the tips, I’ll check your files, but due to my lack of programming skills, I don’t know if they will be any good. I have ran into some books that are a challenge. For example, in one of them, a passage reads, in a cursive font: Thank you for your book, “The Messies Manual.” It has become a major force in my home. (I would handwrite this, but excited handwriting is hard to read.) When I first bought it, my husband thought it was just one more way to procrastinate as I am an avid reader and sat down immediately to read it cover to cover.
Now the OCR’d output:
7″lan~y(Oa 6(P1 y(Par bOO~, “-“le Messier Monuo(.” /~ /as
b9,-,0M9 a mae.(Pi 60xae in my tiame. (l wOa(al1analtV ite
6Laf eXaiLfeLI AanIw1irLin~Lis l aaal /(P ~eaa 7 bt/A n
bit st boa t it , any Aasbana~ t A(~a t irL w04 ?4 One
M610-e way fl P 1(Pa~aLS~inLa~e as / aLm an aviaIxeaa1eX anc>l
-liar a/~wn iMMeaiat e(y t (~ 1e(G7 iT 011efi ~(P a(PV 1.
Good isn’t it? BTW, was this OCR done at Amazon, the publisher or is it the script somehow? If it is Amazon or the Publisher they should be ashamed of it.
But regardless of anything else, I must concede that Topaz has its virtues. Typesetting and page composition are much better here than on mobi or even ePub.
September 15, 2010 at 2:29 am
Hi Rafael,
You wrote:
> Good isn’t it? BTW, was this OCR done at Amazon, the publisher or is it the script somehow? If it is Amazon or the Publisher they should be ashamed of it.
The Topaz scripts do NO ocr. All OCR is the product of Amazon.
Hard to believe I know but true!
And that is not the worst I have seen by far. You should see the absolutely unreadable gibberish Amazon stores in its ocr text field when there are outline fonts being used (especially in titles and headings).
To see if it helps at all, you can pass a command line switch
genhtml.py –fixed-image BOOKDIRECTORY
(just in case that switch begins with two dashes and not one long dash – this website seems to change these from what I write)
to genhtml.py so that it will take any fixed region and create an image of it instead of trying to use the ocr text from Amazon for that region.
This may or may not help.
I sometimes use this to get two different html versions of the books and then merge things I need from the fixed-image version back into the normal version.
And, just for kicks please run the genxml.py program as well and it will create human readable xml files in the xml directory that you can open with any text editor to see the exact ocrtext and fields used for any page. Those are the actual guts of the Amazon format.
September 15, 2010 at 3:18 am
It is hard to believe how Amazon missed an opportunity here. Still playing Devil’s Advocate, as an e-book format, Topaz makes sense, as it allows for typesetting, different font families, and hell, they are even platform-independent, as the glyphs don’t rely on hardcoded fonts. It really could have been an answer/alternative to PDF, but the execution lacks so much it feels someone took a spare weekend, put it together and rolled out without so much as a gamma-testing.
September 17, 2010 at 1:05 pm
Rafael wrote:
> “Still playing Devil’s Advocate, as an e-book format, Topaz makes sense, as it allows for typesetting, different font families, and hell, they are even platform-independent, as the glyphs don’t rely on hardcoded fonts.”
I actually agree with you on this. Some people have issues with Topaz books being slow (page turns) and claim the text is harder to read on the Kindle. I only have two Topaz ebooks in my collection, but if I’m perfectly honest… they look gorgeous and read beautifully on my kindle. If you’ve seen mobi books that use a font that is unsupported by the Kindle, then you’ve seen the ugliness that Topaz could correct.
Unfortunately, topaz fails utterly when you “open the hood” and see the junk that was used to create it.
September 14, 2010 at 2:14 am
Where that error is occurring in the script (774) indicates that the script is having difficulty parsing your kindle.info file.
Can you open that particular book with K4PC?
Is your K4PC registered?
Have you tried reinstalling K4PC/Re-registering and re-downloading that particular book?
September 14, 2010 at 3:27 am
Happens to all my books. Even Mobi ones, though the error is a different one.
Might be because my K4PC is very old. I kept it around and didn’t upgrade because some time ago the script only worked with the first versions. Guess it’s time to upgrade.
I’ll try that.
September 14, 2010 at 2:48 pm
Hi,
It should work on the old version as well (that is the one I have installed on my machine) but upgrading it won’t hurt anything either.
Please verify a few things for me …
– you have Python 2.X installed (not 3.X) and a 32 bit version
– you have installed Kindle For PC in the same account on the same machine that you are trying to run the tools on
– your kindle.info file exists and is located inside your Local AppData folder here:
‘\Amazon\Kindle For PC\
{AMAwzsaPaaZAzmZzZQzgZCAkZ3AjA_AY}\kindle.info’
– your kindle.info file matches the version of Kindle For PC you are using (and not an old version) – you can assure this by removing any old kindle.info files and deregistering and then registering the latest version of Kindle For PC and then make sure the file exists as above.
– you have downloaded the books you want to remove the DRM from to the latest version of Kindle For PC ***after*** properly registering it.
– you have grabbed the latest version of the tools archive (v1.09) and have extracted the entire archive and have not moved or renamed anything (they need to find their lib directories)
– If you have a mobi book too, try double clicking on K4PCDeDRM.pyw and see if gives you a different error message and if so what it says. If this works then the Topaz Tools should as well.
– then try double clicking on TopazExtract_Kindle4PC.pyw and see if it works and if not, if it gives a different error message.
My guess is that you have an older kindle.info file around someplace and that is causing the renaming or moving of any new kindle.info file when you try to register it (but this is a complete wag).
Hope something here helps.
September 14, 2010 at 3:03 pm
Whoops,
In my instructions, I should have said grab the latest tools version 1.9 not 1.09.
Sorry about that.
September 14, 2010 at 3:20 pm
Ok. here is the output from K4PCDeDRM:
K4PC Path = “D:\Tools\MOBI\B000JMKU0Y_EBOK.azw”
Output File = “D:\Tools\MOBI\Decrypt\B000JMKU0Y_EBOK_nodrm.mobi”
Please Wait …
File “lib\k4pcdedrm.py”, line 553
except Exception as message:
^
SyntaxError: invalid syntax
Error: Encryption Removal Failed
and from TopazExtract_Kindle4PC:
Command = “python cmbtc_dump.py”
Topaz Path Path = “D:\Tools\TOPAZ\B001M0MNW0_TOPAZ.azw”
Output Directory = “D:\Tools\TOPAZ\Decrypt”
First 8 chars of PID = “”
Please Wait …
File “lib\cmbtc_dump.py”, line 774
except Exception as message:
^
SyntaxError: invalid syntax
Error: File Extraction Failed
I have done everything else you said. My python is 2.6, and oddly enough, skindle works. I tried it just for the sake of testing, and it decrypted the Topaz (with errors) and Mobi (I assume properly) books.
September 14, 2010 at 3:24 pm
Hi,
One other thing …
To check the version of python that is found first in your path (in case there are other versions on your machine), simply open a terminal session (run cmd.exe) and simply type “python” and hit return.
It should report the version info and then you can use quit() to get out of the python prompt.
I have seen errors on many programs with file io from Python 3.X that do not exist with Python 2.X so that could also be something to check out.
September 14, 2010 at 3:49 pm
Actually, both Python 2.5 and python 2.6 are installed. 2.5 is active.
I’m getting a slightly different error now:
D:\Tools\Topaz_Tools\lib>cmbtc_v2.2.py -d -o book B001GCUJI4_TOPAZ.azw
Traceback (most recent call last):
File “D:\Tools\Topaz_Tools\lib\cmbtc_v2.2.py”, line 899, in
sys.exit(main())
File “D:\Tools\Topaz_Tools\lib\cmbtc_v2.2.py”, line 890, in main
createDecryptedBook(outputFile)
File “D:\Tools\Topaz_Tools\lib\cmbtc_v2.2.py”, line 686, in createDecryptedBook
outputFile = open(outputFile,”wb”)
IOError: [Errno 13] Permission denied: ‘book’
September 14, 2010 at 4:06 pm
Hi Rafael,
Two things:.
1. Your error:
createDecryptedBook(outputFile)
File “D:\Tools\Topaz_Tools\lib\cmbtc_v2.2.py”, line 686, in createDecryptedBook
outputFile = open(outputFile,”wb”)
IOError: [Errno 13] Permission denied: ‘book’
means it is trying to create a file called ‘book’ inside
D:\Tools\Topaz_Tools\lib>
but it is being denied permissions because you do not have write access either to that directory or that a file of that name already exists.
In general, writing any file anywhere inside the tools_v1.9 directory (or its sub-directories) is not a good idea as it my end up overwriting something needed for the python code to work.
2. cmbtc_v2.2 is NOT the same thing as cmbtc_dump.py and running it will in no way allow you to run the next steps. cmbtc_v2.2.py is the fixed original code created by cmbtc and only kept as a reference so that it never gets lost.
This file is not used in the conversion because all it will do is remove the DRM and not dump the book (the thing we had the whole discussion about).
Also,
One reason to get a write error inside the tools directory is that you have not actually EXTRACTED the entire contents of the zip archive and are instead trying to run things form inside the zip archive.
Please double-check that you have actually extracted the entire contents of the tools_v1.9.zip archive and are actually in the extracted directory when you double-click the *.pyw files and NOT inside the zip archive itself – which Windows will in fact open and make it look like everything has been extracted when actually it has not been.
Hope something here helps.
September 14, 2010 at 4:43 pm
some_updates,
thank you for the great help, I am not usually that dumb, and can work around things.
I didn’t know cmbtc_2.2 was deprecated, but thankfully, from the command line cmbtc_dump works.
As for the zip, yes, it is uncompressed in its entirety.
Since I have two Win7 installs, I’ll move the folder to my current install, which I have full rights. Also, I was decompressing things inside the tools folder just for sake of ease, so as not having to type the whole paths, but I’ll set the paths fully.
What is puzzling me, is that the Topaz tools work from the command line, but not from the python window scripts; and that the mobi doesn’t work from either mode.
September 14, 2010 at 5:29 pm
Hi,
You wrote:
> As for the zip, yes, it is uncompressed in its entirety.
Since I have two Win7 installs, I’ll move the folder to my current install, which I have full rights.
Ah 2 different Win7 installs is probably the issue. To work, the tools must be able to find the kindle.info file which is created by Kindle For PC on the same machine and account as the tools are used as well as to which the original books are downloaded to.
You can not remove the DRM from a book that has been downloaded to a different Kindle For PC since its kindle.info file will be different from what it finds on another machine.
To make it clear, please verify that you are using the latest version of Kindle For PC and that you have registered it, that the kindle.info file exists, that you have downloaded the books to it, that you have fully extracted the tools_v1.9.zip archive and that ALL of these are happening on the same machine and in the same account.
> What is puzzling me, is that the Topaz tools work from the command line, but not from the python window scripts; and that the mobi doesn’t work from either mode.
Ah! It sounds like your python install has not been properly done. pythonw.exe should exist and be in the python 2.5 directory that you said is found first in the path. All .pyw programs should be associated with that pythonw.exe executable (or python.exe).
Please verify this by typing just “pythonw” in the cmd.exe window and hitting return (no quotes obviously) and make sure it is running from the very same python installation. Then try the same for “pythonw.exe”, “python”, and “python.exe” (use quit() to get away from the python or pythonw command prompts) and verify they all go to the same Python directory that is found first in your path.
Assuming they do – double-clicking on any of the .pyw files should bring up the GUI program and if everything said above about the same account on the same machine exists, they should run properly (they simply run the command line versions).
If they do not, then you have not properly set your system PATH environment variable to reflect the python installation you want to use. You can go to control panels to check what your system path environment variable is set to and fix it if needed and restart for any changes to take effect.
Something here should hopefully help.
September 14, 2010 at 6:01 pm
some_updates,
I gave up on the windows stuff and will do everything by command prompt. Mobi through the command line is working as well, all the paths are set, and I’ll write a big batch to do everything and don’t bug anyone anymore 🙂
Thanks for the help.
I’ll try my chances with editing Topaz by hand, since the xhtml (or the xml, can’t remember now) has the text along with the pages, it’s easy to compare them. As a side note, on one book I decrypted, the glyphs in the 1000s mapped to regular characters, 2000s mapped to italics, 3000s and 4000s were a cursive alphabet and so on, so it *might* be feasible to make a script or a program that takes that kind of info.
Anyway, not for my low-grade skills.
Again, everybody, thanks for the help!
September 14, 2010 at 3:54 am
Ok, now I don’t know what to do. Upgraded K4PC, nothing. Redownloaded book, nothing. De-registered K4PC. Registered K4PC again, nothing. Downloaded book again, nothing. Same errors.
September 14, 2010 at 1:44 pm
Sincerely stuck here. Did all of the above again, now deleting the old kindle.info, to see if that worked and it didn’t.
This is the output from both command line tries:
D:\Tools\Topaz_Tools\lib>cmbtc_v2.2.py -o book B001M0MNW0_EBOK.azw
Traceback (most recent call last):
File “D:\Tools\Topaz_Tools\lib\cmbtc_v2.2.py”, line 899, in
sys.exit(main())
File “D:\Tools\Topaz_Tools\lib\cmbtc_v2.2.py”, line 785, in main
raise CMBDTCFatal(“No action supplied on command line”)
__main__.CMBDTCFatal: No action supplied on command line
D:\Tools\Topaz_Tools\lib>cmbtc_dump.py -o book B001M0MNW0_EBOK.azw
Traceback (most recent call last):
File “D:\Tools\Topaz_Tools\lib\cmbtc_dump.py”, line 880, in
sys.exit(main())
File “D:\Tools\Topaz_Tools\lib\cmbtc_dump.py”, line 766, in main
raise CMBDTCFatal(“No action supplied on command line”)
__main__.CMBDTCFatal: No action supplied on command line
September 14, 2010 at 2:08 pm
You’re not supplying the action option for the command-line version, so it doesn’t know what you want to do. You can extract individual records with the -r option, or extract the whole book with the -d option… which is what I’m guessing you want to do so:
cmbtc_dump.py -d -o book B001M0MNW0_EBOK.azw
Be warned… most K4PC books have a prc extension (a few have the azw extension) so make sure you are running the script on the book downloaded via K4PC–not the book downloaded to the Kindle or the PC. Also, the newly downloaded book may not be a topaz book anymore–many get updated to mobi later on. The only way to truly determine that it is a topaz file is to open the file in a text editor and look for TPZ near the top.
September 14, 2010 at 3:26 pm
DiapDialer,
Ok, i traced the problem. Using cmbtc_dump things work, but cmbtc_v2.2 they won’t. Still doesn’t explain what’s going on with Mobi decryption as well.
September 17, 2010 at 1:10 pm
A while ago I wrote a program to add a better GUI to Igor’s scripts. Since the Amazon change in the DRM, this no longer works. I have developed a new program in VC++ that incorporates some of the work done by folks here. Since I used their scripts to back up my books I figured I should contribute. Here is a link to the program. As with others it does require that Kindle4PC be installed and will only convert books downloaded through this vehicle. It adds a simple GUI interface that converts 1 or more books at a time and re-names them based on the header information found.
http://pacilio.net/kindle/kindledrm.zip
September 17, 2010 at 1:42 pm
Hi Bill,
Nice work … but where is the source code? I would really hate to distribute just an exe file with no way for others to roll their own, to do what you have done yourself from the skindle code, or to verify there are no back doors, etc
So if you are willing to share the source, I would love to add it to the next release of the tools set if you would allow it.
Thanks
Kevin
September 17, 2010 at 5:24 pm
I want to make sure the basics work and then I’ll post the whole thing
September 20, 2010 at 8:52 pm
I’m with some_updates. I’m reluctant to use it, because of no source.
September 20, 2010 at 11:59 pm
I’m of the “no source, no use” persuasion myself. I appreciate your contribution, Bill… and no disrespect is intended, but I just gotta see the code.
For instance: are your decrypting routines the same ones from Skindle, or did you port mobidedrm-0.16 to C? I ask simply because Skindle has some known issues that mobidedrm doesn’t suffer from.
September 21, 2010 at 10:23 am
I have used the skindle code since it was already ported. I have run it on all my kindle books purchased from amazon without issue. As I said, this is an early release and I have not added extra’s or tried to fix anything in the skindle code. My wrapper does nothing but add GUI and comment out all the printf’s to stderr. I’d like to port the code to an object based class system but would like to have it working well before the port. If there are issues with the skindle code I’d like to solve them before I port it. Once this is all done I’d be happy to post my code.
September 21, 2010 at 11:05 am
There’s a comment by some_updates a little further up the page on this site – https://darkreverser.wordpress.com/2008/02/13/new-blog/#comment-1731 – that details the issues that can cause corruption in skindle’s output.
Some get lucky and never run into the problem with the ebooks they own. The issues wouldn’t arise when running the skindle script, but rather when reading the file that skindle produces.
September 21, 2010 at 11:37 am
Thanks for the info. I looked at the post but it relates to Topaz files. I have no implemented that portion of the skindle script. I don’t own any Topaz books and so I could not check it.
September 21, 2010 at 1:34 pm
No, the bug pertaining to the Trailing Data Entries does indeed apply to decrypting mobi files. That all takes place in the mobi.c file of the skindle source.
At a guess, I’d say that Skindle is based on a pre-0.14 version of mobidedrm. The changelog in mobidedrm-0.16 details at what point the problems with multibyte data and trailing data entries were addressed… and a quick side-by-side comparison of skindle’s mobi.c and newer versions of mobidedrm confirms that those fixes never made it into Skindle.
The changes would need to be in the getTrailingDataEntries function and the end of the writeMobiOutputFile function of mobi.c.
The getTrailingDataEntries function exists in its entirety in mobidedrm. The section in question in the writeMobiOutputFile function would correspond to lines 230-246 in the mobidedrm script.
Hope that helps.
September 21, 2010 at 6:35 pm
DiapDealer, here is what I found. The getTrailingDataEntries seems to have some code that was commented out in the newer mobidedrm. The way the skindle calls the PC1 function, it send in the same buffer as the src and destination. The end result of this is that the trailing chars, if they exist, remain there and are not changed. This is essentially the same as what the .16 version is doing. I did see that the memory was being malloc’d instead of calloc’d so it is possible that the errors could come from an uninitialized location. I’d did change this and comment out the other issue. It still works with my stuff. Does anyone have something that the old skindle didn’t work with?
September 21, 2010 at 7:26 pm
Sorry Bill, I don’t have one of the books that Skindle corrupts. But hopefully someone who does will come along soon so you can test whether you’ve fixed it or not.
My C is not up to snuff at all, or I would have taken a stab at it. I would have sworn that mobidedrm was doing something slightly different at lines 239-241 (the if statement) than Skindle is doing at 358-359, but I guess I’ll have to take a C guy’s word for it. 😉
September 21, 2010 at 7:56 pm
Lately Skindle has corrupted everything I’ve tried. The output file is about 50% larger than the original. I can send you a list of some that are still free at Amazon:
Immortals: The Crossing;
Guns Will Keep Us Together;
Living and Learning with New Media;
The Future of Learning Institutions in a Digital Age;
Confronting the Challenges of Participatory Culture.
If none of those works, I can look out for others. Usually the latest free books that aren’t out of copyright have one or more Topaz in them.
September 21, 2010 at 11:32 pm
Hi Bill,
I eyeballed the code lines that Diap Dealer pointed out. And I think once you remove the code that was commented out in MobiDeDRM 16 from your/skindle code that it should do right thing as long as those are the only two bugs or differences between them.
This is just by eyeballing the skindle C code and Mobidedrm python code in those routines. The malloc versus calloc should not matter since exactly len bytes should be written to the malloc’d memory if things are working.
My 2 cents …
September 22, 2010 at 10:43 am
Thanks to Rafael Oliveira for the books names. As it turns out, all of these books that you reported a problem with are Topaz. I did not implement the Topaz portion but I see that I will have to try. Up until now I’ve never got a book from Amazon that was in Topaz. I’ll get back to you
September 18, 2010 at 10:15 pm
It has worked very well for me, and it’ very nice to use. Thank you.
September 20, 2010 at 4:42 pm
Thanks Bill. Worked beautifully.
September 20, 2010 at 6:48 pm
This is incredible! I have resisted buying Kindle books because it appeared to be so difficult to remove the DRM and convert them. But this worked flawlessly! Thank you!
September 23, 2010 at 1:41 pm
I tried to put the topaz stuff into my code but have several issues. I backed off the GUI version and created a console version in VS 8. I had to create a libz.lib as well which I did with source code v1.2.5. I used the book files from amazon that where reported as not working. One of the files converts but will not open in any other reader and does not even show up when I copy it to my kindle but neither does the original K4PC one. I can download the version for my kndle and it then shows up. The other book errors during conversion in the WriteTopazOutput. getBookPayloadRecord returns a NULL that is not expected. I added the padding code from an earlier post but no change. I’ve attached the project for VS if anybody has it and is more familier with the Topaz.
http://pacilio.net/kindle/KinConsole.zip
September 18, 2010 at 1:18 am
@Bill:
I just tried kindle.exe, and it worked beautifully! It is in my tools folder now.
Thanks,
Lawrence
September 23, 2010 at 1:09 pm
I’ve come across an encrypted Mobipocket file that isn’t decoded correctly by MobiDeDRM 0.16.
It seems that the way utf-8 multibyte characters that split across data blocks are encoded isn’t consistent. My test file (v5/f1) had the extra characters included in the encrypted length of the block. This new file (v7/f5) does not include the extra characters in the encrypted length of the block.
To identify exactly what rules need to be applied I need more sample files.
Specifically, I need get hold of drmed mobipocket files that when decoded report the following versions and flags:
mobi header version=5, Extra Data Flags = 1
mobi header version=5, Extra Data Flags = 5 (or 7)
mobi header version=6, Extra Data Flags = 1
mobi header version=6, Extra Data Flags = 5 (or 7)
mobi header version=7, Extra Data Flags = 1
mobi header version=7, Extra Data Flags = 5 (or 7)
So if anyone has such files, could they please upload them somewhere (the DRMed version) and post a link to them here? For this check I don’t need the PID or the De-DRMed files, just the original DRMed files.
reagrds,
Alf.
September 23, 2010 at 3:14 pm
DiapDealer
In my last post I mentioned problems with the topaz books. I tried these same books with the k4pcdedrm.py script and they all (3) failed to convert. I tried the same test with a mobi and it worked fine. You can look at previous posts to get the names but one was “Guns will keep us together”. You get an invalid file format error.
September 23, 2010 at 5:58 pm
K4PCDeDRM doesn’t decrypt topaz files. I left it out on purpose. Mainly because the decrypted topaz file is unusable by anything except for the Kindle itself and K4PC (which could already read the file).
I point to the Topaz tools for decrypting & converting topaz books to another format.
September 23, 2010 at 6:33 pm
Hi Bill,
Please try running cmbtc_v2.2.py (a python command line tool) on those Topaz Files and please let me know what happens.
cmbtc_v2.2.py can be found in tools_v1.9\Topaz_Tools\lib\
The command switches are as follows:
python cmbtc_v2.2.py -v -d -o NONDRM_FILE_NAME TOPAZFILENAME
The author of this script (with some fixes added) was CMBTC who was the one who deciphered the Topaz DRM system.
Please see if his script can handle any of your problem cases and if not, what error messages they provide and let us know.
If it works, then you should probably use this script to track down any bugs in the skindle topaz treatment.
If not, and if those Topaz files actually open and work on a real Kindle then there are still some issues that have to be worked out with the script. All of the Topaz work began with this script.
Hope it helps.
September 23, 2010 at 7:13 pm
I tried this on the files and it does report success. But I’m not sure what kind of success since, as DiapDealer stated, the file is still not usable by anything other than K4PC. It doesn’t even show up when I copy it to my kindle. As an exercise I took the loooooong way around and used the Topaz tools to unpack the file, convert to svg, convert to html and then use mobicreator to make a .mobi. It works on all platforms so at least I know the book is not corrupt.
September 23, 2010 at 7:27 pm
I’m able to decrypt the file with the original cmbtc script. I have no trouble putting that decrypted file on my Kindle and reading it.
Bill, if I copy the decrypted topaz file to my kindle with a .mobi extension — it doesn’t show up. Could that be what you’re doing? If so, give your decrypted topaz output a .tpz extension and try it on your Kindle.
September 23, 2010 at 7:35 pm
Skindle does indeed blow up on this book with the following error:
Parse Error : Failed payload read of record color:1 offset 0x19565d:0x0
That error is in the Payload function of tpz.c
But we knew that Skindle was buggy to begin with. 😉
September 23, 2010 at 7:43 pm
Hi Bill and DiapDealer,
Out of curiosity I grabbed the “Future of Learning…” book and dumped it with cmbtc_dump and found a new section called “color” in the book.
It appears that since some devices now support color (iPad, ipod touch, etc) that Amazon has added a new section type they called “color”. In this section are stored color images (all jpegs), one for each of the jpeg images stored in the “img” sections.
The issue is that if no color image exists for its corresponding black and white image, it simply stores an empty record in that section.
For example if the original book had 11 black and white images stored in the image section in 11 records, then it will have 11 records in the “color” section. If only the first black and white image (call it image0000.jpg) has a color equivalent (call it colorimage0000.jpg) then its record in the color section will have positive size, all of the remaining records in that section will all have 0 length (after decompressing and removing the drm).
My guess is these 0 length placeholder records (you might call them null records) may actually have positive length after compressing (assuming the compressor attaches magic bytes to the front) and/or after drm.
So my guess is we need to look very closely at these null records and make sure we are properly handling the compressed flag bit and drm flag bit on them.
I will be updating the Topaz Tools to better understand and properly unpack and label these color images (and name them with .jpg) so that users can manually replace the black and white images with their color equivalents if they so desire.
Hope something here helps,
September 23, 2010 at 7:56 pm
Hi Bill,
You said …
> I tried this on the files and it does report success. But I’m not sure what kind of success since, as DiapDealer stated, the file is still not usable by anything other than K4PC.
Yes, all that will be done is that the file will have its DRM removed so it can be read on any Kindle. It still takes a Kindle to read the file since even after removing the DRM because the file itself is actually a binary coded set of xml files using a dictionary based compression to further shrink the file size, and mappings of glyphs to locations to describe the page as well as ocrtext information for searching.
Nothing else knows how to read this except for the Topaz Tools and any conversion of this to another format will not be perfect (unless that format is simply an image (svg or jpeg) of each page of the book.
So all the tool did was remove the DRM since the native format is not at all similar to html. HTML (sometimes with a few extras) is the basis of both mobi and epub books. So converting Topaz to some other format is not straightforward.
So if all you are trying to achieve with your new tool is to replace and fix the Skindle code and add a front-end, you just need to look at how the cmbtc_v2.2 script handles removing the drm and fix the skindle code (which should be very close to correct) to do the same thing.
September 24, 2010 at 10:56 am
DiapDealer, I converted the topaz an put the azw extension on it but it is not showing up in my kindle. Is it possible that the newest kindle has some added security? I am going to try to put the same file on my other kindle (k2) to see what happens.
September 24, 2010 at 12:57 pm
Bill, I don’t think they’ve added security on the Kindle 3. It’s just that topaz files downloaded to the Kindle have always had a .tpz or .azw1 extension…. usually.
I really think if you rename your decrypted topaz file with a .tpz extension, you’ll see that it shows up (and can be read) just fine on your K3 or K2.
September 24, 2010 at 1:07 pm
The books I downloaded through K4PC all came with azw extensions, regardless of format. If I downloaded through Amazon’s page though, they came with azw and azw1 extensions. But the Kindle was able to find them with Skindle.
September 24, 2010 at 1:14 pm
DiapDealer – You are right, I was having a brain drain moment. When you said put a .tpz on it I read .azw in my head. Thanks.
September 24, 2010 at 1:52 pm
I went through the skindle topaz code and added some checks for a null record. It does not appear as though the null adds any length so I just ignore it in the loop. As a result, the books that where crashing now work and appear to be fully readable (I scanned through them with my kindle). As a result, I added this to my program. I have attached links below to the program again and added a link to the project source for VS 8.
http://pacilio.net/kindle/kindledrm.zip
http://pacilio.net/kindle/Kindle_src.zip
September 24, 2010 at 2:03 pm
Hi Bill,
Wonderful.
Did you remember to include the fix by clarknova for the encodeNumber routine used by Topaz?
His suggested fix can be found in the last comment from here:
http://www.openrce.org/forums/posts/1199
I added that same fix (python version) to the cmbtc_dump code used by the Topaz_Tools and to the cmbtc_v2.2.py code shortly after clarknova posted about the bug but I don’t think skindle-06 has this fix yet.
Thanks.
September 24, 2010 at 2:23 pm
I put the fix in as well
September 23, 2010 at 4:01 pm
I got a question about tools.v1.9
It looks like I’m going to have to get a new PC and reload some of my programs.
So I need to know if the Kindle “crowbars” in the latest tools will work with the most recent Kindle For PC.
Thanks for any info.
September 23, 2010 at 6:01 pm
K4PCDeDRM in tools-v1.9 works fine with the latest version of K4PC… as long as the ebook is not a topaz file. For that, the topaz tools work fine with the latest K4PC version as well.
September 23, 2010 at 6:17 pm
Thanks!!!
September 25, 2010 at 9:58 am
k4pcdedrm.py and skindle.exe both seem to miss some possible PIDs
eg. pidHash = SHA1(DSN+kindleAccountToken+extb_records[209]+token)
Either DSN or indleAccountToken are omitted for some titles.
Thus one of 4 PIDS is valid. DevicePID , pidHash, or the two variants
September 25, 2010 at 3:03 pm
Hi,
On which books or which devices have you seen that DSN or KindleAccountToken are omitted? I have seen books that have extb_records and token set to 0’s but that was determined when the book was processed for downloading by Amazon, I believe.
I have never had the PID identification fail for Kindle 4 PC and K4PCDeDRM.py Please provide some specifics here so that people can look at this.
September 25, 2010 at 3:41 pm
Some examples of Amazon book titles where this situation occurs would be greatly appreciated. K4pcdedrm could be modified fairly simply to accomodate this — with some sample books to experiment with — but I’ve not experienced any PID identification problems.
If the DSN is left out of the book specific PID, then I know the device PID will never unlock the file… since the device PID is computed from the DSN.
September 25, 2010 at 6:23 pm
Whether a title gets the variant is not consistent.
Eg. “Predictable Irrationality” B002C949KE_EBOK fails to get PID on one account, works on another.
I verify this with breakpoint on Kindle4PC on PC1 routine.
If you corrupt the book pid, you will see the software try 2 more.
In order, it tries
a. device pid
b. book pid
c. book pid calculated without dsn
d. book pid calculated without user token
September 26, 2010 at 4:51 pm
Hi,
Is this repeatable in that account or just random?
Are you sure you have not downloaded the book to the desktop instead of downloading and opening the book under Kindle 4 PC on that account?
I could see a book that has downloaded directly to the PC (possible with Amazon) but not downloaded for Kindle4PC specifically to not have the same PID.
Please double-check that you can open that specific book in Kindle4PC and that you try to run K4PCDeDRM.pyw on the book, only *after^ you have verified it by loading and viewing it in Kindle4PC.
It is well known that 4 PIDs are used and generated by Kindle4PC, I just have never found one that did not use the book pid as calculated the way skindle/K4PCDeDRM does.
October 12, 2010 at 5:31 pm
Hi,
Any volunteers out there with Barnes and Noble epubs (with their social DRM) that have a Mac running OS X 10.6 (Snow Leopard) and a Nook?
Similar to how IHeartCabbages has modified the ineptepub and key scripts to use OpenSSL or PyCrypto, I have tried to modify both the ignoblekeygen.pyw and ignobleepub.pyw to do the same thing.
Pycrypto is hard to compile on Mac OSX and is rarely available while OpenSSL’s libcrypto is part of every Mac OSX install. so this should hopefully make the IgNoble tools much easier to use on Mac OSX.
I have been able to successfully test the new ignoblekeygen_v2.pyw and it works, but since I am in Canada, I can not buy or download any book from Barnes and Noble (even free ones). So I have no way of debugging or even testing the ignobleepub_v2.pyw script.
So if anyone can help me out with those, I would love to know and I can post the new scripts for you to try and report back.
Thanks!
October 12, 2010 at 6:13 pm
Hi,
Actually, you do not actually need a Nook, just have access to a Barnes and Noble epub file with its social drm, know the name and credit card number you used to buy it, and have access to a Mac running OSX 10.6 or later.
If OpenSSL’s libcrypt is part of Windows, then testing on Windows should be possible as well.
Thanks,
October 12, 2010 at 6:18 pm
I have some B&N digital books. I’ll try and test on a MacBook Pro with OSX 10.6. Let me know where to get the scripts.
Cheers,
Terry
October 12, 2010 at 6:42 pm
Hi Terry,
Thanks,
The first is ignoblekeygen_v2.pyw which you can get from here:
http://pastebin.com/eAZaq1fE
and the second is ignobleepub_v2.pyw which you can get from here:
http://pastebin.com/V1Hzkc57
please grab them, rename them as appropriate and double click to launch them on a Mac.
The ignoblekeygen_v2.pyw should work (it does on my machine).
After running it and building the bnepubkey.b64, then try running the ignobleepub_v2.pyw and see if it properly decodes the book.
If not, then please let me know the exact error message that appears at the top of the tiny window.
Thanks!
ps. if anyone else wants to try these, please do so.
October 12, 2010 at 8:18 pm
Well, I once again put my mouth in gear before my brain! Like Thothamon says all of my B&N books are in .pdb format. So . . . I can’t really offer any help after all.
October 12, 2010 at 8:43 pm
Hi Terry,
Oh well, Thanks for looking.
My call for volunteers is still open if anyone has a B&N epub and would like to test things, please let me know.
October 12, 2010 at 10:51 pm
I’m no help on the OSX front, but I thought I’d let you know that it works fine on Linux… which also comes with libcrypto. 😉
So Windows users are the only ones who need pycrypto installed for most of these scripts to function.
October 12, 2010 at 11:15 pm
Hi DiapDealer,
Wonderful! That means it will work on OS X too.
I hacked up ignobleepub_v2.pyw to only use libcrypto and not pycrypto for this test (see the loader commented out line) which I will fix and post as a final version and include these alongside the originals in the next tools release.
These should also make things easier for your B&N epub plugin if you want to merge them and create a plugin that should work on linux and MacOSX without pycrypto.
Thanks!
October 12, 2010 at 6:31 pm
I have bought over 30 ebooks from B&N but they are all pdbs! Not one ePub. If you point me toward an ePub there (maybe a free one or a cheapie) I can download it to help test….
October 12, 2010 at 7:15 pm
Hi,
It seems for some platforms (Mac’s included) when you do download a book it always seems to be in .pdb format (standard eReader tools work on it) and not B&N epub.
As far as I know, they only provide B&N epubs for Windows users and/or Nook users.
I am not sure what format gets downloaded using the B&N eReader software on an iPad or iPod Touch as again the damn B&N store will not let me buy anything from them since my address (and cc) are both from Canada.
Perhaps they only use the epubs on their own Nook? I am just not sure.
Thanks
October 12, 2010 at 8:52 pm
I had bought books from B&N that were not epub, but I found that if I now go back and download them again, they are now epubs. Maybe that will work for you.
October 13, 2010 at 1:40 am
Nope. Redownloaded about a dozen and they were all still pdb. As another message says, it is likely because I am on the Mac.
October 12, 2010 at 11:49 pm
Hi All,
Okay here are the final versions of the Barnes and Noble epub tools modified to use OpenSSL libcrypto OR PyCrypto.
These should make things easier for Mac OS X 10.6 users and Linux users who will no longer need to add PyCrypto since OpenSSL libcrypto is standard on these platforms. Windows users will still need to add either PyCrypto or OpenSSL to get things to work.
ignoblekeygen_v2.pyw
http://pastebin.com/eAZaq1fE
ignobleepub_v2.pyw
http://pastebin.com/bQ6L1Chk
These tools simply add support for OpenSSL in the same way that IHeartCabbages did for his ineptepub version 5 tools. All I did was port the changes he made in ineptepub version 5 to his version 1 B&N ignoble tools (with some slight additions and modifications). In other words, all kudos should go to IHeartCabbages.
I will be including them in the next tools (v2.0) to be released in a few weeks – after I finish adding preliminary support for color images to the Topaz tools.
Feel free to download and experiment with them on any platform.
October 13, 2010 at 5:47 pm
OK. I opened up Windows XP in a virtual machine and downloaded one of my B&N books. It came down as an epub file. Then I ran the scripts in Mac OS X by simply double clicking on them. They extracted and deDRMed just fine. My only quesiton is: since I do have pycrypto on my machine, did these scripts use it or do it the new way?
Cheers,
T
October 13, 2010 at 6:05 pm
Hi Terry,
From looking at the code that IHeartCabbages invented,
def _load_crypto():
AES = None
for loader in (_load_crypto_libcrypto, _load_crypto_pycrypto):
try:
AES = loader()
break
except (ImportError, IGNOBLEError):
pass
return AES
It will try to open the OpenSSL libcrypto first and if that works (does not throw an exception) then the break gets reached. If instead an exception is thrown then, break will not be reached and the exception is caught and and the loop begins again from the top, and so PyCrypto is then tried.
So it appears form the code that it will use the libcrypto (OpenSSL) stuff if you have both OpenSSL and PyCrypto installed.
Thanks for testing it!
October 13, 2010 at 6:11 pm
Cool. Also, on a lark, I went into the “site-packages” subdirectory in the Python directory and changed the “Crypto” name to “CryptoXXX” and reran everything. It still worked fine. I think that doing that would prevent it from using pycrypto.
Cheers,
TB
October 13, 2010 at 8:17 pm
That’s exactly how I test things like these.
October 14, 2010 at 1:20 am
Some_updates,
I’ve run into a little snag with the ignobleepub_v2.pyw script on linux. When using libcrypto, if I feed the script the wrong keyfile (i.e. created from the wrong username and/or credit card) for an encrypted epub, the program immediately SegFaults. It doesn’t do this with pycrypto (linux or windows). Line 126 of your latest pastebin code seems to be where it blows up. Does anybody know if the same thing happens on OS X?
It’s not that big of a deal, but the plugin I’m working on is setup to handle multiple keyfiles. I trap the errors from wrong keyflles and move on the the next key… but unfortunately, I can’t trap the C SegFault raised by libcrypto.
Any ideas?
October 14, 2010 at 3:02 pm
Hi DiapDealer,
Using a correctly formatted bnepubkey.b64 but with the wrong internal value (ie. I purposely did not give it the right Credit Card Number) on Mac OSX 10.6 the error is properly caught and the message I get back is “Error: Failed to initialize AES key”.
So everything seems to work fine on MacOSX using libcrypto.
Perhaps our libcrypto versions are different.
I am using
lrwxr-xr-x 1 root wheel 21 31 Aug 2009 libcrypto.dylib -> libcrypto.0.9.8.dylib
the 0.9.8 version of OpenSSL.
There are later releases, but Mac OS X has not updated to them.
I don’t think attempting to use an incorrect key (as long as it is the proper length) should ever cause a problem in libcrypto.
I will look further and let you know what I find.
October 14, 2010 at 3:20 pm
Hi some_updates,
It looks like we’re using the same versions (libcrypto.0.9.8.so). I’ve tried to get a backtrace from gdb, but wouldn’t you know, it tells me my stack is corrupted before giving me any valuable info.
I’ll keep investigating, but if I can’t track it down, I can always provide the stripped-down PyCrypto library for the Linux OS in my plugin (like I do for Windows) and leave the libcrypto code for the Mac Calibre users.
Thanks for looking.
October 14, 2010 at 4:27 pm
Hi DiapDealer,
Okay, it seems when a incorrect key is used to decode the internal key, the imperfect internal key can come back with odd/strange blocksize.
I looked at the openssl source code and the only blocksizes supported are 16 (128 bit), 24 (192 bit), and 32 (256 bit).
Others cause set_decrypt_key return value to negative and subsequent use seems to cause the segfault.
I have now tested for the proper blocksize before allowing it to try and set the key.
This may prevent the segfault. Please let me know if version 3 below does the trick.
ignobleepub_v3.pyw
http://www.pastie.org/1221164
October 14, 2010 at 6:32 pm
That fixed it. Thanks!
Now I’m wondering if the same sort of check might be necessary/beneficial for the ineptepub_v5.2.pyw script — if converted to plugin? I saw a few posts mentioning occasional segfaults on I ❤ Cabbages' blog.
Thanks again,
October 14, 2010 at 7:39 pm
Hi,
The ineptepub_v5.2.pyw actually uses two different ciphers (AES in CBC mode and RSA). If it uses AES after trying a bad key, then we should probably use the exact same fix in ineptepub_v2.pyw just to be safe.
There seems to be 15 or more patch levels 0.9.8a through 0.9.8o after the initial release of 0.9.8 available on the openssl site. I would guess that the checks for the proper blocksize were added in one of the later patch revisions to the 0.9.8 code to prevent the segfaulting from improper blocksize. Mac OSX and Linux must simply be using different patch levels of the same library.
Adding the check before trying to set the key and raising an exception if bad, seems to be the right thing to do in this case.
So I would add the following if statement
if (self._blocksize != 16) and (self._blocksize != 24) and (self._blocksize != 32) :
raise ADEPTError('AES improper key used')
return
to this piece of code in ineptepub_v2.pyw
class AES(object):
def __init__(self, userkey):
self._blocksize = len(userkey)
# INSERT CODE HERE
key = self._key = AES_KEY()
rv = AES_set_decrypt_key(userkey, len(userkey) * 8, key)
if rv < 0:
raise ADEPTError('Failed to initialize AES key')
I don’t think it can hurt and it may help.
Take care,
October 14, 2010 at 7:47 pm
some_updates and DiapDealer:
Much of the previous conversation was over my head, but if you’re trying to do what I think you’re trying to do, you have my UNDYING gratitude!!!!
October 14, 2010 at 8:14 pm
Hi zanf,
Actually, all I am trying to do is make my own life easier. It turns out to actually be easier to fix the tools themselves to use openssl than it is to explain to someone who has never used development tools how to install the Mac OSX Xcode tools and build PyCrypto from source (and it does not build cleanly on some versions of Mac OSX) and install it.
DiapDealer is the one who is trying to make Calibre plugins for the tools which in turn will make all of our lives easier since the number of “can’t figure out how to install python” errors will go way down because Calibre comes with python itself.
If things work, DiapDealer may be able to create a B&N epub calibre plugin, and an Adept epub calibre plugin to go along with the mobidedrm plugin and his K4PC plugin. That would just leave the eReader pdb tools and the ineptpdf tools missing plugins. Both of which should be possible as plugins if anyone wants to help out.
Unfortunately the Topaz tools are not to the point that a plugin makes much sense and they never will be given how bad the OCR text can be. Although a tool to create an image only pdf file could be made from the svg output again if anyone wants to take a shot at it.
October 14, 2010 at 10:19 pm
Calibre plugin status:
The B&N Epub plugin is just about tuned and ready to go… An Adept epub plugin shouldn’t be far behind it. I’d just like to ensure that they will play nice together when they are both installed at the same time.
I haven’t had a chance to look at the ereader pdb stuff, but it shouldn’t really pose a problem. I think I even have a few encrypted pdb’s lying around for testing.
October 14, 2010 at 8:36 pm
some_updates:
Well, in my eyes you’re both heroes, LOL! I’m struggling trying to even learn Python (I’ve got something I want to do but it requires Python to do it…) and not having the rate of success I would like.
Also, those plugins would be WONDERFUL if they work out.
October 14, 2010 at 8:49 pm
The K4PC tools and the Calibre plugin work great for me and thank you. But the Windows laptop is by far my secondary machine — I use it only for ham radio and unDRMing ebooks! My main workhorse is my iMac and I have a few other Macs around. All of them have Kindle for the Mac installed. Obviously K4PC Tools will not work on the Mac. (I even tried). But is the DRM on the Kindle for Mac ebooks so much different that this would not be a trivial change of some sort? I feel like Oliver asking for “more” but I can’t help it — it would be great to have those tools on the Mac as well as the PC. Any chance??
October 14, 2010 at 9:18 pm
Hi,
I spent a few days looking at the Kindle For Mac code and it is not a trivial change although structurally it is very similar to Kindle For PC. The key stopping point is what they use on the Mac to replace the damn Windows Wincrypt.dll that has the crypt32.CryptUnprotectData function which is used to encrypt the entries in the kindle.info file.
If I knew that, I could probably get a bit farther but the source code has been completely stripped and statically linked to a modified version of openssl making debugging and tracing very hard to do.
I am sure it is something crypto specific on the Mac OSX side but I have not been able to find what and I have pretty much given up.
I can just run Parallels Desktop and K4PC to do what I want or use Kindle for iPad or iPhone which still work as well.
If anyone can point me in the right direction, I would be happy to follow it up and see if I can get something working. But without that knowledge I am stuck.
October 15, 2010 at 2:42 pm
The info here can’t hurt and may help: http://keystorage.codeplex.com/
Don’t know if the tool itself is applicable, but there is cross-platform crypto info available.
October 15, 2010 at 2:01 am
For years I have been a windows user and have just purchased an Imac. I have followed all the new ways to convert books on windows and have been successful through them all. Now I have to learn the mac. Not sure what to do. Can I do mobidedrm and then calibre? Is there an easier way like with windows? Do I have to use a command line now? Any help would be appreciated. Thanks for all the previous help.
lar
October 15, 2010 at 3:01 am
Where do you buy your ebooks? I now buy 90% of my own from BN.com as they seem the easiest to unDRM on the Mac. They come down as pdb files and you just use the eReaderPDB2PML.pyw script in Tools’ eReader_Tools folder. It should run fine on your Mac as basic Python is installed there. Just double-click it and follow the prompts — it is a very good user interface.
October 15, 2010 at 3:03 am
Oh, of course after you have the PML file you just use Calibre to read it in and convert it to whatever your ereader needs.
October 15, 2010 at 10:35 pm
I downloaded a book. It was pdb. When I double clicked on ereaderpdb2plm.pyw, I got a message that said, “cannot open file”. Is there some application that I should have here. I have all the tools. Tlhanks for your help.
October 15, 2010 at 3:40 am
Thanks Thothamon, I have been buying books from Amazon but have a BN account also so I will try that one. I appreciate the help. Sorta fun learning new stuff.
October 16, 2010 at 12:31 am
Larry, Hopefully someone better versed in this will chime in here to help. I have no idea why that script will not run as I am pretty sure that Python (the script interpreter) comes preinstalled on all Mac OS X machines. You can check to see if you have it installed by going to your applications folder, then your utilities folder, then open the Terminal app. In the Terminal window type python and hit return. If Python is there you will see the version number of python. If it is not there you can download Python from http://www.python.org/download/mac/
but I can’t see why it would not be there. If it is there then right click on the script and as a run choice you should see the Python Launcher App as a choice. Let me know what happens and I wish I was more technical to help you here.
October 16, 2010 at 12:57 am
Hi Larry,
Are you by any chance using Mac OSX 10.5 or earlier?
if so, Please see the following:
Under Mac OSX 10.5.X the Python Launcher program is seriously broken.
The solution is to change the setting in the
/System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python Launcher.app
Go to /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/ and double click to launch the “Python Launcher.app”
For “Settings for file type “Python Script”
make sure the “Run in a Terminal Window” is UNCHECKED (ie. not checked)
For “Settings for file type “Python GUI Script”
make sure the “Run in a Terminal Window” is UNCHECKED (ie. not checked)
Then exit the Python Launcher.app savings the settings.
This is only needed for Python that comes with Mac OSX 10.5 and not with 10.6. Apple fixed this bug in later releases.
Please give that a try.
Also remember to not move or rename any of the tools in the tools_v1.9 directory. They need to find their own lib directories to get things to work.
October 16, 2010 at 1:11 am
Meant to say, the script was not showing.
October 16, 2010 at 1:14 am
I am using OS X version 10.6.4
October 16, 2010 at 1:10 am
Yes, it shows python2.6 as you predicted. However,when I tried to right click on python or python 2.6, all I got was search in spotlight, copy and show inspector. I didn’t get any option to run the python. The script itself was showing and I don’t know if it should or not. Thanks for helping.
October 16, 2010 at 1:22 am
Hi Larry,
To run a python script on Mac OSX do the following:
1. double-click on tools_v1.9.zip
2. open tools
3. open eReader_Tools
4. double-click on eReaderPDB2PML.pyw
This should run the program.
If it does not, then you have set up *.pyw files to be run by some other application.
To fix this, single-click on eReaderPDB2PML.pyw
Then go to the Finder’s File -> Get Info and look in the window that appears.
Click on the triangle in front of “Open with” and make sure that the pull down shows “Python Launcher.app”.
And then hit the “Change All” button so that all *.pyw files open this way.
October 16, 2010 at 1:58 am
Some_Updates, you are right it has b&n at the launcher. I tried to change it by finding applications at that point but python is not in the list. When I type it into the terminal, it shows python 2.6. Still learning how to use a Mac. Like the computer but it takes some time. Thanks for helping.
October 16, 2010 at 2:45 am
Hi Larry,
Single-click on eReaderPDB2PML.pyw
In the Finder->Get Info, go to “Open With” then use the pull down menu to get to:
“Other …”
It will open up a gui to allow you to sellect the app you want to open it with.
The go to:
“Macintosh HD” (or whatever you call you main hard drive)
“System”
“Library”
“Frameworks”
“Python.framework”
“Versions”
“2.6”
“Resources”
and then select “Python Launcher.app”
Then hit the Change All Button.
That should do the trick.
October 16, 2010 at 3:08 am
Ok, I found frameworks . However, no documents are associated with frameworks. The screen to the right is completely blank. I tried to double click but nothing happens. Where would the python.framework be found? Thanks again for the help.
October 16, 2010 at 3:18 am
Larry — What Some_Updates gave you was the path to the Python Launcher application. You need to RIGHT-click on the python script. You will get a menu. The second choice is Open with… Choose that and then choose Other. Now you will get a regular dialog box to open folders to get to files. Follow that path Some_Updates gave you. First you choose your Hard Disk. The you will open the System folder. Then the Library folder. The the Frameworks folder. Then the Python/Frameworks folder. Then the folders one after the next of Versions, 2.6 and finally Resources. In that Resources folder will be the Python Launcher application. That’s your treasure hunt goal, choose it! Then click on Change All and after that all python scripts should open normally for you. Hope you get it going!
October 16, 2010 at 3:21 am
Hi Larry,
You are looking in the wrong spot.
You must go all the way up to the Hard Drive root and then follow this path:
“Macintosh HD”->“System”->“Library”->“Frameworks”->“Python.framework”->“Versions”->“2.6″->“Resources”
and then select the “Python Launcher.app”
to find the proper application.
This is not in your current home directory it is above it at the system level.
October 16, 2010 at 3:34 am
It would help if I would read whats written. Sort of missed the word system. Sorry. Wow, it worked like a charm. Thanks so much to both of you. I’m off and running.
October 16, 2010 at 3:44 am
Larry, good news! Now if the real geniuses can figure the Kindle for Mac problems out (in the other thread here) maybe we’ll be able to save a few dimes per ebooksor sometimes even more given BN prices vs. Amazon!
October 16, 2010 at 12:52 am
For Those Interested in Kindle for Mac,
This is with Kindle for Mac.app Version 1.0.0 Beta 1 (27214)
imac:Applications user$ cd /Applications
imac:Applications user$ gdb “Kindle for Mac.app”
GNU gdb 6.3.50-20050815 (Apple version gdb-1472) (Wed Jul 21 10:53:12 UTC 2010)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions.
Type “show copying” to see the conditions.
There is absolutely no warranty for GDB. Type “show warranty” for details.
This GDB was configured as “x86_64-apple-darwin”…Reading symbols for shared libraries ……………. done
(gdb) b * 0x00b2a56a
Breakpoint 1 at 0xb2a56a
(gdb) commands
Type commands for when breakpoint 1 is hit, one per line.
End with a line saying just “end”.
>x/1s $edx
>c
>end
(gdb) run
Starting program: /Applications/Kindle for Mac.app/Contents/MacOS/Kindle for Mac
Reading symbols for shared libraries .+++++++++++++++……………………………………………………………………….. done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Breakpoint 1, 0x00b2a56a in ?? ()
0xbfffda6c: “3264ZTQK”
Breakpoint 1, 0x00b2a56a in ?? ()
0xbfffda6c: “3264ZTQK”
Breakpoint 1, 0x00b2a56a in ?? ()
0xbfffda6c: “YoMVeJcP”
Reading symbols for shared libraries … done
Program received signal SIGTERM, Terminated.
0x93278142 in semaphore_wait_signal_trap ()
(gdb)
So we should at the very least be able to create an “unswindle” like app for Kindle for Mac.
To verify, I used the last 8 digit PID and PIDCheck.py to get the 10 digit PID and then used MobiDeDrm v16 to remove the drm and it worked just fine.
If anyone knows how to use python to insert breakpoints and read registers in a Mac OSX binary, please let us know.
Have fun!
October 16, 2010 at 6:40 pm
Hi,
Need help from Mac Users? Is the gdb debugger installed on all Macs or must you install the Developer Tools to get it?
If you have a Mac and have NOT installed the Developer Tools would you please do the following:
1. Double-click on Terminal.app (it is in /Applications/Utilities)
2. type the following at the prompt and hit return
which gdb
On my machine it shows it replies with /usr/bin/gdb
but I have the Developer tools installed.
If gdb is installed by default, I think I can create a gui python tool that will determine the correct 10 digit PID under Kindle for Mac and print it.
Then people can use that MobiDeDRM.pyw to remove the DRM.
So it would be a two-step process and not as nice as Kindle 4 PC but it would allow users of Macs to get things working.
But if gdb is not installed by default, then I am not sure how to go about things since ptrace on Mac OSX is seriously crippled and will not let you see the actual register values due to the Mach-O kernel being used.
If you can help, please let me know.
Thanks,
October 16, 2010 at 8:04 pm
On 10.4.11 I get
no gdb in /bin /sbin /usr/bin /usr/sbin
so I think it can’t be installed. But if having the devtools installed will make it work, it’s not too bad as prerequisits go. It’s certainly easier to install than any third-party resource. (e.g. PyCrypto).
And I must say how very impressed I am that you’ve got as far as identifying the PID using the debugger. Excellent work indeed.
Paul
October 16, 2010 at 8:21 pm
Hi Paul,
So do you think it would be worthwhile to have a python gui tool that auto launches “Kindle for Mac.app” inside gdb and upon the user opening a book and then exiting the Kindle for Mac.app” it prints out the 10 digit book PID?
The user would then have to fire up MobiDeDRM.pyw and then navigate to find the correct book in ~/Documents/My Kindle Content/ (not easy since the damn file names are so cryptic) and then enter in a new name and the PID key found to do the actual DRM removal.
This seems like a lot of work but I do not have enough knowledge of the code to find out the name of the file the user chose from their actions so I can’t do it for them. It might be possible inside gdb to do some sort of trace on file io being done but I have not tried that yet.
If I could find the file that was read form inside gdb, I could pass that info to the python script and it could take the pid and the file name and run Mobidedrm automagically.
And of course – all of this would require users to install the Developer tools (which does come as part of the Mac OS X install disk)
I think I could whip something like this up if you think it might be worthwhile.
Also, are you familiar enough with gdb to know how to most easily trace file io?
Thanks,
October 16, 2010 at 8:35 pm
I think it would be an excellent idea. It sounds like it might be possible to get it going with an Applescript, such that the user drop a file on the Applescript, the Applescript runs the new program, which launches Kindle4Mac in gdb, the Applescript sends an ‘open file’ appleevent to Kindle4Mac, retrieves the PID output by the new program, and then calls MobiDeDRM with the file and the PID….
I’m not entirely sure how good Applescript will be at running the program simultaneously with sending off an Appleevent to Kindle4Mac, but I suspect it might work.
Of course, the real problem isn’t the complexity, but that it will break with every update to Kindle4Mac.
But it is a very big step forward, and I’d encourage you to give it a go. Perhaps it will encourage someone else to slog through in the debugger some more, like someone did to get skindle going after unswindle came out.
But I’m afraid my experience with gdb is next to none, so I have no ideas on file io – but perhaps this isn’t necessary if you leave it to an Applescript wrapper?
October 16, 2010 at 6:45 pm
I’m pretty sure gdb isn’t installed by default, but I see if I can find a computer without Xcode to check.
No luck figuring out the replacement for CryptUnprotectData on the Mac yet?
October 16, 2010 at 8:07 pm
Hi,
I have no reverse engineering tools and the IDA Pro tool costs real money. I did find a machodis.pl perl program that combined the output from the otool disassembly with info on cstring data and symbol stubs to at least give a better picture of what is going on. Still, there is virtually no info in the file. I had to search for 0x4e35 (which is 20021) which is one of the constants used in the Pukall Cipher 1 to find that and work my way back up the tree.
Sorry but I so far all I have seen is the EVP_Init function fire (openssl is statically linked into the binary but luckily the “assert strings” are readable and findable) but I have no info yet about what algorithm or what key is being used and even if it is used at all.
So all I could find was the beginning of the Pukall Cipher 1 since it has a unique constant being used in a multiply.
If you have access to better tools, I would appreciate hearing what you found (I already knew about the new string 64) it was easy to see by running “strings” on the executable and eyeballing the results.
October 16, 2010 at 11:54 pm
Kindle4Mac UnSwindle Like Tool
Proof of concept for Mac Users.
You MUST have the Developer Tools (XCode) tools installed to get gdb
Grab the Kindle_4_Mac_Tools.zip from here:
http://www.mediafire.com/?taka33rv60dl9j1
Here are the instructions:
1. Verify you are using Kindle for Mac.app Version 1.0.0 Beta 1 (27214)
(check your about this app when running Kindle for Mac, and then close out of Kindle for Mac)
2. unzip Kindle_4_Mac_Tools.zip
3. double-click on Kindle_4_Mac_Tools folder to open it
4. double-click on K4Munswindle.pyw
You should then see a gui window open
– you should select your “Kindle for Mac.app” if it is not already in its standard location (ie. in /Applications)
– you should hit the … button to select an output directory for the non-drm file
– then hit the start button
Your Kindle for Mac.app should open up automagically
Hit your “Home” button to go home. An then double-click on one of your Kindle mobi books (this will not work for Topaz books!)
This should open the book.
Once open simply hit the “Home” button to go back to home and exit the Kindle for Mac application.
Once you have exited Kindle for Mac you should see mobidedrm.py fire up in the Conversion Log window and if successful you should find your decoded book in the output directory.
Please someone give this a try and let me know what issues you run into.
Thanks!
October 17, 2010 at 3:00 am
Worked 100% for me here. Incredible!! Thank you, thank you, thank you!!!
October 17, 2010 at 5:00 am
Another quick report: Had great success with a more complex try. I used the script to start up Kindle for the Mac. From within KforMac I chose to go to the Amazon store. Bought a book. Had it sent to my KforMac. Used the button in the storefront to go back to KindleforMac and opened the new book once it was downloaded. Back to Home, closed off KindleforMac and the script after all that still took over, undrm’d the book fine. Great stuff!
October 17, 2010 at 10:05 am
Very, very impressive. Works very well indeed. (Mac OS X 10.5.8/Intel)
And no need for any kind of Applescript, all done with pythonw and gdb. Wow. And no compiled code.
An amazing achievement. Congratulations!
October 17, 2010 at 1:14 am
Worked for me! I had associated .pyw with an editor, but Open With Python Launcher worked great.
Thanks!
October 17, 2010 at 12:01 pm
Hi,
Glad to hear it works. I will add in checking the version string from the Kindle4Mac app to make sure it matches, verifying gdb exists, defaulting the output directory to the Desktop, and then release it as part of tools 2.0 after DiapDealer releases ahis new plugins.
One nice thing is that as each new K4M version is released, all that will need to be released is a new gdb kindle command file.
October 17, 2010 at 4:28 pm
First Official Kindle_4_Mac_Tools Release
The first official release of the Kindle for Mac Tools is now here:
Kindle_4_Mac_Tools_v1.0.zip
http://www.mediafire.com/?07byn7kd4c8p16i
Prerequisites:
– Kindle for Mac.app Version 1.0.0 Beta 1 (27214)
(this is the current version at Amazon)
– XCode Developer Tools must be Installed
(see your latest Mac OS X Install Disk for the installer)
The directions for use are:
1. double-click to unzip the Kindle_4_Mac_Tools_v1.0.zip
2. open the Kindle_4_Mac_Tools folder
3. double-click on K4Munswindle.pyw
In the window that opens:
– hit the first ‘…’ button to locate your “Kindle for Mac.app”
if it is not in /Applications
– hit the second ‘…’ button to select an output directory
(defaults to your Desktop)
– hit the ‘Start’ button
After a short delay, your Kindle for Mac.app should open up automagically
In Kindle for Mac:
– hit the “Home” button to go home.
– double-click on ONE of your mobi books (Topaz books will not work)
This should open the book.
Once the book you want is open
– hit the “Home” button and then exit the Kindle for Mac application
Once you have exited Kindle for Mac you should see mobidedrm.py fire up in the Conversion Log window and if successful you should find your decoded book in the output directory.
October 17, 2010 at 8:39 pm
Well, I got good news and bad news.
The good news is that after I located a v1.0.0 Kindle for Mac, this worked like a charm. Great work, thanks.
The bad news is that I started by downloading the current Kindle for Mac from Amazon (since I didn’t have one installed) and got version 1.2.0 and that version didn’t work. It reported “book not found”.
So my question is: how can I determine what displacement to specify in the gdb_kindle_cmds.txt file.
Again, thanks for the work.
Cheers,
TB
October 17, 2010 at 9:03 pm
Hi Terry,
Well that was short-lived!!!!
I will download the new version and see what I can do. Let’s hope they did not change much code.
Thanks for letting me know.
October 17, 2010 at 11:05 pm
New Version of Kindle_4_Mac Tools Version 1.1 to work with the new Kindle.app for Mac OS X.
Took my about an hour or so of work to find and fix things for the new version:
Please get the latest release:
Kindle_4_Mac_Tools Version 1.1
http://www.mediafire.com/?zdbl4k91nvk0157
It supports both
– Kindle for Mac.app Version 1.0.0 Beta 1 (27214)
(this is the original version)
and
Kindle.app Version 1.2.0 (30689)
(this is the current version at Amazon)
Same instructions as before.
Please let me know if it works for you or not.
On my tests it worked just fine.
October 17, 2010 at 11:42 pm
Kindle_4_Mac_Tools Version 1.1 work like a champion!
Many thanks for the effort.
Cheers,
Terry
October 17, 2010 at 9:16 pm
I’m also curious how 0x00b2a56a was figured out 🙂
October 17, 2010 at 11:47 pm
Hi Kee,
Just in case anyone else wants to play with this and help improve it and keep it up as new revisions come out:
First examine the Pukall Cipher code in mobidedrm.py and notice the strrange constant 20021 used in multiply operations.
I thought this unique constant involved in a multiply might be something I could look for in the Kindle code itself.
So I did the following:
1. find on the web machodis.pl and download it. This is a perl program to combine the disassembly output of otool with cstrings and the like.
2. copy /Applications/Kinde for Mac.app/Contents/MacOS/Kindle for Mac to some safe place and call it k4m.dat
3. run the disassembler as follows:
perl ./machodis.pl k4m.dat > diss.txt
And this will take a very long time with lots of warning messages which you can safely ignore.
4. to find the guts of the Pukall Cipher look for that strange constant used in a multiply operation (an imull or imulw). Note 20021 is 0x4e35 in hex.
grep imul diss.txt | grep 4e35
This should give you the address to search for in the diss.txt file.
This is the guts of the Pukall Cipher routine.
Then load up emacs and edit diss.txt . Go to that address and then go to the top of that routine. (ie. 0x00b45c98)
** Please note that 9 digit hex addresses similar to 0x100b45c98 are really just 0x00b45c98 in jmp and call instructions. I have no idea why the extra leading 1 is added to these addresses. **
Then in emacs search for where that routine was called from.
Repeat to work backward up the call chain.
In this case 0x00b45c98 was called from both 0x00b45e78 and 0x00b45eda.
In turn:
0x00b45e78 was called from 0x00b2a530
and
0x00b45eda was called from 0x00b2a56a
and so on.
Then fire up gdb and run Kindle and set breakpoints at each routine in the call chain looking for something that looks like a PID in a routine’s arguments and registers. If it is not there remove the breakpoint and continue. Repeat until you find what you are looking for. (Notice the only time the key will look like a PID is whn you first try and open a book in the Kindle!). Subsequent keys do not look like PIDs.
This is how I came to 0x00b2a56a and found the PID I wanted in the registers.
To get the file name, set a breakpoint at the stub for the dynamically linked in _fopen routine. The easiest way to find its address is:
grep _fopen diss.txt
The $eax register holds the first argument to the fopen routine.
For the newer version 1.2.0 of Kindle.app a similar approach resulted in:
0x00e37c98 (guts of Pukall Cipher)
<- 0x00e37d22
<- 0x00e37f38 <-0x00e37c28
<- 0x00e37fca <-0x00e37be4
And after searching I found the PID in 0x00e37be4 in register $eax
The hard part was actually figuring all of this out the first time
Hope this makes things clearer.
As you can see, the disassembled code is huge and you can't simply just page through it. I am sure someone with a professional disassembler/reverse engineering tools could do a lot more, but I don't have either the time or the money to use such a tool. I could only afford to give it a weekend here or there.
October 18, 2010 at 5:44 am
Great info! Thanks!
October 18, 2010 at 5:31 pm
If standalone Kindles are Linux arm based, then a similar approach used on the binary program to open books on standalone kindles could probably be built.
If anyone has extracted the binary from a standalone Kindle that does the ebook reading, people could take a shot at seeing if the PID can be found by finding the Pukall Cipher in the code and working backwards. If Linux based, then ptrace code could be used to extract the info needed.
It is one way to deal with the problem of finding the PID for standalone Kindles.
Is anyone interested in seeing if something like that might be done for standalone Kindles?
October 18, 2010 at 6:25 pm
Unfortunately, the Kindle’s ereader framework is Java… and it’s Java that’s been terribly obfuscated. I’ve had a peek at it, but I know when I’m in over my head. 😉
October 19, 2010 at 7:00 pm
I’m currently trying to isolate/identify the Kindle’s indexing process. My theory may be way off base, but in order to create a db (which it does) that can then be searched for specific text by the user, wouldn’t that indexing process need access to each book’s PID to get to the actual text of the book?
I’m grasping at straws here, but I was thinking that debugging/decompiling/disassembling that indexing process could prove less cumbersome than attacking the entire ereader framework.
October 17, 2010 at 9:18 pm
First time caller here, saying THANKS!!
I just got an open-box Archos 7 tablet ($159, the one which everyone hates, but which is the only one I can afford) with aldiko installed, an e-reader which requires epub. On first try, eRdr2PML.py (version 0.15) got my B&N-purchased .PDB file into PML, and Calibre handled the rest. I’m positively giddy.
October 18, 2010 at 6:21 pm
I have two Calibre plugins ready for testing. They are based on the I ❤ Cabbages' scripts for B&N Epubs and ADE Epubs (ignobleepub and ineptepub). All credit goes to I ❤ Cabbages for the original scripts.
First up is the Ignoble Epub DeDRM plugin (ignobleepub_v01_plugin.zip). For B&N Epubs. Install it in the usual Calibre plugin manner… the zip file is ready to be installed. (NOTE: works on epub format only, not eReader pdb's purchased from B&N). To configure it, Add your Name and credit card number (just like you did for the original keygen script) to the plugin's customization window in Calibre. Separate the name and CC# with a comma like so: Your Name,1234123412341234. If you have purchased books with more than one credit card/B&N account, you can enter them all by separating them with a colon like so: Your Name,1234123412341234:Other Name,2345234523452345. If you don't like having your CC# on display in Calibre, there is another method explained in the readme/instructions that I will provide a link to.
Second is the Inept Epub DeDRM plugin (ineptepub_v01_plugin.zip). For ADE Epubs. No entering data in the plugin customization for this one. If you have Adobe Digital Editions installed on the same machine (Mac or Windows OS's), it will find the ADE key. If you don't have ADE installed on the same machine as Calibre (or the plugin doesn't find it), check the readme/instruction file for other configuration options.
These two plugin should work on Windows, Mac or Linux and you shouldn't need special software installed. If you only have Calibre installed, they should work (aside from ADE of course).
Ignoble Epub plugin: http://www.mediafire.com/?n5p7l1b9z5yzdy1
Inept Epub plugin: http://www.mediafire.com/?1o28actnx7k2vdk
Readme/Instructions: http://www.mediafire.com/?2os5d3y4qqdta4z
Please read the instructions before posting problems… it may save some time. 😉
October 18, 2010 at 9:13 pm
Great stuff! I just used the ignobleepub plugin with success. When I tried to configure it manually entering name and CC#, I could not get it to work, but when I added my bnepubdey.b64 it was perfect. This is terrific, because, like many others, I often buy books from B&N to read on my Kindle, and they have to go through Calibre anyway, so your fine work is a real time-saver.
Thanks a million.
Just a quick question. What is the likelihood that a book purchased from any vendor might have info to link it to the buyer even after the PID is removed?
October 18, 2010 at 9:46 pm
Hi montalex,
Glad to hear you got it working! As far as the name and cc# not working for you (or anyone that may be reading), just remember that the name you enter is the name that you registered your B&N account with… which won’t necessarily be the same name as on the cc. Also make sure there are no extra spaces before or after the comma. In fact, the name gets converted to all lowercase with no spaces by the script, so you could enter “YourName,1234123412341234” (without the quotes) if that’s simpler for everybody.
As for linking the dedrmed book to the buyer, I wouldn’t see how they could. The two files that relate to the encryption are stripped out of the epub… leaving you with a plain-jane epub that has no identifying features that I can see.
October 19, 2010 at 5:52 pm
Thanks, DiapDealer. Have you had a look at azw2 files? Those are the games for Kindle, like Scrabble. Though I guess there would be no point in removing DRM from those, as they probably wouldn’t work on any other reader. I just want to own my stuff! I managed to purchase Scrabble by changing my account to a U.S. address — I then quickly changed it back to my home country. This is probably not something one should do often — a good way to get banned from Amazon, I’ll bet.
Montalex
October 18, 2010 at 6:55 pm
Forgot to mention that you need Calibre version 0.6.44 or higher for these plugins. Shouldn’t really be an issue as Calibre is now at 0.7.24.
October 18, 2010 at 7:37 pm
I apologize for the horribly formatted instructions. Here are some better ones. http://www.mediafire.com/?7vd0xv42tu76pnl
That’ll teach me to publish documents that I wrote in a python editor on a Linux box!!
October 18, 2010 at 8:30 pm
Hi DiapDealer,
Great Work! I see you had to create two different AES objects in the B&N code to handle the difference between encrypt and decrypt. I should have thought of that approach.
I have created a separate Calibre_Plugins directory in the forthcoming tools_v2.0. I placed your 3 plugins along with a MobiDeDRM_plugin in there along with your updated READMEs. I have then duplicated them in the appropriate tools folder in case people are only interested in one specific tool set.
Is that alright?
Is there a “README” that explains the K4PCDeDRM plugin that you would like me to include? Or should I create one by using just the following:
Installation:
Go to Calibre’s Preferences page… click on the Plugins button. Use the file dialog button to select the plugin’s zip file (K4PCDeDRM-X.XX_plugin.zip) and click the ‘Add’ button. you’re done.
Also do you have by any chance a similar “README” for the MobiDeDRM plugin? How are the extra PIDs added to the plugin? Please forgive me as I am not a calibre user.
Thanks again for all of your hard work!
October 18, 2010 at 8:59 pm
Thanks some_updates. Congrats on your K4Mac breakthrough, by the way!
Yes I think that a separate folder for the plugins makes sense. I have no problem with duplicating them in the appropriate folders, but size could get to be an issue. The Ineptepub plugin is a lot larger than other plugins due to the included external libraries — even though I stripped them to their bare minimums. But do what you think seems best.
I have no Readme for the K4PCDeDRM plugin, but it really is as simple as the blurb you suggested.
As for the MobiDeDRM plugin… the PIDS are entered in the plugin customization window separated by commas (no spaces). It’s not mine, but I wouldn’t think adding a Readme would be stepping on any toes.
I’m looking forward to some feedback on the new plugins. I tested heavily with Linux, Windows (XP, Vista) and OS X 10.6.2, but you never know.
I’m going to play around with your new K4Mac tools, but my Mac is a hacked-up Snow Leopard install on VirtualBox hosted on Ubuntu… it works, but it can be quite painful at times. 😉
October 18, 2010 at 11:23 pm
Hi DiapDealer,
Do you want me to release tools_v2.0.zip now (with your new plugin tools) or wait a few days for you to hear feedback first before I release things. I was not sure which would generate more testing.
Just let me know what you want me to do.
Thanks
October 19, 2010 at 5:06 pm
some_updates,
Go ahead and release the new Tools archive if you’re ready. These new plugins have only been downloaded around 15 times each, so maybe a new Tools release will prompt a bigger response. I say go for it 😉
By the way, sorry I neglected to credit you for the libcrypto code I used for the ignoble epub plugin. I’ll correct that in the next update. That plugin would have ended up a lot larger if I had to include all of the external PyCrypto libs for Linux and Mac users.
Thanks
October 18, 2010 at 7:06 pm
Well dang, I knew I should have waited to buy books… Bought two ADE and one B&N this weekend. Guess this gives me an excuse to buy something else, right? *wink*.
Thanks – y’all are miracle workers!
October 18, 2010 at 7:49 pm
Kindle_4_Mac_Tools Version 1.2
Kindle_4_Mac_Tools_v1.2.zip
http://www.mediafire.com/?8nz9rkg6p9nq23r
New in this release:
– support to identify Topaz Books and print their PID
so that standard Topaz_Tools can be used later if
desired (only works with Kindle Version 1.2.0)
October 19, 2010 at 6:56 pm
New Version of the Tools Available: tools_v2.0.zip
tools_v2.0.zip
http://www.mediafire.com/?i5bca5z1cl7sazk
What’s New in tools_v2.0
– added wonderful new calibre plugins for B&N epubs and Adobe epubs from DiapDealer (see the Calibre Plugins folder for more info). (thank you DiapDealer!)
– reorganized the tools so that the Calibre_Plugins has its own folders and all of the plugins and the READMEs can be found in one place (includes Adobe pub, B&N pub, MobiDeDRM, and K4PCDeDRM plugins)
– added Kindle_4_Mac_Tools (K4Munswindle.pyw) version 1.2 so that Kindle for Mac users can now remove DRM from their ebooks. Please note this requires that the user has installed the XCode Developer Tools (which can be found on their latest Mac OS X install disk).
– modified IHeartCabbages Barnes & Noble epub scripts to use OpenSSL libcrypto for Mac OS X and Linux to remove the requirement for PyCrypto
(thank you IHeartCabbages for the idea and most of the code!)
– fixed a few bugs in the Topaz_Tools and added support for unpacking color_images that the user can use in place of the black and white ones if his/her reading device supports color.
– Updated to the latest versions of Mobiunpack (thank you Paul!)
– tried to add lots of README’s all throughout based on my own notes or from things posted on the Apprentice Alf blog or comments (thank you Apprentice Alf!)
October 21, 2010 at 2:18 am
An Update on Reverse Engineering K4M,
Okay, I had some free time to kill so I looked at things further and I have python code that properly decodes the .kindle-info file on my machine. Unfortunately it only works for my machine.
It seems they have replaced CryptUnprotectData with an AES 256 bit CBC_MODE decryption that requires a Key and an InitialValue.
Once decrypted you then have to run decode(map,data) on the decrypted string with String32 (or charMap1).
This now works. I cheated to get my Key and InitialValue by using gdb to find them. The problem now is how do they determine the Key and the InitialValues they use. I think they are using system specifc information but I am not sure what they are.
Oh well, I have to return to my regular work. So if anyone wants to attack this further, I would be happy to point them at the routines that do the things above so they have someplace to start.
October 21, 2010 at 12:29 pm
Nice work!
I wish I had a spare Kindle to give you… I’m sure we’d have a stand-alone Kindle solution before too long if I did. 😉
October 22, 2010 at 1:37 am
Hi All,
Okay I think I have figured out how to dump the K4M .kindle-info file.
This is not a complete solution and it needs testing to see if it is robust!
So if you have Kindle for Mac installed (any version), please grab the following:
http://pastebin.com/i5b3ZfHN
and download to your Desktop and rename it to
dumpkinfo.py
Then open a Terminal.app session and do the following:
cd ~/Desktop
python ./dumpkinfo.py
And please see if it looks right. I would not paste things here since it is personal information but you should be able to tell if it works for you or not. Please try it and let me know what you find out.
October 22, 2010 at 3:11 am
It looks like it worked here. All of the items in the “clear” contained correct information. Anything else I can do, let me know.
October 22, 2010 at 7:18 am
It looked like it worked here, but ONLY after updating to Kindle 1.2. I think the location of the .kindle-info file has changed, as the name of the application has changed from “Kindle for Mac” to just “Kindle”. (i.e. it’s now in a folder “Kindle” in Application Support/Amazon, not “Kindle For Mac”.)
October 22, 2010 at 1:26 pm
Hi Paul,
Good to know. My test file was actually from the original version (I had copied it into a test folder) so the contents did not change, just the path to where it is located. I can have the script look in both places just to be safe.
We still need to figure out and handle the PID generation process but looking at dumpkinfo.py, and DiapDealer’s K4PCDeDRM.py as a model it should be doable.
Thanks for testing it.
October 22, 2010 at 4:01 pm
Thanks to everyone who tested dumpkinfo.py for me. It helps to know that how I grabbed the Hard Drive serial number works on machines other than mine.
I continued working on it this morning and now I now have a working K4MDeDRM.pyw (with k4mdedrm.py) that is a merge of DiapDealer’s k4pcdedrm.py with dumpkinfo.py (fixed to check both locations for the .kindle-info file).
It seems to work on my machine but I have not really tested it much and I have not tested the calibre plugin functionality at all. I don’t really use calibre much except to convert e-books.
K4Munswindle is still the only place to get a K4M Topaz Book Specific PIDs from since K4MDeDRM only works with mobi not Topaz books.
So it appears we will NOW need a cmbtc_dumpK4M.py script that is a modification of the Topaz_Tools cmbtc_dump.py to work with K4M by merging in info from dumpkinfo.py somehow.
I will try to test it more and post it over the weekend if I don’t run into troubles.
October 22, 2010 at 4:36 pm
some_updates said:
“I continued working on it this morning and now I now have a working K4MDeDRM.pyw (with k4mdedrm.py) that is a merge of DiapDealer’s k4pcdedrm.py with dumpkinfo.py (fixed to check both locations for the .kindle-info file).”
What’s the prognosis for a “One K4*DeDRM.py to rule them all” type solution? 😉
I was curious if you could use Mac’s Carbon python modules to locate the kindle.info file — like I ❤ Cabbages does with ineptepub.pyw to find ADE's activation.dat file?
Great work!
October 22, 2010 at 6:28 pm
Hi DiapDealer,
> “One K4*DeDRM.py to rule them all”
Yes you could merge them into one but there would have to be big a platform test and of course two versions of each of the different functions (including charMaps, GetUser, decryption routines, libcrypto being needed, etc).
It might be easier to split the file down into smaller segments (multiple python files) and import them depending on platform.
How would that impact plugins? Can the plugin have multiple *.py files in it or must it be one single file? If we can use with multiple files, we can pull out the mobidedrm code as well as all of the platform specific code so that we can maintain mobidedrm.py and the platform specific pieces separately?
So it might look like:
k4pcmdedrm.py
k4mroutines.py
k4pcroutines.py
mobidedrm.py
all with just one front end gui.
As for where the .kindle-info file is, I can do a unix find if the file is not where it is supposed to be but it really isn’t needed because that is what the Library/Application Support directory is for on Mac OSX.
I will look at the code but what I know about Carbon and Cocoa you could put in a thimble.
I am really an old C / Unix / Linux kind of guy who cut his teeth on an old DEC pdp-11 with punch tape and IBM mainframes with punch cards. So I basically treat MAc OSX like a Linux box when developing tools for it.
If you want an advance copy of k4mdedrm.py to see what I did, just PM me on Mobileread or e-mail me.
October 22, 2010 at 8:53 pm
Hi DiapDealer,
I checked the website and calibre will automatically add any of the other .py programs to the sys.path that in finds in the plugin zip.
So we could make a main program k4dedrm that has something like the following structure:
if sys.platform.startswith(‘win’):
import from k4pcsutils GetUserName, GetVolumeSerialNumber, CryptUnprotectData, KInfoPath, charMap1, charMap2, charMap3, charMap4
if sys.platform.startswith(‘darwin’):
import from k4mutils GetUserName, GetVolumeSerialNumber, CryptUnprotectData, KInfoPath, charMap1, charMap2, charMap3, charMap4
Then grab my version of decode (since the main one has a bug in it that does not work with charMaps that are only 32 chars long) and a few other little things I added (generation of 10 character pids) or changed.
and the bulk of the rest of it should be identical or nearly so.
If we write standalone code to grab exth (easily borrowed from mobiunpack.py) we can do the full PID generation without having to pull apart mobidedrm into pieces. (we could simply invoke a completely unchanged mobidedrm.py by adding a simple non-main function entry point to it.
What do you think? I can take a stab at it and then pass it along to you for debugging on the windows side of things.
Let me know if you are interested.
October 23, 2010 at 12:12 am
I’m certainly interested. If you get something that works OK on OS X (stand-alone and calibre plugin) I can test it for you on windows.
The only tricky part with importing optional (non-standard) python modules in a calibre plugin is WHEN you import them. The extra .py files in the plugin’s zip file are only available for import during the ‘run’ function of the plugin class.
That’s why you see me using globals quite a bit in my plugin code. The non-standard imports outside of that function will barf when the plugin is called. So they would all have to be surrounded by try->except->pass clauses (so the standalone script can still function while keeping calibre from gagging) and then the imports would have to be repeated in the plugin’s ‘run’ function (for the plugin to function). With globals to make the dual-headed portions of the script play nice with each other.
It could end up being more trouble than it’s worth, but I’m definitely willing to give it a shot.
October 22, 2010 at 5:15 pm
Speaking of locating the kindle.info file, any chance of having K4*DeDRM be able to use multiple kindle.info files (presumably in different places)? Between the opening of the UK Kindle store and Amazon dropping the $2 RoW tax, I now find myself using three different Amazon accounts for Kindle purchases (a US-registered account for US-only titles, a UK-registered account for UK-only titles, and a Canadian-registered account for US/Canada titles, which are often cheaper if purchased “from” Canada, since the Agency agreement doesn’t seem to be in effect there).
At the moment I just de/re-register K4PC depending on which account I’m buying from/de-DRMing, but a solution that allowed it to check multiple kindle.info files (like MobiDeDRM allows multiple PIDs) would be really sweet…
October 22, 2010 at 6:34 pm
Hi,
It might be doable. At the very least, in the GUI front-end a person could be asked to locate the kindle-info file they want to use and then run things that way.
As for doing that in a plugin by submitting the different kindle-info files, I am not sure. I really haven’t looked at calibre plugins closely, but DiapDealer should be able to answer that.
You could always make 3 copies of the plugin (one for each account) and hard code the path to the kindle-info files in each one and call each of them from 3 different front-ends: K4PCDeDRMUK, K4PCDeDRM_USA, and K4PCDeDRM_Canada.
October 24, 2010 at 4:47 pm
Announcing: Kindle_4_PC_and_Mac_Tools
(the one K4 tool to rule them all)
Kindle_4_PC_and_Mac_Tools.zip
http://www.mediafire.com/?ie44b77fqlekz8f
We have merged the wonderful K4PCDeDRM tool with the recently reverse engineered “Kindle for Mac” information to create one new tool and one new Calibre plugin called K4DeDRM that works on both Windows machines that run “Kindle for PC” and Mac OS X machines that run “Kindle for Mac”. This new tool should work with any version of “Kindle for *” as they run based off the kindle info file contents.
This tool currently replaces the K4PCDeDRM tool and plugin but does not actually add any additional capabilities so please keep using that tool if you already have it installed.
For Mac OSX users, K4DeDRM replaces K4Munswindle. Please note K4Munswindle works just fine so keep using it if you already have it installed or install them both, they will not interfere with each other.
K4DeDRM does add in the capability of using a it as a Calibre plugin so that if you are a Calibre user, you no longer have to worry about how to run the tools after installing the plugin.
Have Fun!
some_updates and DiapDealer
October 25, 2010 at 12:17 am
Wow. You are awesome!!!!!!! I tried it and it works like a charm. Thanks for all your work. I’m just amazed. Thankyou, thankyou, thankyou.
October 24, 2010 at 6:33 pm
The calibre plugins are wonderful – thanks.
I’m having a problem with the latest, though, on my Mac. I’ve installed K4DeDRM, but when I try to add a file, I get the “loading encrypted version” error.
I tried the stand-alone version of K4DeDRM, and as soon as I click on the box to select the output file, I get errors in the terminal window. I’ll guess that’s what’s causing the problem in the plugin, too, but I’m not positive.
I’ve successfully used the stand-alone version of K4MacdeDRM, so I think I have all the necessary tools installed, but it would be great to get the plugin working, too.
How can I help? Do you need me to post my output from the error?
October 24, 2010 at 8:17 pm
Hi,
Please run the K4DeDRM gui version on that same book and let me know exactly what it says in the Conversion Log window. Simply cut and paste everything in that window and then edit it to remove any personal info that shows up (if any even exists).
I am wondering if the file you chose is a Topaz file which can not be decrypted by this tool.
Thanks for testing it.
October 24, 2010 at 8:32 pm
Thanks for getting back to me. I hope I’ve sanitized this the right amount – I removed my userid and the title of the book. I think I caught all occurrences.
Looking at it some more, I see two separate problems. It’s not a Topaz file – I can decrypt it fine using the Mac standalone GUI but not the combined one.
First – If I don’t try to specify an output directory, the deDRM fails – here’s the output from the conversion log –
Conversion Log
Command = “python k4dedrm.py”
K4M/K4PC Path = “/Users/********/Documents/My Kindle Content/B0030GE2ZU_EBOK.azw”
Output File = “test”
Please Wait …
Traceback (most recent call last):
File “./lib/k4dedrm.py”, line 381, in
sys.exit(main())
File “./lib/k4dedrm.py”, line 373, in main
file(outfile, ‘wb’).write(mobidedrm.getUnencryptedBook(infile, pid))
File “/Users/********/Dropbox/Kindle Tools/tools 2.0 – October 2010/Kindle_4_PC_and_Mac_Tools/lib/mobidedrm.py”, line 258, in getUnencryptedBook
strippedFile = DrmStripper(data_file, pid)
File “/Users/********/Dropbox/Kindle Tools/tools 2.0 – October 2010/Kindle_4_PC_and_Mac_Tools/lib/mobidedrm.py”, line 225, in __init__
raise DrmException(“no key found. maybe the PID is incorrect”)
mobidedrm.DrmException: no key found. maybe the PID is incorrect
K4DeDrm v1.0 provided by the work of many including DiapDealer, SomeUpdates, IHeartCabbages, CMBDTC, Skindle, DarkReverser, ApprenticeAlf, etc .
DSN: 5SSCPz77r1tS5YtJ5xnSnM7EnzrYS8Pxrx6bS7nd
Device PID: C7ZFXPHGE5
Account Token: e11660cc97697f28d83176a4d8605af8218f00c1
Pid for The Book: g/DbOnGeZM
MOBI header version = 6, length = 232
Extra Data Flags = 10
Error: Encryption Removal Failed
Second problem – as soon as I click to specify an output directory, the process terminates – this is the output that appears in the terminal window
2010-10-24 13:18:25.079 Python[92388:d07] An uncaught exception was raised
2010-10-24 13:18:25.079 Python[92388:d07] *** -[NSCFArray insertObject:atIndex:]: attempt to insert nil
2010-10-24 13:18:25.085 Python[92388:d07] *** Terminating app due to uncaught exception ‘NSInvalidArgumentException’, reason: ‘*** -[NSCFArray insertObject:atIndex:]: attempt to insert nil’
*** Call stack at first throw:
(
0 CoreFoundation 0x00007fff81921cc4 __exceptionPreprocess + 180
1 libobjc.A.dylib 0x00007fff88fa90f3 objc_exception_throw + 45
2 CoreFoundation 0x00007fff81921ae7 +[NSException raise:format:arguments:] + 103
3 CoreFoundation 0x00007fff81921a74 +[NSException raise:format:] + 148
4 Foundation 0x00007fff88166c2a -[NSCFArray insertObject:atIndex:] + 119
5 AppKit 0x00007fff8315d90d -[NSNavOutlineDelegate _highlightedNodes] + 165
6 AppKit 0x00007fff8315ecd7 -[NSNavOutlineDelegate prepareForReloadChildrenForNode:] + 65
7 AppKit 0x00007fff83148fb8 -[NSNavDataSource _prepareForReloadChildrenForNode:] + 54
8 AppKit 0x00007fff8314774f -[NSNavDataSource _handleChangedAllChildrenInExpandedNode:] + 25
9 AppKit 0x00007fff8314d5a4 -[NSNavDataSource observeValueForKeyPath:ofObject:change:context:] + 748
10 AppKit 0x00007fff8314fd39 -[NSNavFBENode _sendChildrenChangedWithDictionary:] + 350
11 AppKit 0x00007fff833e9057 -[NSNavFBEQueryNode _resetQueryForChangedAttributes:andSendChildrenChanged:] + 191
12 AppKit 0x00007fff83147b5b -[NSNavDataSource _rawChildrenOfNode:] + 140
13 AppKit 0x00007fff83147bcc -[NSNavDataSource _processedChildrenOfNode:] + 36
14 AppKit 0x00007fff8314da80 -[NSNavDataSource _cachedChildrenForNode:createIfNeeded:] + 280
15 AppKit 0x00007fff8315dfa8 -[NSNavOutlineDelegate _cachedChildrenForNode:allowInteraction:] + 211
16 AppKit 0x00007fff8315cffa -[NSNavOutlineDelegate outlineView:numberOfChildrenOfItem:] + 84
17 AppKit 0x00007fff82dcb750 -[NSOutlineView _expandItemEntry:expandChildren:startLevel:] + 209
18 AppKit 0x00007fff82dc73f5 -[NSOutlineView numberOfRows] + 198
19 AppKit 0x00007fff82dc7ae8 -[NSTableView _verifySelectionIsOK] + 128
20 AppKit 0x00007fff82dc7a4c -[NSTableView _tileAndRedisplayAll] + 375
21 AppKit 0x00007fff82dcaedd -[NSOutlineView reloadData] + 352
22 AppKit 0x00007fff8315e701 -[NSNavOutlineDelegate reloadChildrenForNode:] + 74
23 AppKit 0x00007fff83149008 -[NSNavDataSource _reloadChildrenForNode:] + 43
24 AppKit 0x00007fff83147773 -[NSNavDataSource _handleChangedAllChildrenInExpandedNode:] + 61
25 AppKit 0x00007fff8314d5a4 -[NSNavDataSource observeValueForKeyPath:ofObject:change:context:] + 748
26 AppKit 0x00007fff8314fd39 -[NSNavFBENode _sendChildrenChangedWithDictionary:] + 350
27 AppKit 0x00007fff833e9057 -[NSNavFBEQueryNode _resetQueryForChangedAttributes:andSendChildrenChanged:] + 191
28 AppKit 0x00007fff83147b5b -[NSNavDataSource _rawChildrenOfNode:] + 140
29 AppKit 0x00007fff83147bcc -[NSNavDataSource _processedChildrenOfNode:] + 36
30 AppKit 0x00007fff8314da80 -[NSNavDataSource _cachedChildrenForNode:createIfNeeded:] + 280
31 AppKit 0x00007fff83145591 -[NSNavBrowserDelegate _cachedChildrenForNode:logonOK:] + 422
32 AppKit 0x00007fff831440a4 -[NSNavBrowserDelegate browser:numberOfChildrenOfItem:] + 27
33 AppKit 0x00007fff82e33a68 -[NSTableView numberOfRows] + 157
34 AppKit 0x00007fff82dc7ae8 -[NSTableView _verifySelectionIsOK] + 128
35 AppKit 0x00007fff82dc7a4c -[NSTableView _tileAndRedisplayAll] + 375
36 AppKit 0x00007fff82f22952 -[_NSBrowserTableColumnViewController setView:] + 233
37 AppKit 0x00007fff82f231a0 -[_NSBrowserTableColumnViewController loadView] + 848
38 AppKit 0x00007fff82d1c8e3 -[NSViewController view] + 38
39 AppKit 0x00007fff82f22e02 -[NSBrowserColumnViewController setMenu:] + 38
40 CoreFoundation 0x00007fff818d3304 -[NSArray makeObjectsPerformSelector:withObject:] + 516
41 AppKit 0x00007fff82f21550 -[NSBrowser setMenu:] + 100
42 AppKit 0x00007fff82f22d72 -[NSView menu] + 91
43 AppKit 0x00007fff82f2226d -[NSBrowser addColumn] + 1786
44 AppKit 0x00007fff82f21a36 -[NSBrowser loadColumnZero] + 89
45 AppKit 0x00007fff83145382 -[NSNavBrowserDelegate loadAndRestoreCurrentBrowsingNodePath:selectedNodes:] + 64
46 AppKit 0x00007fff8314a79d -[NSNavDataSource _loadAndRestoreCurrentBrowsingNodePath:selectedNodes:] + 105
47 AppKit 0x00007fff831499e8 -[NSNavDataSource setFileListMode:] + 220
48 AppKit 0x00007fff831d133c -[NSSavePanel _readFileListMode] + 259
49 AppKit 0x00007fff831d27a2 -[NSSavePanel runModal] + 62
50 Tk 0x00000001010ac983 TkAboutDlg + 5916
51 Tcl 0x00000001006a8559 Tcl_CreateInterp + 4820
52 Tcl 0x00000001006a9769 Tcl_EvalObjv + 66
53 _tkinter.so 0x000000010068e243 Tkapp_CallDeallocArgs + 11477
54 Python 0x0000000100016ff3 PyObject_Call + 112
55 Python 0x000000010009654e PyEval_EvalFrameEx + 20380
56 Python 0x0000000100096cce PyEval_EvalCodeEx + 1803
57 Python 0x000000010009535e PyEval_EvalFrameEx + 15788
58 Python 0x0000000100096cce PyEval_EvalCodeEx + 1803
59 Python 0x000000010009535e PyEval_EvalFrameEx + 15788
60 Python 0x0000000100096cce PyEval_EvalCodeEx + 1803
61 Python 0x00000001000388e1 PyClassMethod_New + 1748
62 Python 0x0000000100016ff3 PyObject_Call + 112
63 Python 0x000000010009654e PyEval_EvalFrameEx + 20380
64 Python 0x0000000100096cce PyEval_EvalCodeEx + 1803
65 Python 0x00000001000388e1 PyClassMethod_New + 1748
66 Python 0x0000000100016ff3 PyObject_Call + 112
67 Python 0x00000001000269df PyClass_New + 1575
68 Python 0x0000000100016ff3 PyObject_Call + 112
69 Python 0x00000001000909db PyEval_CallObjectWithKeywords + 175
70 _tkinter.so 0x000000010068c126 Tkapp_CallDeallocArgs + 3000
71 Tcl 0x00000001006a405b TclInvokeStringCommand + 105
72 Tcl 0x00000001006a8559 Tcl_CreateInterp + 4820
73 Tcl 0x00000001006e02eb TclStackAlloc + 6637
74 Tcl 0x00000001006e6fae TclStackAlloc + 34480
75 Tcl 0x00000001006a9ad7 TclEvalObjEx + 714
76 Tk 0x00000001010301dc TkCreateMainWindow + 6699
77 Tcl 0x00000001006a8559 Tcl_CreateInterp + 4820
78 Tcl 0x00000001006a9769 Tcl_EvalObjv + 66
79 Tcl 0x00000001006a9906 TclEvalObjEx + 249
80 Tcl 0x0000000100718d74 TclObjGetFrame + 738
81 Tcl 0x00000001006a8559 Tcl_CreateInterp + 4820
82 Tcl 0x00000001006e02eb TclStackAlloc + 6637
83 Tcl 0x000000010071a9e6 TclObjInterpProcCore + 1623
84 Tcl 0x00000001006a8559 Tcl_CreateInterp + 4820
85 Tcl 0x00000001006a91a9 Tcl_CreateInterp + 7972
86 Tk 0x000000010100afee Tk_BindEvent + 5234
87 Tk 0x000000010100e234 TkBindEventProc + 342
88 Tk 0x0000000101014857 Tk_HandleEvent + 1431
89 Tk 0x0000000101014960 Tk_HandleEvent + 1696
90 Tcl 0x000000010070bfcb Tcl_ServiceEvent + 147
91 Tcl 0x000000010070c2bf Tcl_DoOneEvent + 314
92 _tkinter.so 0x000000010068abe9 init_tkinter + 1132
93 Python 0x0000000100095187 PyEval_EvalFrameEx + 15317
94 Python 0x0000000100096cce PyEval_EvalCodeEx + 1803
95 Python 0x000000010009535e PyEval_EvalFrameEx + 15788
96 Python 0x0000000100096cce PyEval_EvalCodeEx + 1803
97 Python 0x000000010009535e PyEval_EvalFrameEx + 15788
98 Python 0x0000000100096cce PyEval_EvalCodeEx + 1803
99 Python 0x0000000100096d61 PyEval_EvalCode + 54
100 Python 0x00000001000ae65a Py_CompileString + 78
101 Python 0x00000001000ae723 PyRun_FileExFlags + 150
102 Python 0x00000001000b023d PyRun_SimpleFileExFlags + 704
103 Python 0x00000001000bc286 Py_Main + 2718
104 Python 0x0000000100000e6c start + 52
)
terminate called after throwing an instance of ‘NSException’
Abort trap
October 24, 2010 at 9:31 pm
Hi,
If you do not specify an output file name other than typing in test, the error you get on this book is:
raise DrmException(“no key found. maybe the PID is incorrect”)
mobidedrm.DrmException: no key found. maybe the PID is incorrect
So this program did NOT find the right pid.
Let’s verify that it is not the right PID.
The one K4DeDRM found for this book was: g/DbOnGeZM
Please run K4Munswindle on the same book and look in the conversion log and see what it says the proper PID to use is?
They they are different then perhaps you have more than one .kindle-info hanging around?
Please open a Terminal.app and do the following for me:
cd ~
find ./ -name “.kindle-info”
and let me know how many it finds and where it is located (edit out your username) before posting.
As for the second problem, not being able to use the output … buttun. That worries me as I an not recreate this at all.
The error was that python tried to insert ‘nil’ into a gui widget and died as a result. I just can’t see how it can generate a ‘nil’ to begin with
010-10-24 13:18:25.079 Python[92388:d07] An uncaught exception was raised
2010-10-24 13:18:25.079 Python[92388:d07] *** -[NSCFArray insertObject:atIndex:]: attempt to insert nil
2010-10-24 13:18:25.085 Python[92388:d07] *** Terminating app due to uncaught exception ‘NSInvalidArgumentException’, reason: ‘*** -[NSCFArray insertObject:atIndex:]: attempt to insert nil’
*** Call stack at first throw:
This woudl mean that the tkwidget that was supposed to allow you to navigate to select a place for the file returned a NIL before it even opened.
What version of Python do you have installed, and what Mac OSX version?
Thanks,
October 24, 2010 at 9:53 pm
K4Mac gives a different PID – XKMQ2ajWNG. Only 1 .kindle-info file
.//Library/Application Support/Amazon/Kindle/storage/.kindle-info
python is 2.6.1 on Mac OS 10.6.4
Anything else?
Thanks
October 24, 2010 at 10:18 pm
Hi,
Yes – if you only have one .kindle-info file then the PID generated is wrong for one of two reasons:
It is not getting the right user name or the right Hard Disk Serial Number. Both are needed to calculate things.
So please open a Terminal.app and do the following:
whoami
env
Please verify that the response form whoami is exactly identical to the value for
USER in your environment (output of the env command) and that the value of LOGNAME (also in the output of the env command) is the same as well.
Next please verify that you only have 1 hard disk drive and what its serial number is by running the following command in Terminal.app
ioreg -r -c AppleAHCIDiskDriver
And look in the output and let me know how many times this exact string appears:
“Serial Number” =
Is it more than once? Does the Serial Number you find have any non-numberic characters in it? Does it match the same value you see for the Serial ATA hard drive you get from Apple’s System Profiler?
These two pieces of information are needed to properly use the .kindle-info along with data from the book itself to create the proper book specific PID.
Something must be different about it?
As for the other problem, I can not get the gui to fail in the way yours is doing.
For some reason tkFileDialog.asksaveasfilename is not working on your system and is returning nil.
I can work around this by having it ask for an output directory instead (like K4Munswindle does). I will search the web to see if anyone else has reported a similar problem.
Thanks for helping me figure out what is up so that I can get it fixed for others as well.
October 24, 2010 at 10:33 pm
The whoami, and environment variables all match. I only have one hard drive, but the serial number isn’t strictly alpha-numeric – there are spaces(?) and a ‘-‘
| “Serial Number” = ” WD-WX81A30K4508″
The system profiler shows the same serial number, but again, it appears there are leading spaces since it doesn’t line up in a nice table form with the rest of the data – it’s hard to show without a screenshot.
Thanks for working on this – it’s going to be great to have this for Calibre. I can only hope that I’m NOT the only person having problems – I could always do it the long way.
October 24, 2010 at 10:56 pm
Hi,
Neither is a problem. The username is the same so it should be the one that Kindle for Mac used when it set things up and your serial number is like mine (with leading spaces and a dash) which I strip and it works fine with my machine.
This is very strange indeed.
Please grab dumpkinfo.py from here:
http://pastebin.com/i5b3ZfHN
and download to your Desktop and rename it to
dumpkinfo.py
Then open a Terminal.app session and do the following:
cd ~/Desktop
python ./dumpkinfo.py
Does the login information there look right? Is your name in the kidnle account info look correct?
If that works, and your username is correct (no leading spaces or trailing spaces there right), and your serial number is corect, then dumpkinfo.py should dump the .kindle-info file for you and should work. It it does then k4dedrm should work as well.
Please let em know if you see anything funny.
October 24, 2010 at 11:58 pm
Hi Zoe,
I found a bug that has never been fixed in tkFileDialog.asksaveasfilename in that it can sometimes return an object and not its string representation. The workaround is easy (which is why it has probably never been fixed).
Open K4DeDRM.pyw in any text editor and look for the following routine:
def get_outpath(self):
Then change the following lines
outpath = tkFileDialog.asksaveasfilename(
parent=None, title=’Select Unencrypted Mobi File to produce’,
defaultextension=’.mobi’, initialfile=initname,
filetypes=[(‘Mobi files’, ‘.mobi’), (‘All files’, ‘.*’)])
to use the str() command as follows:
outpath = str(tkFileDialog.asksaveasfilename(
parent=None, title=’Select Unencrypted Mobi File to produce’,
defaultextension=’.mobi’, initialfile=initname,
filetypes=[(‘Mobi files’, ‘.mobi’), (‘All files’, ‘.*’)]))
ie. you need to wrap the entire function call in the str() function (so don’t forget the extra closing parenthesis at the end.
Then save it (and try not to change anything else.
Then give it a try and see if it lets you use the … button on the output or not.
If that works for you I will add the fix. If not, I we rework things to use the select an output directory approach used in K4Munswindle instead.
Of course, the problem with the pids is the only reason that the calibre plugin is not working correctly and we still have to figure out why.
Thanks
October 25, 2010 at 12:05 am
Okay, the dumpkinfo output looked fine, so I decided to start from scratch. I uninstalled the Kindle app (including all the support/plist files and .kindle-info) and redownloaded it. Deleted all the books and downloaded again. This time, the standalone K4deDRM worked (which I can understand), but it also didn’t crash when I clicked the output directory selector (that I don’t understand!)
The plugin, however, still isn’t working for me.
October 25, 2010 at 12:25 am
Hi Zoe,
> Okay, the dumpkinfo output looked fine
That is good to know. That means we are getting the username and hard drive serial number stuff correct. That means that this tool should work for you.
> I uninstalled the Kindle app (including all the support/plist files and .kindle-info) and redownloaded it. Deleted all the books and downloaded again. This time, the standalone K4deDRM worked
Please try the dumpkinfo.py program again. Is anything different from the last time? Anything at all?
I am glad to hear that it works.
> but it also didn’t crash when I clicked the output directory selector (that I don’t understand!)
This does not surprise me. It seems there is a long running bug in that widget kit tkFileDialog.asksaveasfilename() routine can sometimes seemingly at random return something that it thinks is an object and not a string. See my previous post about a workaround to fix this for all time (hopefully).
> The plugin, however, still isn’t working for me.
Arggh! This I do not understand. Are you running calibre on the exact same account as you have the Kindle.app (and the .kindle-info file)? You have to, otherwise the plugin can not find the .kindle-info file and will not be able to open the books.
Please try the calibre plugin again but make sure you have grabbed a freshly downloaded from Amazon (with the new Kindle.app you just installed) book and not some previous copy of it you might have lying around.
So open the book that you want to convert in normal Kindle.app and read a few pages just to verify that it works. Then go and grab that exact same book from your Documents/My Kindle Content/ directory and try it in Calibre. Please make sure you do NOT have the mobiplugin installed as well as it can interfere since both work on mobi books.
Please let me know if that works or not.
Go to your Documents/My Kindle Content/ and only use that book
October 25, 2010 at 12:43 am
Hi Zoe,
If you have mobidedrm calibre plugin installed you might want to read this comment from DiapDealer on the ApprenticeAlf blog:
Basically it says you might get a bogus import error message if you have k4dedrm plugin and mobidedrm plugin because the k4dedrm works and fixes it and the second (the mobidedrm) one gets passed an already unencrypted file so it generates a bogus error message. If you go and look in calibre you should see an unencrypted file.
Maybe that is what is going on?
October 25, 2010 at 1:00 am
It wasn’t a bogus error – I tried to read it with the calibre viewer and it failed – but the original mobidedrm was what was causing my problems. If I disable MobideDRM, I can succesfully add K4Mac books – yea!!!
I guess this was user error, for which I apologize.
I’m thrilled to have this all working – thanks for your patience.
October 25, 2010 at 1:29 am
Hi,
Not user error at all. The K4DeDRM plugin imports mobidedrm.py (we thought its own copy) but I think instead it is getting something from the mobidedrm plugin instead.
We might have to revert back to how K4PCDeDRM did things to prevent the conflict from being an error. K4PCDeDRM did not import mobidedrm.py at all and instead copied the key code pieces right into its own internal code.
We could make K4DeDRM do the same thign and that might prevent any problems caused by having both plugins installed at the same time.
I will talk with DiapDealer to see if he sees the same issue on his machine as I am not much of a Calibre user except as a book converter.
Thanks for the bug report and thanks for working through it with me. I am glad you got things working.
October 25, 2010 at 2:43 am
This is my mistake… I forgot all about people using MobiDeDRM and K4DeDRM at the same time. I ran into this issue with the ignobleepub and the ineptepub plugins being installed at the same time. It’s pretty simple (if a little bit ugly) to fix. It’s not a python import problem or anything, but rather, it’s an issue of the MobiDedrm plugin failing just a little too gracefully.
Calibre doesn’t process its plugins serially: i.e. when two plugins run on the same filetypes (azw, prc, mobi), the output of the first plugin doesn’t get passed to the input of the second… the second plugin starts over with the original encrypted file.
The last plugin that runs successfully, determines which file gets imported. In this case, K4DeDRM is successfully removing the encryption… then MobiDeDRM comes along and and “successfully” determines that it can’t decrypt the file, so it “successfully” hands the original file (encrypted) to Calibre. Basically, the plugins are being a little too polite.
The solution is to make the MobiDeDRM plugin “Bomb out” if it can’t decrypt the book. Right now, it’s returning the path_to_ebook and we need to make it return nothing, or an error, or just about anything but “path_to_ebook”. We should probably do the same with the K4DeDRM plugin in case another plugin comes along later, and has the same issues.
So until we can get updated versions of the MobiDeDRM and K4DeDRM plugins, the solution is to disable one or the other if you happen to have both installed.
My bad…
October 25, 2010 at 5:57 pm
Hi DiapDealer,
So instead we want something like the following, right?
def run(self, path_to_ebook):
from calibre.gui2 import is_ok_to_use_qt
from PyQt4.Qt import QMessageBox
PID = self.site_customization
data_file = file(path_to_ebook, 'rb').read()
ar = PID.split(',')
for i in ar:
try:
unlocked_file = DrmStripper(data_file, i).getResult()
except DrmException:
if is_ok_to_use_qt():
d = QMessageBox(QMessageBox.Warning, "MobiDeDRM Plugin", "Error decoding: %s\n" % path_to_ebook)
d.show()
d.raise_()
d.exec_()
raise Exception("MobiDeDRM Plugin: Error decoding ebook")
else:
of = self.temporary_file('.mobi')
of.write(unlocked_file)
of.close()
return of.name
Is that right? If so I can fix the mobidedrm_plugin and the k4dedrm_plugin to raise and exception (as the Calibre file plugin documents sugggest) instead of returning a value.
Can you do the same for the B&N and AdeptEPub ones if neeeded?
Thanks
October 25, 2010 at 7:28 pm
Yes, that’s correct.
The B&N and Adept Epub plugins already take this into account. I fought with them a long time to figure it out before I released them. I can’t believe it didn’t occur to me while working on the K4DeDRM plugin!
Sorry for the snafu.
October 25, 2010 at 8:08 pm
some_updates,
Even with these fixes, if mobidedrm runs first, the user is still going to get the erroneous “Error decoding” message pop-up. Even though K4DeDRM will eventually successfully import the decrypted book. Or vice versa if k4dedrm runs first.
The only thing that can be done about that is to pick one of the plugins to always run first (priority = 100) and get rid of the Qt pop-up from that plugin’s code.
But then, if a user only uses one of the plugins, they may not get the failure message at all if they’re using the one with no notification.
It’s a bit of a catch-22.
October 25, 2010 at 8:48 pm
It seems to me that the only solution is to make K4DeDRM able to also decode non-Kindle ebooks. There must be some EXTH in Kindle books that could be detected, and then either we get the kindleinfo file and use the info in that, or we use the PIDs entered in the customisation dialog.
This has the advantage of moving the MobiDeDRM into the K4DeDRM code, and I suspect it can also be made to run from the command line with a little tweaking, thus totally replacing MobideDRM.
October 25, 2010 at 10:39 pm
Hi Pual,
Actually that is now easier to do with the K4DeDRM since it actually now imports has its own copy of mobidedrm.py inside it s zip (actually moidedrm v17 – that has slight modifications to v16 to allow it to be imported easily). We could take the customization part from the current mobidedrm plugin to get a list of the user’s fixed pids, and then look for the kindle-info file.
If the kindle-info file exists, try its PID first, and if that fails, in turn try the pids from the customization part.
That way, as you said, we could eliminate the need to keep/maintain a separate mobidedrm_plugin.
I think this is quite doable unless I am missing something here. DiapDealer, what do you think?
October 25, 2010 at 10:56 pm
Yes, it’s quite doable.
On the plugin side of things, just create an array of PIDs (from the kindle.info flle if it exists AND from the plugin customization string) and try them all in a for loop. You wouldn’t technically have to differentiate between Amazon ebooks or encrypted mobi’s from elswhere. The PID will either be correct or it won’t be. If it exhausts all of the PIDs without a winner… then throw the error message.
October 25, 2010 at 10:48 pm
I have a new Calibre plugin for everyone to try. Based on the erdr2pml.py script by The Dark Reverser. It decrypts secure pdb files and imports them into Calibre as pmlz archives. Install it, customize it by adding your name the last 8 digits of the credit card used to purchase the ebook (separate the two with a comma): Your Name,12345678. NOTE: do NOT put quotes around your name.
If you have more than one credit card, separate the data with a colon: Your Name,12345678:Other Name,23456789
The file size is a little big because I’ve included the psyco python libraries (compiled for each platform) to try to gain as much speed as possible. The original was pretty slow without them. If your system can use them, it will. Unless you have a 64-bit version of Calibre, it should be able to make use of them.
Also note that calibre cannot read the meta-data of the original encrypted pdb, so you may want to make sure that your ebooks are named in such a way that calibre can find at least the Title and Author from the filename.
The plugin can be found here: http://www.mediafire.com/?kqr373hjkp5r7j8
And the README can be found here: http://www.mediafire.com/?xvngtbdobu5kds2
October 25, 2010 at 11:19 pm
Worked perfectly here, THANK YOU!!!!!
October 26, 2010 at 3:49 am
I downloaded the file but it doesn’t downlaod as a zip file for some reason. I’m on a new Imac and other things have downloaded as zip files. Not sure what is wrong. Any ideas? Thanks for all the help and work.
October 26, 2010 at 10:40 am
Larry,
The second link for the README file is a txt file (unzipped). The first link is the actual plugin — and it SHOULD download as a zip file. What is it downloading as on your machine?
October 26, 2010 at 11:55 am
It downloads as eReaderPDB2PML_v01_plugin. It is not a zip file. Doesn’t seem to have an ext. And I’m not able to choose it from the Calibre plugin window. Hope this makes sense. I have downloaded zip files before. Don’t know why this happens. Thanks for your help.
October 26, 2010 at 12:25 pm
In the plugin, there is a readme.txt and eReaderPDB2PML_v01_plugin.py but there is no zip file.
October 26, 2010 at 12:39 pm
Yes, I seem to remember that OS X will automatically unzip zip-files when downloaded by clicking on a hyperlink. Hopefully an experienced Mac user will come along to give you some advice here. I don’t know if you can “right-click” and choose “save as” an a Mac, but I’m pretty sure there must be a fairly simple solution… I just don’t know it. I’ll look into it.
Sorry
October 26, 2010 at 12:50 pm
Larry,
In the meantime, you should be able to zip everything back up.
Go into that folder (eReaderPDB2PML_v01_plugin). There should be three folders and three files in there. Highlight everything (six total items) and right-click (or “whatever click” on a Mac that brings up the menu) and choose “Compress”. The name of the zip file doesn’t really matter, but make sure you save it as a zip file. You should then be able to import that resulting zip file into Calibre.
Hope that helps in the meantime.
October 26, 2010 at 1:35 pm
Hi Larry,
If this is with Safari, then do the following:
In Safari->Preferences->General : UNCHECK “Open “safe” files after downloading.
For safety reasons, auto-running anything downloaded (“safe” or not) is simply a bad idea. The first thing I do on all of my family’s Macs is turn off this “feature”.
October 27, 2010 at 2:09 am
Some_updates, I think the problem is with the installing the installation of command line tools. I did go there and clicked on install but nothing really happended. So I went ahead, maybe they were installed when I clicked on them and ran the terminal with calibredb and this is what happened:larry-kettells-iMac:Desktop larrykettell$ calibredb add 9780786541096_Cath_Eleventh.pdb
Running file type plugin eReader PDB 2 PML failed with traceback:
Traceback (most recent call last):
File “site-packages/calibre/customize/ui.py”, line 176, in _run_filetype_plugins
File “”, line 58, in run
ImportError: No module named erdr2pml
Backing up metadata
Notifying calibre of the change
larry-kettells-iMac:Desktop larrykettell$
Thanks again for all the help.
October 27, 2010 at 12:57 am
Thankyou both for the help. I changed the preferences and it downloaded fine. However, although the program works outside of Calibre, I haven’t yet got it to work with the plugin. This is what happens :larry-kettells-iMac:Documents larrykettell$ /Users/larrykettell/Documents/Conversion\ tools/Books/Input calibredb
-bash: /Users/larrykettell/Documents/Conversion tools/Books/Input: is a director
I went to Calibre to install command line tools but not sure anything happened. Or what I should do from there. What an adventure. New computer and new tools. thanks for all your help.
October 27, 2010 at 1:53 am
Hi Larry,
Please read the README file that comes with the plugin. Please make sure you have following the installation instructions and the configuration instructions.
For the eReader PDB plugin, these are:
Installation:
Go to Calibre’s Preferences page… click on the Plugins button. Use the file dialog button to select the plugin’s zip file (eReaderPDB2PML_vXX_plugin.zip) and click the ‘Add’ button. You’re done.
Configuration:
Highlight the plugin (eReader PDB 2 PML under the “File type plugins” category) and click the “Customize Plugin” button on Calibre’s Preferences->Plugins page. Enter your name and last 8 digits of the credit card number separated by a comma: Your Name,12341234
Both the name and the last 8 digits of the credit card number have to match those you used when you bought the eReaderPDB book.
If you are still having trouble, then the first thing is to install the calibre command line tools on Mac OSX as follows:
Go to the ‘Preferences’ page and click on the miscellaneous button, you’ll see the option to install the command line tools.
Once they are installed, copy the book you want to remove the DRM from to your Desktop. Then open Terminal.app (it is in /Applications/Utilities) and then in Terminal.app type in the following:
cd ~/Desktop
calibredb add YOUR_EBOOK_NAME_HERE
The see what if any error message it comes back with.
October 27, 2010 at 2:45 am
Larry,
Did you re-download the zip file after all your earlier troubles… or did you try to zip all of the files back up (like I suggested you might try)?
Because this:
> “Traceback (most recent call last):
File “site-packages/calibre/customize/ui.py”, line 176, in _run_filetype_plugins
File “”, line 58, in run
ImportError: No module named erdr2pml”
sounds like there’s a file missing from the plugin’s zip file.
I would suggest uninstalling (removing) the plugin… downloading the Tools2.1 archive… unzipping it… and installing the eReaderPDB2PML plugin that can be found in the “Calibre_Plugins” folder.
October 27, 2010 at 2:56 am
Good idea. I removed everything and downloaded the tools again and went through the process and it worked. Thanks everyone for all the help.
October 27, 2010 at 3:10 am
Yay!
Glad to hear it’s now working for you. 😉
October 26, 2010 at 2:38 pm
guys, first of all a BIG THANK YOU for all the hard and great work done 🙂
now, I tried to add the new pluggins from the tools package v2.0 and I’m getting an error with MobiDeDRM_0.16_plugin.zip (the others gave no problems); here is the error message I’ve got:
ERROR: Unhandled exception: BadZipfile:File is not a zip file
Traceback (most recent call last):
File “site-packages\calibre\gui2\preferences\plugins.py”, line 150, in add_plugin
File “site-packages\calibre\customize\ui.py”, line 377, in add_plugin
File “site-packages\calibre\customize\ui.py”, line 65, in load_plugin
File “zipfile.py”, line 693, in __init__
File “zipfile.py”, line 713, in _GetContents
File “zipfile.py”, line 725, in _RealGetContents
BadZipfile: File is not a zip file
I’m still using Calibre version 0.7.18… but it should work on version 0.6.44 or higher, right?
Do you have some ideas? Please? 🙂
Thanks!!
October 26, 2010 at 3:23 pm
Soalla,
First, make sure that the original mobidedrm plugin is what you really need. It won’t help you with Kindle4PC or Kindle4Mac ebooks (or files that you’ve pulled off of your Kindle). It will only work for encrypted mobi’s that you already have a valid PID for. If that’s what you need… fine. But otherwise, you should wait for the newest version (hopefully soon) of the plugin that will work for ALL situations (except the files pulled from a Kindle).
If this is the one that you KNOW you need, it sounds like you haven’t unzipped the Tools 2.0 archive. Windows will allow you to browse inside zipfiles (like a folder) and Calibre will allow you to select the plugin zipfile within the zipfile. If that is the case, you need to unzip the tools archive and THEN try to import the plugin into Calibre.
If you have indeed unzipped the tools archive properly, then you may have a corrupt zip file. In this case you should download a fresh version of the Tools2.0 archive, unzip it, and try to add the plugin again.
Hope something in all that rambling helps! 😉
October 26, 2010 at 3:36 pm
Soalla,
Forget most ofl the nonsense that I spouted above.
The mobidedrm_0.16.zip included in the Tools2.0 archive is indeed unable to be added to calibre. It seems like a valid zip file… but calibre certainly doesn’t recognize it as one.
Same with the one from the Tools1.9 archive. 😦
I’ll have to check it out.
October 26, 2010 at 3:45 pm
If you’re certain that the MobiDeDRM_0.16_plugin.zip plugin is what you want/need… you can try to download it from here. http://www.mediafire.com/?juk4ua6a5xnarqn
October 26, 2010 at 4:01 pm
thanks again, DiapDealer 🙂
I’ll try it as soon as possible!
October 26, 2010 at 3:55 pm
thanks for your fast reply, DiapDealaer 🙂
In fact I’ve been using most of the scripts for a long time with different ebook formats I’ve been buying from several different ebook stores (sadly less and less each passing month – even today waterstones emailed me saying they will cease to sell ebooks to my country…) to use on my different devices.
anyway, I never used the plugins, only the executable versions, but today I decided to give them a try… and got that error (btw, I know I must unzip all the tool, etc, etc, as you suggested on the first reply :))
so, glad to know the problem is not specific of my machine and installation, but sorry that you need to work a bit more on the plugin… I’m sure you’ll have a solution for the problem in no time! and thanks again for all the time and effort!
October 26, 2010 at 4:08 pm
Hi,
Just a thought, but on many windows machines you can’t easily see the extension and perhaps his web browser has “unzipped” the file already? Similar to Larry’s issue?
I looked in tools v2.0 in the Calibre Plugins folder and run unzip test on it:
unzip -t MobiDeDRM_0.16_plugin.zipArchive: MobiDeDRM_0.16_plugin.zip
testing: MobiDeDRM 0.16_plugin.py OK
No errors detected in compressed data of MobiDeDRM_0.16_plugin.zip.
so it is a proper zip file.
I also checked the copy in the Kindle_Mobi_Tools folder inside tools version 2.0 and it passes as well:
unzip -t MobiDeDRM_0.16_plugin.zip
Archive: MobiDeDRM_0.16_plugin.zip
testing: MobiDeDRM 0.16_plugin.py OK
No errors detected in compressed data of MobiDeDRM_0.16_plugin.zip.
So something is either corrupting them upon download or they are being unzipped in some way by the act of downloading (a setting in the browser?)
October 26, 2010 at 4:26 pm
It’s a real puzzler. My windows machine recognizes it as a valid zip file (Windows built-in or 7-zip), but Calibre doesn’t like it. If I unzip it and rezip it… calibre is OK with it.
I haven’t had any trouble with the other plugin’s zipfiles in the archive, but calibre doesn’t like THAT one for some reason.
October 26, 2010 at 4:36 pm
yep, the version you posted for me 😀 worked well, DiapDealer, no problem adding it to Calibre 🙂
the version from tools 2.0 didn’t, even after un-ziping it and re-ziping…
and my browser is FF in windows (XP, btw), my machine identifies the file as a zip file, I can uncompress it as all other zips… the other plugins in the folder all were added to Calibre without issues…
to top it, meanwhile I tried it on another machine, having re-downloaded the whole package, and using a more recent Calibre version, and it still failed…
October 26, 2010 at 7:33 pm
For the record… the only plugins that require a minimum Calibre version are the IgnobleEpub, IneptEpub, and the eReaderPDB2PML plugins. For these three, you need Calibre 0.6.44 or higher.
I think I forgot to mention that for the eReader plugin.
October 26, 2010 at 8:10 pm
Announcing version 2.1 of tools:
tools_v2.1.zip
http://www.mediafire.com/?hmxshqx1166166e
What’s New in tools_v2.1
– added yet another wonderful new calibre plugin this time for eReader PDB style books. See the Calibre Plugins folder for more info. This works on Windows, Mac OS X, and Linux. (Thank you DiapDealer!)
– announcing a new tool K4MobiDeDRM (and plugin) that combines MobiDeDRM, K4PCDeDRM, K4DeDRM, K4MUnswindle, all into one program and one plugin. This will make things easier for volunteers to keep up support in the future and prevent calibre plugin conflicts. With this release MobiDeDRM and its plugin, K4PCDeDRM and its plugin, and K4DeDRM and its plugin have all been obsoleted and removed. This works on Mac OSX, Windows, and Linux. (Thank you DiapDealer and some_updates)
***
Please note if these now obsoleted tools are working for you, then by all means keep using them. There is no additional functionality in this new tool that can’t be done with these older tools. The change simply unifies the code bases and makes support easier.
***
***
Calibre plugin users should remove the MobiDeDRM plugin, K4DeDRM plugin, and K4PCDeDRM plugin to prevent conflicts if installing the new K4MobiDeDRM plugin
***
– updated to the latest version of MobiUnpack – my last attempt to update it was incorrect. Again, for all platforms. (Thanks Paul!)
Have Fun!
October 28, 2010 at 4:52 pm
Hi! I just want to thank everyone for this fantastic tools v2.1: I managed to use it despite being a very basic windows user. I have just bought today my first Kindle ebook for K4PC, and wanted to deDRM it in order to be able to use it on whatever kind of ebook reader I wanted. After having spent most of the a good couple of hours reading this thread, I obviously understood nothing of the technicalities, but however downloaded the tools v2.1 and after a couple unsuccessful attempts using first the newest K4MobiDeDRM.pyw and then unswindle_v7.pyw, I succeeded cracking the DRM with the Topaz tools (yep, it turned out it was a Topaz file, I thought it was AZW…), following the instructions on the relevant Readme.
The fact that even a pc illiterate like me could do that says a lot about the sterling work you guys have done! Many thanks again!
October 30, 2010 at 6:51 pm
THANK YOU!!!
Since Fictionwise has refused to update their iPhone app to something that works, I’ve been stuck without the 100s of books I purchased there. Your plugin for Calibre works perfectly, and I can load them onto my new Kobo reader. Converting from PDB to Epub works like a charm.
Seriously, a thousand thanks.
October 31, 2010 at 5:10 am
Hi,
Amazon has released Kindle for Mac version 1.2.1 build 30781 today.
So I have updated Kindle_4_Mac_Unswindle to deal with it.
Until I figure something else out K4Munswindle is the only tool that can provide the Topaz PIDs for use with the Topaz_Tools.
If you use K4MUnswindle and have upgraded to the latest Kindle for Mac you will need this update.
***Please remember to disable auto updating in the Kindle for Mac Preferences to prevent unwanted updates.
Kindle_4_Mac_Unswindle_v1.3.zip
http://www.mediafire.com/?0ojga8mo1mfig89
November 1, 2010 at 1:57 pm
Thank you very much. After months of frustration I can now use my e-reader of choice to enjoy my purchased E-Books on. Nice job……..well done!
October 31, 2010 at 9:14 pm
Thank you…I can finally read my books on the device of my choice.
November 1, 2010 at 5:50 pm
Hi,
Because so many DRM epubs are improperly zipped, I have created an unzip.py program (based on one used by ActiveState Python) that actually uses the local filenames in place of the central file names.
Here is the unzip.py program:
http://pastebin.com/K9UgBe7H
We should be able to incorporate this python code into our epub plugins to unzip the original into a temporary directory and then properly rezip it to create an epub that will import properly to calibre and will load into other tools properly.
Please test this on your system and let me know if it works or not. It should work as it is on Mac OS X and on Linux. I am just unsure about Windows.
November 1, 2010 at 8:10 pm
some_updates, tried it in Windows XP and it worked perfectly 🙂
I used a DRMed epub first, and it was unziped correctly (and all the html parts were still encrypted); then I unDRMed said epub and used your sript – it unziped again perfectly, with all the correct hrml parts and the rest of the files… I think this was what you wanted, right?
November 1, 2010 at 9:28 pm
Hi soalla,
Yes, thanks for testing it. I ran more tests of my own and it failed on the very zip archives I wanted it to work on (at least on Mac OSX) which are DRM epub files where the local filename and the central filename for one of the xhtml files do not match.
The version you have did not successfully read the data for that one problem file.
I have rewritten it and now I have a version that works when the problem with fielnames exists (at least I hope so). I have also written an zipfix.py program using the guts unzip.py to take a bad drm epub (or zip) in and write out a properly fixed epub (or zip) for the files whose names do not match.
I just wish Adobe would not produce epubs with broken zip files, but they do.
Thanks for testing it on windows. If that version works, then the new version will as well.
November 2, 2010 at 1:57 pm
Hi DiapDealer,
Okay, I polished up a zipfix.py program and posted it here:
zipfix.py
http://pastebin.com/KMN06pjX
For the command line the syntax is:
python zipfix.py INPUTZIPFILE OUTPUTZIPFILE
It will copy from the INPUTZIPFILE to a new OUTPUTZIPFILE fixing any local file name versus central file name issues along the way.
It can easily be imported into any other python program and used as well.
If possible, for the next release of the Adobe Adept and B&N epub plugins, I would like to incorporate this code into the plugins so that people do not have to worry or even think about if their DRM epub is a proper zip or not.
Please let me know what you think.
Thanks
November 3, 2010 at 3:26 am
Is there any news on de-drm’ing post-2.5 firmware files? I keep reading post saying to use K4PC, but that isn’t an option with subscriptions. Is anybody working on a way to de-drm them? Is there already a solution I’m not familiar with?
Thanks!
November 3, 2010 at 1:34 pm
I seem to recall an announcement by Amazon, that subscriptions are going to be available to the various free Kindle4(whatever) apps, soon. I would imagine that the various K4* scripts will be able to decrypt them at that time.
As to the post-2.5 Kindle files… there is nothing I know of, yet. And even if a solution were to be found, I suspect that “hacking” your Kindle would be a prerequisite for that solution to work properly.
November 4, 2010 at 3:32 pm
Hi,
There is now a new version of K4MUnswindle (version 1.4) that works with the latest Kindle.app version 1.2.2 (30814) and earlier builds.
*** You MUST remember to uncheck the “auto update” in the Kindle.app preferences, otherwise this program will simply stop working upon the very next update. ***
This will probably be my last release of the K4MUnswindle tool as I simply can’t keep up with the number of builds that Amazon is producing.
*** You MUST also have a recent set of Developer Tools installed. You can find the XCode/Unix tools on your very latest Mac OS X Install DVD, and then use Apple System Updates to update it to an even newer version if one is available.
Kindle_4_Mac_Unswindle_v1.4.zip
http://www.mediafire.com/?e1l1rf8vd3dyyls
Have Fun!
PS:
If there is yet another build, please convert to using K4MobiDeDRM which works fine for normal Kindle for Mac books.
For Topaz books, ONLY this program will tell you the PID you should use with the Topaz Tools. I hope to have a fix for that in the near future and then K4Munswindle will be officially retired.
November 5, 2010 at 11:12 pm
First, thanks for a great set of tools – using ereader2html to work my way through a whole library of books and wondering if there is any way to amend the script to run on an entire directory? Have worked up a script to auto rename the books 1.pdb, 2.pdb, 3.pdb etc. They all have the same account details and unlock code.
November 6, 2010 at 6:03 pm
Hi,
you could try this:
In my python folder I have all the docs I want to convert and 2 empty folders called “Done” and “Converted”
create a .bat file in the python folder (where you ereader script is) and paste the following into it changing the name and cc details. Save the file and then double click on it and it should do the following:
find all *.pdb files
convert them
move all the original files to the “Done” folder
create in the “Converted” folder an individual folder for each converted book with the folder name = name of the book
It works for me so hopefully it might work for you. (Saves time when you have multiple books to do at the same time).
@echo off
Echo You are about to process the following files
Echo .
dir *.pdb /b
Echo .
pause
echo .
for %%f in (*.pdb) do (
echo Processing %%f
–python ./ereader2html.py %%f ~/%%~nf “Caroline Moloney” 01996600
python ./ereader2html.py %%f ~/%%~nf “Caroline Moloney” 00172252
move %%f “C:\Python26\Done”
rename C:\Python26\Converted\%%~nf\book.html %%~nf.html
echo .
)
pause
November 14, 2010 at 3:47 pm
Thanks both – am on Win so the batch file solution would work better (although i have a Linux media server with python that I could use at a pinch).
Have created the batch and it runs fine – finds the files, but I keep getting error message “the syntax of the command is incorrect”. Will have a play around and see if I can resolve this. Thanks again for the steer.
November 14, 2010 at 4:00 pm
Fixed it – for anyone who is trying to use the batch file, you need to make sure there are no spaces in your file names before running the file. Working very well, thanks.
November 6, 2010 at 6:42 pm
On Linux… make sure your ereader2html.py script is in your folder full of pdb’s… then open a terminal, cd into that pdb folder, and type the following command:
find ./ -name “*.pdb” | tr \\n \ | xargs -0 -I {} python ./ereader2html.py {} “Your Name” 12345678
Change the name and cc# of course.
November 6, 2010 at 10:24 am
First of all, thanks to everyone who has worked so hard on these tools. Admirable efforts.
I am working through a set on instructions to try and establish how best to strip the DRM from Amazon books, and it has been suggested that I use K4MobiDeDRM.pyw after installing Python and PyCrypto.
The Readme file for K4MobiDeDRM.pyw says that I need to enter the 10 digit PID for my Kindle (which is a Kindle 3) – how do I find this? When I tried searching Google about this, I couldn’t find an answer – only unanswered questions from people who couldn’t find their PIDs on K3s.
Can anyone help? Thanks!
November 6, 2010 at 2:06 pm
Right now, the only Amazon ebooks that it’s possible to strip the DRM from, are ebooks that are downloaded to one of the free Kindle4* reading apps (Kindle4PC, Kindle4Mac, etc). The 10-digit PID is not necessary for ebooks downloaded to K4(PC/Mac)… the K4MobiDeDRM script will find the proper PID from the K4(PC/Mac) installation. The 10 digit PID is OPTIONAL (to be entered for ebooks purchased from non-Amazon sources — or a few other special cases).
If you are trying to strip the DRM from the ebook that was actually downloaded to your Kindle 3 device (and you transferred it to your PC via USB)… forget it. It can’t be done right now. You need to install and register the Kindle4(PC/Mac) program, download the ebook to your PC from your “Archived Items” (within the application) and run the K4MobiDeDRM script on THAT file (it can be found in the “My Documents\My Kindle Content” folder.. or something similar on a Mac) No PID needs to be entered.
If you determine that you absolutely NEED the PID for your K3, use the KindlePID.pyw script (found in the “Mobi Additional Tools” folder of the Tools_v2.1 archive). You enter your K3’s serial number to determine your PID.
November 7, 2010 at 10:17 pm
I’m getting an error when trying to de-DRM a NookStudy pdf I bought (man it’s *nuts* that you can’t put it on the Nook unless you remove DRM). The error I get is: unsupported security handler `EBX_HANDLER’.
FlateDecode: decoding error: incorrect header check.
Does anyone know how to solve that problem?
November 8, 2010 at 12:12 pm
Are you using version 7.4 of ineptpdf.pyw?
It should be able to deal with the EBX security handler.
The link to the 7.4 script is at the bottom of the post on Apprentice Alf’s blog – http://apprenticealf.wordpress.com/2010/02/12/ineptkey-ineptepub-ineptpdf/
November 12, 2010 at 1:33 am
A New Version of the tools is now available:
tools_v2.2.zip
http://www.mediafire.com/?ey9rmzzyd9x16pr
What’s New in tools_v2.2
– a new version of MobiDeDRM (v18) that now supports V7 Mobi files and can handle 8 digit or 10 digit PIDs. MobiDeDRM is the engine which drives all of the Kindle DRM removal tools. This updates also improves K4MobiDeDRM, Unswindle, and K4Munswindle who all make use of it. (Thanks to Apprentice Alf!)
– added support for using OpenSSL with the eReaderPDB tools which dramatically cuts the decrypt time from 40 seconds down to half a second or less for a typical ebook. FYI: this makes the calibre plugin much faster.
(Thanks to some_updates and DiapDealer)
– added support for OpenSSL on Windows to the standalone versions of IHeartCabbages wonderful ineptepub, ineptkey, ignobleepub, and ignoblekeygen (Thank to DiapDealer and some_updates)
– created a unified TopazExtract.pyw tool that now works for Kindle for PC, Kindle for Mac, Kindle for iPad/iPhone/iPodTouch, and for users who have known fixed PIDs. This tool supports the use of multiple kindle-info files if available. A VERY nice job! (Thanks to DiapDealer!)
– updated K4Munswindle to support both versions of the “1.2.2” release of the Kindle for Mac. This will be the last change for K4Munswindle. It will be removed completely in the next release. K4Munswindle users should migrate to the very latest K4MobiDeDRM and latest Topaz Tools for working with their Kindle for Mac ebooks. (Thanks to some_updates)
– updated K4MobiDeDRM and plugin to allow use of multiple kindle-info files if they are available. Users of the plugin need to simply put the different kindle.info files into Calibre’s configuration directory (like the keyfiles for the epub) to make use of this feature. Also updated to MobiDeDRM v18 (Thanks to DiapDealer! and ApprenticeAlf)
– added an ePub_Fixer tool that handles the issue that many DRM ePubs have of not properly creating the ePub/zip container. Simply select any DRM ePub that is broken like that, and a new ePub (still with DRM) will be created that will allow the Adobe Adept and B&N ePub tools to work properly to remove the DRM (Thanks to some_updates)
– updated ignoblepub plugin for Calibre to automatically fix improperly created epubs and to use OpenSSL on windows if available (Thanks to DiapDealer)
– updated ineptepub plugin for Calibre to automatically fix improperly zipped epubs (Thanks to DiapDealer!)
**Note**
OpenSSL now can be used to speed up Adobe ePub, Barnes & Noble ePub, and eReader conversions, replacing the need for PyCrypto completely. The use of OpenSSL improves the DRM removal speed for both standalone tools and Calibre plugins.
Windows users should install OpenSSL on their system to see these improvements. See the Win_OpenSSL_0.9.8o.txt file for details how to install this library.
Mac OS X and Linux users have this library installed by default on their systems.
*****
**Note**
Calibre plugin users should remember to remove the older MobiDeDRM plugin, K4DeDRM plugin, and K4PCDeDRM plugins to prevent conflicts if installing the new K4MobiDeDRM plugin
*****
November 12, 2010 at 3:09 am
I cannot get the new 2.2 K4MobilDeDRM_plugin)V01.1.zip to work for me. The older Tools 2.1 K4mobdedrm_V01_plugin.zip works fine. If I uninstall the old one and install the new one then the books are not unDRM’d. If I then uninstall the new plug-in and reinstall back the old one it works fine again. Any idea why the new one might not be working here? (I do not have the possible conflicting plug-ins installed). I am running System 10.6.5 and the latest Calibre. What information do you need?
November 12, 2010 at 3:15 am
Yes My Fault!
I broke the K4MobiDeDRM plugin when I rezipped it up!
So that of course leads to an ammended release:
Please grab:
tools_v2.2a.zip
http://www.mediafire.com/?8mzrezba1bvmuqd
It is the same as tools_v2.2 but with a fixed K4MobiDeDRM plugin.
I apologize for any inconvenience!
Thanks for catching that.
November 12, 2010 at 5:19 pm
Yep, 2.2a works fine, thank you, thank you!
November 13, 2010 at 4:12 pm
Hi,
I have now seen more than a few posts on Mobileread and other places that think these tools are for pirating books or posting the books publicly for everyone to grab.
No matter what we write in the comments and how many times we say it, there appears to be a group of people who simply think it is alright to steal.
The following is taken from the k4mobidedrm.py script:
# engine to remove drm from Kindle for Mac and Kindle for PC books
# for personal use for archiving and converting your ebooks
# PLEASE DO NOT PIRATE EBOOKS!
# We want all authors and publishers, and eBook stores to live
# long and prosperous lives but at the same time we just want to
# be able to read OUR books on whatever device we want and to keep
# readable for a long, long time
So I would like to propose that we actively ADD **more** owner, user info in comments to every file we remove the DRM from if it is not already present.
During conversion it is very easy to add/include in non-displayed html, in comments, or in metadata information that identifies the specific user who is running the scripts, and for Kindle for PC/Mac the full e-mail address and account token. Similar things could be done for eReader books, Kindle books, Topaz books (cmbtc scripts add in a whole new record type with this info), Adept epubs, B&N epubs, etc. This simply adds information that many times is stored inside the original ebook in some form or another anyway (inside unique identifiers, in patterns of errors introduced, in unique metadata info, encoded in jpegs or images, etc).
None of this will impact in any way the use of these tools to convert books for another device someone owns, or for archival purposes. It will only impact people who think it is okay to steal and who really don’t have the skills to convert the books without these tools or who are too lazy to search for and find and remove the information. The last thing we need is more script kiddies running around doing more damage than they are already doing.
On the down-side, professional ebook pirates will obviously have the skills to remove this added information but it does force them to develop/maintain their own tools and not simply use ours straight out of the box.
Any thoughts on this? Is this something we should do? If not, why not?
Thanks,
some_updates
November 26, 2010 at 5:53 pm
Since nobody has responded, I’d like to throw in my resounding support for this idea. It is completely in in line with the ideals of most of the community.
November 27, 2010 at 5:59 pm
Hi,
You and I seem to be alone on this issue. I do think putting personal information in into each book as html comments, or pml comments or adding metadata, or even an zero size image with the info inside of it, would be a good deterrent from people simply posting the books they have removed DRM from.
But before I make any changes I would like to see a braoder base of support for this idea.
Thanks,
November 27, 2010 at 6:13 pm
I think this is a great idea. There is no reason for anyone to object, unless they plan to do something shady with the book.
November 27, 2010 at 6:58 pm
It seems to me that those who write the scripts should make the decision about what they add to them. From what i can see, you all really believe strongly, as do most of us, that they should not be used for purposes other than those stated. If those who use them for other purposes don’t like it, well, let them write their own scripts. The rest of us are just fine with anything you add. And, thank you all again for all the work you have done making this possible.
November 13, 2010 at 5:15 pm
I have a couple of Mobi files that absolutely will not get un-DRMed (and yes, they were obtained legitimately, I’m just trying to get them onto my iTouch, which calls for converting through Calibre to ePub). Help, please?
November 13, 2010 at 5:31 pm
Hi Seanette,
A couple of possibilities:
1. Was the source of the mobi ebooks “Kindle for Mac”, “Kindle for PC”, “Kindle for iPad/iPhone/iPod Touch” or something else. Please note, if something else, then removing the DRM is probably not possible as standalone Kindles have changed the way they do DRM.
2. Are the files corrupt? Can you open them in Kindle for Mac, Kindle for PC, or Kindle for iPad/iPhone/iPodTouch? Do they open? Can you read them?
3. Some Mobi ebooks simply do not have DRM on them (even if bought from major ebook stores? To check, please try loading them directly into Calibre and see if you can view the contents? Or try unpacking them with MobiUnpack (part of the tools archive)
4. Some mobi books are actually Topaz books and not true mobi books. To find out simply copy the ebook and then try to open the copy in any text editor (Word, etc). You should see lots of garbage. If you look near the top, if your see the string TPZ0, then these are Topaz books and the only way to convert them is to use the Topaz Tools.
Something here should do the trick.
November 13, 2010 at 5:42 pm
These files were purchased from mobipocket.com several years ago. 🙂
They’re in PRC format, in case that helps.
Calibre reports a decoding error when I add them to the library, and simply will not view them, in both situations reporting a DRM error (I had to go through a logon process to open them in Mobipocket Reader, since I’ve changed computers between purchase and now).
No Topaz reference in the file when opened in WordPad (Notepad flat-out refused to deal with a PRC).
November 13, 2010 at 5:53 pm
Hi,
If they can be viewed in the Mobipocket reader then if you look in the Mobipocket reader software you will find the official reader PID. You can use that PID to unlock those files. It is typically found in the “settings” inside MobiPocket reader. It has been a long time since I have run that software so you will have to hunt around to find it.
Once you know the PID, you can add it to your K4MobiDeDRM plugin for calibre or use the standalone K4MobiDeDRM.pyw program and add that PID to remove the DRM. Take the book directly from where the Mobipocket ereader software stores it to make sure it is a version of the ebook that has properly been encoded with that PID.
November 13, 2010 at 6:21 pm
And it turned out directories were the solution. I kept trying to bring the files in from my downloads directory, didn’t even think about the one Mobi was storing the files in. OK, my brain hit a bad sector there. 🙂
Calibre is now happily converting away. Thanks!
November 15, 2010 at 3:16 pm
Thank you very much for your hard work! I’m very happy that work in continuing on optimizing both amazon (MobiDeDRM) and epub scripts.
I absolutely hate drm, and use these tools to strip it off my purchases before loading the clean mobi files onto my kindle. Thanks very much for helping me control the ownership of my digital media!
November 17, 2010 at 11:52 am
Hi,
got problems using the Calibre plugins from the tools_2.2A to decode an AZW book that I just purchased on Amazon.
I used the command line as suggested somewhere above, and this is the output I got (with some data removed…):
F:\documents>calibredb add Sunstorm-asin_B000FCK2BU-type_EBOK-v_0.azw
K4MobiDeDRM: Calibre configuration directory = C:\Users\Frank\AppData\Roaming\ca
libre
K4MobiDeDRM: Kindle info file kindle.info found in config folder.
DSN: (removed)
Device PID: (removed)
Account Token: (removed)
Pid for Sunstorm: reMMfVgwYC
MOBI header version = 6, length = 228
Extra Data Flags = 2
DSN: (removed)
Device PID: (removed)
Account Token: (removed)
Pid for Sunstorm: reMMfVgwYC
MOBI header version = 6, length = 228
Extra Data Flags = 2
Running file type plugin K4PC, K4Mac, Mobi DeDRM failed with traceback:
Traceback (most recent call last):
File “site-packages\calibre\customize\ui.py”, line 176, in _run_filetype_plugi
ns
File “”, line 562, in run
Exception: K4MobiDeDRM plugin could not decode the file
Backing up metadata
Notifying calibre of the change
F:\documents>
… can you please help me with this ?
thx.
November 17, 2010 at 12:34 pm
ok, sorry for this, I got it working now.
I didn’t find clear instructions on how this all works yet (your blog is hug, I have problems searching in it… 🙂 ).
So: if I understand correctly, my mistake was that I tried to convert the AZW file directly from the Kindle.
I must first transfer it to Kindle for PC, then locate the correct file and add that to Calibre. Right ?
So there’s no way to directly add a protected file to Calibre from the Kindle (over USB conenction) ?
November 17, 2010 at 4:27 pm
Hi Frank G,
Yes, you must download the file to your Kindle for PC (look in the archive if need be) in order for the tools to have any chance of removing the DRM.
Please note, you will get the same error message (or nearly the same) if the book in question is a Topaz Book. Topaz books need a separate more involved set of programs to convert the books. You can open a copy of the file in any text editor (like Word) and if you see the string “TPZ0” near the top, then you have a Topaz book and should use different tools.
You may find it easier to read the Apprentice Alf blog since they regularly trim the comments to keep it easier to follow.
November 17, 2010 at 1:14 pm
The links to Kindle_4_Mac_Unswindle_v1.4.zip don’t work. Is there another download site?
November 17, 2010 at 2:59 pm
stingo, the link to tools_v2.2a.zip still works and it contains the Mac tools 🙂
you may get it from some_updates post on November 12, 2010 at 3:15 am
November 18, 2010 at 4:21 pm
Hi Mac OS X Users,
You may want to check out the latest topic on the Apprentice Alf Blog!
http://apprenticealf.wordpress.com/
Alf just announced a DeDRM AppleScript Droplet, that combines B&N eReader, B&N epub, Adept EPub, Kindle for devices with fixed PID, and Kindle for Mac DRM removal all in one simple drag and drop tool.
No need to install anything. Simply drag and drop your ebooks (or even folders of ebooks) onto the droplet and up pops a non-drm version of your ebook! Even works with DropBook if you have it installed. I don’t think ebook DRM removal could get any easier for Mac OS X ebook owners.
I have now installed it on my own desktop.
Perhaps if someone out there has some Windows developer experience, a similar drag and drop tool could be built (maybe even include ActiveState Python 2.6 and OpenSSL in the drag and drop application itself) to make install and use completely painless?
If anyone out there is familiar enough with scripting done on Windows and would like to take a shot at a similar drag and drop tool, please let us know here so interested people can get involved.
Thanks,
November 18, 2010 at 8:03 pm
If this drag/drop DRM remover does get written for Windows, please have nothing to do with ActiveState Python. if it needs a Python install, please make it use Python 2.6.x from Phyton.org. Most people have that one installed. We don’t need to have two Python installs. Plus, I do feel that Python from Python.org works better and is what most people use when writing Python code.
November 18, 2010 at 8:36 pm
Hi JSWolf,
Based on what data?
Ever since the Dark Reverser and IHeartCabbages wrote and released the original scripts, the recommended Python for Windows has been the free version of ActiveState’s Python 2.X (32 bit).
The problem has been that the official Python for Windows (from python.org) installer is a very minimal install missing many of the most popular external libraries, sometimes even missing tK support and/or pythonw. That was the original reason people have been recommending Active State Python for Windows for the last couple of years.
I have never hear of the Phyton release? Are you sure it is what most Windows users have installed?
Even the official python.org website has a link to ActiveState’s ActivePython whereas I see no link to Phyton’s release.
You can check out:
http://www.python.org/download/
to see for yourself.
Some objective data on the number of Windows users with ActiveState vs Official Python.org vs Phyton.org would be interesting to see.
November 18, 2010 at 9:51 pm
Not to mention that the ActiveState Python release for Windows more closely resembles the release of Python that already comes pre-installed on OS X and Linux.
But basically… the Python interpreter is fairly universal. The only difference being what libraries are included with any particular release.
November 18, 2010 at 11:48 pm
Phyton is a typo for Python. There was no way to go back after posting to fix the error. I was thinking it won’t matter as people will understand it.
Anyway, most advice I’ve seen for Windows is to use Python from Python.org. I’ve not seen anyone stating to use ActiveState Python. Doesn’t really matter to me that ActiveState is closer to an OSX or Linux Python.
The thing is, if you make sure the code runs on Python.org’s Python, it will run on ActiveState. The reverse is not true. So really, bundling ActiveState will eventually make it so we end up with incompatible scripts. This is not what we want at all.
November 19, 2010 at 12:55 am
Hi,
Yes, I didn’t catch the typo. Either way, I have already had a number of reports of straight Python.org 2.7 installs on Windows not working with the tools we have right now because of missing pieces that almost every other version of Python has. That alone is enough to move to a distribution that support python windows specific extensions better since many of the scripts require Windows extensions to collect the data the needed to remove the DRM.
Here is what the official Python.org wiki.python.org has to say about ActiveState ActivePython:
[QUOTE]
ActivePython
The ActivePython Python distribution is a distribution focusing on an easy install and use. It provides builds for a number of platforms: Linux, Windows, Mac OS X, Solaris, HP-UX and AIX. On Windows, ActivePython also includes Win32All (a.k.a. pywin32).
You can find this distribution in:
http://www.activestate.com/activepython.
…
ActiveState is one of the most well known and one of the longest running suppliers of Python distributions, tools, and support.
[/QUOTE]
So it appears python.org seems to think this is a very good distribution as well.
And if people only want one installation of python as you say (even though calibre comes with its own python installation – so maybe 2 ;-), you would think that Windows users would want the one that is most complete. Please note, incomplete is not the same as incompatible. Both pythons are compatible, the user just has to add in more things with the standard install to get a more complete installation.
Either way the point is moot until some Window’s developer steps forward to create the drag and drop app.
November 19, 2010 at 2:13 am
“Doesn’t really matter to me that ActiveState is closer to an OSX or Linux Python.”
Fair enough, but it does matter to the developers of these tools in order to maintain cross-platform functionality. I’m not sure if any of these tools could be made to work with the stock Windows python distro from python.org (without downloading and installing the necessary missing modules, that is)… and even if they could, it would mean maintaining two different versions of each script (one for Windows and one for OS X and Linux). Something I know I’m not interested in doing.
November 19, 2010 at 10:37 pm
Mobipocket Blues:
Some recent downloads of this format *seem* to decrypt, but…
The DRM-infested download displays fine in MobiPocket desktop reader. Both MobiDeDRM_0.18.py and K4MobiDeDRM.pyw produce files with small blocks of characters missing or corrupted.
November 20, 2010 at 9:52 am
As some_updates mentions, this sounds like a problem with file flags. Posting more details over at my blog on this post http://apprenticealf.wordpress.com/2010/09/23/mobipocketkindle-request-for-help/#comments would be really useful. (And I can probably sort out a fix for the files that are causing you problems.)
— Alf.
November 19, 2010 at 10:40 pm
One further note. both programs mentioned came from the 2.2a toolset.
November 20, 2010 at 1:17 am
Hi,
When it decodes what flags and version are the files? This info is printed when the drm removal just starts. Look in the conversion log window.
November 20, 2010 at 1:21 am
Hi,
See this link for more info
http://apprenticealf.wordpress.com/2010/09/23/mobipocketkindle-request-for-help/#comments
November 23, 2010 at 3:08 am
Hi, I have a real problem with some mobipocket files that I purchased from Books on Board around a year ago. I purchased over 50 legitimate books, all DRM’d and downloaded them onto 3 devices – my laptop, Bebook One and my PC at work. I now have an ipad and wanted to strip the DRM so I could convert them to epub and read through iBooks.
I have changed jobs since and do not have access to my work computer anymore and, as I had since deleted the device in my Books on Board account, I cannot retreive the PID for my old work PC. I then got to thinking that I would re-download all the books without the PID embedded or whatever for my old work computer and that way when I ran the Mobidedrm plug in for calibre I would not need to enter the PID for that PC in the ‘customize’ window.
All went well until I tried to download 13 of my more than 50 books and, since I purchased them, BooksonBoard has changed the geographical rights to download these books. I live in New Zealand and apparently these 13 books are not allowed to be downloaded where I live. I have contacted BoB who just told me it was the publishers policy which is a lot of s*$#t because I could go to kindle, kobo or fictionwise and purchase these books again, in the same format, released by the same publisher (most of them are doubleday). What I think the problem really is, that the online retailers are protective of their rights to sell in certain areas and I think BoB lost the argument so to speak.
Anyway, I’m now stuck because the .mobi files I have of these books were downloaded with the PID of my old computer which I cannot access anymore and so mobidedrm does not remove all the DRM. Does anyone have any suggestions short of spending another $164 to get another copy of these books?
Thanks 🙂
November 23, 2010 at 3:25 am
Hi,
Without the PID used to buy the books originally, you are out of luck. You could try calling BoB and asking what PIDs were previously registered to your account.
Short of that you could write python code to generate and try all possible pids using capital letters and digits for the first 7 chars and $ for the 8th folloed by the 2 checsum chars and run it continuously to see if you can brute force things (assume 26 letters and 10 numbers that would only be 36 raised to the seventh power different possible PIDs.
Now they use small and caps and digits and all 8 positions so brute forcing will no longer work with new style PIDs.
November 24, 2010 at 3:14 am
Hi,
WOW!! I am so new to all this code stuff….actually I’m an accountant and I thought that it was an absolute accomplishment when I managed to get the mobidedrm plug in customized and working on calibre with some of my recent downloaded mobi files….to me that was like I had won the lottery for a while….I almost couldn’t believe that it had worked….I hadn’t even heard of Python until a couple of weeks ago and I still have no idea what it actually is or what it does….thanks heaps for taking the time to suggest that though, it is so very much appreciated that you thought up something to help…. 🙂
November 23, 2010 at 9:00 am
You could try using a US based proxy server to download your books.
Failing that, upload one of your encrypted books somewhere and post a link here. I have some brute force code that might work to recover your old PID, although it takes about a week of solid computation.
— Alf.
November 24, 2010 at 3:20 am
Hi there,
Thanks for suggesting that I use a US based proxy. Our work’s computer guy was in my office just now and I told him about this and he says that he uses a couple of different US based proxy servers and he’ll help me try it out…..I am going to write to BoB (again!!) and ask if they can give me the PID history on my account first…..to me that is the easiest option so I’ll give that one a go first….wow thank you so much for offering to run it through your code….that is so kind of you to offer help like that to a total stranger….kind offers like that do not happen very often and I am truly grateful for your suggestions and offer of help. 🙂
November 23, 2010 at 1:54 pm
Or you could just go to your Account on BooksOnBoard.com and then find your registered PIDs by going to the Mobipocket Device Controls. There you can see all previously registered PIDs.
At least if you still have your login info.
November 24, 2010 at 3:24 am
Hi Joerg Mosthaf,
Great minds think alike….that’s what I thought of too and I tried that idea first, but I had been far too efficient (or so I thought). I logged into my BoB account and deleted the PID for my old work computer shortly after I left there…..I just did not realise how important the PID is….I do now of course!!!!
I am going to write to BoB though and ask if they can give me the PID history on my account thought. I’ll let you all know how I get on with this problem…..:-)
November 24, 2010 at 3:34 pm
Hi All,
😉 BTW … IANAL
Any thoughts on adding an EULA to the tools scripts?
Perhaps:
By downloading or using these tools the user agrees to all of the following:
1. these tools are for his/her personal use only
2. that they will ONLY use these tools to convert between ebook formats and reading devices, for enabling specific features not otherwise possible (such as read aloud), and for archiving the ebooks to prevent loss when the ebook vendor ceases either support or operations.
3. that they agree to NEVER use these tools to steal or pirate ebooks in any way, shape, or form.
4. that no ebook converted by these tools will be posted to publicly available websites or shared via peer-to-peer networks for the purposes of illegally sharing the book.
It is an idea … I wonder if an EULA of this sort could reduce or limit liability of the authors/distributors of such tools in countries where there are laws against their creation and sharing.
Probably not worth the hassle since people here never do that anyway and because no one ever reads the EULAs but something like this (but obviously written by someone with some expertise in the matter) might be useful?
November 24, 2010 at 3:47 pm
Distributing de-drm software is either illegal or not. If illegal, it’s either a civil matter or a criminal matter.
Neither of those questions will be affected by a EULA.
In the US is seems that distributing de-drm software without personal gain is a civil matter.
I’ve forgotten if it’s even a civil matter anywhere else. I sort-of think not, but IANAL and people shouldn’t rely on my vague statements, but make their own decisions.
The only prosecution I can think of is the Sklyarov/Elcomsoft case. (And they were found not guilty.)
November 27, 2010 at 8:26 pm
I don’t think that a EULA would do any good at all. No matter how you look at it, the DMA is a Draconian measure that makes deDRM scripts illegal to program (for a US citizen) and illegal to link to (from a US server) and makes the use of deDRM scripts to be questionable for a US citizen. This, of course, happening in the Land of the Free. If it comes down someday to a court fight by a Publisher conglomerate against some poor user the EULA will do no more (or less) for him or her than a simple under-oath statement and a good attorney. Think of an unlicensed handgun that comes with a note saying that you will not loan it to anyone, that you will only use the firearm tool for target practice, and that you agree never to rob or shoot someone with it. I also think that the EULA could be misconstrued by a prosecuting attorney as being a sort of built-in help file indicating illegal uses the software COULD be used for (i.e. a don’t throw me in the briar patch kind of argument.) Anyway, I hate to be negative but I don’t see an upside to a EULA. It seems to me it might be better to include with every script a petition that can be printed out, signed and mailed to your Senator or Congressman asking them repeal the unfair, impossible to implement, and unconstitutional DRM Act itself. This is only going to last another two years or so and then when millions and millions of people suddenly discover they cannot transfer their $500 ebook library to some breakthrough ereader then they will decide to vote the rascals out. We need to encourage the rascals now to change the awful Laws they made or realize they WILL be voted out at some point if this continues. Just some thoughts.
November 28, 2010 at 1:10 am
Adding metadata or a file inside the eBook is not really a good idea. Most eBook formats would have to be decompiled and then the new information added and the eBook recompiled. There;s a good chance that something could get screwed up. Also, ePub is trivial to remove anything put in. Really, it’s not a worthy effort for most eBook formats these days. And the only format t might be is too easy to fix.
November 28, 2010 at 1:46 am
Hi,
I have been looking more into this. It actually seems quite easy to do. There are many python scripts to change bits inside images to encode data (inserting the User’s Name and/or Mac Address, or …) that can be put into any available image file and not be seen or detected, or in the thumbnail image of the book or into any added jpeg, gif or png spacer file, or binary encoded into a metadata element, or simply added as a book producer metadata field, or in comment strings in pml or html, or in sub-fields of the unique-id or ….
Detection and removal of these would not be easy for most non-programmers and they would not in any way interfere with the book itself. In fact, the best way might be to use something like the evercookie approach and put it in many places both obvious and non-obvious and let the user worry about finding them all.
BTW: I have seen/found code that creates unique typos or replaces spaces with different “space” characters that allow any version of an ebook to be tracked down to its original source. I assume if I have found these, that someone must be using them. My guess is some book publishers are already doing these or other tricks. Perhaps that explains why we see all of these silly typos in even new release ebooks!
So I do not think adding the decoding user’s info to the book need impact the ebook negatively in any way and is really quite easy to do with pml, epub, topaz, and mobi books and I think it can be done in a way that even experienced ebook types will never find all of the occurrences.
I am searching the web now for more way to encode information into binary data files without impacting them (the whole field of steganography is very much related to what we are thinking about). Luckily many of these tools are in pure python or can easily be ported to python.
So I don’t think ease of detection, or impact on ebook is in any way a good argument if we do things right.
November 28, 2010 at 6:27 am
Interesting email discussion I have been having with the customer support team (now there’s an oxymoron….no support been even attempted by them so far) at BooksOnBoard.com. Seems the whole thing is related to sales taxes believe it or not.
Apparently it goes like this:
“The publishers restrictions are due mainly to sales tax issues. for example, UK based retailers are required to pay a 17% VAT tax on all eBook downloads. This of course puts them at a disadvantage as no one in the UK will buy their books if they can order them for 17% less abroad. Additionally, in the case of the publishers, for example, there is a Random House UK and a Random House US, the UK branch of that publisher wishes to see all sales within the UK credited to their side of the company. Agency publishers are requiring all retailers not only to pay the sales taxes on any files sold within their own taxing nexus, they are preventing them selling them internationally. ”
So there you have it. Of course I’m waiting for their reply when I pointed out that my taxes were paid already therefore there should be no restriction on the books I have already purchased.
Another interesting issue is:
“Distributors, in an effort to remain in business, have had to agree to the publishers new pricing and territory schemes, and now prevent the download of titles from their servers based on geography. We agree that what has occurred does not seem legal, and know of at least two US states where lawsuits are being prepared against this new system”
So court cases are looming. Seems that publishers have locked their books up so tight they are now courting law suits. Watch this space i’d say.
November 30, 2010 at 1:55 am
I apologise for troubling you as I am an IT novice and didn’t know where to send this query. Please could you tell me where I can learn how to deal with (amazon) Topaz files. I don’t klnow whether the file has another persons Kindle PID in it. Of course I can find my own, but cannot read the file on my kindle DX & want to fix it.
Thank you
November 30, 2010 at 2:51 am
Hi,
Try reading http://apprenticealf.wordpress.com for info on Topaz Books. There is no fixed PID for Topaz Books, so you will have to install either Kindle for PC or Kindle for Mac and download your copy of that Topaz book so that the software has some chance of removing the DRM.
More can be found on the ApprenticeAlf Blog.
December 3, 2010 at 3:55 pm
Hi,
There is a new version of the “tools” archive available today:
tools_2.3.zip
http://www.mediafire.com/?2ibexwv6a7d5saw
Changes in tools version 2.3
———————-
– DeDRM 1.2.zip now included. This is a Mac OS X specific drag-and-drop applet that runs the tools for you. New in this version is support for PDF files (separate download needed). Mac OS X users no longer need to worry about python or scripts at all. (Thank you Alf!) If anyone wants to take a shot at a similar thing for Windows, please let us know, we would love to include it in a future release.
– K4MobiDeDRM bug fixed that prevented removal of DRM used with some older Mobi files – those without exth sections (Thank you Apprentice Alf and DiapDealer!)
– new version of MobiDeDRM (version 19) is now included that fixes some issues with trailing data in Version 6 type mobi files with specific flags. MobiDeDRM is the engine which drives all Kindle/Mobi drm removal and is part of many of the tools (Thanks Alf!)
– a new version of the K4MobiDeDRM Calibre plugin v01.3 that includes the bug fixes and new mobidedrm code (Thank you DiapDealer).
– all tools have been modified to work with python 2.5 exception handing code. If anyone finds other Python 2.6 specific features, let us know and we can take a stab at removing them as well. The code here should hopefully work with Python 2.5, Python 2.6, and Python 2.7 (Thank you Alf, DiapDealer, and some_updates!)
December 5, 2010 at 8:13 am
I had to switch computers and am now having difficulties with the kindle for pc tools working. I can get the B&N and adobe tools to work but not the mobi kindle for PC tools, this is both through tools 2.3 and through the calibre plugins.
New computer – windows 7 64 bit, running python 2.6.6 32bit. Kindle for PC is version 1.3.0 (30884). Using the K4PC archives, I pulled several files onto the new computer. Using those new files that are specific to the new K4PC, I tried K4MobiDeDrm but was not able to un-encrypt any files. I even tried LZSkindle which had always worked previously on the older computer. No better results.
Conversion log below is from a file that I know is not a topaz file as I un-encrypted it successfully with an earlier version of tools & K4PC on the old vista machine. This file used below in the log was downloaded through the K4PC archives onto the new machine. I tried multiple other files but had the same results.
Conversion Log
Command = “python k4mobidedrm.py”
K4PC, K4M or Mobi Path = “C:\Users\Elizabeth\Desktop\B003U4VASO_EBOK.azw”
Output File = “C:\tools_v2.3\Kindle_Mobi_Tools\K4_Mobi_DeDRM_Combined_Tool\B003U4VASO_EBOK_nodrm.mobi”
Kindle.info file = “”
PID list = “”
Please Wait …
K4MobiDeDrm v1.2 provided by the work of many including DiapDealer, SomeUpdates, IHeartCabbages, CMBDTC, Skindle, DarkReverser, ApprenticeAlf, etc .
DSN: tt7JnJtrSx5S6ztMtfS87zrvPkSkPh7C5MPdrJ7Y
Device PID: I6LYSL576L
Account Token: 9b9a30534e2f785731dfe64a28ff2d92374596ef,18401b32cb58e58327dc01c05e268fcbd58d5dce,9310bff92f3862ef380b7f1b646153f5e9bb0b20,3f1f17ef12e6c4d47a995c74f32d8c7c486372ec,7bc83baa7ddd9b339ae9dc1c5c25592e4be48f15,7990bace81a597c492732d4e262c511ea7bdebb6,51eb93e4cbe897db9b0d80232174d2e13709afb9,e4b5fa033a463565e25a19b3f472da5d75c21e9d,7d335ba5b222861aa0bf268b581d1a2deab18309,2cc88326107547c3ad56f6bcad9155982fb9b3a3
Pid for Dark Embers: A Dragon’s Heat Novel: Qhs0exrTDH
MOBI header version = 6, length = 232
Extra Data Flags = 2
Trying: “”
Error trying default K4 info: no key found. maybe the PID is incorrect
Error: Could Not Unencrypt Book
Error: Encryption Removal Failed
I am appreciative of any help you can give. Hopefully my problem is something obvious that I’m forgetting and not related to the version of K4PC that I had to load onto the new computer.
Thanks to all of you for all your work with these scripts. I would be crying if I was locked into 1 file format and 1 machine.
Thanks again, Beth
P.S. Since any resulting files are for my use on my machines, I would have no problem with some identifying feature being added to files when un-encrypted.
December 5, 2010 at 2:06 pm
It certainly sounds like you have your ducks in a row. 😉
I see that the ebook is located on your Desktop… can you verify that it originated from the “My Documents\My Kindle Content” folder on THAT computer? Perhaps you have multiple versions of that ebook that are being switched. You can select the file (with K4MobiDeDRM) directly from “My Documents\My Kindle Content” to be 100% sure.
Also, what flavor of Python are you running? ActiveState is the recommended distribution for these tools.
I use the same version of K4PC, so we can eliminate that, but I don’t have Windows 7 to test with.
December 5, 2010 at 3:07 pm
Using ActiveState Python on both machines.
I just tried several books straight from “my kindle content” with the same results. See below.
Later today I’ll pull from archives a file that hasn’t been “read” by the “original” K4PC. I’ll also go back to the original K4PC on the vista machine with the same tools and see what happens. I thought that each K4PC created PIDs dependent upon pulling the book from their archive but this is the first time I’ve had to set up another K4PC so I haven’t confirmed that yet. Has anyone else had to add K4PC to another machine yet?
Thanks for confirming that my process is correct. Any other thoughts are welcome.
Conversion Log
Command = “python k4mobidedrm.py”
K4PC, K4M or Mobi Path = “C:\Users\Elizabeth\Documents\My Kindle Content\B002T18VFY_EBOK.azw”
Output File = “C:\tools_v2.3\Kindle_Mobi_Tools\K4_Mobi_DeDRM_Combined_Tool\B002T18VFY_EBOK_nodrm.mobi”
Kindle.info file = “”
PID list = “”
Please Wait …
K4MobiDeDrm v1.2 provided by the work of many including DiapDealer, SomeUpdates, IHeartCabbages, CMBDTC, Skindle, DarkReverser, ApprenticeAlf, etc .
DSN: tt7JnJtrSx5S6ztMtfS87zrvPkSkPh7C5MPdrJ7Y
Device PID: I6LYSL576L
Account Token: 9b9a30534e2f785731dfe64a28ff2d92374596ef,18401b32cb58e58327dc01c05e268fcbd58d5dce,9310bff92f3862ef380b7f1b646153f5e9bb0b20,3f1f17ef12e6c4d47a995c74f32d8c7c486372ec,7bc83baa7ddd9b339ae9dc1c5c25592e4be48f15,7990bace81a597c492732d4e262c511ea7bdebb6,51eb93e4cbe897db9b0d80232174d2e13709afb9,e4b5fa033a463565e25a19b3f472da5d75c21e9d,7d335ba5b222861aa0bf268b581d1a2deab18309,2cc88326107547c3ad56f6bcad9155982fb9b3a3
Pid for What A Dragon Should Know: XKz1pVkcL5
MOBI header version = 6, length = 232
Extra Data Flags = 2
Trying: “”
Error trying default K4 info: no key found. maybe the PID is incorrect
Error: Could Not Unencrypt Book
Error: Encryption Removal Failed
December 5, 2010 at 4:49 pm
ElizabethN said: “I thought that each K4PC created PIDs dependent upon pulling the book from their archive but this is the first time I’ve had to set up another K4PC so I haven’t confirmed that yet.”
This is correct. The same ebook will have a different PID for each K4PC installation that it’s downloaded to.
It’s probably a foregone conclusion, but have you verified (on the new computer) that K4PC can open the ebook(s) with no issues?
December 5, 2010 at 4:52 pm
Hi,
Please note that the CryptUnprotect data used in deciphering the kindle info file is machine specific. Worse yet, Amazon uses information from the hard drive serial number so it is hard drive specific (if your hard-drive ever dies and you replace it, you will have to redo everything).
On your new machine:
1. please verify that you can open and read that exact book (perhaps, it only updates books to the new version when they are actually opened).
2. then try with K4MobiDeDRM with that exact book you just opened and flipped through on the K for PC app on the new machine
3. If that fails, please try one other thing. In K 4 PC, please delete that same book, and then download a new copy from your archive if need be, and then try using K4MobiDeDRM.
Please let us know what you find out.
Also, one other thing to check, could your new computer have more than 1 hard disk drive in it? I had a very similar bug report to yours but it turned out the machine actually hard two hard drives in it and K4MobiDeDRM got the serial number from the wrong one!
Thanks,
December 5, 2010 at 8:59 pm
On your new machine:
1. please verify that you can open and read that exact book (perhaps, it only updates books to the new version when they are actually opened). – done, opens just fine on the new K4PC.
2. then try with K4MobiDeDRM with that exact book you just opened and flipped through on the K for PC app on the new machine – done, still getting unable to unencrypt errors.
3. If that fails, please try one other thing. In K 4 PC, please delete that same book, and then download a new copy from your archive if need be, and then try using K4MobiDeDRM. – done, no change, still getting unencrypt errors
4)Also, one other thing to check, could your new computer have more than 1 hard disk drive in it? – done, only 1 hard drive in this machine.
December 5, 2010 at 9:12 pm
Hi Elizabeth,
Okay so the problem must be related to the two kindle info files.
Pleas keep the one that was modified/accessed most recently ( I think it was the one without the extra numbers in the name) and then remae and move the other kindle info file to someplace safe and out of the way.
Then repeat the set of experiments you just ran and report back.
I am almost positive the drm removal code is finding the wrong file.
So let’s force it to use the same one actual K4PC is using.
December 5, 2010 at 9:12 pm
Ok, K4MobiDeDRM (from tools 2.3) still works on the old machine and decrypts K4PC files. Old machine is vista 32bit with activeState python 2.7. Same version of K4PC as new machine.
I’m going to finish decrypting files with the old machine then I’m going to deregister the vista K4PC to see if that will force the win7 K4PC to become the default. Should I deregister the new version as well then just activate only the win7 K4PC? Or is this related to the 2 kindle.info files (see earlier post)?
Thanks again.
December 5, 2010 at 9:21 pm
The de-registering of K4PC on your old PC will have no effect. You can have a number of registered K4PC installations with no problems.
December 5, 2010 at 9:32 pm
The de-registering of K4PC on your old PC will have no effect. You can have a number of registered K4PC installations with no problems. — that’s good to know. I won’t waste time de-registering etc.
Will try this suggestion by SomeUpdates first, regarding the 2 kindle.info files.
Please keep the one that was modified/accessed most recently ( I think it was the one without the extra numbers in the name) and then remae and move the other kindle info file to someplace safe and out of the way.
Then repeat the set of experiments you just ran and report back.
If still no response, then trying DiapDealer’s suggestion of deleting all books, deleting both kindle.info files and re-registering the win7 k4PC.
I will try to keep new comments at the “end” of the blog to keep things less confusing. I normally read through all the comments chronologically in the rss feeder & via email.
December 5, 2010 at 4:21 pm
Just as a long shot, check that your K4PC is logged in – when you open it, does it ask you for your email/password to your Amazon account? If it doesn’t – if it already has your account saved – try manually specifying the kindle.info file – use the … box next to Kindle.info file (Optional) and navigate to Users/Elizabeth/AppData/Local/Amazon/Kindle for PC/(longrandomalphanumeric) and select the kindle.info file there – it should be around 11k.
December 5, 2010 at 8:52 pm
“If it doesn’t – if it already has your account saved – try manually specifying the kindle.info file.”
The new K4PC does automatically log in when I open the program. I went to the kindle.info file but am confused as I have 2 files in the alphanumeric folder. One is an info file and the other is a 145235457 file, i.e. kindle.info.20101129.145235457 this file was created and last modified when I set up K4PC.
Google could not tell me anything about this type of file. does the file perhaps tell amazon that this is a second K4PC? Should I try deleting it and see what happens?
the kindle.info file has the most current date and time. My old vista machine only has one kindle.info file.
December 5, 2010 at 10:33 pm
In my experience multiple kindle.info files, with the extras being kindle.info.date.number results from having registered/de-registered more than one Amazon account on a single installation of K4PC. Looking at my own installation (I alternate between three Amazon accounts), the kindle.info file you want should be the larger of the two. Have you tried manually specifying both files? (ie. specify kindle.info, if that doesn’t work, specify kindle.info.20101129.145etc.)? If that doesn’t work, another vote for delete both and re-register K4PC.
December 5, 2010 at 9:07 pm
To be honest, I would probably delete all books from K4PC, delete both of those kindle.info files and try re-registering K4PC. Then download a book from your archives and try again. That other file may be a back-up of some kind from a previous registration. Not sure, but definitely irregular.
A quick thought: you did actually INSTALL K4PC on this machine, right? What I mean is… you didn’t just move your K4PC installation from the old PC to the new one did you?
December 5, 2010 at 9:22 pm
What I mean is… you didn’t just move your K4PC installation from the old PC to the new one did you? – No, I went to Amazon and downloaded K4PC from their site.
Trying SomeUpdates suggestion of temp removing the irregular file and repeating the previous experiments.
December 5, 2010 at 9:37 pm
My posts are overlapping with some_updates’s at times. By all means, stick to one suggestion at a time. Too many cooks can sometimes spoil the soup. 😉
December 5, 2010 at 10:40 pm
Success!
I suspect that my K4PC was corrupted somehow, maybe during installation.
To get tools to work with K4PC, I did the following
1)delete all books from K4PC
2)de-registered K4PC, which would have deleted all of the books as part of the de-registering process
3)closed K4PC
4)deleted any files remaining in my kindle content
5)deleted kindle.info files
6)opened K4PC
7)registered K4PC
After registering, I moved some books from archives, validated that they opened in K4PC then ran K4Mobidedrm. Files decrypted with no problem. Also ran calibre plug-in and decrypted with no problem.
I also tried all of the above without de-registering but had no success.
A big thanks to everyone, especially DiapDealer, Some_Updates and Pardoz, for their help. I know that I would probably not have figured this out on my own.
Keep up the great work. Thanks again.
December 5, 2010 at 11:08 pm
I would guess that (for whatever reason) K4PC was using the funky kindle.info.whatever file to open the books, but the script was attempting to use the regular kindle.info file… and failing.
You might have been able to force K4MobiDeDRM to use the other file with success, but it’s probably better to start from a clean, standard K4PC install, anyway.
Glad it’s all working now!
December 7, 2010 at 1:17 am
The new Google Bookstore — anyone know how to dedrm the books? I was able to find (for my Mac) the Google Book Downloader app. When I buy a book on Google and enter the book’s bought ID it does download the entire ebook as a pdf (as opposed to not buying a book and using the unbought book ID which downloads every 5th page). So I wind up with an encrypted pdf file. I thought I could use the ineptpdf script from Alf’s site but that does not work. It says that the book has been undrm’d and saves it with that in the title. However, it is still DRM’d. I suppose this is because Google is not using the Adobe DRM scheme but one of their own? Any ideas?
December 7, 2010 at 1:36 am
According to the Google ebook store, it will download as a pdf or epub that is supported by ADE readers such as the Nook (B&N epub format is now actually a supported Adobe epub format) as well as the Sony Reader.
So as long as you download to one of those devices, you should be getting an Adept epub or pdf. I am not sure what a Google Book Downloader.app does?
If you do get an .acsm file, you should load it on your Adobe Digital Editions, and it should download and encode a book that can be decoded by Adobe tools.
This is all according to their website. Being from Canada, I am not allowed to purchase anything on the Google ebook store.
December 7, 2010 at 2:11 am
Wow, apologies for a rather stupid message! I am very, very unfamiliar with Google Books. Turns out that if you just click on the book covers as I was doing in the My Library view you only get a Read Now choice, no download to device. Double-clicking takes you to the Book View page that does feature that download to device choice where you can get an easily dedrm’d ePUB file! Not knowing that about downloading to devices led me on a merry chase after some older apps. Anyway, all is well now and I have the book I bought on the device I want and th ereader I like best. Thanks!
December 8, 2010 at 12:40 am
Hey 🙂
I use toolsv2.3, python 2.7, kindle4PC 1.0.1.
The output of TopazExtract.pyw – if I define the output folder with the trailing “\” is:
Kindle.info file = “”
First 8 chars of PID = “”
Please Wait …
Files successfully extracted
and the output folder gets created. But it is empty.
If I run TopazExtrac.pyw omitting the trailing “\” for output folder, I get the following error:
Kindle.info file = “”
First 8 chars of PID = “”
Please Wait …
Traceback (most recent call last):
File “lib\cmbtc_dump.py”, line 691, in
sys.exit(main())
File “lib\cmbtc_dump.py”, line 659, in main
dkey = getBookPayloadRecord(‘dkey’, 0)
File “lib\cmbtc_dump.py”, line 316, in getBookPayloadRecord
raise CMBDTCFatal(“Parse Error : Invalid Record, record not found”)
__main__.CMBDTCFatal: Parse Error : Invalid Record, record not found
DSN: 5b79606G7z6dS4P3Ph6rrMnttE7Cr85kn3tMtxrM
Device PID: xxx (8 digits)
Account Token: xxxx (looooong number)
Book PID: xxx (8dgits)
Error: File Extraction Failed
Some_Update suggested in September to open the prc-file in notepad and look for “topaz” or so near the top of the file. I checked that: there is no string similar to that. Only squares with an occasional random character.
What can I do?
Thanks heaps!
December 8, 2010 at 12:58 am
Hi Silver,
> open the prc-file in notepad and look for “topaz” or so near the top of the file. I checked that: there is no string similar to that. Only squares with an occasional random character.
Not “topaz” but “TPZ0”. If you open the file in Word (or some other editor) and do NOT see “TPZ0” near the top, then the file is NOT a Topaz file.
If it is NOT Topaz, then none of the TOPAZ_TOOLS are correct to use with it.
Simply run the K4MobiDeDRM.pyw script to deal with regular mobi (non-Topaz) files. The Topaz Tools are only needed and only work for Topaz Files.
December 8, 2010 at 10:21 am
Thank you for your help, Some_Updates!
The output of is :
Kindle.info file = “”
PID list = “”
Please Wait …
K4MobiDeDrm v1.2 provided by the work of many including DiapDealer, SomeUpdates, IHeartCabbages, CMBDTC, Skindle, DarkReverser, ApprenticeAlf, etc .
Trying: “”
Error trying default K4 info: invalid file format
Error: Could Not Unencrypt Book
Error: Encryption Removal Failed
For some reason I don’t have a kindle.info file on my XP32-SP2.
Can I try something else? What am I doing wrong…?
Thank you again for your help!
December 8, 2010 at 10:42 am
I forgot: TPZ0 is not found in the prc. These are the first lines I see:
偔ず挆搄捩Ŵ踁姊挀朆祬桰艳踄惊ឨ踀翲⢮輀⾡檡輀⇃箮輀⓲撨退ႛ岭退瓈䦮退䗷鄀垠⊪鄀Nj䶨鄀図熨鈀侜䊯鈀冮鈀狺ҩ錀纣㮮錀䇒Ꭹ錀峻熭鐀喩ᾪ鐀糓悯销撃ኯ销纲梭销滠暮阀岏皩阀媹㖮阀៨犭需ᆖ⾮需䣄㚯需۴㮭頀䦡▭頀盎㊯頀ヾ綮餀㖭ʭ餀㿚䦬騀ႇ岮騀璵ⶮ騀⧤₮鬀冒ޮ鬀惀檭鬀勮㾬鰀ᦛ岧鰀緂皭鰀篰䲭鴀侞■鴀精ધ鴀䢮鸀应憯鸀䟄⚯鸀痳⦨鼀⚜綧鼀⯄ᒯ鼀䟳妮ꀀ⢢䢮ꀀ磐㚭ꀀ㛾⮯ꄀ榭➮ꄀᣜ㲩ꈀ岅綮ꈀ憴ᶨꈀયꌀᢌᶩꌀ㶵亭ꌀᏣ䲭ꐀ析撮ꐀ县侮ꐀ⫮மꔀ㶜⺮ꔀ珊殩ꔀ更ꘀޤযꘀᣓনꘀ⧻㚮꜀枩㺮꜀ⷘኯꠀ䞇⒭ꠀ玴㊮ꠀⷣҭ꤀㦐咮꤀ᖿᾭ꤀㳬梭ꨀⲚ憫ꨀᗆᾮꨀ㳴垮ᮣᆯ㓒ᾮ가宀抭가䖮璬가䇛榭관㊉劭관ಷ⒩관㣠妮글ᦏ嚨글瞷碮글矦㺭꼀㶔ޯ꼀䳃綮꼀凲䮩뀀⒜⾮뀀寊岦뀀㿱亮넀ᖠ㊮넀俎Ⴏ넀柽沮눀宬嶭눀䃚ᶨ대斂⾯대Ჲ䮮대濠ᖯ됀ಐ嚮됀檾⊮됀ᓭ粭딀ᢛ冬딀燇檮딀揶檮똀喥⦯똀ەʧ똀ჼ徨뜀瞤纮뜀緓窮렀羂렀₲䶮렀痠溧뤀殈窭뤀涶䮭뤀䃤䖬먀එ箭먀Ⴟ䂩먀壨⢯묀࢘묀⧆䪯묀糵ឯ밀Ქኩ밀㟎ᮮ밀导斮봀䦫ギ봀˚ᶭ븀⢇㮭븀沴ᖬ븀ૡ劭뼀斎⚨뼀ᒷ䦨뼀曟ྮ쀀纍Ⲯ쀀㎼庯쀀⒯섀䞛䢮섀ᣊ瞭섀ᶩ숀㺡§숀䟈ᒮ숀擶㞮쌀⒥⒮쌀凓窮쐀咂䚭쐀⎰碮쐀ⓟ岮씀䂧씀劵断씀䃣ྦྷ였墊¨였憲䮣였㗖瞬윀㖃䚨윀ҬᲮ윀⧚⪪저岄妮저㺳㚭저締ᚮ준微준ҾƮ준ᚮ쨀厭쨀ৈᲭ쨀嶭쬀ᒣ粦쬀⦮쬀䯸ޮ찀宦䢭찀Ⳕ沭촀ↂ梫촀ኮ촀㏛䂬츀粇츀ᾶಭ츀㓣岭케ᦑᚮ케㢿垮케ᣮ㚬퀀垚ڮ퀀曈碬퀀柵撮턀咤䎭턀⃒䂭턀槿箭툀涭⪭툀⃛熭팀᪉䎭팀暶䖮팀㓥➮퐀撓咧퐀䆻Ⲯ퐀盩犮픀熘픀ׅ㎮픀䇳妬혀⎠播혀澮혀ࣽ₭휀ㆪ玬휀⦮待枧侭妧㇕ޭ䆂⾧禩⊨ⓒ㞥擷֮犥ឬዒ玬箫ካ⪫䗖岫⪂㪬涮断寜枬䮉⒭碶㮫㳢ᦧ庉㞭ặ羠⛘嚬օƮླ宧珚᪦ᚁ墦瞧ᾮῖ碭₄炭ᦲᾭ䇟䚭ႍ掦粳岧懛撨亄㒬ற溦˘抧淿ࢧ约座旎抄僓争⟝⦈姥悉䋯ᚈ懷ᦈ㺈䪈厒㦈ᖛ斈Τ侈宬沉傶劈挀椃杭⮿⎁叀徯㟰亦ગiꍸw鏨ꁴ 듨脂熆確骁 헪萆精-폫 뗬葠`뫬악￬蕣 藭혇 蔗(큄N뇮萗M뗮腩ࢠ盕淣梹ኅ羾䛮䪭㯝ઋ䊄冏ᮽ燌嶄发㷫ᖽ㫨咥䊅᮫岻糦咅嗬峍㚺境Ꮎ⏉㮇᪄媋仉ⷕ咄ۚ˿ෙ㷭俆㺄ዋ壡豈澬熅豈斲ꦃpP뗽蔯뫽,韾腧㦀▘抆ಟ握脀肀葴Z肁厅掂脀袀밻p肁ヅ掄脀쪀 肁⧵禄脀婢茧’肁叽粂脀肁0膁৯痽脀萃 芁Ᏹბ脀슃萨5莁拆拍脀钄葉 蒁涘櫝脀蕜s蒁哼⚃脀ﾄ쥿>薁䋉ކ脀쾅d蚁㞰羄脀떆茻E蚁ֹ緐脀誇萇P螁岎惗脀荁 螁曩叡脀쮈萾d袁⟐ײ脀슉茱h要Ệ埗脀鶊艺誁纠ᓥ脀蚋蘗c讁羌䧤脀葍@讁ዶ寢脀葲貁ঢ়ஃ脀 趁⟅殅脀쮍躁㒥䊆脀ꮎ辁Ɣ燊脀葷:辁㛣櫆脀ꪐ营O邁禮玼脀葱y邁濰᪨脀馑萎醁➝▨脀양葑a醁㟊羭脀茻!醁懻ᾫ脀Ꞓ蔅J銁咬篜脀覓葔3鎁ಎ䞗脀ꖓ願H鎁▾ઃ脀솓舴暟脀脟 ࡣ敭慴慤慴脁蕄dգ瑯敨Ų閁㋨ኼ挀瀄条荥脬꒖腋E隁ᚦd隁§㞂脀ꦖ蘽w隁㪰d隁⒱炞脀킖蔚a隁ǖ沘脀퍳 鞁ᧂ冭脀顰$颁᪈殭脀뚘옋W颁棼羈脀薙깭X馁䮴槚脀辚0骁烷旱脀鲁旝盩脀잝ﭡW麁㻃ᇰ脀뎟ꕕk龁䛙媭脀螠;ꂁ柷䗬脀mꊁ◇绮脀뚣﬩sꒁ⊲蒁.ꖁ嚶峹脀낦문Nꚁ೬¬脀颧 ꢁ㒐淕脀bꦁ࿎ᇩ脀랪Zꮁڙ磖脀밄 겁બグ脀t궁㫏揞脀꺮W꾁俱脀ﺯ(낁ϥ᷷脀~늁⫌埤脀놳2뒁㾋檥脀놴蘯U뒁સ䲨脀@떁⋏狸脀좶D랁撻㋠脀鲸+릁䶏芁몁枑膁3뮁ₓ⣺脀趼N벁⋲巤脀ힽ뺁ເᒇ脀잾긨 뺁㫵媛脀醿눚r뾁ዄ᯳脀럀솁䶧䯣脀诂V슁竩䇮脀蝁6쎁緟ᢲ脀鋄脛ᦇ脀駅}욁㶒绠脀B잁抦脀豱좁箇宦脀껈Z즁㲓秸脀賊Z쪁ᯯ濪脀q첁߂ᇧ脀꧍ 캁䎐檜脀귎괳y캁㋛忢脀뻏>킁审棛脀톁槞囘脀럒굅$튁濤嫝脀싓J풁ᾨ糄脀ꤡ)햁傖烘脀D횁პ䗓脀뇗瞏䓮脀ﻘR᧱䯡脀틚.ề濬脀껜
⚟濹脀駞(䦍擯脀勉a懮脀숁9䂩ࢧ脀탢b㚵峔脀諤 ⫯叡脀퇥쬃I劜斆脀ꏦ긽>ǒ仮脀샧z喣槭脀釩p㮂櫯脀䗑脀ꇬ4ႀ䗎脀컭ꉜ/ዱ泣脀헮Z暽忎脀賰擴滥脀띙r劒⏖脀홼j涿ⷩ脀꧴ꬡ`ࣕ⓬脀쇵z㒰脀ﵑCᮨ䗼脀ꓹﵧ嬨澡䃷脀駻&ﲁ掌箤脀뇼굥ﲁ滞满脀샽:ﺁ⒞脀켵jﾁ⛅燤舀ꪀ 膂䊟ǰ舀辂#节瓹㻄舀뺃鰹莂俚篬舀임 薂掤៚舀ﾅI蚂凯柊舀몇ꜿB螂曬舀캈B覂㺼僟舀鲊騕誂㎶碫舀퀲讂㺲ⓚ舀貌趂Ҁ工舀Y躂䣍䛧舀떏邂⾡喷舀D醂嚴竦舀鮒$銂˼痢舀{钂Á᮱舀訢+钂哼㺮舀ꮕ９
隂ⶪ蚁6鞂檰念舀Ꚙ@馂ប舀讚3骂Ӱ篫舀f鲂珐痭舀뺝E麂㮬ۡ舀趟kꂂ㪁ࣲ舀졉ꆂ徻⺱舀ﴔꊂリ嫰舀ꒂ⫂糭舀낥Ꚃ㦏ヵ舀蒧 Ꞃ知೮舀풨nꦂ翊ᳱ舀벪 ꮂ䚪檧舀튫됷A겂羆ǵ舀ﲬf궂瓦㋮舀햮s꾂➻芁”낂傽肁y놂傾対舀붲뎂厗舀ﶳ?뒂俟㏩舀즵휉4뚂䒠殣舀쒶B랂羥Ӟ舀蒸,뢂㷯៧舀횹Z몂㲸㫬舀꒻墳&벂⪟ӯ舀躽p뺂ⲇ㗼舀莿ｨB삂ㆃ膁?솂瞄ᷓ舀1슂厃惱舀쎂叭壸舀A얂竚粤舀ￅﱽ+욂⿼ⷼ舀腣璀舀杻腞ᲂ舀ﳉﴁt쪂糹芁4쮂㟼淶舀ﰫ
춂㳯ᛵ舀p쾂僒ዹ舀쯐L톂㲨䎦舀쿑}튂વ䋱舀ꛓd풂㺗矱舀觕屮P횂ᎄ懽舀臗ﵻ힂ᗿ凊舀짘깭!ᗸ⧩舀x䓒㷥舀룛n綦䧶舀鷝/ϻ嶦舀ꇞ 㟼舀臠囬ϻ舀褐
瓡膁C㻣声舀퓤혝gʩ舀퓥⫈囱舀뫧d犓⧪舀ﻨ濫1嫷˾舀ϟ巾舀腧嚇舀ｄ~䧥螁[⯭˜舀짰꠴D翱舀.䛣ᯯ舀틳Q䂰ᇦ舀雵3ዾঠ舀黶ꄢ>枿狴舀듷褐榮ۯ舀鷹 廒ᮋڪ舀뗺괨%廒哢௬舀컻~ﲂ殴䫳舀ꣽTﺂវ㳨舀蓿腚羀茀薀｠+膃ኅ㯞茀흔+芃ڻ᧰茀ꮃ蒃䚎ຝ茀ꮄ걛-蒃࿘⣒茀ꪅU蚃᪑◟茀(螃痉毦茀낈覃涛Ꮻ茀螊ꌇ誃Ẫ㗦茀邋흚
讃滧䫑茀릌+趃熕仓茀 躃毌淰茀붏D邃⪡㗢茀莑4醃⇳Ǡ茀펒괩-鎃嶀茀输띤鎃ۇ೬茀뎔1閃冥ò茀鞖멘隃拑羥茀흨:鞃⧏ᷲ茀솘ｍ{馃俁◫茀겚T鮃嚤⯶茀鮜5鶃䒉㫡茀A麃䷖粮茀薟념I龃₷勷茀꺠ﭹEꆃ䖪⇪茀钢ꙭ ꊃႻ犲茀^ꎃ滝ᷧ茀얤씒 ꖃ㦊䲹茀쒥zꚃත滰茀麧휂#ꞃ䮶茀ꮨ5ꦃ㪞㫻茀馪嗢 ꮃₔ䪫茀뾫둱;ꮃ㏴ǥ茀]궃᾿⳪茀ꦮ꾃殘⏨茀膰’낃䏜ድ茀톱,늃྿⣧茀ꚳ 뒃傔ǵ茀覵c떃䋦烚茀솶 랃䂸ۥ茀鶸B뢃ᛶ䯣茀鹨y릃棸䆮茀Ꞻ쨰U몃ೲ梆茀赻h뮃檆璽茀쒻ﵥf벃勂螁r붃䯊茀삾쭠 뾃炋茀蓀!삃㿟⧦茀엁g슃嶴薁w쎃宺㳽茀룄O얃璩悲茀꽛욃暋⾁茀跆脜*욃䶎䒇茀集萘 욃ᾚi욃ྛ]䁤搄捩t$戄潯ݫ敶獲潩൮汆睯摅瑩ㅟ楟ᑤ⼵㔱㈯〰‸㨴㈲ㄺ‱䵁䘒潬䕷楤彴弱敶獲潩ݮ⸵⸸⸲ര潔慰彺敶獲潩ͮ⸰ጵ潗摲敄慴汩摅瑩ㅟ楟ᑤ⼵㔱㈯〰‸㨹㠳㐺‶䵁團牯䑤瑥楡䕬楤
I can read it on K4PC. callibre does not recognize it.
Possibly, this is a very old prc file. I think the first e-format of this book was published as early as 2005 or so. maybe, this is a newer version.
Amazon doesn’t sell it (anymore?).
I looked for it just now and found an e-version at Barnes&Nobel. does that ring any bell for you? I inherited this book. can’t ask the previous owner where he got it from.
December 8, 2010 at 1:04 pm
If you can read an “inherited” ebook in K4PC, then the ebook is not DRM’ed (unless you inherited the entire PC with K4PC and all the books).
If Calibre determines that it’s a Topaz book, then I’m inclined to trust it. If it’s a DRM’ed Topaz book, then you should try the Topaz tools. But be warned that they won’t help you if you didn’t buy the ebook and download it via K4PC on the machine that the tools are installed on.
If it’s an “inherited” DeDRM’ed Topaz book (like the evidence is leaning towards), then you are out of luck. Calibre, nor any of these tools will help you convert it.
December 8, 2010 at 12:06 pm
Callibre output when you try to convert it to E-pub ends with this:
File “site-packages/calibre/ebooks/mobi/reader.py”, line 262, in __init__
ValueError: This is an Amazon Topaz book. It cannot be processed.
December 8, 2010 at 2:59 pm
Hi,
All topaz books first 4 characters of the binary file are “TPZ0”. If this file does not start with “TPZ0” then it is NOT a Topqz file.
That said, you will also find the string “BOOKMOBI” if it is a Mobi book.
Neither of these show up in the file snippet you posted. Perhaps the editor you used to open the file thinks it is encoded in some strange way and therefore messed up how it is displayed.
I would try and open a copy of the file in Word (or OpenOffice.org Writer) and look near the very top of the file either for “BOOKMOBI” or “TPZ0”.
December 8, 2010 at 3:15 pm
Hi Silver,
Diapdealer is right! I looked at the calibre code and it does the following:
if raw.startswith(‘TPZ’):
raise ValueError(_(‘This is an Amazon Topaz book. It cannot be processed.’))
So your book **does** start with TPZ so it is a Topaz Book. But someone has already used some tool (probably skindle) to remove the DRM from the book and “gave” the book to you.
Unfortunately, you can not use the Topaz Tools to convert that book since the tools expect all Topaz books to have DRM on them (they all do). So only the true owner of the book can remove the DRM properly and convert the book.
So you are out of luck here unless you can program in python and are willing to try and change the Topaz Extract code to work with “inherited” Topaz books. I am not willing to do that as only the books true owner should be able to convert it.
Sorry I can’t be any more help here.
December 8, 2010 at 3:31 pm
hey 🙂
no, you guys are awesome!
in Word it shows TPZ0 right at the start!
I’ll try to read the scripts and see how far I can go.
Thank you soo much for your input!
December 9, 2010 at 11:40 am
Hi,
I just got a very strange Mobipocket book – Great Expectations by Charles Dickens (Yes, I know I can just get it at project Gutenberg). It was a freebie at ebookmall.com and is supposed to be a mobipocket book. I can read it in the Mobipocket Reader (Windows 6.2.608) (it says “encrypted file”) but neither the calibre-plugin nor the MobiDeDrm0.19 script can decrypt it. I used the PID from my Mobipocket Reader and the MobiDeDRM script says:
“Error: invalid file format”
It seems to be some kind of binary, I can see “Great Expectations” and “TEXtREAd” in the first line of the file (using VI).
Very strange.
December 9, 2010 at 3:55 pm
Hi,
The book you have is an old format TEXtREAd prc book that is content encoded but is not a true Mobi book becasue it does not have mobi headers.
The only tool that still exists that seems to work with these older books is pdbshred.
You can find it hidden away in the trunk of the checkout of the source code to mobiperl (perhaps only in the older versions?). Look for a pdbshred archive.
You will probably have to play around with perl and compilers and things to get the tool to work.
You would simply be better off getting the Project Gutenberg version of the book and save yourself the hassle.
December 11, 2010 at 9:08 pm
I’ve written a simple tool to identify all Amazon Topaz ebooks in a specific directory.
You can download FindTopazFiles.pyw here: http://hidemyass.com/files/fZYPa/
There always seems to be confusion since Topaz books downloaded to K4PC/Mac can have almost any extension (.azw, .azw1, .prc, tpz). While the .azw1 and .tpz extensions are fairly easy to identify, the others are not (without opening the files in an editor).
To run the tool with the GUI frontend, just double-click on the ‘FindTopazFiles.pyw’ file and select the folder where all of the ebooks in question are located. Then click the ‘Search’ button. The program will list the file names of the ebooks that are indentified as being Topaz. You can then isolate those books and use the Topaz tools to decrypt and convert them.
You can also run the script from a command line by supplying the folder to search as a parameter: python FindTopazEbooks.pyw “C:\My Folder” (change appropriately for your particular O.S.)
** NOTE: This program does NOT decrypt or modify Topaz files in any way. It simply identifies them for you. **
December 11, 2010 at 9:45 pm
Like a lot of brilliant ideas, obvious in hindsight 🙂 Great idea, nice work – should come in handy for identifying any stray Topaz files that slip past the usual filter (look at the book’s Amazon page: if it lists a file-size and a page-count, it’s in .MOBI format; if it just lists a page-count, it’s in .TPZ format. Not 100% guaranteed, but it’s always worked for me.)
December 17, 2010 at 8:31 am
A friend of mine has a Kindle2. She sent me an azw file, and I have the device PID from her (Kindle 2). When I run the tool, I get:
MobiDeDrm v0.18. Copyright 2008-2010 The Dark Reverser.
MOBI header version = 6, length = 232
Extra Data Flags = 2
Error: no key found. maybe the PID is incorrect
1) Is there any way for me to deDRM their file, or do they have to install the old version of Kindle for PC on their machine and run the tool themselves?
December 17, 2010 at 9:14 am
The file from the Kinde 2 cannot be dedrmed. If your friend wants to loan you the book, they will need to do the dedrming first.
Please use the dedrm utlities responsibly.
December 17, 2010 at 4:22 pm
I have a book from Amazon which is free and I wanted to read it on my favorite ereader so I tried to dedrm it to no avail. The file is a free download at
I am able to have it sent to Kindle for the Mac and then use the Unswindle tool which takes the drm off of the book BUT loses all of the book’s interior graphics and all of the text formatting making the book unusable. When I try to use the DeDRM 1.2 app it simply fails to dedrm the book at all. Any ideas?? (All this because I would rather use Stanza!)
December 17, 2010 at 4:37 pm
Unfortunately, only free in the US, so I can’t try it myself. I’d suggest:
(1) Get DeDRM 1.3
(2) Delete from Kindle for Mac
(3) Re-download and try again.
If that doesn’t work, try running te k4mobidedrm script manually in terminal and report any errors from that.
December 17, 2010 at 8:04 pm
I have a similar issue with another ebook (not free unfortunately). All formatting is gone after DeDRMing. No errors are reported, just no formatting whatsoever.
I was able to download the title Thothamon linked to and I get the same results after decrypting.
My workaround was to use Mobiunpack on the decrypted mobi and then use MobiML2Html. After importing the html into Calibre, I was able to convert to mobi and all formatting was restored.
I’m not sure what’s going on with these two particular ebooks, but they probably deserve a look-see from someone who has more intimate knowledge of mobi internals.
December 17, 2010 at 10:29 pm
Hi,
If MobiUnpack and conversion to html worked then the DeDRM process worked just fine and did nothing to the formatting. All of the formatting and images exist just fine inside the DeDRMed mobi files, otherwise the MobiMl2HTML.pyw would not have been able to produce any of the formatting.
So I bet the problem is that Calibre is for some reason is losing or stripping the formatting and images.
Have you looked to see what the intermediate formats are like inside Calibre (there is a way to have Calibre output all intermediate forms).
If we can show it is Calibre, then we can file a bug report.
December 17, 2010 at 10:32 pm
Hi,
If MobiUnpack and conversion to html worked then the DeDRM process worked just fine and did nothing to the formatting. All of the formatting and images exist just fine inside the DeDRMed mobi files, otherwise the MobiMl2HTML.pyw would not have been able to produce any of the formatting.
So I bet the problem is that Calibre is for some reason is losing or stripping the formatting and images.
Have you looked to see what the intermediate formats are like inside Calibre (there is a way to have Calibre output all intermediate forms).
If we can show it is Calibre, then we can file a bug report.
The book Thothamon pointed to was not free in Canada either. Can you post your ebook privately for us to test with and track down what is happening. I will be sure to delete it once we are done.
Thanks
December 21, 2010 at 8:21 pm
My guess is the problem is due to differences in the mobi xml ‘standards’ and lxml which Calibre uses. There are several issues where mobi is extremely lax about how elements should be ordered – e.g. block level and inline elements – mobi allows blockquote tags inside tags as an example. lxml doesn’t allow this sort of thing, and when the mobi based html is passed to it it gets re-arranged to meet current standards. I can imagine this might result in lost content in rare cases, but in my experience it mostly screws up formatting.
Calibre has had some recent fixes checked in where the html gets pre-processed to make lxml happier. It’s very likely some scenarios weren’t covered though. MobiML2Html probably has these workarounds built-in.
December 21, 2010 at 8:23 pm
tag got stomped on in my example – mobiml allows blockquote tags inside of p tags, lxml and current standards don’t allow this.
December 21, 2010 at 9:27 pm
Hi,
Interesting. Although MobiML2HTML was originally based on the calibre code, I did not like the requirement to add lxml and so wrote my own simple tag parser to replace that requirement. As long as tags are in open/close pairs, the code I wrote for MobiML2HTML should work just fine.
Interesting. I am not sure how to move this forward. DiapDealer did find that if you simply do a calibre convert from mobi to mobi it will fix things and calibre will have the tags and images back.
In some cases the problem seems to be related to svg code added for cover images to expand them to fill the screen. This is common in epubs and thiat particular mobi may have been created from an epub and carried that svg code with it.
December 17, 2010 at 9:17 pm
DiapDealer, thanks that worked for me as well. I would never have thought to do that on my own though!
December 21, 2010 at 8:03 pm
Attention standalone Kindle owners …
Some recent work has resulted in code we think will work with standalone Kindles ***even those that have been updated to firmware > 2.5***.
So if you have a standalone Kindle and Python 2.X installed (for Windows please use ActiveState’s Active Python 2.6 or 2.7 for 32 bits), and know your Kindle Serial number (typically 16 characters long and it begins “B00…”) and are willing to test things for us, we would greatly appreciate it.
Please grab K4X_Tool.zip from here:
K4X_Tool.zip
http://www.datafilehost.com/download-865d310d.html
Please use a book copied directly from your standalone Kindle.
Extract K4X_Tools from the zip archive completely and then do the following:
1. double-click on K4MobiDeDRM.pyw
2. In the window that opens, hit the first ‘…’ button to locate your ebook you have copied form your standalone Kindle
3. Then hit the second ‘…’ button to select an output directory for the unlocked file
4. Then add in your 16 digit standalone Kindle Serial Number and hit the ‘Start’ button
After a short delay, you should see progress in the Conversion Log window indicating is the unlocking was a success or failure.
We have done only limited testing with Kindle 2’s updated to the most recent firmware. We really need more Kindle 2 and Kindle 3 testers. This script should also work for Kindle 1’s as well.
Please report any successes and for failure, please copy the output of the conversion log window and paste it here (please XXX out any personal information such as your Kindle Serial NUmber).
Thanks for your help in testing it. If it all works, future calibre plugins and DeDRM applications will incorporate this technique into their next releases (and the next tools release as well).
December 21, 2010 at 8:28 pm
Works great! First I dragged a drm’d book from my Kindle 2 (2.5.6) to the desktop (on my iMac, System 10.6.5). That undrm’d without any problem. I then went to the Manage My Kindle area and chose to download one of my drm’d purchased ebooks from my account to my computer — choosing my Kindle when asked. That file I also put on my desktop and the new script undrm’d that very quickly as well. Very, very nice work! My only suggestion is the height of laziness that it remember the Kindle serial number so it does not have to be entered each time.
December 21, 2010 at 8:38 pm
Hi,
Actually, there could be a prefereneces file added and the setting could be kept. I am will not that for a future release.
BTW: is you don’t hit quit, you can run again as many times as you like by just pointing it at a new book and hitting the start button again. You only need to enter the serial number once until you finally quit out.
December 21, 2010 at 8:42 pm
If I could spell correctly it would have said “I will note that for a future release”.
It also would have said “if you don’t hit quit”
Oh well spelling was never my forte.
December 21, 2010 at 8:34 pm
Wow!
I just tried this out in a bit of a regression test and it worked perfectly. I am running version 2.5.4 on a US Kindle 2 but since I am not in the whispernet coverage area I was able to escape the blessings of the latest format ebooks. In other words I am still receiving my purchased ebooks from Amazon in the older format that has always worked with the tools. My test today just confirmed that this new tool did not break anything.
December 21, 2010 at 8:44 pm
Hi Yecam,
Glad to hear it. We actually try both the old fixed PID and the newer book specific one since some Kindles have both older and newer books on the same device.
Thanks for testing and letting us know it works.
December 21, 2010 at 9:58 pm
Works on my K3G!
December 21, 2010 at 10:09 pm
Great!
The reported failure was because the user tried to use the old KindlePID.pyw to find a fixed PID and that old tool was not updated to handle recent Serial Number nor was it even needed!
I am so glad to hear it actually works on K3’s.
Thank you! Thank you!
December 21, 2010 at 10:17 pm
I was the PID user with the failure. 🙂
December 22, 2010 at 7:37 am
Worked flawlessly with my K3G, latest firmware. I didn’t have any DRM-infected books on the K3 to copy over and test (gee, wonder why?) and I’m currently out of wireless range, so I grabbed a half-dozen recent purchases from Amazon, redownloaded them with the “Save to my computer and transfer manually” option, and they all decrypted perfectly.
December 21, 2010 at 9:37 pm
Arrghh!!
We have had reports of a failure with K3 machines. If anyone has a K3 device, please test this and let us know what the error messages (if any ) says.
Thanks!
December 21, 2010 at 9:42 pm
I’m crossing my fingers that it was a case of someone trying to use the kindlepid.py script from the Tools package. It hasn’t been updated with all the latest Kindle 3 serials and WILL fail.
Hope, hope, hope.
December 21, 2010 at 10:17 pm
Hi DiapDealer,
You were right! It works so I think we can go ahead after the holidays with a new DeDRM and new calibre plugins and then a new tools release.
It is also nice to know that the refactored code works too. I do think having too many boxes for the user to fill in is an issue with the K4MobiDeDRM tool and we should split it as you first suggested.
We still have to deal with the Topaz side of things for standalone Kindles. I will throw something together tomorrow and send it to you for testing. I will probably refactor that code as well just to simplify things. Maybe we can even get a plugin for Topaz going as well (even though the result will need to be cleaned up in many cases).
Good news!
December 21, 2010 at 10:23 pm
Since We’ve had reports of success on K2’s and K3’s… I’ll post the calibre plugin version of the same tool for testing.
http://www.datafilehost.com/download-88946d0e.html
This a plugin zipfile ready to be added to calibre. There is no need to unzip it first.
Just like the stand-alone tool, you need to give it your Kindle’s 16 character serial number. Just like many other plugins, enter the serial number in the plugin’s customization window. If you have more than one serial, separate them with a comma (you can also enter 10 digit “fixed” PIDS here for special cases… just make sure all entries are separated by commas).
** NOTE ** There is NO need to generate a PID for your Kindle with the kindlepid.py script to use this plugin (or the stand-alone tool).
Please report successes or failures here or on Alf’s blog.
December 21, 2010 at 11:23 pm
I tried the plugin with all 3 of my Kindles. I added all 3 serial numbers in the customization box. I used a different book from each Kindle so I wouldn’t get confused.
Kindle 1 – success
Kindle 2 US [2.5.6] – failure
Kindle 3G [3.0.3] – success
I tried the same Kindle 2 US book with the original tool and it still failed.
I’ll move the book in question to the other 2 Kindles to see if there is a problem with the book itself (it is a .azw book)
December 21, 2010 at 11:28 pm
I think there is something wrong with that particular book (I haven’t put it on the other Kindles yet). I picked 3 other books from the K2 and they all worked well!
December 21, 2010 at 11:30 pm
I use the plugin successfully on K2 2.5.6 files. Are you sure the book that failed isn’t a Topaz file?
December 21, 2010 at 11:40 pm
Thanks for trying all three Kindle types, by the way. Thoroughness is much appreciated when it come to testing. 😉
December 21, 2010 at 11:52 pm
It is a mobi book. I opened it with notepad++ and it says “BOOKMOBI”.
The book decrypts just fine when imported from K1 and K3G. I don’t know what happened with K2, maybe something got corrupted.
December 22, 2010 at 5:12 pm
Just installed the new plug in for Calibre. It works perfectly for my K3 with 3.0.3 firmware. I like this much better than using K4PC app. I simply download azw files to computer then import them into Calibre. How do you think Amazon’s next update will effect your efforts? Especially when they start book lending. Hows that gonna work?
December 22, 2010 at 9:23 pm
Just to report that the plugin for Calibre also working well for me with Kindle 3 (v3.0.3). Thanks to all who contributed.
December 22, 2010 at 10:41 pm
What about non-amazon DRM mobi files? Will this work with them or do I need to still have a plug-in for that?
Some experimenting has shown that it probably doesn’t work with non-amazon .mobi files and I foolishly removed MobiDeDRM from calibre when I installed this because it seemed to be conflicting with it.
I take out a lot of ebooks from my library and a good portion of their older collection are DRM’ed mobis. I have to de-DRM them to be able to read them on any of my readers.
December 22, 2010 at 11:04 pm
Hi,
Yes it works with non-amazon Mobi files as well. You do need to know the PID that you purchased the book with (ie. thge MobiPocket PID) and add it to the customization for the plugin. See the readme on adding PIDs to the plugin customization.
December 22, 2010 at 11:05 pm
BTW: This plugin package includes the latest MobiDeDRM as part of the plugin and uses it to do the DeDRM removal once the right PID has been found.
December 22, 2010 at 11:14 pm
Hi Marcy,
Yes, this plugin should still work with non-Amazon DRM’ed mobi’s. Just like your old mobidedrm plugin did. For non-Amazon stuff, you just need to supply the same PID that was already working for your library books (whatever PID you were using with the older plugin). The new plugin will accept serial numbers and PIDS, as long as they are separated by commas.
That is the intention, anyway… so please let me know if it doesn’t work for you so we can fix it — or at least get you set back up with the old plugin.
December 22, 2010 at 11:36 pm
I did enter the PID I use for the book and it didn’t work. I found a very old copy of MobiDeDRM calibre plug-in and re-installed it and was able to de-DRM the book.
I entered the info as Kindle serial number, then comma, then the mobi PID. I tried doing it with and without a space before the mobi PID. Is there something else I should try/do?
December 23, 2010 at 12:01 am
Hi Marcy,
Yes, if you would try adding the book to calibre via the command-line tools “calibredb add your_ebook.mobi”. That will produce a lot of debugging output that you should copy & paste here. That will give us a better idea of what exactly is going on. Check the output and XXX-out anything that might seem too personal.
No need to uninstall the MobiDeDRM plugin in the meantime, just disable it in calibre while you’re testing the new plugin.
December 23, 2010 at 12:43 am
Here you go:
C:\Users\Marcy>calibredb add Downloads\Sabriel.prc
Traceback (most recent call last):
File “site-packages\calibre\customize\ui.py”, line 350, in get_file_type_metad
ata
File “site-packages\calibre\customize\builtins.py”, line 209, in get_metadata
File “site-packages\calibre\ebooks\mobi\reader.py”, line 837, in get_metadata
File “site-packages\calibre\ebooks\mobi\reader.py”, line 295, in extract_conte
nt
DRMError: Sabriel
Backing up metadata
Notifying calibre of the change
C:\Users\Marcy>
The book is added, but not de-DRM’d.
December 23, 2010 at 1:14 am
Hmmm, it appears that the new plugin didn’t even try to process that particular file. Are you sure the new plugin is installed/enabled?
You can leave both installed. Just make sure the new one is enabled (and has the PID, forget the serial for now) and the old one is disabled when you add the book via the command-line tools.
Then repost the output.
December 23, 2010 at 1:20 am
Oh dear. I was stupid. I had disabled the plug-in to de-DRM my book and forgot to re-enable it.
Here is the new message with the plug-in actually enabled. Same result however, book is added, but not de-DRM’d.
C:\Users\Marcy>calibredb add Downloads\Sabriel.prc
Traceback (most recent call last):
File “site-packages\calibre\customize\ui.py”, line 350, in get_file_type_metad
ata
File “site-packages\calibre\customize\builtins.py”, line 209, in get_metadata
File “site-packages\calibre\ebooks\mobi\reader.py”, line 837, in get_metadata
File “site-packages\calibre\ebooks\mobi\reader.py”, line 295, in extract_conte
nt
DRMError: Sabriel
K4MobiDeDRM: Calibre configuration directory = C:\Users\Marcy\AppData\Roaming\ca
libre
No EXTH record type 209 – Perhaps not a K4 file?
Running file type plugin K4PC, K4Mac, Mobi DeDRM failed with traceback:
Traceback (most recent call last):
File “site-packages\calibre\customize\ui.py”, line 176, in _run_filetype_plugi
ns
File “”, line 624, in run
File “”, line 404, in getKindlePid
TypeError: cannot concatenate ‘str’ and ‘NoneType’ objects
Backing up metadata
Notifying calibre of the change
December 23, 2010 at 1:51 am
Hi DiapDealer,
I think we need to protect against rec209 and token being None in the Serial Number routines to allow it to work with older books with no exth
Something along the lines of:
# Parse the EXTH header records and use the Kindle serial number to calculate the book pid.
def getKindlePid(pidlst, rec209, token, serialnum):
if rec209 != None:
# Compute book PID
pidHash = SHA1(serialnum+rec209+token)
bookPID = encodePID(pidHash)
bookPID = checksumPid(bookPID)
pidlst.append(bookPID)
# compute fixed pid for old pre 2.5 firmware update pid as well
bookPID = pidFromSerial(serialnum, 7) + “*”
bookPID = checksumPid(bookPID)
pidlst.append(bookPID)
return pidlst
December 23, 2010 at 2:09 am
Hi some_updates,
I had a feeling it was the non-existent 209 record (or token) that was causing the issue. The debugging output supports that. Your solution would appear to fix that issue for non-amazon mobi’s (which may still have an EXTH Header, but won’t have a 209 or 208 record type.
Marcy,
I’ll make a few changes and post another version of the plugin for you to test shortly… if you don’t mind.
December 23, 2010 at 2:19 am
DiapDealer,
Of course I don’t mind testing. I’m so grateful for all the work you’ve done on these tools. They let me actually own the books I purchase.
They also make my library’s large collection of DRM’d mobi files usuable for me. There are so few readers capable of reading these files that they really are only usuable if de-DRM’d first. You can see that in the library’s usage numbers—70 of the 428 epubs are available, but 779 of the 839 mobi books are available. Most people probably don’t know how to get these onto their readers.
December 23, 2010 at 2:43 am
OK Marcy,
Try this updated plugin and see if it works for you. If not…. well… just post the debug output again. 😉
I have high hopes, though!
http://www.datafilehost.com/download-c9b4f1d3.html
December 23, 2010 at 3:17 am
That fixed it! Thanks so much.
May I buy you two a book in appreciation for all your work?
December 23, 2010 at 3:31 am
Appreciated, but unnecessary… I work for karma. 😉
Thanks for helping debug the issue.
December 22, 2010 at 12:21 am
Deleted and re-added the problem book to K2 and now it decrypts properly. So I do think the file was corrupted.
December 22, 2010 at 2:51 pm
Success with K4X_plugin.zip! I tried it with my Kindle 2i, version 2.5.3.
I was a little mystified at first, because I couldn’t find the plugin after installing it, then I simply added my PID to the KforPC, etc. plugins, and it worked perfectly.
Bravo!
December 22, 2010 at 5:07 pm
Did you add your ‘PID’ or your serial number? The PID will only work for ebooks on your k2 that were purchased before the upgrade to 2.5. The fixed PID won’t work for newer Amazon purchases, or re-downloaded older purchases.
Basically the old fixed kindle PID is just not needed anymore for Amazon purchases.
The serial number is the only thing required. From that, the old PID is calculated (for the special case of older ebooks on k1’s & k2’s) and the new per/book PID is calculated.
December 22, 2010 at 8:30 pm
Sorry. Yes, I meant my serial number, not PID. And it worked with a book purchased recently, long after upgrade to 2.5.
It is great to have the Calibre plug-ins, because I put everything I buy through Calibre because my Calibre folder is a Dropbox, so all of my books get stored and stripped in one shot. I love my Kindle, but who knows whats coming in the next couple of years?
Thanks for helping me to own what I purchase.
December 23, 2010 at 10:17 pm
I downloaded the latest Tool set, v2.3, and loaded Python 2.6.6.17 win64. As I have Calibre 0.7.34 running, I decided to add the plugins eReaderPDB2PML_v02_plugin.zip, ignobleepub_v01.1_plugin.zip and K4MobiDeDRM_v01.3_plugin.zip.
The K4Mobi worked in Calibre. However, the other two did not. I was able to de-drm the books using the tools outside of Calibre and they both worked fine. They just did not work in Calibre. It just loaded the books in without being de-DRMed.
Here’s the diagnostics for you for eReaderPDB2PML_v02_plugin:
calibredb add angela-warrior.pdb
Running file type plugin eReader PDB 2 PML failed with traceback:
Traceback (most recent call last):
File “site-packages\calibre\customize\ui.py”, line 176, in _run_filetype_plugins
File “”, line 58, in run
File “c:\users\jusy\appdata\local\temp\calibre_0.7.34_tmp_bq8hfl\calibre_0.7.3
4_ajjnocplugin_unzip\erdr2pml.py”, line 62, in
File “c:\users\jusy\appdata\local\temp\calibre_0.7.34_tmp_bq8hfl\calibre_0.7.3
4_ajjnocplugin_unzip\openssl_des.py”, line 20, in load_libcrypto
File “ctypes\__init__.py”, line 353, in __init__
WindowsError: [Error 193] %1 is not a valid Win32 application
Backing up metadata
Notifying calibre of the change
Here’s the diagnostics for ignobleepub_v01.1_plugin.zip.
calibredb add thegirlwhokickedtheh_9780307593672.epub
Running file type plugin Ignoble Epub DeDRM failed with traceback:
Traceback (most recent call last):
File “site-packages\calibre\customize\ui.py”, line 176, in _run_filetype_plugins
File “”, line 287, in run
File “”, line 174, in _load_crypto
File “”, line 88, in _load_crypto_libcrypto
File “ctypes\__init__.py”, line 353, in __init__
WindowsError: [Error 193] %1 is not a valid Win32 application
Backing up metadata
Notifying calibre of the change
I even tried installing Pyton Win32 thinking the problem was with the Win64 version but got the same result. I hope you can figure out the problem. Otherwise, I’ll de-drmized the books in their home directories and then import them into Calibre.
Thank you so much for doing such a great job!!!
December 24, 2010 at 3:29 am
Very strange… if I had to guess, I’d say that you have OpenSSL installed on your Windows machine. But you don’t have the full version of OpenSSL installed – which provides the crypto libs. You probably have the lite version installed – which doesn’t include the crypto libs.
Two solutions here: 1) Uninstall the lite version of openssl that you have installed. At which point, the plugin will use pycrypto instead (which comes with calibre for Windows).
2) Uninstall the lite version of oppenssl that is installed and install the full version (preferably 0.9.8) – which is going to give you much better performance (with the eReaderPDB2PML plugin, anyway) than pycrypto.
What version of Windows are you running? And what version of OpenSSL (if you know it)?
December 24, 2010 at 4:50 am
Or maybe you have the 64-bit version of OpenSSL installed… which would explain how you were able to get the stand-alone scripts to work. You had a 64-bit version of Python (not recommended for these tools, BTW) calling a 64-bit version of the OpenSSL crypto libs (because I’m guessing you didn’t get a 64-bit version of PyCrypto).
But when it comes to the plugins; calibre for Windows is a 32-bit application that comes with its own 32-bit version of Python. Calibre doesn’t care (and doesn’t use) the version of python you installed. It’s self-contained.
I don’t have a 64-bit version of Windows to test on, but I’m guessing that calibre’s 32-bit version of Python is having trouble with the 64-bit OpenSSL library.
The bottom line is 32-bit. If you want to have the best opportunity for ALL of these tools (stand-alone and plugins) to work well for you, you need 32-bit Python. And you need 32-bit PyCrypto OR 32-bit OpenSSL.
December 26, 2010 at 6:05 am
Thanks, DiapDealer for replying. I’ll have to dig around. I have Calibre installed on my Dell Studio 15 which has Windows 7 on it. From comments by my brother, I think I have a laptop running 64-bit as he tells me to generally chose this version when 32 and 64 is offered. Do you have any recommendations for me to dig through my laptop to figure out what my problem is? You mention OpenSSL. Would you know where I need to look. I’m still learning Windows 7. I’ve downloaded both 32 and 64 Python software. I originally installed 64 and standalone tools ran fine. When I tried the plugins and they did not work, I downloaded 32 Python and installed that. The standalone tools still worked but Calibre plugins did not. Did I have to uninstall 64 Python before installing 32 Python because I did not do that?
December 26, 2010 at 10:27 am
The open SSL bits can be found in the following location as a exe
http://www.slproweb.com/products/Win32OpenSSL.html
you also need the Visual C++ 2008 Redistributables
December 25, 2010 at 6:16 am
Ok guys need some help. I have some old .pdb ebooks and would like to use them on a new device. They are DRM protected and as of yet I haven’t found anything that works to remove it. I understand that you could use ereader2html.py with Python to remove the DRM and convert it to html, but the script seems to be gone. Any help would help me greatly. 😀
December 25, 2010 at 11:40 am
Visit:
http://apprenticealf.wordpress.com
It has links to the tools you need.
December 26, 2010 at 6:06 am
A windows version of those scripts are available in the toolset that Some_updates posted. I know because I used them last week and they work.
December 25, 2010 at 5:52 pm
Another problem with a book that, apparently, Calibre is not treating properly. This was THE GOLDEN AGE by Michal Ajvaz which I purchased at Amazon. It deDRM’d OK with the usual tools. But whenj I loaded it into Calibre, again, I saw a book that had lost all formatting and was unreadable. I did the workaround with using the MobiUnpack on the unDRMd book and then using MobiM2HTML and then converting that file to an ePub with Calibre. But how can we communicate to Kovid that Calibre seems to be messing up the latest books once they are unDRMd? I’m not very technical so am unsure how to communicate this error report (if that is what it is) and hope someone more qualified here will pick this up. Anyway, Happy Holidays to all!
December 26, 2010 at 4:49 pm
Hi Thothamon,
The only way to deal with this issue is for us to create a standalone test case that illustrates the problem that has nothing to do with drm. The way to do that is to compare the actual mobiml generated after dedrming the file with the xhtml version that when imported does not show the problem.
So we would need someone to remove the drm from one of those problem books and post the nodrm version (privately to one of us) – no public links here so that we can unpack it and see what is actually going on.
Once we know that we can create a test book (the simplest possible) that recreates the problem and then file a bug report with calibre and attaching that test case.
So if you want me to get involved pm me (KevinH) on Mobileread with a location to grab the nodrm version of that book and I will take a look at creating a test case.
December 26, 2010 at 8:42 pm
Hi — Well, I sent you a PM on Mobileread but not sure if that is working. Keeps saying 0 in my Sent Messages. But I sent myself a test one that worked and it still says 0 in sent messages so go figure. Anyway, I do not have a place to post the file so if you can recommend one or I can do email if we exchange addresses (Obviously proimising to keep each 0thers’ secure). Anyway let me know there (if you indeed get the message) or here (if not). Thanks.
December 26, 2010 at 8:51 pm
OK, just remembered I have a premium Rapidshare account and uploaded the file there. I sent you a link via Mobileread PM but, again, keeps saying no sent messages. Please let me know if you got it or not.
December 27, 2010 at 3:00 pm
HI Thothamon,
Would you try one thing for me? Using the **DRM** version of the problem book, load it on your Kindle, Kindle for PC/Mac, etc and go to the **end** of the book. Do you actually see a copy of the cover of the book at the very end?
There is svg code at the very end of the book but it references a file that is not in the book itself and it does not reference it by record number (as it does all of the other images in the book). So I was wondering if this was left over by some epub to Kindle automated conversion of if in fact this svg code was actually processed by a real Kindle/Kindle for Mac/ etc.
So please let me know if you see a copy of the cover of the book at the very very end.
Also, a simpler workaround than using MobiUnpack exists. Once you have the book in calibre and you see the markup go away. simply to a Mobi to Mobi conversion inside Calibre and all of the markup will return.
Thanks,
December 27, 2010 at 5:40 pm
Nope, I do not see a cover at the end of the book. However, I do note that the location of the last page I can view is (on Kindle for Mac) “4,483 of 4,484.”
December 27, 2010 at 5:47 pm
No cover at the end when viewed directly on the Kindle as well.
December 27, 2010 at 9:17 pm
Hi Thothamon,
That confirms it then. The problem is that some Mobi’s are being created form epubs and the producers are not stripping out the incorrect svg at the end of the or the beginning. I think Alf came to pretty much the same conclusion.
So I will send a patch to Calibre that will strip out the svg:svg tags in Mobi’s and see if Calibre/Kovid will except it.
Until then, simply import it into calibre and the do a calibre mobi to mobi conversion and then all should be well.
Thanks,
December 26, 2010 at 10:01 am
I got a new laptop for Christmas and decided to reorganise…
Installing openssl helped with the 64 bit error (from above) but now I have ended up with a load of epubs (converted from pmlz) which have every paragraph after the first indented. I have tried playing around with the setting for text justification but no joy. Anyone got any ideas? (originally PDB books)
December 26, 2010 at 4:57 pm
Hi Caroline,
Yes, I have found that calibre does not keep to the original as well as I would like when importing many types of files. If you look at the standalone tools (not the plugins) you can run eReaderPDB2PML.pyw on one of the books that have that problem. Then run Pml2HTML.pyw and point it at the directory where you unpacked the book too. Once it completes, simply open the resulting html in any browser and look to see if it has the same problem with the formatting. If so, then the issue is in the pml. If not, then the issue is in calibre.
If it bothers you enough you can import that html made from Pml2HTML.pyw into Sigil and get much finer contraol over the conversion to epub. This is what I do as the formatting changes literally drive me crazy. I want it to look as close to the original as possible.
December 30, 2010 at 2:59 pm
Has the Encryption Changed on B&N Epub books? Nothing has changed on my name registered to the B&N nor has the credit card info. I am using tools 2.4. I have never had problems before removing DRM. I redid they key file with ignoblekeygen.pyw thinking maybe something got corrupted. When ignobleepub.pyw is executed it does so with no errors and says successful. But the resulting file still looks encrypted. using python 2.6 and pycrypto-2.1.0 (both 32 bit version). I was able to purchase and remove drm on on book on Dec 27th but since then none will work. I do have a gift card on the account but it appears to not be charged. I am a loss at what to look at.
December 30, 2010 at 4:55 pm
I heard that the Nook can now check-out books from the library, and am wondering is they didn’t switch to the adobe scheme as a result. If this is the case, the inept tools should to the trick for you (http://i-u2665-cabbages.blogspot.com/2010/07/updates.html).
December 30, 2010 at 5:09 pm
Already tried that. The ineptkey_v5.2pyw retrieved my key from Adobe. Then ran ineptepub_v5.3.pyw prompted for the book and the output file. You get Error: Problem decrypting session key. So I don’t thing the encryption is the same as B&N.
December 30, 2010 at 6:11 pm
Hi Ed,
One problem that happens sometimes is that the files inside the epub have non-ascii file names. This same bug exists in ineptepub. When there are non-ascii file names, the code that matches the list of files that need decrypting will not match the files themselves causing some files not to have the drm removed.
Typically this happens with international (non-US) books. Could this be the problem with this book. If so, we have a fix in the pipeline.
December 30, 2010 at 6:27 pm
Hi Ed,
If it is not the international file names issue, then I would be happy to take a look at things. I would need a copy of the .epub and the *.b64 one-way-hash file. Please do not send or post your name or credit card information as I would not want to be responsible for such info. The *.B64 is a one way hash (there is no way to get your name or password from the hash itself).
Please post those files inside a zip archive and PM the location to “KevinH” on Mobileread, if you want me to figure out what is going on and get a fix.
Please do not post a public link.
Thanks,
December 31, 2010 at 1:16 am
Hi Ed,
Your books already had the DRM successfully removed. I was able to open both of them in Sigil and in ADE without any trouble. What made you think that your DRM removal attempt failed?
You should be good to go.
December 30, 2010 at 8:48 pm
Probably not it but did you try the script ePub_Fixer.pyw on the file first? I had some problems with B&N books that fixed….
December 30, 2010 at 10:46 pm
Thanks Kevin,
I did try the ePub_Fixer.pyw before and no luck. I will zip up the files and send you a message later tonight.
December 30, 2010 at 6:30 pm
Call for Mobi with Type 1 encryption (no pid is needed).
If anyone has a MOBIBOOK file (an old .prc or .pdb format) that has Type 1 encryption (that does not need a pid but that will not decode in MobiDeDRM, I would love to have a copy of it for testing purpposes.
I have an experimental version of MobiDeDRM with some code changes but no way to test or debug the changes without a test case of some sort.
Please PM “KevinH” on Mobileread if you have such a file that you are willing to provide as a test case.
Thanks
December 31, 2010 at 3:12 pm
Announcing a Major Upgrade of the tools
tools_v3.0.zip
http://www.datafilehost.com/download-e98eb36e.html
Changes in tools 3.0
1. Added support for standalone Kindles even with firmware >= 2.5
2. New integrated KindleBooks gui tool – handles both Topaz and Mobi Books (renamed from K4MobiDeDRM)
3. Updated version of ineptepub_v5.4.pyw that fixes a bug that caused it to fail with non-ascii chapter names
4. Updated version of ignobleepub_v3.2.pyw that fixes the same bug that caused failures with non-ascii file names
5. New version of MobiDeDRM (v21) that now handles lists of pids
6. Updated version of eReaderPDB2PML (erdr2pml) which allows pycrypto use as well to speed up decryption
7. Lots of Calibre Plugins Updated:
– new K4MobiDeDRM plugin that handles Topaz books and Kindle books
– new ineptepub plugin with the latest bug fixes for non-ascii filenames
– new ingnobleepub plugin with the latest bug fixes for non-ascii filenames
– new eReader2PML plugin to allow use of pycrypto for faster decrypting
8. Incorporated inept_pdf_tools.zip into the tools set
9. Topaz Tools now integrated into KindleBooks so no longer a separate tool
10. Updated version of MobiML2HTML that strips out leftover svg code from KindleGen epub to mobi conversions.
11. New version of DeDRM 1.4.app (for Mac OS X) that has all of the above bug fixes and now supports for standalone Kindles and Topaz ebooks.
Happy New Year!
from your “tools” developer team:
Apprentice Alf, DiapDealer, and some_updates
January 1, 2011 at 2:02 pm
Wow! Very impressive! So the K4MobiDeDRM plugin will now remove DRM from Topaz, but Calibre will not convert Topaz, right? I still have to go through the steps previously required to get to mobi, yes?
Happy New Year Apprentice Alf, DiapDealer, and some_updates, and many, many thanks!
January 1, 2011 at 8:46 pm
What a wonderful New Year’s present.
Would it be possible to allow ‘batch’ processing, or is that just too difficult given what has to be done.
Many thanks,
January 1, 2011 at 9:04 pm
Hi,
I believe that calibre can import a folder of ebooks. And on Mac OSX DeDRM allows you to drag and drop a folder of ebooks. On Windows, I am not sure. I do know there are ways to create a .bat file that can automate the python scripts for you and I believe they were posted on this blog somewhere, so if you search within this blog with your browser you should see bat files commands to automate the process of running a python script.
January 11, 2011 at 12:35 am
Some_updates:
Many, many thanks for the modifications that change the output filename to the actual title of the book. It works quite well, and does virtually all I could ask. Well, I could always ask for more, but realistically that would be being greedy.
I would love it if there was a way for the tool to remember the output directory, last input directory and the serial number(s)/PIDs, but that is easy enough to work around with a simply text file that I can copy and paste from.
Many, many thanks. You have worked wonders yet again.
Mike
January 3, 2011 at 6:21 am
Kindle 3 owner here: everything works great. I just used the Calibre plugin to decrypt my whole kindle library at once. It’s not a huge library at all, with just a dozen books or so, but it works as a charm.
As a side note, both the plugin and the standalone tool were tested under Linux. I know it shouldn’t make much difference, being Python, but it worths the regiter.
Also, I would love to test and retest anything for help, and even more – I am an experiences programmer (mostly C) learning Python, and would love to help while learning, even if it is just about polishing and testing and bug chasing.
Actually, I can’t stop thinking about makine a quick and dirt preferences file, just for storing the serial number (and as a bonus the input and output folder). Will try to post a patch here this weekend as a “thank-you” note.
January 3, 2011 at 3:29 pm
Hi, Aylons,
We would welcome the help. Our target is for everything to work cross-platform: Windows, Mac OSX, and Linux.
So if you do plan to put together a patch for preference file support, please use the official directories to store any data for each platform.
On Mac OSX, this information is stored in:
{$HOME}/Library/Preferences/ and can be a file or a folder of other files that uses an unique name similar to com.apprenticealf.KindleBooks or org.darkreverser.KindleBooks.
So you could make com.apprenticealf.Kindlebooks be a folder and store a .KindleBooks text file there or create an xml based plist.
On Windows, typically use registry keys to find the current user’s directory and then look in Local Appdata:
similar to how it is done in k4pcutils.py
regkey = winreg.OpenKey(winreg.HKEY_CURRENT_USER, “Software\\Microsoft\\Windows\\CurrentVersion\\Explorer\\Shell Folder\
s\\”)
path = winreg.QueryValueEx(regkey, ‘Local AppData’)[0]
kinfopath = path +’\\Amazon\\Kindle For PC\\{AMAwzsaPaaZAzmZzZQzgZCAkZ3AjA_AY}\\kindle.info’
You could create your own app directory and text based preferences file there.
For linux, As you know I have seen many many styles of preferences. So it is up to you. But whatever you choose should be stored in the user’s {$HOME} someplace, perhaps in a .org.alf.Kindlebooks file or whatever you like.
Something that worked along those lines and that lived in its own pure python implementation that could be imported and that is class based and that can both read and write preferences for any program would be wonderful!!!!
Thanks and welcome to the team. We can use all the developers we can get!
January 5, 2011 at 2:08 am
What would really be nice is to have the file name changed to what is actually inside the amazon book. I would also request that the file extension be changed to either prc or mobi.
Of course, if I could program well enough, I could do it, but I cant…
It would be most appreciated….
January 5, 2011 at 2:44 am
Hi Mike,
Do you mean the title, the author, a combination? The current file name is actually the ASIN number for the book which completely specifies it. Another way to achieve this right now is to use the K4MobiDeDRM calibre plugin and then export the books from Calibre and it will generate folders with author name and change the book to be the title.
January 5, 2011 at 5:11 am
Someupdates:
I would like to see the title, as it is listed “whatever Amazon for PC shows it as” and perhaps with a .prc or a .mobi.
Basically put all the information you can for the dumb user – as in me – so it is easy to move the files to somewhere else.
January 5, 2011 at 7:25 pm
Hi Mike,
Go to: http://pastebin.com/e7MfHtYz
and hit the download link and rename the downloaded file to:
k4mobidedrm.py
Then copy that file over its namesake in KindleBooks\lib\ (or KindleBooks/lib/ depending if Linux/MacOSX or Windows).
Then try converting one of your books. This version will try to create the file name using the title of the book.
This is easy for most latin based languages but fails for asian and other languages. Converting their titles to proper filenames can be a real issue (and one that calibre handles well but that is too much work for what we do here).
Please let me know if this is what you are thinking about.
January 4, 2011 at 10:56 am
Hi,
Tried out tools 3.0. I cannot get the K4 plugin to work with Calibre under Linux (Fedora 14). It just doesn’t seem to register with Calibre.
However the KindleBooks.pyw script works a bliss with Amazon’s ‘corrupted’ files copied over from my Kindle 3 Wifi. Finally fair use access to the books I paid for.
Thanks a lot, your efforts are much appreciated!
JS
January 4, 2011 at 11:36 am
By “just doesn’t seem to register with Calibre” do you mean that when you click on the Add button, nothing seems to happen except that the path field clears?
In which case, the plug-in has been successfully added. Unfrotunately Calibre provides no explicit feedback when a plug-in is successfully installed.
After clicking on Add, check the File type plugins list. You should find the K4PC, L4Mac, Mobi DeDRM (0.1.7) plug in the list. Just add in your Kindle’s serial number in the customisation field.
January 1, 2011 at 3:24 pm
Nevermind! I found the readme text under Kindle Books. I guess that using the plugin with Topaz will create the same 3 files as would the GUI version.
Thanks again.
January 1, 2011 at 4:11 pm
Hi montalex,
Unfortunately no. A Calibre plugin can only forward one file (not 3), I asked just to make sure. So when you use the plugin in Calibre, all you will get in the html version of the book which is what most people want. The svg images and the xml will not be generated by the plugin version. If you want those, you will have to use the gui KindleBooks.pyw program.
Hope this makes things clearer.
January 1, 2011 at 10:57 pm
some_updates – I installed your latest Kindle plugin (K4Mobi v1.7) in Calibre (v0.7.35) and customized it with my Kindle serial number. I can add a book from Kindle to Calibre without any messages but when I try to view the book in Calibre it says it is protected by DRM. I have tried adding the file from Kindle, and copying the file to my laptop and then adding, with the same results. Here is the info from a command line add:
C:\>calibredb add crossing.azw
K4MobiDeDRM: Calibre configuration directory = C:\Documents and Settings\xxx.xxxxxx\Application Data\calibre
MOBI header version = 5, length = 232
Extra Data Flags = 1
Running file type plugin K4PC, K4Mac, Mobi DeDRM failed with traceback:
Traceback (most recent call last):
File “site-packages\calibre\customize\ui.py”, line 176, in _run_filetype_plugi
ns
File “”, line 285, in run
File “C:\Documents and Settings\xxxx.xxxx\Application Data\calibre\plugins\K4PC, K4Mac, Mobi DeDRM.zip\kgenpids.py”, line 305, ingetPidList
File “C:\Documents and Settings\xxxx.xxxx\Application Data\c
alibre\plugins\K4PC, K4Mac, Mobi DeDRM.zip\kgenpids.py”, line 280, in getK4Pids
File “C:\Documents and Settings\xxxx.xxxx\Application Data\c
alibre\plugins\K4PC, K4Mac, Mobi DeDRM.zip\kgenpids.py”, line 101, in getKindleInfoValueForKey
File “C:\Documents and Settings\xxxx.xxxx\Application Data\calibre\plugins\K4PC, K4Mac, Mobi DeDRM.zip\kgenpids.py”, line 91, in getKindleIn
foValueForHash
KeyError: ‘AbaZZ6z4a7ZxzLzkZcaqauZMZjZ_Ztz6’
Backing up metadata
Notifying calibre of the change
I appreciate any help you can give me!
January 1, 2011 at 11:44 pm
I’m not some_updates, but I’ve played around with the MobiDeDRM code a little.
It looks to me like it’s crashing when trying to get a possible key from your Kindle for PC installation. This might be happening because you don’t have Kindle for PC installed, and it’s a bug. Or it might be happening for some other weird reason (and still a bug).
If you don’t have Kindle for PC installed, simply installing it and registering it to your kindle account might work around the bug.
January 2, 2011 at 12:10 am
Installing Kindle for PC did it – THANKS!
January 2, 2011 at 12:26 am
For the record… you shouldn’t have had to install K4PC (but I’m glad you did to at least verify the bug). We’ll have to take a look at that.
January 2, 2011 at 12:22 am
Hmmm,
Good advice from Paul. If you don’t have Kindle for PC installed, that error definitely smells like a bug. If you do have Kindle for PC installed and registered (and are still getting the same error), you might try temporarily renaming your kindle.info file to kindle.info.bak (or something other than .info) and see if that works. If it does, it may still be a bug (but one that doesn’t affect me on my Windows machine).
The plugin works for me on Linux without K4PC installed, but to be perfectly honest… I’ve never tested it on Windows without K4PC at least installed. Everybody’s gotten so used to needing K4PC/Mac to do everything, that it never even occurred to me to try it.
January 2, 2011 at 1:29 am
Hi,
This is strange. I think the user’s account must have had a kindle.info file at one time or another. The parseKindleInfo call is done inside a try block to catch IOError exceptions from trying to open a nonexistent file. It should not proceed after that and should in fact return a null pidlist from that function.
The funny thing is that from where the code bombed out it had actually successfully read the MazamaRandomNumber from the kindle.info file. It is just that the kindle.info file did not belong to the current user and so the trying to load a account token failed with a key error.
Perhaps they tried to remove Kindle for PC at one point and did not remove the kindle.info file?
I am not sure how we can work around a broken or partial Kindle for PC installation.
January 2, 2011 at 2:00 am
some_updates said: “The parseKindleInfo call is done inside a try block to catch IOError exceptions from trying to open a nonexistent file. It should not proceed after that and should in fact return a null pidlist from that function.”
That’s what I determined, as well. I agree that if it’s a broken K4PC install, there’s very little we can do.
I am going to uninstall K4PC on one of my machines and see what gets left behind. If I can duplicate the same situation, maybe we can account for it somehow, but I’m leaning toward it being a freaky broken install/uninstall of K4PC.
January 2, 2011 at 4:01 am
Using latest Tools 3.0 and Calibre 7.35 I am having the same problem importing Topaz books into Calibre. All three of them loaded into Calibre without errors, but cannot be opened. I see:
ERROR: Unhandled exception: WindowsError:[Error 1155] No application is associated with the specified file for this operation: u”E:\\Documents\\Books\\Calibre\\XXXXXXe (122)\\XXXXX.azw1″
Traceback (most recent call last):
File “site-packages\calibre\gui2\actions\view.py”, line 144, in view_triggered
File “site-packages\calibre\gui2\actions\view.py”, line 179, in _view_books
File “site-packages\calibre\gui2\actions\view.py”, line 45, in view_format
File “site-packages\calibre\gui2\actions\view.py”, line 80, in _view_file
File “site-packages\calibre\gui2\actions\view.py”, line 71, in _launch_viewer
File “site-packages\calibre\gui2\__init__.py”, line 666, in open_local_file
WindowsError: [Error 1155] No application is associated with the specified file for this operation: u”E:\\Documents\\Books\\Calibre\XXXXXX\\XXXXXX(122)\\XXXXXX.azw1″
Error:Unhandled exception ” message.
If I try to convert the books I see:
“WARNING: Could not convert some books: Could not convert 1 of 1 books, because no suitable source format was found”.
I tried installing K4PC with no effect.
January 2, 2011 at 1:34 pm
We need to see the the output generated when you try to ADD the ebook to calibre from the command line. That will give us a better idea of what is going on. “calibredb add XXXXXX.azw1”
Obviously, if the ebook still has an .azw1 extension… the decryption didn’t work. It sounds like the K4MobiDeDRM plugin (from the tools package) may not be installed. Can you verify that a plugin called “K4PC, K4Mac, MobiDeDRM (0.1.7)” is showing up in calibre’s list of installed plugins (under File type plugins)?
January 2, 2011 at 2:50 pm
Hi michaelB
Ys the output from running the calibredb add YOURFILE.azw1 in a terminal (cmd.exe) window would be a big help. The problem may be an unknown xml tag. The Topaz format is not completely reverse engineered yet and possibly you have books that have something that the converter does not YET know how to handle.
Please post the info you see from running the calibredb add BOOKFILE.azw1 (as described in the plugin README file at the end) so that we can see what errors happened.
Thanks,
January 2, 2011 at 5:07 pm
I did the command line add. I must comment that the 01.7 plugin failed on all 3 Topaz books I tried, but converted about 20 other .azw books without incident
E:\Temp>calibredb add “XXXXX.azw1”
K4MobiDeDRM: Calibre configuration directory = C:\Documents and Settings\Admin.E
EEPC1000\Application Data\calibre
Running file type plugin K4PC, K4Mac, Mobi DeDRM failed with traceback:
Traceback (most recent call last):
File “site-packages\calibre\customize\ui.py”, line 176, in _run_filetype_plugi
ns
File “”, line 284, in run
File “C:\Documents and Settings\Admin.EEEPC1000\Application Data\calibre\plugi
ns\K4PC, K4Mac, Mobi DeDRM.zip\topazextract.py”, line 171, in getPIDMetaInfo
UnboundLocalError: local variable ‘keysRecordRecord’ referenced before assignmen
t
Backing up metadata
Notifying calibre of the change
January 2, 2011 at 5:36 pm
Hi,
That is strange indeed!
Here is the exact error message:
It occurred here: topazextract.py, line 171, in getPIDMetaInfo
The error was:
“UnboundLocalError: local variable ‘keysRecordRecord’ referenced before assignment”
That means we tried to use keysRecordRecord before we assigned anything to it.
What makes this strange is that the code does the following:
def getPIDMetaInfo(self):
keysRecord = None
KeysRecordRecord = None
if ‘keys’ in self.bookMetadata:
keysRecord = self.bookMetadata[‘keys’]
keysRecordRecord = self.bookMetadata[keysRecord]
return keysRecord, keysRecordRecord
and line 171 is the return statement. And the very first thing we do is set the values of keysRecord and KeysRecordRecord so there is no way it can be referenced before assignment via either path through the routine. If you have no ‘keys’ record it should return None, None. If we have a keysRecord then keysRecord should be set properly. And if the bookMetadata does not have a key that matches keysRecord, then it should return a key error. Neither of these seems to have occurred.
MichaelB:
have you edited the metadata in this Topaz file at all? Have you removed the DRM from this Topaz book using some other tool?
Just to verify is there any way you could remove this book from the device and download a clean copy of it from your Kindle archive and try again with the clean copy and let us know if you get the exact same error message. Also, do all 3 Topaz books show the same error message?
And is this Topaz book from a Kindle for PC, Kindle for Mac, standalone Kindle, Kindle for iPhone/iPodTouch/iPad or ?
I guess we could always change the code to look at follows just to be safer:
def getPIDMetaInfo(self):
keysRecord = None
KeysRecordRecord = None
if ‘keys’ in self.bookMetadata:
keysRecord = self.bookMetadata[‘keys’]
if keysRecord in self.bookMetadata:
keysRecordRecord = self.bookMetadata[keysRecord]
return keysRecord, keysRecordRecord
I have just never seen a case that has keysRecord set but not a keysRecordRecord and that is not the error we received (if that was the case we should have received a “key error exception but none happened.
Wow! This is strange.
January 2, 2011 at 5:53 pm
Hi,
Not so strange. The bug was right in front of me.
We set “KeysRecordRecord” to None not “keysRecordRecord”!!!
Argh! You must have an older Topaz book and it did not have either in its exth area and it triggered that bug.
I will build a new calibre plugin and tool for you to test to see if that works for you without that bug.
I will post it a link here when it is ready.
Thanks for your bug report!
January 2, 2011 at 6:06 pm
Hi michaelB,
Okay please grab the following zip archive:
KindleBooks_Tools.zip
http://www.datafilehost.com/download-1f05b438.html
Unzip it and grab the K4MobiDeDRM_v01.8_plugin.zip from inside and install it in Calibre and then retry with your Topaz books and please let us know if that fixes your issues.
Thanks again for your bug report!
January 2, 2011 at 6:40 pm
O.K. New plugin allowed one of the three files to decode. The other two acted the same as before. One of the failed was the book added above. These files are from a K1. I got new copies from the archive backup but not fresh from Amazon.
January 2, 2011 at 6:54 pm
Hi,
For each failed book please post the results from the calibredb add command so that we can see what is going wrong and why.
It simply can’t be the *exact* same error as above since that typo was fixed in the new version of the plugin.
January 2, 2011 at 10:06 pm
The 2 Topaz filesgenerate the same add files:
E:\Temp>calibredb add “XXXXX.azw1”
K4MobiDeDRM: Calibre configuration directory = C:\Documents and Settings\Admin.E
EEPC1000\Application Data\calibre
No EXTH record type 209 or token – Perhaps not a K4 file?
Trying: TY14HHPH
Trying: 3FYN43C*
Trying: HUJLWKV*
——————————————
Error message from Calibre
Error opening file
_______________________
Running file type plugin K4PC, K4Mac, Mobi DeDRM failed with traceback:
Traceback (most recent call last):
File “site-packages\calibre\customize\ui.py”, line 176, in _run_filetype_plugi
ns
File “”, line 309, in run
Exception: K4MobiDeDRM plugin could not decode the file
Backing up metadata
Notifying calibre of the change
January 2, 2011 at 11:14 pm
Hi MichaelB,
Thanks!
The problem is I am not sure that this is a bug. It is simply saying no pid worked for these two files. Since these books came from a Kindle 1, you have a fixed PID based on your serial number.
It should be 3FYN43C* or HUJLWKV*
So please try running KindlePID.pyw. It can be found in tools_v3.0 inside the Mobi_Additional_Tools folder. It is a gui tool. If you have Python installed on your machine, all you have to do is double-click to run it. Then enter in your 16 digit Kindle 1 serial number (it typically begins B00…) and let it calculate a fixed PID and let us know if it matches either of the two PID’s above.
If it doesn’t then something strange is going on. Also double-check that this book did in fact come directly from your Kindle 1 (and not a version of Kindle for PC) and that you have inserted the proper Serial Number into the plugin customization as instructed in the README.
Please let us know what you find out.
January 2, 2011 at 10:58 pm
Update: Sucess!
Other two books converted after I remembered to put the PID back into the plugin. I had removed it while playing around and just left the serial numbers in the plugin. Used the command line and saw no errors, used Calibre also no errors.
Good work gentlemen!
January 2, 2011 at 11:18 pm
Hi MichaelB,
But the serial numbers should have been enough! The same code that calculated the fixed PID for your Kindle 1 is part of this codebase. It should have calculated the same PID from that same serial number. If not, perhaps we do have a bug.
Please compare the PID you just added with those in your error report? Are they different? One of those should match it.
I am glad it worked for you but I am thinking there must still be a bug here. As long as you had the correct serial number in the customizations that this book came from, it should have matched one of those PIDs.
January 2, 2011 at 11:53 pm
The PID that worked was ITM3UK8*KW, however KindlePID will not calculate a PID from the K1 serial number (error message). KindlePID worked fine for my K2 serial number (HUJLWKV*). I don’t recognize TY14HHPH – not something I entered.
This K1 was a replacement unit, perhaps KindlePID does not like the serial number format. But if so how did I calculate it in the first place? A earlier version of KindlePID or one I previously hacked?
January 3, 2011 at 1:19 am
The rest of the story …
KindePID.pyw cannot deal with serials B101…… I modifed an old version of kindlepid.py and it worked fine, BUT, I discovered that the PID that worked for me was from the original Kindle1, not the PID from the K1 serial I was using. By chance, I accidentally had saved the old PID in place of the new one (3FYN43C*) and that is the one that was needed to decrypt the Topaz files. So mystery solved
January 3, 2011 at 2:22 am
Hi MichaelB,
Thanks that explains it. So no bugs except in kindlepid.py.
You can fix the one inside tools_v3.0\MobiAdditional_Tools\lib\ by editing kindlepid.py to change startswith(“B00”) to startswith(“B”).
I will make that same change in kindlepid.py so that KindlePID.pyw works for everyone for the next release.
Thanks again for reporting the bugs and working with us to resolve them.
January 3, 2011 at 2:09 pm
I’m still not sure if there IS a bug. To my knowledge, there is no Kindle serial number that starts with the prefix B101.
B001 = Kindle 1
B002 = Kindle 2 U.S. (Sprint)
B003 = Kindle 2 International (AT&T)
B004 = Kindle DX U.S.
B005 = Kindle DX International
B006 = Kindle 3 3G + WiFi Graphite/White (U.S. and Canada) NOTE: Some European cities may receive this serial number as well.
B008 = Kindle 3 WiFi
B009 = Kindle DX Graphite
B00A = Kindle 3 3G + WiFi Graphite (Europe)
January 3, 2011 at 2:34 pm
Well, mine does. B101 was not uncommon for K1s and caused problems for Kindlepid even back then in the Old Days..
January 3, 2011 at 2:48 pm
I wonder if the B101 serial numbers were only given to replacement Kindle 1s, and had the same internal PID as the old Kindle?
January 3, 2011 at 6:27 pm
I think that’s very possible Paul. Actually MichaelB, I think I’d pretend that the second character was a zero (B001 instead of B101) and see if kindlepid.py comes up with the same PID that properly decoded the book(s) in question.
January 4, 2011 at 12:07 am
It was a replacement for my wife’s original unit with an entirely different S/N. In fact using the complete B101….. string generates the proper PID. If you ignore or change any character you get a different PID.
January 6, 2011 at 8:52 pm
New Version of the tools is now available:
tools_v3.1.zip
http://www.datafilehost.com/download-74a4e2fc.html
Changes in tools 3.1
This is primarily a bug fix release. The only new features are support for type 1 encrypted Mobi ‘TEXtREAd’ books and ‘BOOKMOBI’ (very experimental) books. These books do not require a PID but they are encrypted.
The following bugs were fixed:
1. fixed bug where remove of temporary directories used in the conversion process was causing errors on Windows (impacts eReader and KindleBooks)
2. new DeDRM 1.6.app with a bug fix for preferences for ineptpdf and all of the other bug fixes mentioned here.
3. fixed bug on Windows and Windows64 systems that have multiple incompatible OpenSSL libleay dlls on their system. On Windows, PyCrypto will now be tried first and then OpenSSL. This impacts eReader, ineptepub, ineptpdf, and ignobleepub books on Windows and Windows64 machines
4. fixed bug in mobidedrm that causes failures with some very old Mobi books that do not have an exth (extended header).
5. fixed bug in mobiunpack that caused images in old mobi ‘TEXtREAd’ files to be ignored
6. fix bug that showed funny boxes in the Conversion Log window for many of the gui tools on Windows platforms.
7. skindle has been completely dropped from the tools zip archive because it has not been actively supported for a very long time and gives incorrect results in some cases
8. Added a ReadMe_First.txt to help new users to get quickly oriented and started.
These bug fixes change things in almost every tool, so there are new versions of all of the Calibre plugins as well.
Your Development Team: ApprenticeAlf, DiapDealer, and some_updates
January 9, 2011 at 8:33 pm
I’m having trouble decrypting a Kindle book on Ubuntu Linux.
I am successfully running the latest K4PC 1.3(30884) using wine1.3. I can download and view an ebook and have a kindle.info file in the wine C drive.
I have tried using KindleBooks.pyw specifying the path to kindle.info and get this:
Please Wait …
MOBI header version = 6, length = 232
Extra Data Flags = 2
Processing Book: Cloud Atlas
global name ‘openKindleInfo’ is not defined
global name ‘openKindleInfo’ is not defined
Crypto Type is: 2
… not suceessful No key found. Most likely the correct PID has not been given.
Error: Encryption Removal Failed
I also tried using k4dumpkinfo on the kindle.info and just get:
Unknown Record
Any ideas? Is k4mobidedrm working on Linux?
January 9, 2011 at 10:12 pm
I’ve never successfully got a newer version of K4PC working under Wine. But if you have K4PC working, the reason none of the scripts seem to work is because you are using Ubuntu’s Python. If it has any chance of working, you’re going to have to install Active State’s Python under Wine.
Wine Python and Wine K4PC stands a chance, but I’ve never had any luck with it… and frankly, gave up. Ubuntu’s Python knows absolutely nothing about your Wine C: drive, Wine Volume Serial, Wine Registry, or Wine K4PC installation… nor should it.
The only successful decryption of a K4PC ebook I’ve personally experienced on Linux (under Wine) involved unswindle.
The newest version of the tools will work fine on Linux (using files copied directly from the Kindle and providing the serial number), but K4PC file decryption on Linux… I can’t get it to work without unswindle.
January 10, 2011 at 1:27 am
OK, so I wiped out my current Wine installation and did a fresh “virgin” install of 1.3. The newset version of K4PC does indeed work “out of the box”. However, even installing ActiveState’s Python under Wine does not allow me to decrypt the K4PC files. The kindle.info file can be parsed, but something in the way the volume serial is being generated (or the username) is not allowing the script to build the proper PID.
I’ll have to do some more poking around, but it’s far easier to just use the files from my Kindle and provide the serial number, rather than run K4PC AND python under Wine.
January 10, 2011 at 2:46 am
Hi,
The problem is probably getting the volume serial number. It is actually hard to do on most Linux boxes unless you are root and can run hdparms or sdparms.
If you look in k4mutils.py you will see I enabled environment variables to be used to override the username and serial number. If we do the same thing for k4pcutils.py (or some other override) methods, you may be able to tweak things to get it to work.
My 2 cents,
January 10, 2011 at 5:52 am
The Wine configuration allows you to actually SET the Windows volume serial. I’ve tweaked the scripts and I can see that the serial I configure is being used to attempt to build the different PIDs, but none of them seem to work.
Part of the problem, I think, is that you have to have the Volume Serial set BEFORE you install K4PC (or at least before you register it) for things to work properly. I don’t think you can just adjust the Serial # and re-download the books — but I could be wrong.
January 10, 2011 at 1:29 pm
Hi DiapDealer,
Yes, that makes sense. The first time you use and register K4PC it does check your Username and Serial Number to create the dsn and account token that is sent back. They need that information there in order to encrypt the file properly so that the reverse of the process gets you the book pid.
Have you tried unregistering and then reregistering after setting the volume serial number to the correct value and see if that works.
Strange….
January 10, 2011 at 5:32 pm
Yes, I’ve set the serial and then registered/installed K4PC with no luck.
Too be honest, I’m not sure there is a “correct” Volume Serial. Whatever you enter in the configuration is what any Windows application (including K4PC) will “think” the serial is.
The Wine configuration accepts a hex value for the Volume serial… the GetVolumeSerial function (in the scripts) returns the proper decimal equivalent.
I’ve tried entering a valid Windows Volume serial (copied from my Vista machine and converted to hex) and it is encoded/hashed exactly as it would be on the Vista machine. Same for the user name.
I’ve dumped the values in the kindle.info file, and all the values/names appears kosher. Which tells me the the Wine equivalent of crypt32.dll is functioning as expected.
It just seems that the DSN and AccountToken that gets sent to Amazon to encrypt the files is NOT the same as what the CryptUnprotectData function is returning from the kindle.info file (even though the values look fine).
I’m baffled… but not too concerned, since I’m not relying on K4PC files any more. 😉
January 7, 2011 at 5:19 pm
Still getting failed to determine book path even using v7….
January 7, 2011 at 5:40 pm
We might need just a tad more information in order to help you correct whatever problems you’re having. 😉
What Operating system are you using.
What tool/ebook-type are you trying to work with?
Command-line or GUI?
If command-line, exactly what command did you type?
What’s the “exact” error message you’re getting?
January 7, 2011 at 6:11 pm
Hi Marcelo,
Are you using unswindle by chance? If so, this was posted on the IHeartCabbages site at the very end of the newest comments:
Jason said…
For those who are getting the “could not determine book path” error, close the Kindle4PC app, and go to your task manager and kill the KindleDRM task. Then run the unswindle.py script.
We really do not maintain unswindle (I HeartCabbages) does. We maintain the other tools including K4MacUnswinlde, KindleBooks, etc.
You should be easily able to use KindleBooks on your Windows PC (or the K4MobiDeDRM Calibre plugin) in place of unswindle to remove the DRM from your books. The only real reason to use unswindle anymore is for Linux wine users to run Kindle for PC.
January 10, 2011 at 7:38 pm
Hi All Windows Users,
Testers needed!
I have put together an experimental DeDRM drag and drop tool for Windows. This is very experimental and only proof of concept. It still requires the user to have Python properly installed as well as PyCrypto or OpenSSL.
So if you have a Windows box with both of these installed, we really need your help testing things.
Download DeDRM_WinApp.zip from here:
http://www.datafilehost.com/download-75cdd2c7.html
Once downloaded move it to your Desktop. Extract the zip archive fully. Open the folder and inside you should see a DeDRM_Drop_Target bat file ShortCut and a folder DeDRM_lib.
To initialize the tool, simply double-click on the DeDRM_Drop_Target shortcut. It should open a Preferences screen that allows you to select your adeptkey.der, bnepubkey.b64, add in fixed pids, add in serial numbers, and even add in additional kindle info files (only needed if you have two Kindle for PC installations). Then hit the “Set” button.
You should be ready to go.
Try dragging and dropping any number of drm ebooks unto the DeDRM_Drop_Target shortcut file (you can also drop folders of ebooks).
You should see a window show up with an activity bar and a scrollable window to show the results so far.
It should create nondrm version in the same directory that your ebooks were dragged from.
Please try this and let me know what problems you run into.
Thanks
January 10, 2011 at 10:13 pm
Hi,
I’ve downloaded it, I’ve changed DeDRM_Drop_Target.bat so it points where launch_dedrm.bat is located. After that, the program finishes with some error. If I run it from the command line, the result is:
Traceback (most recent call last):
File “DeDRM_app.pyw”, line 523, in
sys.exit(main())
File “DeDRM_app.pyw”, line 481, in main
prefs_array = getPreferences()
File “DeDRM_app.pyw”, line 195, in getPreferences
import ineptkey
File “C:\Python26\ineptkey.pyw”, line 20
from __future__ import with_statement
SyntaxError: from __future__ imports must occur at the beginning of the file
I’ve run it in Windows 7 and all the other tools work.
January 10, 2011 at 10:22 pm
Hi Terisa,
Thanks for trying it. The import ineptkey should have loaded ineptkey.py not ineptkey.pyw but it doesn’t matter, they were exact copies of each other. The problem is the error message is very strange. Line 20 in that file is a comment. In fact all of the lines before the “from __future__ import with_statement” are comments so it is the first line.
Very very strange.
Oh well. Throw it out. It obviously is in need of a lot more work.
Thanks for reporting back what you found.
January 10, 2011 at 10:39 pm
Hi Terisa,
I just saw this …
> File “C:\Python26\ineptkey.pyw”, line 20
My code tried to import ineptkey.py from its own DeDRM_lib folder but in fact you have copied all/some of the previous tools python scripts and placed them in C:\Python26\ and these are being found and loaded before the latest ones.
In general I would recommend you keep all of your python code OUTSIDE of the C:\Python26\ directory. Otherwise, any time a changed version of the file is released in the tools, when you run them they will find the older versions in C:\Python26\ by mistake instead of their own copies unless you manually remove all of the older python python files and replace them with newer ones.
Your ineptkey.pyw is one such problem file. It should really be replaced with one that does have the line
from __future__ import with_statement
as the first runnable non-comment line in that file.
January 11, 2011 at 9:05 am
Hi,
Sorry, I didn’t remember those scripts. I’ve already moved them, I’m tried again (with the modified shortcut) and it has worked well for Kindle and ePub books (those are the books I have). By the way, thanks about the trick of drag and drop to a bat file, I didn’t know it.
January 11, 2011 at 1:31 pm
Hi Terisa,
You can throw out that first attempt. I retabbed the entire main program and have found a way for a bat file to actually act as a proper drag and drop target instead of a shortcut.
So version 2 of that software will be out tomorrow with the fixes and an improved bat file so that all the user should need to do is unzip it and put the application wherever they want. They can then drag and drop to the new bat file or even easier create a shortcut to it on their desktop and never have to worry where they install it.
I will make an announcement when version 2 is available. If that passes peoples test we can start to worry about how to either convert them to exe files or package them with their own interpreter.
Thanks again for testing.
January 12, 2011 at 12:06 am
Hi Windows Users,
We need Beta testers for the beta version of DeDRM_WinApp_v1.1.zip! (This version should actually work!)
We have put together a DeDRM drag and drop tool for Windows. This software is beta level. It still requires the user to have Python properly installed as well as PyCrypto or OpenSSL.
So if you have a Windows box with both of these installed, we really need your help testing things.
New with this version are a number of bug fixes, replacement of the short cut with a bat file that allows the application to be moveable. You can now move the DeDRM_WinApp_v1.1 folder anyplace you like and then create a shortcut to the DeDRM_Drop_Target to leave on your Desktop.
Download from here:
DeDRM_WinApp_v1.1.zip
http://www.datafilehost.com/download-81b2a8e8.html
Once downloaded fully extract the zip archive. Open the folder and inside you should see a DeDRM_Drop_Target (bat) file and a folder DeDRM_lib.
To initialize the tool, simply double-click on the DeDRM_Drop_Target. It should open a Preferences screen that allows you to select your adeptkey.der, bnepubkey.b64, add in fixed pids, add in serial numbers, and even add in additional kindle info files (only needed if you have two Kindle for PC installations). Then hit the “Set” button.
You should be ready to go. Try dragging and dropping any number of drm ebooks unto the DeDRM_Drop_Target. You can also drop folders of ebooks.
You should see a window show up with an activity bar and a scrollable window to show the results so far. It will create nondrm version in the same directory that your ebooks were dragged from. The first few runs will be slow while the python code inside is byte compiled. Speed should then improve.
Please try this and let us know what problems you run into.
January 13, 2011 at 10:48 am
Hi,
There is (I have) a problem when the document is not in the same drive as the App.
I think you should also add the change to the drive letter where the APP is (%~d0).
%~d0 && cd %PWD%\DeDRM_lib && start /min python
or
cd %PWD%\DeDRM_lib && %~d0 && start /min python
Thanks for all the tools.
January 14, 2011 at 4:07 pm
Hi Ramon,
I am not much of a windows user so please bear with me …
If you drag and drop a file from one drive onto a bat file on a different drive, the bat file should still be able to determine where it is located and from there invoke the right python script. Where the data file you are dropping on to it comes from should be captured properly by the path.
So your change is saying that when the dragged file comes from another drive, the current working directory of the bat file is not enough and that the drive letter of the bat file must be prepended for the working directory to actually work. Is that correct?
Why if I tell something to cd to C:\MYPATH does the cd command not take the drive info from the path itself? That must be a weird Windows specific path thing.
I will add it to the next release of the pure python version. It would be interesting to see if the binary version has the same bug. I am not sure how I would deal with that.
Thanks,
January 15, 2011 at 9:04 am
Hi,
I use windows 7, so I don’t know how it atcs in other windows.
I know in linux (and probably in mac) there are no drives, so I don’t know how it may afect to the script.
I made some tests, and when you drag-and-drop the bat file “is executed” in the drive where the document is (f.e. Y:) (and not in the drive where the document is (f.e. C:).
To change from Y: to C: you only have to input “C:”.
In this case, using the “%~d0” “command” is enough, since “%~d0” is the “letter:” of the drive where the script is.
Regarding your question, if you input “CD C:/mypath/”, you are changing the working dir in C (so if you afterwards change to C:, you will be in “c:/mypath/”), but you are still in “Y:”.
Let’s say that in windows every drive has its own “working dir” stored somewhere, that you can change, but you are only working in one of those “stored” working dirs. Then, when you change from one drive to another, then you go to the working dir of that destination drive (and not to the root dir).
Maybe I have been too cryptic.
January 15, 2011 at 3:59 pm
Hi again,
I have made some more investigations, and in windows the /d parameter also changes the drive.
So using /d in cd:
cd /d %PWD%DeDRM_lib && start /min python DeDRM_app.pyw %*
also works, and no %~d0 is required.
I have tested using cd /d in linux but it gives an error, so I don’t know the best option for the script to run in linux as well as in windows (maybe some if/else condition) in the batch file.
January 15, 2011 at 4:36 pm
Hi Ramon,
Thanks for the explanation. I will use the /d switch for the official release. The Linux version will use a bash shell script (or no script at all) to launch. I only need the .bat for Windows.
Thanks,
January 13, 2011 at 5:15 pm
I had a chance to test DeDRM_WinnApp, and it worked perfectly on Kindle and B&N epub, but failed on Adobe DRM. I had success with an Adobe epub using ineptepub_v5.5, but the same book failed to convert using DeDRM_WinnApp. I am using the same adeptkey.der for both utilities, so I don’t know why this would not work.
Thanks for all of your great work!!! I just had to replace my Kindle, and because of your tools I am able to just copy my books over to my new device, rather than re-downloading my whole library.
January 13, 2011 at 6:00 pm
Hi Montalex,
Thanks for testing it and reporting back. I will look into the epub code and see why it is messing up. I probably am not passing the right path to the key to ineptepub.
Thanks,
January 12, 2011 at 3:21 pm
Hi,
Feedback anyone on DeDRM_WinApp_v1.1???
Did it work for you? Does it work well enough to continue development? Do Windows users even use Drag-and-Drop interface tools? They are quite common on Mac OS X machine, but perhaps there is no demand for such a tool on the Windows side of things?
Are we barking up the wrong tree?
Any feedback at all about its layout, ease of use, interface, usefulness, etc from Windows users would be welcome so that we know if we need to take this a step further or just give up on it.
Thanks,
January 12, 2011 at 8:50 pm
Hi,
I’ve checked against ePub files, B&N ePub files, Mobi Kindle files and Topaz Kindle files (sorry, no lit or pdb 😉 ) and it has worked well at any case. And I like a lot the drag and drop method 😀
Best regards
January 12, 2011 at 9:37 pm
Hi Terisa,
Thanks again for testing this updated version. I am glad it is working well for you. So we have 1 vote for DnD and 1 vote against DnD. Hopefully others will chime in with their thoughts.
ps. lit is not supported – sorry about that. As a longtime Mac and Linux user I never ever purchased a lit book and the solution to remove the drm seems to require a Windows box.
Perhaps if we could attract a Windows developer to help out here, we could add support for that format as well.
Thanks again,
January 14, 2011 at 3:49 pm
Hi Terisa, Ramon, Montalex, Kosst,
There is one more version to try, if you would. This version has been compiled to a binary version but it does not have an installer. So download it and extract it and move it anyplace you like.
Unfortunately, right now, to actually use it you have to create a shortcut from the “DeDRM_app.exe” inside the downloaded folder (there will be a bunch of other exe, dll, and pyd files inside the folder) to your Desktop. Then drag and drop away or double-click to set your preferences.
if anyone has a machine without Python on it testing it there as well would be an added bonus.
Please don’t move anything out of that folder as it will cause it to barf and die. The compressed zip is over 9 meg in size.
I personally can not see why anyone would want to run a binary when they can just install ActiveState Python 2.7 win 32 free community edition and run from the source scripts and get the added advantage of running other python based programs as well.
The only advantage of this version is that it should work on any Windows box even those without Python, Pycrypto or OpenSSL and it has a minor bug fix.
Please give a whirl and let me know if it works for you or not. If so, once we get a volunteer Windows developer, they can take over what we have designed and improve it, add an installer, etc.
We will include a pure Python version of this tool in the next tools release but we really don’t want to get in the business of adding binaries to the tool sets (both for download size and safety reasons). That is why we are looking for some Windows person to volunteer to help out and host and support a binary version if enough people seem to like the idea (and if it actually works).
Here is the link:
http://www.datafilehost.com/download-3f036c95.html
Please do not advertise this link to your friends as this is still an experimental build and the download link will go away in a couple of days.
If any of you know of any Windows based developers who might be interested in helping out here or at Alf’s, please let us know.
Thanks,
January 15, 2011 at 7:52 am
Hi,
I’ve used the tool for ePub, B&N epUB, Kindle PRC and Kindle Topaz and it has worked. Python is installed at my computer, however. It works well but I think I’ll go by the other version 😉
January 12, 2011 at 4:22 pm
I use the tools all the time but I couldn’t get this to work for some reason, just got the failure message. I selected by bnepubkey.b64, selected and used it on a B&N book. Failure.
But it’s still a beta and a great effort.
As for your last question, being an avid Windows user I don’t use a drag and drop like that much and as a feature I’d probably prefer just a menu select. I very much liked all the tools being in one and the interface was great for that.
Finally (and I don’t even know if this is possible) is it possible to make this portable? Preferably where I don’t have to take the Pyton Lib with me.
January 12, 2011 at 4:51 pm
Hi Kosst,
Thanks for trying it. Exactly what error message did your receive? The exact error message should appear in the Log window. If not you can find a full log file in your Local Appdata in the DeDRM folder there. It seems some B&N epubs are created and stored improperly (not to epub standard) and so give the software some issues. We are working to fix that B&N bug right now.
So info on the exact error message would help.
Thanks also for your feedback on Drag and Drop in general. You would prefer an application that you double-click and it asks you for one book or a single folder and then it goes to work from there? Is that correct?
As for portable on your own system, you can move that folder anyplace you want and it should still work. You can simply create a shortcut to the De_DRM_Drop_Target and put it on your Desktop.
As for moving it to machines that do not have Python installed? Python is universally installed on every Mac OS X and Linux box. Downloading and installing ActiveState’s Active Python 2.7 for 32 bits is easy and painless for Windows boxes. There are also simple binary installers for OpenSSL and PyCrytpo for Windows users. So I am not sure why Windows people seem so afraid of Python in general and installing these packages. Windows users actually do that every time they download and install Calibre!
That said, there are tools to compile python scripts to stand-alone exe applications, but the resulting exe files end up being huge in size. For example, the compressed size of DeDRM_WinApp_v1.1.zip is under 100k bytes (tiny). But a simple hello-world program in python compiled with py2exe ends up being close to 6 meg in size. So we might be talking 50 meg or more to have a standalone version of DeDRM_WinApp! Plus it is a real pain to have to compile them using py2exe (that tool is not well documented) and it has a tendency to need a lot of support dlls.
Or simply keep everything as python scripts and include in the DeDRM_lib a version of Python 2.7 itself (this is what Calibre does).
It is sad that modern Window’s boxes do not include support for industry standard scripting languages like Python, Perl, Ruby, PHP, etc as part of their default install.
Thanks again for your feedback. If enough people like it or want it, we can try to use the py2exe approach and see how big the damn thing turns out to be.
January 12, 2011 at 6:46 pm
Alright here’s the log:
***
Processing: filename.epub … Failed
‘python’ is not recognized as an internal or external command,
operable program or batch file.
***
Like I said I use the tools and even the Calibre plugins with no problems. Let’s see…you are correct on the GUI, it’s just more common then drag and drop. On being portable, I meant putting it on a flash drive and giving it to others, I use portable Python so its not an issue for me but i know plenty of windows users who’ve never even heard of it. I’m an occasional Linux user so I’m not scared of it but its not that common for Windows users have to deal with that programming language.
Finally as to compiling it all into a single application, my Python folder is about 50 MB and I only use it for DeDRMing files. So while there would be no saving space there would be a huge jump in portability (also providing some internal saving of the various keys) not to mention ease of use for beginners.
January 12, 2011 at 7:35 pm
Hi Kosst,
That error means you have not added Python to your system environment path properly.
You can test this by running cmd.exe.
Once it opens simply type python and hit return. You will get the same error message. Then try typing python.exe and python should start up.
The reason the tools work is that we check for “Python” being in the PATH environment variable or not that and then try to work around the broken installations.
Oh well, I guess I will add the same trick to these files.
Thanks again for testing it and the feedback.
January 12, 2011 at 7:29 pm
Hi,
I am having trouble decrypting a tpz file. I use windows 7 x64, have the latest version of the tools and calibre. I tried adding the file to calibre using the command line but there is some error taking place.
Here is the log:
http://pastebin.com/BXVagArJ
Any help would be appreciated. Thanks.
January 12, 2011 at 7:46 pm
Hi Sebastian,
Unfortunately, that Topaz book has a bit of xml that the topazextract parser has never seen before and that confuses it and it barfs trying to parse something it does not understand.
If you want me to fix this bug I would be happy to since it will benefit others.
I need you to post the original file someplace for me to grab and post me the link.
I will grab it and run the tools and figure out what new xml exists and add support for that in in a new Python file that you can use and test by overwriting the file of the same name in the DeDRM_lib.
Once we have the fix verified and tested, we can add the bug fix to the next release of the tools.
Thanks,
January 12, 2011 at 7:52 pm
Great, thanks.
Here is the book: http://dl.dropbox.com/u/4478637/Book.azw
Thanks again
January 12, 2011 at 8:43 pm
Hi Sebastian,
Yes it was a new xml tag interestingly enough called .
I was able to change the parser to understand it.
I have posted a new version of a python script called “convert2xml.py”, here:
http://pastebin.com/qXccHR2W
Hit the Download link at the top and then rename the downloaded file to convert2xml.py.
Copy it and move it into the DeDRM_lib folder so that it replaces the file of the same name.
Then try the DeDRM tool.
If it succeeds you should end up with 3 zip archives. You should be able to convert the _nodrm.zip version to a Mobi format by importing it as a zip file into Calibre and converting to Mobi or epub.
Since this book has lots of equations and things and the OCRtext Amazon has is not perfect, I would also keep the _SVG.zip archive.
You can unzip the _SVG.zip archive and open the index xhtml file in any good browser to check exactly what a specific page of the book should have looked like. I would especially check equations and things to make sure they did not get messed up.
Please let me know if this works for you. If so, we will include it as part of the next release of the tools.
Thanks again for the bug report and helping us track it down.
January 12, 2011 at 11:20 pm
Hi,
I tried the new file you sent and it works great.
As you said though, the conversion isn’t as clean as the one on azw files.
So, I have a follow up question. Is it possible to have a tpz file with no DRM? I understand that only kindles read tpz files, but I wonder whether it is possible to have a tpz file that could be opened in several kindles.
Thanks again for your help.
January 12, 2011 at 11:46 pm
Hi,
It is by we don’t support that format because it can only be used to pirate books. You might want to search inside the tools 3.1 zip for a cmbtc_v2.2 python script.
January 12, 2011 at 11:51 pm
Thanks again for all your help.
And I do understand not supporting piracy. Just for your inner peace, you are not helping a pirate but a guy with 4 siblings. We all have kindles and read books together. Unfortunately, some books have restrictions and can only be downloaded for 4 users and because of the system, I can’t even lend them.
After playing around with the DnD version, I am not sure if I am for it or against it.
I personally like it because it is easy for me to find the ebooks in my computer and also because I have 2 screens so dragging isn’t a problem.
But for people that have trouble finding, for example, the My Kindle Content folder, an app like LZKindle that opened K4PC and allowed them to select the book to unlock might work better. Also, I don’t mind DnD when I am sitting in front of my desktop but I hate doing so when working with a touchpad and smaller screen in my laptop.
But I do like the app and hope you find that Windows developer! Thanks again and good luck.
January 13, 2011 at 12:24 am
Stripping DRM out of Topaz doesn’t necessarily means piracy. I have an American and a Brazilian account. I can use both on my I-devices, but I only use the Brazilian on my Kindle. That means Topaz books I purchase (or download as freebies) can’t be read on the Kindle.
January 13, 2011 at 2:04 am
Hi Rafael,
No it means making copies of a book that can only be read on Kindles (or Kindle for something) and can not be converted to any other format. Our tools are to support format conversion and archiving by the legitimate owner. There are special cases but …
January 13, 2011 at 2:33 am
Some Updates, I know that, and I have already come to terms on not having all my books on the kindle. Just wanted to point out that it is a tool I would benefit from, and not pirating.
Anyway, I try not to judge, that’s what consciences are for. Too bad so many people lack them. Our loss.
Keep up the great work!
January 13, 2011 at 3:49 am
The post above with http://pacilio.net/kindle/kindledrm.zip will deDRM Topaz.
January 13, 2011 at 6:15 am
Thanks Davo. That’s very helpful
January 12, 2011 at 7:34 pm
By the way, I used the Drag-n-drop tool and it shows the same error I pasted above.
January 13, 2011 at 1:35 pm
I am completely new to this un-drming Kindle azw’s. I got pissed off because I was reading Christopher Hitchens’ The Portable Atheist which has loads of great quotes and I assumed all of my highlights were being copied to my clipping file. After finishing the book I saw that a clipping limit had been reached, and though my highlights were there the majority of my clips were not. I was furious. So I started looking for tools to remove the clipping limit. After an afternoon I finally stumbled on this blog which was current – many of the tools out there were out of date and did not work for my Kindle 3 (3G + Wi-Fi), gprs).
I was already using Calibre (latest version as of now) to manage my eBook library (all my free books) and Sigil to edit them, correct or enhance them. So I downloaded and installed Python 2.6.4 and pycrypto 2.1.0 for Windows 32 bit. I run Vista. The installation of Python and pycrypto went very smoothly – I merely clicked my way through all the default installation settings.
I unzipped the DeDRM_WinApp_v1.1.zip into its own directory and dragged an azw file from my “Documents\My Kindle Content” directory (I already had Kindle for PC installed and running), and found out that “python” was an unknown command from an error immediately given by the drag-and-drop batch script. I had double-clicked on the batch file to initialize the utility and was presented with a form asking for all sorts of things. I did not have my Kindle with me and had read somewhere on this blog that the utility in any case only works with files downloaded by Kindle for PC, which do not need a PID or serial number specified, so I filled in nothing and merely clicked “Save”.
I realized I needed to add the location of “python.exe” to the PATH environment variable, which I did by opening “Computer | (right click) Properties | Advanced system settings | (button) Environment Variables… | System Variables” and editing the “Path” variable to include “C:\Python26\;” right after the system directory paths. No sweat. Then I opened a command window (“Start Button | type “command” and press Enter”) and typed “python” which found and opened the python console. I Pressed Ctrl-Z and Enter to exit Python. I was now set to try the drag-and-drop again.
I dragged an azw file onto the “” file and the utility reported success! Feeling somewhat bolstered I dragged the entire “Documents\My Kindle Content” directory onto the batch file, and waited with bated breath. It took a while (Lord of the Rings was included in my modest list of a dozen files) but in the end every single one of my azw files had its accompanying “_nodrm” version. Hooray!
I imported them into Calibre and was able to read all of them with the native Calibre reader (but only after adding a .mobi version because Calibre kept on using Kindle for PC to open the azw-only files).
So, great utility! I immediately used Calibre to add better cover images and enhance various meta-data for the books (reviews, synopses, etc.). I can now store my Kindle purchases in Calibre and read them on any device I choose! I think you guys have done a brilliant job, and I am grateful for your ongoing efforts in development and for keeping this blog.
One question: one of the books was de-drmed into three files! They were an “_nodrm.zip”, an “_SVG.zip”, and an “_XML.zip”. What was that about? I imported the “_nodrm.zip” into Calibre and converted it to .mobi, so that was no problem. I can read the .mobi format book fine in Calibre. This book had large graphic images that served as the first letter of each chapter and section headings that were also in fact images, not letters. Did that have something to do with it?
OK, one more question: how likely is it that my Kindle for PC (currently version 1.2.0, Aug 16 2010) will not allow de-DRMing if I update it to the latest version?
January 13, 2011 at 1:42 pm
IGNORE THE CRAP ABOVE. I WILL SHORTLY POST THE SAME WITH CORRECTIONS!
January 13, 2011 at 1:43 pm
CORRECTED AND UNOBFUSCATED VERSION OF PREVIOUS POST!!!
I am completely new to this un-drming Kindle azw’s. I got pissed off because I was reading Christopher Hitchens’ The Portable Atheist which has loads of great quotes and I assumed all of my highlights were being copied to my clipping file. After finishing the book I saw that a clipping limit had been reached, and though my highlights were there the majority of my clips were not. I was furious. So I started looking for tools to remove the clipping limit. After an afternoon I finally stumbled on this blog which was current – many of the tools out there were out of date and did not work for my Kindle 3 (3G + Wi-Fi).
I was already using Calibre (latest version as of now) to manage my eBook library (all my free books) and Sigil to edit them, correct or enhance them. So I downloaded and installed Python 2.6.4 and pycrypto 2.1.0 for Windows 32 bit. I run Vista. The installation of Python and pycrypto went very smoothly – I merely clicked my way through all the default installation settings.
I unzipped the DeDRM_WinApp_v1.1.zip into its own directory and double-clicked on the “DeDRM_Drop_Target.bat” batch file to initialize the utility and was presented with a form asking for all sorts of things. I did not have my Kindle with me and had read somewhere on this blog that the utility in any case only works with files downloaded by Kindle for PC, which do not need a PID or serial number specified, so I filled in nothing and merely clicked “Save”. I then dragged an azw file from my “Documents\My Kindle Content” directory (I already had Kindle for PC installed and running), and found out that “python” was an unknown command from an error immediately given by the drag-and-drop batch script.
I realized I needed to add the location of “python.exe” to the PATH environment variable, which I did by opening “Computer | (right click) Properties | Advanced system settings | (button) Environment Variables… | System Variables” and editing the “Path” variable to include “C:\Python26\;” right after the system directory paths. No sweat. Then I opened a command window (“Start Button | type “command” and press Enter”) and typed “python” which found and opened the python console. I Pressed Ctrl-Z and Enter to exit Python. I was now set to try the drag-and-drop again.
I dragged an azw file onto the drop target and the utility reported success! Feeling somewhat optimistic now, I dragged the entire “Documents\My Kindle Content” directory onto the batch file, and waited with bated breath. It took a while (Lord of the Rings was included in my modest list of a dozen files) but in the end every single one of my azw files had its accompanying “_nodrm” version. Hooray!
I imported them into Calibre and was able to read all of them with the native Calibre reader (but only after adding a .mobi version because Calibre kept on using Kindle for PC to open the azw-only files).
So, great utility! I immediately used Calibre to add better cover images and enhance various meta-data for the books (reviews, synopses, etc.). I can now store my Kindle purchases in Calibre and read them on any device I choose! I think you guys have done a brilliant job, and I am grateful for your ongoing efforts in development and for keeping this blog.
One question: one of the books was de-drmed into three files! They were an “_nodrm.zip”, an “_SVG.zip”, and an “_XML.zip”. What was that about? I imported the “_nodrm.zip” into Calibre and converted it to .mobi, so that was no problem. I can read the .mobi format book fine in Calibre. This book had large graphic images that served as the first letter of each chapter and section headings that were also in fact images, not letters. Did that have something to do with it?
OK, one more question: how likely is it that my Kindle for PC (currently version 1.2.0, Aug 16 2010) will not allow de-DRMing if I update it to the latest version?
January 13, 2011 at 2:24 pm
Hi,
> one question: one of the books was de-drmed into three files! They were an “_nodrm.zip”, an “_SVG.zip”, and an “_XML.zip”. What was that about?
See this link for a description of what the 3 zip files are all about:
> one more question: how likely is it that my Kindle for PC (currently version 1.2.0, Aug 16 2010) will not allow de-DRMing if I update it to the latest version?
I did a clean install of Windows 7 and installed the latest Kindle for PC from yesterday and all worked. Of course, no promises on later versions.
January 13, 2011 at 2:44 pm
I have opened the tools zip for the first time and see that under the Kindle tools the readme explains all about Topaz files and aswers my question about the three zip files. I will check out your link, too, thanks. Quite a steep learning curve but not beyond the pale 🙂
Good news on the K4PC: I will upgrade to the latest version then, thanks.
January 15, 2011 at 1:16 am
Works for Adept epubs although I do still have Python installed. Great tool (half the size of my Python installation) but I’m still not used to the drag and drop but I can live with it.
January 15, 2011 at 5:26 am
Hi Kosst,
Glad to here the binary version works. Thanks for testing it.
January 18, 2011 at 5:25 am
Hello again,
Here is another TPZ book that I haven’t been able to deal with. I have tried both the using the DnD Kindler and also the Python script. But it hasn’t worked.
Here is the log:
http://pastebin.com/Pw8Gi2VY
And here is the original book:
http://dl.dropbox.com/u/4478637/Book.tpz
Any help would be great. Thanks a lot!
January 18, 2011 at 12:40 pm
Sebastian:
The book you posted is not encrypted. It opens with Kindle for PC.
January 18, 2011 at 3:34 pm
Hi Sebastian,
Yes, Julian is correct. That book has been passed through skindle or some other program and the DRM has already been removed. Unfortunately, the record offsets were messed up during the removal process (there are minor bugs in skindle that can hit and mess things up) and the table of offsets in the file is no longer correct. It appears Kindle for PC is either robust enough to ignore these inconsistencies or you simply haven’t hit the page yet that triggers the bug (K4PC reads data for only the current page at one time). When you try and convert it, data from all pages and all glyphs must be read and one of them is not correctly specified. Your only choice is to go back and find the original DRM version of the book.
January 18, 2011 at 7:37 pm
You are absolutely right. Thank you very much Julian.
January 18, 2011 at 7:39 pm
Thanks to both (I posted the comment above before I was done typing). I was making a mistake and what I thought was the original file was actually a file that had been put through skindle.
I downloaded the original file and DeDRM had no problems handling it.
January 18, 2011 at 1:10 pm
Hi All,
Given Alf’s decision to retire, I thought it best to get the latest versions of everything packed up into “tools_v3.2.zip”.
You should grab it from here:
tools_v3.2.zip
http://www.datafilehost.com/download-92989482.html
New in this release:
– new DeDRM_WinApp_v1.2, a pure Python Drag and Drop Tool for Windows that offers the same functionality as the DeDRM 2.0.app that Mac Users have available, Use the README to install it properly.
– new version of Mobidedrm 0.26, which now resetting of clip limits to 100% and resetting of the TTS to enable it if present in the book. Also includes support for Type 1 encrypted TextRead and BOOKMobi style books
– new version of KindleBooks that uses the new MobiDeDRM and that now names books according to their title where possible so that users can figure out which B0* book is which. Includes a fix for a new tag called _tag for Topaz books.
– new K4MobiDeDRM calibre plugin which includes the new MobiDeDRM and the Topaz bug fixes. PLEASE REMOVE ALL EARLIER VERSIONS.
– updated to include Alf’s latest DeDRM 2.0.app for Mac OS X
– bug fixes for ePubFixer to actually handle B&N epubs without a proper mimetype set correctly
This archive includes all of the most recent versions of all of the files just so everyone can get access to them in case no further tool sets get posted.
January 18, 2011 at 8:24 pm
Thank you very much! It works like a charm and it gets well the name from a Kindle book. And yes, I think it’s good to get this version. It’s a pity Alf’s decision but it’s normal because of such “notoriety”.
January 22, 2011 at 5:55 pm
Well, I guess we should have expected it. The kindle plug-in is no longer working. I got some books from Amazon (AZW) and they imported just fine. Since all the notoriety, I decided to try and convert them to epub. Got “this book has DRM”. Tried downloading the book to kindle4pc – same result. Tried to convert to mobi – same result.
Luckily, the pdf and epub plug-ins are still working. Congratulations Amazon – no purchases from you until I can undrm after purchase.
January 22, 2011 at 6:09 pm
I suspect you might have multiple versions of the Kindle plugin installed, or else you didn’t delete the old (drmed) ebook from calibre before re-importing with the filter in place.
To confirm that the dedrm still works, give it a go with the KindleBooks stand-alone tool.
January 22, 2011 at 6:38 pm
Hi DLR,
FYI: Just tried the latest Kindle for PC with the K4MobiDeDRM calibre plugin on Windows 7 and it worked with no problems for me.
As Paul suggested, maybe multiple versions of K4MobiDeDRM plugin is confusing things, or your .kindle-info has changed or gotten corrupted. Did you upgrade Kindle for PC lately? Any other recent changes?
January 22, 2011 at 7:18 pm
Thank you, gentlemen. I know you said REMOVE PREVIOUS VERSIONS, but I figured if I didn’t upgrade to 3.2, I could leave mobidrm 8, and all the other old stuff in Calibre. Well, I cleaned everything out (still using pdf and epub from versions and versions ago – if it ain’t broke) and it works just fine! Thank you so much – i probably would have wasted most of today fooling around with it. When I should be reading, so I meet my goal of 100 books in 2011! Have a great weekend, Debra
January 18, 2011 at 2:01 pm
All code was removed from pasteBin!
Need new links!
January 18, 2011 at 2:04 pm
So is he going to shut down the site, do we need to find another?
January 18, 2011 at 2:17 pm
He says he gave the password to some other people, so I guess it’s their decision as to what happens to it now.
My guess is that it will live on in the comments, rather like this blog has!
January 22, 2011 at 7:57 pm
Can anyone tell me how to load this page in under 10 minutes? I am running Windows 7, and after 10 minutes I get box asking me if I want to disable a script which is making it run slow. I say yes, but there is no option to permanently disable it.
January 22, 2011 at 8:24 pm
I’d suggest subscribing to the comments RSS feed so all you see is new comments as they come in, rather than trying to wade through the morass or wait 10 minutes for the page to load.
January 22, 2011 at 8:52 pm
Thanks, I did. But I still have to load the page to comment. But it is worth it – thank you everyone!
January 25, 2011 at 11:04 pm
I’m running Windows 7 64. The page loads in about 30 seconds. It sounds like you are running the add on no script in firefox.
February 2, 2011 at 8:18 pm
Hi All,
A pure bug-fix release of tools_v3.3.zip is now available:
tools_v3.3.zip
http://www.datafilehost.com/download-3a9f67eb.html
Changes in this version:
1. internally synchronized file versions across the many tools to prevent as much code duplication as possible
2. any incorrect epub zip archives input are now fixed during the conversion process in the ignobleepub and ineptepub Calibre plugins
3. fix for using alternative kinfo files on Mac OS X for KindleBooks and K4MobiDeDRM Calibre plugin
4. fix for Topaz ebooks that use blank fixed regions as spacers to force page breaks. This fix impacts KindleBooks, K4MobiDeDRM Calibre Plugin, DeDRM 2.1.app and DeDRM_WinApp)
5. fix typo and add creation of a last.log feature to help improve debugging of errors for DeDRM 2.1.app
6. fix to use underscores to replace spaces in filenames to make it easier to use other command line tools. This fix impacts KindleBooks, DeDRM 2.1.app, DeDRM_WinApp
7. fix for missing failure messages and longer conversion dialog for long file names; restructured to allow apps location to be determined internally to make running it easier on Windows for DeDRM_WinApp
—-
This is a bug fix release, barring new major-bugs (or major updates from IHeartCabbages, Dark Reverser, etc) , tools_v3.3.zip should be the last version of the tools archive needed. They all work, are reasonably robust, and are near feature complete. Help is still available here as well as the ApprenticeAlf site.
I can’t seem to say this enough: Please ONLY use these tools for converting your personal ebooks between platforms, enabling TTS, and for making versions for archival purposes. Please do not steal ebooks! We want authors, publishers, and bookstores to all make a profit so that more great books/ebooks are forthcoming.
February 6, 2011 at 4:53 pm
I downloaded the latest tools package and generated the key for the bnepubkey. When I try to use it I get a “key cannot be the null string” error. What am I doing wrong?
February 6, 2011 at 5:22 pm
Hi,
So you used the ignoblekeygen_v2.3.pyw script, is that correct? If not, that is the only keygen program that works anymore. The older ignoblekey.pyw no longer works (and hasn’t for a long time). Did you remember to input your name on your credit card (and double-check it to make sure it is an exact match to your account info on the B&N website), and your credit card number (not just the last 8 digits, the whole thing unfortunately).
What size is the bnepubkey.b64 file? Perhaps the file was not properly created?
February 7, 2011 at 4:30 pm
Thanks for the tips. Two things I think I did wrong: I only used the last 8 digits of the credit card and I may not have used the v2.3 script. I just deleted the old versions and installed the tools v_3.3 on my home computer. My office computer still has the old scripts on it and it works fine. I’ll check the script version and try the full credit card number. Happily, I still have all the old version of the tools.
February 8, 2011 at 10:03 pm
Hi All,
Unfortunately, yet another bug fix release is needed
tools_v3.4.zip
http://www.datafilehost.com/download-385939ba.html
Bugs fixed
1. updated zipfix.py to add in mimetype file if missing in an epub
2. bug fix for ineptpdf when len(bookkey) == length + 1
3. KindleBooks fix for Topaz ebooks with glyphs that have 0 for dpi
4. KindleBooks fix for Topaz books with &,>,< chars in the Title/Authors
5. fix for initialization bug in DeDRM 2.1.app which prevented its use by new users who had not already added the configuration information
As a result the following have all been updated:
– DeDRM 2.2.app (Mac OS X: DeDRM Drag and Drop Tool)
– DeDRM_WinApp_v1.4 (Windows: DeDRM Drag and Drop Tool)
– ePub_Fixer
– Adobe_PDF_Tools/ineptpdf_v7.9.pyw
– KindleBooks
– Calibre Plugins
– ineptepub_v01.6_plugin.zip
– ignobleepub_v01.5_plugin.zip
– ineptpdf_v01.3_plugin.zip
– K4MobiDeDRM_v02.3_plugin.zip
February 17, 2011 at 2:08 pm
Have latest K4PC, Calibre, and tools v3.4. Neither the Calibre plugin nor the DeDRM drop are working for me. Both fail with an error in line 252 in getPIDMetaInfo in mobidedrm.py, KeyError: 148.
February 17, 2011 at 2:42 pm
Hi,
Yes, we have seen similar repots. Please try with the test version of the Calibre plugin K4MobiDeDRM (version 2.4) that can be found here:
http://www.datafilehost.com/download-4dd9dd4e.html
Please remember to remove the older plugin and to hit “Apply” after installing the newer version in Calibre.
Then try again and let us know if this fixes your issue.
February 17, 2011 at 3:31 pm
It works! Fantastic. Thank you very much. I also tried copying the python files over the same ones in the DeDRM drop target lib folder and that effectively fixed that so it now works as well. Thanks!
February 17, 2011 at 4:55 pm
Hi All,
Yet another bug fix release.
tools_v3.5.zip
http://www.datafilehost.com/download-d8b42a6e.html
It seems that Amazon recently changed something in how it uses metadata in its Kindle ebooks, that triggered a bug in the MobiDeDRM code. This bug has been tracked down and fixed. This change in MobiDeDRM means the following are updated:
KindleBooks (GUI Kindle ebook Tool)
K4MobiDeDRM_v02.4_plugin.zip (Calibre Plugin)
DeDRM 2.3.app (Mac OS X Drag and Drop Multi-Tool)
DeDRM_WinApp_v1.5 (Win Drag and Drop Multi-Tool)
MobiDeDRM_0.27.py
Hopefully this will be the last update needed (famous last words?)
February 18, 2011 at 3:50 am
I just downloaded this and it worked like a charm! Thank you so much!!!
February 18, 2011 at 2:01 pm
Hi,
Thanks for your hard works. I have used tools_2.3 before and now tools_v3.5. I have just tested the new version with 1 Topaz ebook from Amazon, and I noticed that the new version does not work well as the old one.
Topaz ebook to be tested: http://www.amazon.com/dp/B001JEPVZU
I used tools_v2.3 to convert to HTML and then used Mobipocket Reader to convert to PRC file. With tools_v2.3, the resulting PRC file is 12MB in size and everything is OK. But with tools_v3.5, the resulting PRC file is 43MB and some page in the ebook was not formated correctly.
And I think there should be something wrong in the tools_3.5. Please correct me if I am wrong.
Sorry for my bad English.
February 18, 2011 at 4:48 pm
Hi,
Please try again with the latest tools (3.6) and let me know if that helps. They only real change other than bug fixes and things like that has been to use svg images for fixed regions. This increases the file size but more acurately converts the book. You can convert all of the svg images to png at screen resolution if you want to reduce the file size back to close to what it was previously.
Also, if the OCR conversion is not good, simply use the svg images to create an image only pdf.
February 19, 2011 at 4:22 pm
Hi,
Thank you for your response and the new version. I have tried the new one (v3.6), but the resulting PRC file is the same in big size (43MB vs 12MB) and some error in the ebook content. If you do need more information, I will post some screenshots. Thanks for your concern.
February 19, 2011 at 6:39 pm
Hi Ho,
How did you get a PRC file from a Topaz ebook conversion? You must be using the Calibre plugins and converting it within Calibre?
If so Calibre probably has a hard time converting some svg images resulting in errors. If so, this is a calibre issue not a conversion issue.
As I said, the increased file size is **not** a bug. It is a result of using internal svg images for fixed regions in the html version. This is exactly what the official Kindle reader does with fixed regions and it reduces the number of errors.
These extra svg images are nice because they can be incorporated directly into epubs and be perfectly rescaled to fit the screen of the device. Unfortunately, mobi/prc books do not support svg and so calibre attempts to auto convert these images which can cause problems. They do take up more disk space but if really needed they can be converted to fixed size png files and downsampled to screen resolution to get back that space. You can use any image tools (inkscape for example) to do the conversion and downsampling.
No Topaz conversion is perfect. There are lots of OCRText errors in some books. The conversion can do nothing about that. That is why the DeDRM* and KindleBooks tool produces both the _nodrm and SVG zip archives (as well as an XML zip archive).
So try with KindleBooks instead of the calibre plugins and then unzip both the _nodrm and _SVG zip archives and take a look at the files inside with any good web browser (Firefox for example or Safari). How do they show up in the browser? Do the pages show any new errors not in the older version? They should not. Open the SVG .xhtml files with a web browser and look at the exact images of the pages to check things.
If the html version has too many bugs use the _SVG output and create an image only pdf file from it if you want exact page copies of the book.
Hope this explains things.
February 20, 2011 at 2:15 am
Hi,
I just used the Mobipocket Reader and Import -> HTML Document to produce PRC file from HTML file (book.html).
I know that the increasing in file size is not a bug, however with the “same same” content, if the resulting file size is too big, it’s also a problem. For example, with the Topaz ebook I have mentioned above, the original file size is 3.5MB, the PRC file when used tools_v2.3 is 12MB – which is acceptable, and 43MB when using tools_v3.5 or tools_v3.6.
Futhermore, some contents are missing in HTML file when using tools_v3.6. I will show you some screenshot:
– I used the above ebook (“Essentials of UMTS”) as the input file.
– Here is the SVG image: http://img189.imageshack.us/img189/9232/svgimage.jpg
– Here is the HTML content by using tools_2.3: http://img713.imageshack.us/img713/7224/htmlv23.jpg
– And here is the HTML content by using tools_3.6: http://img403.imageshack.us/img403/2504/htmlv36.jpg
As you can see in these screenshots, “Equation (4.2)” is missing in the HTML content that produced by using tools_3.6, and it still exists in HTML file produced by tools_v2.3 with some OCRtext errors, that are acceptable.
Lastly, a good news is the lastest 3.6 version can handle some Topaz ebook that cannot handle by older versions (3.5 and before).
February 20, 2011 at 3:04 am
Hi Ho,
Again, you can control the size difference by simply controlling how the *.svg files in the image directory are converted.
As for the missing info, that must be a bug but I have to see exactly what the xml file says to be sure. So please either **privately** post the original Topaz ebook and the proper BOOK pid OR **privately** post the entire _nodrm.zip, _SVG.zip, and _XML.zip archives someplace and pm “KevinH” on http://www.mobileread.com/forums with the link or links. He will pass them along to me and I will try to see how what is broken and try and fix it.
February 20, 2011 at 4:52 am
Hi,
Thank you for quick response. I will upload the files you need soon and will send links to KevinH on mobileread.com forum.
One more question, in order to control the size of the resulting ebook, how can I control the *.svg files to be converted? (as you have said)
Thanks
February 20, 2011 at 5:29 am
Hi Ho,
Any increase in size i because of svg images that represent fixed regions in the book. To reduce the size, unzip the _nodrm.zip archive and look in the img directory for PageXXXX_YYYY.svg files. Each of these files represents “fixed regions” in the book that have been converted to svg images.
If you do not want to convert to epub, you can use many different software packages to convert these images from svg to png while reducing the resolution (dpi) to the screen resolution. One such program is “Inkscape” it is a free download (see http://www.inkscape.org). It is a gui program that supports a command line interface. So it is quite easy to create a .bat or .sh file that converts each of these svg images to png files downsampled to 72 dpi or 90 dpi. Others include OpenOffice.org Draw, and Scribus. I personally use Inkscape and automate the process with a shell (batch) script to convert all of the .svg files to .png files while keeping the same root name. It is then simple to edit book.html and do a global replace of ‘.svg’ with ‘.png’ and all of the links will work. Once converted you can throw out all of the .svg files in the img directory.
February 20, 2011 at 6:33 am
Hi,
Thank you for your advice, now I know how to control the ebook size by converting images to reduce image resolution.
For your information, I have just upload the files you need and send links (by private message) to KevinH on mobileread.com forum. Please contact him to get these links.
Best regards.
February 20, 2011 at 2:37 pm
Hi Ho,
I have the link that was passed along and was able to open the rar and process the ebook. There is no bug. Your html conversion software simply does not understand how to convert or process svg images.
To see this, simply unzip the _nodrm.zip archive and open book.html with a “good” web browser such as Safari or Firefox … do *not* use IE. Then do a search for “4.2.4 Cell selection procedure” and you will clearly see both equations are present and the conversion worked. The second equation is one of those fixed regions that has been converted to an svg image.
You either only examined the book.html with an old/poor web browser that does not support svg images or those images were lost when you used MobiPocket to import and convert the html.
The correct solution is use a “good” web browser to view the book.html (such as Safari or Firefox) or conversion software that understands svg images (such as Calibre) or pre-convert the svg images to png files if your conversion software does not (such as Inkscape) etc.
I understand that svg images are an issue for you due to increased size and not being properly converted by your software, but they are a standard part of the epub specification and html5 and there is free conversion software (such as Calibre) that understands how to read and convert ebooks with svg images properly.
Your best bet is to use a shell or bat file with inkscape to convert to png and downsample to 72dpi or 90dpi ***before*** trying to convert it with Mobipocket and then all images should work for you.
For most Topaz ebooks, the fixed regions are often horribly messed up since they typically depend on spacing, have ocr issues, and can not be reflowed. That is why using scalable vector graphic images (svgs) for these regions makes the most sense.
Hope this explains things better.
February 20, 2011 at 3:42 pm
Hi,
– As you can see in my screenshots, the is no “Equation (4.2)” in book.HTML file that produced by using tools_v3.6
– You can check this again by open the _nodrm.zip archive that I have sent you.
(link: http://rapidshare.com/files/448887531/B001JEPVZU_EBOK_Essentials_of_UMTS_nodrm.zip)
– But this equation is present in book.html file that produced by using tools_v2.3, as you can see in my screenshot above.
– In these screenshots, I used FireFox 3.6.13 to open the book.html files.
– My problem is, the “Equation (4.2)” is missing in book.HTML file, before converting to PRC file, not after converting.
February 20, 2011 at 4:44 pm
Hi Ho,
Yes, I unzipped your _nodrm.zip file and opened book.html with my Safari web browser and did a search for “4.2.4 Cell selection” and I see TWO equations.
The second is actually an img tag pointing at img/page0220_0000.svg. You can go to your img directory and open that specific svg file and you will see your equation.
So I will repeat what I said earlier. There is NO bug. Your web browser is simply NOT showing you any svg images when you view it.
To prove this to you, simply open book.html in any **text** editor and search for “4.2.4 Cell selection” and and you will see the link to the svg image:
img src=”img/page0220_0000.svg” alt=””
Then open that image in Inkscape and you will see your second equation.
I am sorry your web browser will not show it. I am not sure why? But the second equation is there and the software works as shown by your own zip archive.
Another way to prove this to you is to import the _nodrm.zip directly into calibre and view it with the Calibre viewer and your equations will appear. Your software just does not know how to deal with image tags when the images are .svg files.
That is a bug in your viewing software, not in the Topaz conversion.
February 20, 2011 at 5:06 pm
Hi Ho,
BTW: What web browser are you using? I have tried Safari, and Firefox 4 (beta) and both show TWO equations and seem to work fine with img tags with .svg images.
This is with Mac OS X 10.6.6. If you are on Windows, please try with Firefox 4 (I think they are up to Beta 11 or higher now) or Safari for Windows. I do not think IE supports svg files. I do not have Firefox 3 so I do not know if it works on that browser or not. Your file **DOES SHOW BOTH EQUATIONS** with both Safari and Firefox 4.
Please confirm that you see both equations when you open book.html with Safari or Firefox 4.
February 21, 2011 at 1:55 am
Hi,
Thank you for your clarification. I will try to update my browser so that it supposts
SVG images.
If I understand clearly, in the older version of your tool (tools_v2.3), it can do OCR
conversion for the “Equation (4.2)”. The newer one (tools_v3.6) cannot recognize this
Equation as a text, so it just puts this Equation as an SVG image. Is this correct?
If I am right, that means the old version (tools_v2.3) can do OCR conversion better
than the new one?
And one more question, how do I use the Calibre to import _nodrm.zip archive to Calibre
as an ebook? Do I need to install any plug-in from your tools? And what command used to
import ZIP archive?
Tks and best regards.
February 21, 2011 at 2:13 am
Hi Ho,
No, you completely misunderstand. That is not correct. Any OCR is done by Amazon and built into the book itself. There is no loss of capability with any version. The version in 3.6 simply uses svg graphics for fixed regions like your more complicated second equation. These svg images are glyph based and not ocr based and so are **exact/true** copies of what is seen on the page when viewed by an official Kindle reader, and so do not suffer from any errors that might exist in the internal ocr. The changes are actually a big improvement in accuracy for complicated equations and fixed regions like tables. Once you get a working browser you will see the differences.
As for Calibre, you simply import the _nodrm.zip file as if it was an ebook itself. Calibre is smart enough to open the zip, find the html, css, images, and opf/metadata info to create the book. If you have in enabled in your Calibre preferences, you can set Calibre to use its own internal viewer for zip files and it will show up just like any other ebook.
February 21, 2011 at 4:56 am
Hi some_updates,
Thank you for your kindly support. I will try to use Calibre and upgrade browser for better dealing with SVG images.
Best regards
February 21, 2011 at 1:22 pm
Hi some_updates,
I have used Calibre to open _nodrm.zip archive as an ebook and convert it to MOBI format and it works like a charm. The “Equation (4.2)” is there and the MOBI file size is also too small (6.2MB for the tested ebook above). Although there are still some OCR errors, but it is acceptable.
Once again thank you so much for your kindly support.
February 18, 2011 at 4:44 pm
Hi All,
It seems that Amazon also recently changed something in how it uses metadata in its Topaz ebooks similar to changes it made for normal Kindle ebooks.
So “Yet Another” bug fix release.
tools_v3.6.zip
http://www.datafilehost.com/download-aa179896.html
Updates:
KindleBooks (GUI Kindle ebook Tool)
K4MobiDeDRM_v02.6_plugin.zip (Calibre Plugin)
DeDRM 2.4.app (Mac OS X Drag and Drop Multi-Tool)
DeDRM_WinApp_v1.6 (Win Drag and Drop Multi-Tool)
MobiDeDRM_0.28.py
Sorry about all of the bug fix updates coming so fast. Hopefully this will be it for a long time.
February 20, 2011 at 8:04 am
Some_Updates, Is this the best place to get program updates or do you have a site?
February 20, 2011 at 8:10 am
some_updates appears to have access to Apprentice Alf’s blog
http://apprenticealf.wordpress.com/
and new versions get mentioned there in updates to the first post.
February 20, 2011 at 2:44 pm
HI,
As Paul noted. Updates are posted on the ApprenticeAlf site and here (The Dark Reverser’s New Blog). Due to the huge size of this blog, I storngly recommend using the ApprenticeAlf blog since it loads and updates much quicker than this blog.
February 21, 2011 at 9:20 pm
thank you for the link. I’ve bookmarked both sites 🙂
February 20, 2011 at 4:20 pm
Am I missing something? Didn’t ApprenticeAlf retire in January?
February 20, 2011 at 5:22 pm
Hi Julian,
Alf did retire but there were a number of us involved in the support so he gave us access to the site in case we wanted to keep it going. Since it is still legal in Canada to distribute DRM removal software, we will try to keep up support. But be forewarned, that Canada is debating a bill at this very moment (C-32) that if passed unchanged will make it illegal to distribute DRM removal software in Canada, and if that happens, I will no longer be updating the tools and the site may close completely.
So I strongly recommend keeping things up to date and removing the DRM from all of your books now so that you can make html archive copies for yourself that can be converted to other formats in the future and not waiting until later. Amazon and others do change how things are done making the tools quickly obsolete unless someone will continue to update them. Case in point is the most recent changes to the use of metadata in the DRM process done by Amazon.
Hope this makes things clearer.
February 24, 2011 at 7:07 am
I am trying to remove the DRM from my Kindle books I have purchased. I have downloaded Calibre and tried to download your mobydedrm file and the location is not found…can you help me please? Thanks in advance! I am very new to converting the files and don’t know much about it.
Thanks!
February 24, 2011 at 8:24 am
Get the latest tools package from http://apprenticealf.wordpress.com/
(scroll down to the first (by date) post for the links)
Install the calibre plug-in.
Done.
February 25, 2011 at 4:38 am
Hi All,
Another bug fix release.
tools_v3.7.zip
http://www.datafilehost.com/download-a0cecfba.html
A bug in MobiDeDRM that triggers improper handling of Trailing Data for specific Kindle/Mobi ebooks has been found and fixed.
In addition, a small bug with preferences in the Mac OS X DeDRM app has been found and fixed
This change in MobiDeDRM means the following are updated:
KindleBooks (GUI Kindle ebook Tool)
K4MobiDeDRM_v02.7_plugin.zip (Calibre Plugin)
DeDRM 2.5.app (Mac OS X Drag and Drop Multi-Tool)
DeDRM_WinApp_v1.7 (Win Drag and Drop Multi-Tool)
MobiDeDRM_0.29.py
February 27, 2011 at 10:11 pm
Dude – thank you soooooooooooo much for your work.
March 19, 2011 at 4:31 pm
Not working for me, why?
March 19, 2011 at 6:39 pm
Hi Jaja,
If you want help, please tell us what programs you have tried, what type of ebook it is, where you bought it, what exact error message you received, whether you followed the directions in the ReadMe text files, what versions of the reader software you are using, whether you can actually read that exact file with the official software, etc.
No one can help without much more info.
February 27, 2011 at 11:24 pm
Hi again,
As I said here some months before, thanks again for your effort.
Thank you for making things working in easy ways,
🙂
Again thanks,
Oscar
February 28, 2011 at 2:30 pm
Plugin for K4PC – Trouble Shooting Question: Sorry that I am a self-taught newbie |& cannot get the newest Calibre plug-ins to convert but could you explain “Open a command prompt (terminal) and change to the directory where the ebook you’re trying to import resides. Then type the command “calibredb add your_ebook.azw”. Don’t type the quotes and obviously change the ‘your_ebook.azw’ to whatever the filename of your book is. Copy the resulting output and paste it into any online help request you make.” What and where is a Command Prompt terminal?
I am trying to convert a purchased azw ebook from my Kindle for PC version 1.4.1 (31629) thru Calibre.
Sorry for being to ignorant…………………
February 28, 2011 at 2:51 pm
Start Button/Run…/cmd
You might find it easier (quicker page loading) to ask at Apprentice Alf’s blog.
http://apprenticealf.wordpress.com/2010/02/11/
February 28, 2011 at 8:46 pm
Hi mmwhite,
A few common things to check for:
1. That you are using the very very latest version of the tool, which is tools_v3.7 at the time of this writing to get the very latest Claibre plugins
2. double-check that you have properly loaded the K4MobiDeDRM plugin by going to Calibre->Preferences->Plugins->FileType Plugins and verify the K4MobiDeDRM plugin is listed and that there are not any older versions listed that could interfere
3. double-check that you have installed Kindle for PC on the exact same machine and same account on your Calibre program.
4. Make sure that your take the ebook you want to convert directly from your “My Kindle Content” folder that is inside your “My Documents” folder
5. Make sure that that the specific ebook you want to import to Calibre does not already exist in the Calibre database. If that ebook already exists in the Calibre database, you have to delete it from Calibre in order for the plugin’s version to get loaded.
If after all of that, you still have trouble, do as Paul (and the README) suggested and use Start Button->Run->cmd.exe and manually enter the calibredb add YOURBOOK.azw command and paste its output here or on the ApprenticeAlf site.
Please let us know what you find out.
March 1, 2011 at 2:23 pm
OMGOODNESS!!!! You guys ROCK!!!
It was step #5 that I did not do. It was my “test” .azw book to see if the new plug-ins worked. So, I deleted the book – reinstalled it and it is now all converted. Thank goodness! >| as I was not real sure how to do the last “If after all that…” paragraph.
Anyway }}}HUGS{{{{
March 1, 2011 at 7:55 am
Question about tools 3.7 and windows xp home.
I run calibre and the tools for decryption on two different personal computers. Both have the latest version of Calibre 7.47 as of writing. Both have python 2.7 installed. Up until yesterday both machines decrypted .azw files just fine. Was using v3.2 of tools. Tonight the xp home computer did not work with books I just bought. So I downloaded v3.7. New plugin didn’t decrypt. So tried the new kindlebooks and it still did not decrypt. Downloaded v3.7 to my Windows 7 machine and both the plugin and kindlebooks decrypt my newly purchased books. Went back to the xp machine, deleted v3.7. redownloasded and tried again, still no luck on that machine.
Any ideas please? Thanks so much.
March 1, 2011 at 2:48 pm
Hi,
Please double-check a few things …
1. Make sure that you are not trying to decode a file taken from one machine on the other. This will not work. All files to be processed must come from the “My Documents\MyKindle Content\” folder on that machine.
2. Please verify that the K4MobiDeDRM Calibre plugin is correctly installed and that there is not an older version still there that is interfering.
3. What does the KindleBooks error message say (copy and paste the Conversion Log window contents (after changing any personal information)?
4. Try deleting a book from inside Kindle for PC on the xp machine and re-downloading it from your Kindle “Archive”. Does that fresh copy of the book work with the tools 3.7?
The error message should tell us something? If all else fails, you might try removing the Kindle for PC program (using a good uninstaller) from your machine and reinstalling a fresh copy and reregistering it and trying again.
March 17, 2011 at 7:03 pm
I am also having problems with a new Topaz book with 3.7 tools, all of which are failing. Here is what I have tried and the error messages (personal info removed) all on my Mac:
1) I first tried with DeDRM 2.5 that returned:
B00359FZHQ_EBOK.azw couldn’t be decoded:
Traceback (most recent call last):
File “/Users/myname/Desktop/tools 3.7/DeDRM 2.5.app/Contents/Resources/k4mobidedrm.py”, line 242, in
sys.exit(main())
File “/Users/myname/Desktop/tools 3.7/DeDRM 2.5.app/Contents/Resources/k4mobidedrm.py”, line 237, in main
return decryptBook(infile, outdir, k4, kInfoFiles, serials, pids)
File “/Users/myname/Desktop/tools 3.7/DeDRM 2.5.app/Contents/Resources/k4mobidedrm.py”, line 128, in decryptBook
pidlst = kgenpids.getPidList(md1, md2, k4, pids, serials, kInfoFiles)
File “/Users/myname/Desktop/tools 3.7/DeDRM 2.5.app/Contents/Resources/kgenpids.py”, line 304, in getPidList
pidlst = getK4Pids(pidlst, md1, md2)
File “/Users/myname/Desktop/tools 3.7/DeDRM 2.5.app/Contents/Resources/kgenpids.py”, line 279, in getK4Pids
kindleAccountToken = getKindleInfoValueForKey(“kindle.account.tokens”)
File “/Users/myname/Desktop/tools 3.7/DeDRM 2.5.app/Contents/Resources/kgenpids.py”, line 103, in getKindleInfoValueForKey
return getKindleInfoValueForHash(encodeHash(key,charMap2))
File “/Users/myname/Desktop/tools 3.7/DeDRM 2.5.app/Contents/Resources/kgenpids.py”, line 92, in getKindleInfoValueForHash
encryptedValue = decode(kindleDatabase[hashedKey],charMap2)
KeyError: ‘ZcB00HbABC0wb6bI0dBXBu0a0S0M0-bH’
2) Then I tried with Calibre using the K4mobiDeDRM 2.07 plug-in. That read in the book but when I try to view it, it says that the book is an Amazon Topaz book (which is odd because I thought the plug-in was supposed to decode Topaz).
3) I tried using Unswindlw for the Mac but that seems totally broken now and returns an error that the version of Kindle.app is not known. (Unless I am mistaken though this tool was said to be going away).
4) The script FiundTopazEbooks does see and identify the file as being a Topaz book — so the format is not totally changed.
Where can I go from here? Thanks!
March 17, 2011 at 7:46 pm
Quick Add: I was able to find a link to the older Kindle for Mac 1.2.2 so I can use Unswindle for Mac again (changed the autoupgrade pref now) but that also fails. It does work on other books although for some reason I must start the older Kindle app with the script, choose the book, and then it will not decrypt until I actually close the Kindle app (!) which is wrong I know but it works).
March 17, 2011 at 8:11 pm
OK, sorry for the convoluted messages. I tried again after reinstalling the older Kindle app and this time UnSwindle for the Mac used the 1.2.2 Kindle App and returned that it was a Topaz book but this time gave me the PID. I entered the PID into DeDRM and DeDRM did decode the book ok! Yay! But I have a feeling I still did this a hard way. If you have a book that is Topaz what is the best way to proceed with the current tools? Thanks again.
March 21, 2011 at 6:14 pm
This error means you have a corrupted kindle info file (or more than 1 kindle info file).
This can happen from disk corruption. Please run disk repair on your main disk to fix any problems. If the problem persists, then follow the faq entry on the ApprenticeAlf site regarding failures after upgrading, changing Kindle for Mac.
March 18, 2011 at 7:25 pm
Hello, I have bought and downloaded an Amazon e-book (kindle for PC) and it’s topaz format and no success in removing DRM from it. Here is the error produced:
Processing Book: The Wizard of Oz and Other Narcissists: Coping with the One-Wa
y Relationship in Work, Love, and Family
Trying: J2DD6NV4
Trying: 4zzZb+4M
Book Key Found!
Processing Section: img
. . . . . .
Processing Section: color
. . . . . .
Processing Section: other
.
Processing Section: dict
.
Processing Section: glyphs
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . .
Processing Section: page
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . .
Processing Section: metadata
.
Successfully Extracted Topaz contents
Updating to color images if available
Creating cover.jpg
Processing Dictionary
Processing Meta Data and creating OPF
Traceback (most recent call last):
File “D:\Programas\Kindle\tools_v3.7\tools\Calibre_Plug ins\K4MobiDeDRM_v02.7_p
lugin\topazextract.py”, line 439, in
sys.exit(main())
File “D:\Programas\Kindle\tools_v3.7\tools\Calibre_Plug ins\K4MobiDeDRM_v02.7_p
lugin\topazextract.py”, line 395, in main
tb.processBook(pidlst)
File “D:\Programas\Kindle\tools_v3.7\tools\Calibre_Plug ins\K4MobiDeDRM_v02.7_p
lugin\topazextract.py”, line 269, in processBook
rv = genbook.generateBook(self.outdir, raw, fixedimage)
File “D:\Programas\Kindle\tools_v3.7\tools\Calibre_Plug ins\K4MobiDeDRM_v02.7_p
lugin\genbook.py”, line 331, in generateBook
authors = meta_array[‘Authors’]
KeyError: ‘Authors’
March 18, 2011 at 8:38 pm
It looks to me from that error message at the end, that the problem is that the book doesn’t list any authors in its metadata, and the script assumes that there’s always at least one.
I haven’t seen some_updates around recently, but I suspect he could fix this easily. Hopefully he’ll be back in the not too distant future and do just that.
Otherwise, you could always have a go yourself – look at line 331 in genbok.py and see if there’s a way to comment out the bits about an author.
March 18, 2011 at 10:28 pm
What’s your msn messenger address Paul? I’m not familiar with python programming but I don’t know if the author of those scripts will ever fix it.
March 18, 2011 at 10:54 pm
Great idea, it did fix it! I had to remove the meta_array[‘Authors’] and meta_array[‘ASIN’] but it did fix it and I got the book unprotected! thank you! I believe the author will have to fix the script so that it checks wether or not there is the meta data author and ASIN.
March 18, 2011 at 11:13 pm
You bought an Amazon book that doesn’t have Author or ASIN metadata? Hopefully it’s just a freak mistake, but that sounds rather odd.
Anyway, I’m sure the plugin and standalone programs will be updated to accommodate the issue at some point.
March 28, 2011 at 8:56 pm
Hi All,
A bug fix/robustness release.
tools_v3.8.zip
http://www.datafilehost.com/download-e8a06eee.html
Minor update to:
– fix issues with very strange Topaz ebooks that do not specify the Authors or ASIN metadata values
– fix issues related to multiple or corrupted or disabled kindle-info files left over from bad uninstalls/updates done for previous Kindle for PC and Kindle for Mac installations.
The following are all updated in this release:
KindleBooks (GUI Kindle ebook Tool)
K4MobiDeDRM_v02.8_plugin.zip (Calibre Plugin)
DeDRM 2.6.app (Mac OS X Drag and Drop Multi-Tool)
DeDRM_WinApp_v1.8 (Win Drag and Drop Multi-Tool)
April 1, 2011 at 9:24 am
Under Linux i get the following errors with 3.8 on every book I tried.
python2 KindleBooks/lib/k4mobidedrm.py -s B00XXXXXXXXXXXXX Book.azw /tmp
K4MobiDeDrm v2.8 provided by the work of many including DiapDealer, SomeUpdates, IHeartCabbages, CMBDTC, Skindle, DarkReverser, ApprenticeAlf, etc .
MOBI header version = 6, length = 232
Extra Data Flags = 2
Processing Book: Book
Traceback (most recent call last):
File “/tmp/tools/KindleBooks_Tools/KindleBooks/lib/k4mobidedrm.py”, line 242, in
sys.exit(main())
File “/tmp/tools/KindleBooks_Tools/KindleBooks/lib/k4mobidedrm.py”, line 237, in main
return decryptBook(infile, outdir, k4, kInfoFiles, serials, pids)
File “/tmp/tools/KindleBooks_Tools/KindleBooks/lib/k4mobidedrm.py”, line 128, in decryptBook
pidlst = kgenpids.getPidList(md1, md2, k4, pids, serials, kInfoFiles)
File “/tmp/tools/KindleBooks_Tools/KindleBooks/lib/kgenpids.py”, line 294, in getPidList
kInfoFiles = getKindleInfoFiles(kInfoFiles)
NameError: global name ‘getKindleInfoFiles’ is not
With version 3.7 everything went fine. (Python 2.7.1)
April 1, 2011 at 1:29 pm
Whoops! It appears Linux users have been excluded with the latest revisions in 3.8. Stay tuned for a fix – and revert back to 3.7 in the meantime.
This bug should only affect Linux users.
April 1, 2011 at 2:12 pm
Try replacing the /tools/KindleBooks_Tools/KindleBooks/lib/k4mobidedrm.py file with this modified one, and see if it fixes the issue (the 3.8 version of the tools):
http://www.datafilehost.com/download-6c371288.html
This will affect Linux users of the Calibre plugin as well, so please let me know if it works for you, so I can get a fixed plugin out there as well.
April 1, 2011 at 2:23 pm
Hi Carl and DiapDealer,
My Bad! I should have tested it on my linux box, but I did not.
Sorry about that. So much for being a robustness improvement release!
DiapDealer’s fix should do the trick but please verify that for him and he and I can get a fixed release out there (maybe a tools_v3.8a.zip?) with a new plugin that will work for everyone.
Again, my apologies for breaking things.
April 1, 2011 at 4:14 pm
YES!
Now it’s working(tested with 5 different books). This was really a fast.
Thank you very much.
April 1, 2011 at 4:42 pm
Not a problem 😉
Thanks for testing and reporting back.
April 1, 2011 at 5:12 pm
Hi All,
Well its seems our robustness changes broke things for Linux. DiapDealer was able to fix the problem so we are now releasing:
tools_v3.8a.zip
http://www.datafilehost.com/download-8210f840.html
If you are not running Linux and already have tools_v3.8.zip, then you do *NOT*need this release. This 3.8a release is made specifically to fix things for Linux users (although it will work fine for everyone).
April 2, 2011 at 8:03 pm
I have a question about DRM on active content AZW2 files. They are games like Scrabble and chess. I purchased these games for my Kindle with a U.S. account. I no longer have the U.S. account, and I have purchased a second Kindle. The games are still listed in my archived items folder, but can only be downloaded to the first Kindle. I tried using Kindle plugins to remove DRM, and the games were added to Calibre with no problem, but DRM was not removed.
Does anyone have any ideas about removing DRM from active content AZW2 files?
April 2, 2011 at 8:16 pm
Hi,
I don’t think it is DRM per se, I think each executable is simply digitally signed in some way that ties it to an account, but I am not positive of that.
I would ask in the Kindle Developer’s sub-forum of the Kindle forum on Mobileread. People there should be able to help.
http://www.mobileread.com/forums/forumdisplay.php?f=150
April 3, 2011 at 1:00 pm
Thanks, some_updates. Actually, right after I posted my question here, I went again to my account on Amazon, and the games were available to my new Kindle. Previously, the drop-down download menu for the games did not include the new Kindle, as all of those for the books did. It actually makes me slightly paranoid about who is monitoring this forum… But I am happy to have my chess and scrabble for Kindle. Thanks.
April 18, 2011 at 12:03 pm
I downloaded Kindle for the first time and this appears to be version 1.4. Kindle for Mac version does not work – as this is higher than whats posted in the README of 1.2.2.
Any update plans in the near future?
April 18, 2011 at 1:07 pm
Unswindle won’t work with Kindle for Mac 1.4. But all the other tools do. Get the latest tool archive from http://apprenticealf.wordpress.com/ and on the Mac I’d recommend using the DeDRM Applescript, or the calibre plug-ins.
April 21, 2011 at 11:59 pm
Hi to everyone.
I am trying to export some of my books to my Calibre library and I have reached a dead end… I wonder if some one please could give me a new idea these are my details.
– I have bought the books from the ipad kindle app but I try to transform the files downloaded directly from the Kindle for PC program (I say this in case it may have some importance)
– My calibre version: 0.7.54, kindle for pc version: 1.5.0, book format azw, calibre plugin: K4MobiDeDRM_v02.8_plugin.zip
– I have read many instructions for installing and using the pluging and i think I have done it right (this video I think gives the best summary: http://www.youtube.com/watch?v=iq5veXQ60xA&feature=related)
– I can import the books fine into calibre but when i try to convert them it a window opens saying that they are DRM protected. Not protected books are finely taken
-One last thing… when I try to remove the DeDRM by not plugin means, I get the message that “Crypto Type is: 2” “Error: No key found. Most likely the correct PID has not been given” Actually I have not put any as it is explained that it is not needed for kindle for PC…
Any idea what to do next please?
April 22, 2011 at 1:54 am
Hi Delosari,
You may not have installed the plugin correctly, or your kindle info file created by Kindle for PC may be broken or corrupted, or you may not have used the correct version of the ebook file (needs to come form your “My Kindle Content” folder), or you may have already imported the DRM version into Calibre and Calibre will not let you re-import the same book twice. To fix these possible issues please use the following steps:
1. completely uninstall and erase your Kindle for PC application, download a new copy and install it properly. (this will fix any corruption problems)
2. Fire up the newly installed Kindle for PC, register it, and go to your Kindle “Archive” and download all of your books.
3. In Calibre, remove the DRM copies of the ebooks from your Calibre Library, because Calibre will not allow you to import a book that already exists in its Library.
4. In Calibre, go to Preferences->Plugins and cliCk on the File Type Plugin triangle and verify the K4MobiDeDRM plugin is there and properly installed
5. Using Calibre try to import one of the .azw ebooks from your “My Kindle Content” folder. This is an important step as only the copies of the book inside of your “My Kindle Content” folder have been properly encrypted by Kindle for PC to allow the decryption to succeed. Do NOT use the files taken from your Kindle for iPad app!
6. Using Calibre, try to open the newly imported file for viewing.
If this fails, remove the ebook from Calibre and then run cmd.exe and use the “calibredb add YOUREBOOK.azw” command as described in the K4MobiDeDRM ReadMe txt file to generate a more detailed bug report and post that in the comments at the bottom of the http://apprenticealf.wordpress.com.
April 22, 2011 at 12:04 pm
Hi Some_updates, thanks for your detailed reply! I have followed your points but I did not manage to make it work, however thanks to your explanations I manage to make the terminal work. This is what I got:
C:\Users\USER\Delosari\calibredb add BOOO47DVIEI_EBOK.azw
A is not a valid Kindle serial number or PID.
C is not a valid Kindle serial number or PID.
7 is not a valid Kindle serial number or PID.
1 is not a valid Kindle serial number or PID.
U is not a valid Kindle serial number or PID.
G is not a valid Kindle serial number or PID.
M is not a valid Kindle serial number or PID.
7 is not a valid Kindle serial number or PID.
4 is not a valid Kindle serial number or PID.
X is not a valid Kindle serial number or PID.
K4MobiDeDRM: Calibre configuration directory = C:\Users\USER\AppData\Roaming\calibre
MOBI header versión = 6, length = 232
Extra Data Flags = 10
The Kindle.info files has not been found.
Crypto Type is: 2
Then a window opens with the title: K4MobiDeDRM Plugin and the message: “Error decoding: C:\Users\USER\Documents\My Kindle Content\B0047DVIEI_EBOK.azw”
Once I close it I get the message in the terminal:
Running file type plugin K4PC, K4Mac, Kindle Mobi and Topaz DeDRM, failed with traceback:
Traceback (most recent call last):
File “site-packages\calibre\customize\ui.py”, line 133 in _run_filetype_plugins
File “calibre_plugins.dummy1._init_”, line 341, in run Exception:K4MobiDeDRM plugin could not decode the file Backing up metadata
Notitying calibre of the change
A new azw is created in calibre and in the My kindle Content folder is created but when I try to open it, the Kindle Pc opens it.
I will post this in the website you gave me.
Thanks for your help.
April 22, 2011 at 12:26 pm
HEADS UP!!
The newly released Kindle for PC v 1.5.0 appears to break shit. The current scripts cannot locate the kindle.info file… and neither can I.
So do NOT upgrade to 1.5.0. If you already have, revert to version 1.4.1. Who knows what they’ve done or even if it’s fixable. Version 1.4.1 can be found all over the net, but if you have trouble locating one… you can download it from http://www.datafilehost.com/download-5c24bb7a.html (this is a link to a binary install package from a random guy on the internet. I’m not trying to trick anyone, but do yourself a favor and scan it before installing).
I have a feeling that the new German K4PC is 1.5 as well; can someone verify?
April 22, 2011 at 2:40 pm
For clarification: This appears to only affect the PC version of the “Kindle for *” app. I’ve been told that the new Kindle for Mac version (1.5.1) continues to work as expected with the current tools.
April 22, 2011 at 2:55 pm
Thank you my hero.
April 22, 2011 at 1:14 pm
I just finished installing python 2.7.1 on my computer, the 64-installer; my question is which pycrypto to install? For both 2.1 & 2.3, I only see 32bit Windows and I don’t have a 32bit computer. Thanks
April 22, 2011 at 1:43 pm
For all of these tools, the 32 bit version of python is highly recommended (and many times required). I would uninstall the 64-bit version… then install ActiveState’s 32-bit version of Python 2.7.1 and the corresponding 32-bit version of PyCrypto.
Alternatively, you could use Calibre and the plugins found in this tools archive and avoid installing Python or PyCrypto altogether.
April 22, 2011 at 6:15 pm
I went ahead and took your advice and installed the Calibre Plugins. When I tried to create my keys for epub, it could not find the software but when I had python installed the file automatically turned into a python file. Now I’m getting a error asking me to install pycrypto when I double click to create my key. What am I doing wrong?
April 22, 2011 at 6:58 pm
Well yes, If you want to use the stand-alone scripts to create your own key-files, then you will need python and pycrypto (or OpenSSL) installed.
But with the plugins, you don’t HAVE to create your own keyfiles. The inept_epub_plugin will create its own key-file from your ADE installation the first time you import an epub to Calibre. And the ignoble_epub_plugin (B&N) can be configured by entering your name and cc# in the plugin’s customization window.
Your choice. But if you do install python…install the 32-bit version. It won’t hurt if you decide to use the plugins AND install python and pycrypto, either.
April 22, 2011 at 1:56 pm
Hi Some_updates and DiapDealer ! The version 1.4.1 you gave me solved my problem.
Thanks a lot!
May 4, 2011 at 10:42 pm
All this talk about K4PC 1.5.0 made me curious about what I had. Since the new Calibre plug-ins, I rarely use it. I still have 1.1.0. It works fine. Is there any reason to update it to 1.4?
May 5, 2011 at 4:15 am
Hi,
Not unless there is some K4PC feature you truly need/want. You would have to check out the “features” of each release of K4PC to see what you might be missing. I use K4PC 1.4.1 with no troubles at all with the latest tools. All earlier versions of K4PC should work just as well.
May 5, 2011 at 12:58 pm
Thanks some updates. I really don’t use it, just keep it around incase the other tools fail. Thanks for all you do to untie our purchased books.
May 30, 2011 at 5:44 pm
Does any one get the message invalid stored block lengths with B&N epub decryption?
May 30, 2011 at 5:50 pm
never mind.. my own problem. Thanks for the great piece of software 🙂
June 2, 2011 at 3:41 pm
Hi,
If anyone running Linux and a recent wine (1.3.20 or 1.3.21) is willing to test a slightly modified version of the KindleBooks tools under wine, please let me know.
A workaround for a long-running bug in that prevented the tools from working with K4PC and Windows python under wine has been added but it needs testers:
Test Requirements:
1. Linux
2. Wine dev releases 1.3.20 or 1.3.21
3. Some versions of winecfg have a bug in setting the volume serial number, so create a .windows-serial file at root of drive_c to set a proper windows volume serial number (8 digit hex value for unsigned integer).
cd ~
cd .wine
cd drive_c
echo deadbeaf > .windows-serial
Replace deadbeef with whatever you want but I would stay away from the default setting of ffffffff
4. Only after setting the volume serial number – download and install under wine K4PC version 1.4.1 for Windows. Register it and download from your Archive one of your Kindle ebooks
5. Download and install under wine ActiveState Active Python 2.7 for Windows 32bit
6. Download and unzip tools_v3.8a.zip from the usual places
7. Download wine_k4pcutils.py from:
http://www.datafilehost.com/download-2d3580a9.html
and rename it to k4pcutils.py and use it to replace the file of the same name in KindleBooks_Tools/KindleBooks/lib/
Then give things a try and let me know if it works or not for you.
June 4, 2011 at 7:20 am
Wow! Thank you so much! It works beautifully here, I have tried it with two books. I am not using *pure* wine but crossover impersonator, though.
I advice disabling the automatic updates immediately after first installing KPC 1.4.1. If not, it will update itself to 1.5 automatically the next time it’s started and of course the script won’t work with that one.
June 3, 2011 at 6:55 pm
Hi —
Something is broken sort of, I say sort of because the books still DRM. But here is what is happening: I download a Kindle book with the azw extension and pop it onto DRM 2.6. That app returns the following error:
couldn’t be decoded:
K4MobiDeDrm v2.8 provided by the work of many including DiapDealer, SomeUpdates, IHeartCabbages, CMBDTC, Skindle, DarkReverser, ApprenticeAlf, etc .
MOBI header version = 6, length = 232
Extra Data Flags = 10
Processing Book:
Keys not found in /Users//Library/Application Support/Amazon/Kindle/storage/.kindle-info
Crypto Type is: 2
File is encoded with PID VheAcMLNFI. (Diff PID each book)
Decrypting. Please wait . . . . done
But the book does turn into a nodrm mobi file even though the above message has the “Bother” and “Save” buttons.
I am on a Mac with Kindle 1.4.0. I also noticed the Calibre plug-in no longer works at all. Is this a problem?
June 3, 2011 at 7:39 pm
Soemhow you have two .kindle-info files on your system. The one that doesn’t work is an old one from a previous installation or version of Kindle for Mac, and doesn’t have the right keys in it. Luckily, the DeDRM Applescript uses code that tries all the .kindle-info files that it finds, not just the first one.
The error reporting ought to be updated so that a successful decode doesn’t get reported as an error just because a bad .kindle-info file was also found.
You can stop these error reports by deleting the bad .kindle-info file – you have the path to it in the error report. Note that it’s an invisible file, so you might have to use the terminal to delete it.
June 3, 2011 at 9:48 pm
Well, I deleted the bad .kindle-info file but I still get the error message although the book unDRMs. Here is why]at I get now:
TITLE.azw couldn’t be decoded:
K4MobiDeDrm v2.8 provided by the work of many including DiapDealer, SomeUpdates, IHeartCabbages, CMBDTC, Skindle, DarkReverser, ApprenticeAlf, etc .
MOBI header version = 6, length = 232
Extra Data Flags = 10
Processing Book: TITLE
Crypto Type is: 2
File is encoded with PID VheAcMLNFI.
Decrypting. Please wait . . . . done
Any way to get rid of this error for me?
June 4, 2011 at 7:37 am
It’d be a lot easier to continue this at Aprentice Alf’s blog.
I think that message shouldn’t appear unless there was an error code returned from the script. Are you sure you’ve posted the entire message (bar the hiding of details)?
June 4, 2011 at 3:09 pm
Yes that is the entire message. Note that the Keys Not Found line has disappeared after removing the kindle-info file that it referenced. I thought Alf’s blog was abandoned a while ago? I could check there but if we could continue this here that might be easier than trying to start up there and explain all that went before? I can if you want tho.
June 4, 2011 at 3:15 pm
OK, have also posted on AA’s blog (never looked to see the comments were actually current).
June 4, 2011 at 7:26 pm
Works for me!
I am installing Lucid (ubuntu 10.4) as Karmic (9.04) support just got dropped. I am using wine 1.3.21 from WineHQ.org
I can testify that python.org’s python 2.7 does indeed not work…I had installed it previously as I had used ineptkey.pyw and ineptepub.pyw to convert some google books to mobi.
June 4, 2011 at 10:38 pm
I am trying to download the latest version of ereader2html, but I get “Invalid paste ID when I click the link. Where is the file ?
June 4, 2011 at 10:40 pm
I am trying to download the latest version of ereader2html, but I get “Invalid paste ID when I click the link. Where is the file ?
[to make a different comment|
June 5, 2011 at 8:52 am
Get the latest tools from Apprentice Alf’s blog.
http://apprenticealf.wordpress.com/
June 14, 2011 at 7:22 pm
Announcing a major upgrade to the “tools”
tools_v4.0.zip
http://www.datafilehost.com/download-2b574fa7.html
MD5 (tools_v4.0.zip) = 1798494dbe0085cdfc03c081d9348d19
Improvements in tools_v4.0.zip
—————————————–
– Support for Kindle for PC version 1.5.0 and later has been added which required a reverse-engineering effort and a rework of the code
(warning: experimental – very limited testing has been done)
– All plugins have been updated to the new Calibre plugin interface that requires at minimum Calibre 0.7.55 but work **best** with Calibre 0.8.1 or later
– New for Android Users!: A Kindle for Android patch that allows MobiDeDRM.py to be used with a patched version of Kindle for Android 3 to effectively create an “Unswindle for Kindle for Android” (Thanks “Me”!)
– Linux wine users can now successfully use KindleBooks/K4Mobi Calibre Plugin under recent Wine versions (1.3.18-1.3.21). Requires Windows K4PC and Windows Calibre (or Windows ActiveState Active Python 2.7 for 32bit for the KindleBooks tool) to be installed under wine. The bug in wine which caused the problem with earlier tools should be fixed in wine 1.3.22
– Bug fixes to the Topaz htmlz ebook stylesheets and move to use .htmlz archives versus .zip archives
– Bug fix for DeDRM 2.7.app for Kindle ebooks that have the substring “error” or “Error” in the title of the book which confused the error detection code in the app.
– minor interface changes (no functionality changes) for ineptpdf and mobidedrm to ease its use as imported modules by other python code
Includes new versions of almost everything:
* Adobe_PDF_Tools (minor update to sync with plugin version)
* eReader_PDB_Tools (minor update to sync with plugin version)
* Calibre_Plugins
– eReaderPDB2PML_v05_plugin.zip
– ignobleepub_v01.6_plugin.zip
– ineptepub_v01.7_plugin.zip
– ineptpdf_v01.4_plugin.zip
– K4MobiDeDRM_v03.1_plugin.zip
* DeDRM 2.7.app (all in one drag and drop tool for Mac OS X)
* DeDRM_WinApp_v2.0 (all in one drag and drop tool for Windows)
* KindleBooks (gui tools for Kindle ebooks)
* Kindle_for_Android_Patch (New!)
* MobiDeDRM.py (to version 30, a minor interface update)
June 14, 2011 at 7:39 pm
There is a small problem with DeDDRM_WinApp_v2.0 (at least with the older version) when you tried to decrypt a Mobipocket file (so .prc extension, no mobi), because it confused input and output files. I made some minor changes to avoid this.
June 14, 2011 at 8:20 pm
Hi terisa,
Dragging and dropping files in my tests worked (and no name conflicts since “_nodrm” was added to the output file name) but it obviously doesn’t for you.
So what file and line numbers need to be changed? I would be happy to include the fix in the next release.
June 14, 2011 at 8:21 pm
I had trouble using the inept plugins after upgrading to Calibre 0.8.5. I tried deleting the .der keyfile, and the file was never regenerated, as if the plugins ceased to run. I was able to get it working again using an old 0.8.3 Calibre install.
June 14, 2011 at 8:33 pm
Hi DaveO,
You probably downloaded Calibre 0.8.5 for Windows immediately after it was posted. It had a bug in it (incomplete pycrypto files inside of Calibre). Kovid was alerted and fixed the Calibre 0.8.5 version very soon after release but did not change the release number. So please download a fresh copy of Calibre 0.8.5 for Windows and all should be fine again.
June 14, 2011 at 8:43 pm
OK, I was about to post that the new tools were working in Calibre 0.8.5, but it probably has more to do with the fixed 0.8.5 install than the tools. Thanks some_updates.
July 3, 2011 at 6:44 pm
Hmmm. I bought a couple of B&N epub books today and the latest version of tools won’t decrypt them. I used my full credit card number to generate the key. Have they done something new to their DRM?
July 3, 2011 at 7:06 pm
OK, never mind. It appears it’s just one particular book. The others worked fine. The one book isn’t that important.
July 8, 2011 at 8:21 am
Hey there. Looking forward to being free to do with the books I’ve bought what I’d like to do. However, when I click on the link above for the mobidedrm.py the pastebin page I get taken to says ‘unknown paste id’ and the page is blank. Where did it go?
July 8, 2011 at 9:11 am
The original scripts are long gone. Much improved scripts, and tools to make them easier to use, are freely available at Apprentice Alf’s blog:
http://apprenticealf.wordpress.com/
The link to the tools archive is in the earliest message on the blog.
July 11, 2011 at 5:31 am
I changed computer so have to reinstall all the tools I successfully installed in the past. However I am finding it much harder this time round. I downloaded tools 4.0. However when I run ineptkey_v5.4 I get this error:
Traceback (most recent call last):
File “C:\Users\Marcy\AppData\Local\Temp\Temp2_tools_v4.0.zip\tools_v4.0\Adobe_ePub_Tools\ineptkey_v5.4.pyw”, line 449, in main
success = retrieve_key(keypath)
File “C:\Users\Marcy\AppData\Local\Temp\Temp2_tools_v4.0.zip\tools_v4.0\Adobe_ePub_Tools\ineptkey_v5.4.pyw”, line 346, in retrieve_key
with open(keypath, ‘wb’) as f:
IOError: [Errno 13] Permission denied: ‘adeptkey.der’
Any suggestion of what to do next would be appreciated.
July 11, 2011 at 4:40 pm
Hi Marcy,
Yes, you can see from the error message that you are trying to run the script from within a zip archive or other reserved location I am not sure how that even works at all. That causes the write failure.
Starting with the tools_v4.zip archive, right click on it and select “Extract All”. Then *move* to the newly created unzipped tools folder and then navigate from within that.
IHMO, Windows is crazy to allow the File Manager to browse inside of zip archives and even run things without first extracting it from the zip. That has caused no end of troubles for many users.
July 11, 2011 at 5:52 pm
Argh!!
I never even realized that. And this new version of Windows has different icons…the zip archive looks just like a folder to my old eyes.
Thanks so much! Problem solved.
July 15, 2011 at 12:56 am
BN epub – no luck, not working (mac) …. tried all sorts of combinations, command line, calibre plugins, file-drop…
July 15, 2011 at 5:02 am
I don’t know if this will help but….
BN has converted all their books to Nook format. I went to BN.com/ebooks and to my library and downloaded a few test books which are now in pdb format. I found that (on my Mac) they undrm’d fine if I dropped them onto DeDRM 2.7. But the Calibre plug-in did act funny. It DeDRM’d the books but made them into PMLZ files. If I tried to view such a PMLZ file in Calibre it opened another instance of the same PMLZ making two (or multiple) copies of the book in Calibre as unreadable and self-copying PMLZ files. However, Calibre had no problem in converting the PMLZ to EPUB which then worked normally. Just adding this to try to help as I am surely NOT an expert here.
July 15, 2011 at 7:46 am
The B&N website won’t download ePubs on Macs, unless you fiddle with the browser to get it to claim to be a Windows browser. Instead it downloads eReader (.pdb) files. You’ll need to use the eReader plugin to decode them.
eReader files can’t have their DRM simply stripped like other formats. It’s necessary instead to decode them into the original PML (Palm Markup Language) source file. This is what the Calibre plug–in does, but also zips up the source and images in a zip file – PMLZ.
The DeDRM drag&drop app does the same, but if it finds the DropBook application on your Mac, it uses that to re-compile the PML source back into an eReader (.pdb) file.
It’s probably best to ask this sort of thing over at Apprentice Alf’s blog, where there aren’t thousands of comments slowing the page load.
July 18, 2011 at 2:23 am
Announcing a new version of the tools
tools_v4.5.zip
http://www.datafilehost.com/download-07764c4f.html
See http://apprenticealf.wordpress.com/for the details.
MD5 (tools_v4.5.zip) = 8ab842d3dc7694b41de633eb52a7bb85
Changes in the tools
– Added support for Kindle for PC 1.6 and Kindle for Mac 1.6.
Therefore there are new versions of:
KindleBooks
K4MobiDeDRM Calibre Plugin
DeDRM 2.8.app
DeDRM_WinApp
Please read the following tool developers’ note:
As we have said many times, we are strongly against piracy. But we are for “fair-use” rights.
Pirates can and do create ebooks even if no digital copy of the book exists (Harry Potter anyone?). They have been doing that for a long time. Pirates can always fall back on scanning a hard copy and creating a pdf.
Therefore, DRM can not really do anything to prevent actual ebook piracy.
The truth is that DRM is actually about enabling vendor lock-in. Forcing you to stay with the vendor given your large investment in ebooks so that they can enforce a license agreement to “rent” you the book at prices far above “rental” prices and approaching or surpassing full paper book prices. Vendors like Amazon want to take away your “fair use” rights. They do not want you to be able to archive the book as html (so it never gets lost) and they do not want you to convert it to your favourite reader format.
The problem is Amazon is now getting nasty about it.
In the space of a few short months, their developers have completely revamped their “secure storage” of the information needed to remove the DRM on both of their Kindle for PC and Kindle for Mac applications.
They have now added code to their Kindle application that tries to prevent a debugger from attaching, detects when a debugger is being used, and literally changes the path the code takes on the fly depending on whether the process with that pid is being debugged or not. So any use of a debugger will result in deliberately incorrect information/code paths being selected with incorrect data DRM data being used which will result in ebooks that won’t open triggering the need to “re-register” with Amazon.
It has been a good fight to protect our fair use rights, but given the fast approach of encrypted binaries and sandboxing, coupled with now active detection/counter-measures for using a debugger, it is a fight we are very soon going to lose. I hope Amazon and others like them are proud of themselves.
July 27, 2011 at 4:47 pm
So far, anyway, BN books are easy to convert. I fell out of love with Amazon a while back due to their questionable corporate ethics and bought a Nook Color. Yes, I pay local sales tax, but I’m fine with that because the money goes to schools, police, firefighters, etc. And, I still get the selection I want. Not only that, but I rooted my Nook and have a pretty decent little Android tablet for a lot less money than most tablets are selling for. I highly recommend the switch if you are fed up with Amazon.
July 18, 2011 at 5:41 am
Thanks, Some Updates. I was a very loyal Kindler, but I think my next ereader will be something else. I do not like the idea of anyone knowing what I do with something i have bought. I have never used these tools to pirate, only to save my books in a format that will be accessible to me.
July 18, 2011 at 2:12 pm
If I never update my Kindle for PC program, will it continue to work w/present plug-ins?
July 18, 2011 at 11:17 pm
Hi mmwhite,
Not necessarily. At any point Amazon could decide that you must upgrade to see/read/access new purchases.
Until they do, you should be okay.
July 19, 2011 at 1:30 pm
Thanks – you all are the greatest! Got a solution for the economy? just kidding around…………….
July 19, 2011 at 9:07 pm
I know that just stripping the DRM from a TPZ file – without converting it – is frowned upon because it is associated with piracy.
But my case is fair. I bought a $120 ebook from amazon (http://amzn.to/oakTNS) because of work. Unfortunately, I upgraded 2 of my computers, bought a new kindle and I realize I have only one download left of this book.
I’d really like to get a tpz file without the drm to protect my investment in the future. The last version of the tools decrypts and extracts the file. However, I haven’t been able to just remove the DRM even by tweaking the cmbtc script at the right places as an error that reads KeyError thumbnail clippingLimit etc pops up.
Any help would be greatly appreciated. And I am happy to show the receipt for the book to prove that I am being – for once – an upstanding citizen.
July 19, 2011 at 9:11 pm
I think you may have misunderstood the Amazon download limits. You can only have a limited number of devices registered to your account at any one time, but you ought to be able to re-download the book to any of them, I think. Just de-register the devices you no longer own from your Amazon account web pages, and you should be fine.
July 26, 2011 at 1:27 am
I am running linux but am using wine to run python for windows and i got this:
Traceback (most recent call last):
File “Z:\home\gayle\Downloads\pastie-1030388.py”, line 850, in gui_main
unswindler = Unswindler()
File “Z:\home\gayle\Downloads\pastie-1030388.py”, line 727, in __init__
self._mobidedrmpath = self._get_mobidedrm_path()
File “Z:\home\gayle\Downloads\pastie-1030388.py”, line 730, in _get_mobidedrm_path
basedir = sys.modules[self.__module__].__file__
AttributeError: ‘module’ object has no attribute ‘__file__’
any help is great
August 7, 2011 at 5:45 pm
Tools updated:
– tools_v4.6.zip, see http://apprenticealf.wordpress.com for the links and changes information.
– Calibre_Plugins_using_Old_API.zip, see the latest comments in the last post at http://apprenticealf.wordpress.com for the link and information.
August 14, 2011 at 6:09 am
Title…
This is my Excerpt…
August 23, 2011 at 9:06 pm
press release games…
New Blog « Darkreverser’s Weblog…
September 6, 2011 at 5:29 am
kindle dx videos…
New Blog « Darkreverser’s Weblog…
September 6, 2011 at 2:00 pm
Tools updated:
– tools_v4.7.zip, see http://apprenticealf.wordpress.com for the links and changes information.
NOTE: Because of the sheer size of this blog, this will be the last post about the tools we will be making here. Your best bet if you are interested in the tools or have questions is the ApprenticeAlf blog (see the link above).
October 18, 2011 at 7:42 pm
java programming lesson…
[…]New Blog « Darkreverser’s Weblog[…]…
October 19, 2011 at 7:45 am
iPhone 5 Jailbreaking…
[…]New Blog « Darkreverser’s Weblog[…]…
October 29, 2011 at 2:52 pm
Japanese Technologies…
[…]New Blog « Darkreverser’s Weblog[…]…
October 31, 2011 at 7:16 am
Article Marketing click here…
[…]New Blog « Darkreverser’s Weblog[…]…
November 1, 2011 at 6:46 pm
money problems…
[…]New Blog « Darkreverser’s Weblog[…]…
November 5, 2011 at 8:10 pm
I have a Windows computer and been using the Tools Removal Calibre Plugins since the beginning. A friend of mine just bought a Sony Reader (we live in different States), so I’m talink her through the steps via the telephone while we are both sitting in front of our computers. She has a MAC. I’m stuck in File type plugins. It seems she does not have that so I’m unable to tell her where to put the calibre plugins zip files. Obviously the steps after downloading Calibre and the Tools files are different from my Windows computer. Could someone tell me what to do for MAC or point me to steps to get the Calibre plugins Removal tools for a MAC instructions. Thanks
November 5, 2011 at 10:07 pm
For support with the tools see http://apprenticealf.wordpress.com
November 12, 2011 at 2:14 am
buy kindle in store…
[…]New Blog « Darkreverser’s Weblog[…]…
November 16, 2011 at 6:39 am
highest vertical jump…
[…]New Blog « Darkreverser’s Weblog[…]…
November 16, 2011 at 9:10 am
nook and kindle…
[…]New Blog « Darkreverser’s Weblog[…]…
November 28, 2011 at 11:02 am
dx format…
[…]New Blog « Darkreverser’s Weblog[…]…
December 2, 2011 at 4:19 pm
what are kindle books…
[…]New Blog « Darkreverser’s Weblog[…]…
December 11, 2011 at 6:23 am
denled…
[…]New Blog « Darkreverser’s Weblog[…]…
December 11, 2011 at 3:54 pm
pinupfiles…
[…]New Blog « Darkreverser’s Weblog[…]…
December 14, 2011 at 5:43 am
kindlecoversstore…
[…]New Blog « Darkreverser’s Weblog[…]…
December 15, 2011 at 12:56 pm
kindle gift card…
[…]New Blog « Darkreverser’s Weblog[…]…
December 20, 2011 at 1:38 pm
gadget | gadgets | electronics | cell phones | cameras | software | mp3 players | ipod | stereos | televisions | health monitors | kindle | mp3 downloads | audio | video | camcorders | computers | printers | scanners | fax machines | kindle cases and…
[…]New Blog « Darkreverser’s Weblog[…]…
December 31, 2011 at 2:02 am
free virus software…
[…]New Blog « Darkreverser’s Weblog[…]…
December 31, 2011 at 8:12 pm
Which Kindle…
[…]New Blog « Darkreverser’s Weblog[…]…
January 2, 2012 at 10:37 am
fake documents verify…
[…]New Blog « Darkreverser’s Weblog[…]…
January 5, 2012 at 1:10 am
nullled script…
[…]New Blog « Darkreverser’s Weblog[…]…
January 10, 2012 at 5:59 pm
I have a topaz book that gave the following error.
Successfully Extracted Topaz contents
Updating to color images if available
Creating cover.jpg
Processing Dictionary
K4MobiDeDrm v3.9
Error: invalid literal for int() with base 10: ‘8.synthetic.synthetic’
DRM Removal Failed.
Processing Meta Data and creating OPF
Processing StyleSheet
Error: Encryption Removal Failed
February 25, 2012 at 10:22 am
prescription sports eyewear…
[…]New Blog « Darkreverser’s Weblog[…]…
March 16, 2012 at 7:21 am
wha is a blog…
[…]New Blog « Darkreverser’s Weblog[…]…
March 20, 2012 at 6:15 am
folding tables…
[…]New Blog « Darkreverser’s Weblog[…]…
March 22, 2012 at 10:27 am
vectramind hyderabad…
[…]New Blog « Darkreverser’s Weblog[…]…
April 18, 2012 at 5:38 am
electric vehicles for sale…
[…]New Blog « Darkreverser’s Weblog[…]…
April 20, 2012 at 8:57 am
Funs.co is the leading social entertainment destination powered by the passion of fans. Music, movies, celebs, TV, and games made social….
[…]New Blog « Darkreverser’s Weblog[…]…
April 20, 2012 at 4:42 pm
Fashion News and Scoops- Celebrity scoops, pictures and tattoos | Fashion News And Scoops…
[…]New Blog « Darkreverser’s Weblog[…]…
May 6, 2012 at 1:27 pm
I purchased ebooks in the old rocket ebook format .rb. I have searched and searched but have not been able to find a method that will allow me to convert mobi or epub.
June 29, 2012 at 7:18 am
hey dont fear please spread truth about war on terror LIE george bush LIE 9/11 = BIG LIE !!!!!!!!!
July 4, 2012 at 7:59 pm
Wow that was unusual. I just wrote an incredibly long comment
but after I clicked submit my comment didn’t show up. Grrrr… well I’m not writing all that over again. Regardless, just wanted to say excellent blog!
September 7, 2012 at 1:29 pm
Peliculas HD Putlocker…
[…]New Blog « Darkreverser’s Weblog[…]…
October 6, 2012 at 7:55 pm
I am actually delighted to glance at this web site posts
which includes tons of valuable facts, thanks for providing these statistics.
October 11, 2012 at 12:58 am
Hey there! Would you mind if I share your blog with my twitter group?
There’s a lot of people that I think would really enjoy your content. Please let me know. Many thanks
April 8, 2013 at 2:46 am
These days people are carrying their laptops and notebooks
everywhere. The mouse pad is complete, and should look stunning.
Aluminum, stamped metal, crown molding featuring heavy-duty all aluminum construction is also
available for use as a decorative accent.
April 10, 2013 at 4:03 am
Once you find how each of these words will be represented
in your language, you will often end up with
hundreds if not thousands of different version of just one verb.
At the time that by age 6 or 7 years of age just
telefonsex don’t get what, to more advanced thinkers, is stunningly obvious.
April 14, 2013 at 4:32 pm
Muy bueno. Me gusta mucho el modo en que se expone. Un abrazo!
April 17, 2013 at 1:57 am
Excellent article. I will be facing some of these issues as well.
.
April 19, 2013 at 3:22 am
WOW just what I was searching for. Came here
by searching for touch screen gloves
May 6, 2013 at 5:27 am
Thanks , I have just been looking for information about this subject for a long time
and yours is the greatest I’ve came upon so far. However, what about the conclusion? Are you sure concerning the source?
July 10, 2013 at 1:51 pm
Hello, always i used to check weblog posts here in the early
hours in the dawn, for the reason that i love to find out more and more.
July 11, 2013 at 7:10 pm
Find more there:
[…]I am now not sure where you are getting your info, however good topic.[…]
August 4, 2013 at 12:05 am
If you are going for finest contents like I do,
simply visit this website all the time for the reason that it provides
quality contents, thanks
October 2, 2013 at 6:06 pm
This was a great article. Thank you for sharing it was informative.
February 20, 2014 at 3:03 am
Sadly all pasties have been removed…would love to know where to find the new scripts…when you have time.
March 10, 2014 at 7:40 am
Hello very cool web site!! Guy .. Excellent .. Amazing ..
I will bookmark your website and take the feeds also?
I am satisfied to search out a lot of helpful info here within
the put up, we need work out extra techniques in this regard, thank you
for sharing. . . . . .
April 4, 2014 at 3:13 am
Permanent installations in vehicles gave way to the portable Bag Phones, built with
a cigarette lighter plug. The etiquette of business may be the pair of written and unwritten rules of conduct that produce social interactions run more smoothly.
Keep up your eyes and ears open for any new site polices regarding this
issue.
May 26, 2014 at 9:18 pm
The Ships’s Voyages
I believe know-how just causes it to be even worse. Now there is a channel to hardly ever care, now there wouldn’t become a probability for them to find.
June 13, 2014 at 9:16 pm
hey dont fear please spread truth about war on terror LIE george bush
December 3, 2014 at 5:34 am
[…] New Blog – Darkreverser’s Weblog | Just another WordPress … – Feb 13, 2008 · I ran across a problem with the ereader2html script and footnotes. As mentioned previously, the footnotes are not being decrypted with the rest of the book…. […]
January 12, 2015 at 1:42 pm
[…] New Blog – Darkreverser’s Weblog | Just another WordPress … – Feb 13, 2008 · Hi all, I’ve finally made a blog to describe my reversing endeavours. I’m not used to blogging, so I might not write so much. Anyway, here’s the latest …… […]
May 5, 2016 at 5:25 pm
My business partners required a form a few weeks ago and were informed about a great service that has an online forms database . If you are interested in it too , here’s the link: http://goo.gl/BS8xki
March 8, 2018 at 1:45 pm
Hey great website you are running here. It seems you have tons of subscribers
May 19, 2018 at 7:04 pm
on the topic of DEDRM, I can’t seems to remove DRM from Microsoft new DRM ebook format with ext mspeub under edge web browser, with De-DRM tools for calibre
I talking about the ebooks from Microsoft store that have an new DRM format, I would like to remove the DRM from the epub files so I can convert them to read on my kindle hardware. below more info on a post about this topic.
would like to ask about Microsoft new DRM for their eBooks in the Microsoft store, is their an app to remove the DRM from MSEPUB files , under their directory called AppData\Local\Packages\Microsoft.MicrosoftEdge_8****\LocalState\library_books_***** I would like to copy the ones I buyed into my kindle but of course this new DRM format by Microsoft does not seem to be DEDRM yet, I don’t no if anyone’s working on an app that would allow be to transfer this ebook type onto my kindle paperwhite from calibre app in windows 10. calibre seems to read the ebook title if I rename msepub to epub, but its still in an DRM format when imported into calibre and the DeDRM_tools don’t work for this new MSEPUB format yet, also inside its directory is 3 folders Licenses, Covers Common. please write me back if and when an app to remove microsofts new DRM Tools can remove this new DRM format, since I would like to be able to use this ebooks in something else than just edge books in windows itself, seems to be an DRM epub format, with the ext. of msepub files when extract with 7 zip show some of the structure but of course I can’t yet remove the DRM from these epubs even extracting the files from it but I hope that info helps get DeDRM_Tools updated with an app to take care of microsofts new DRM format, again I was really hoping that I could copy the epub into calibre and convert them into kindle format for my kindle paperwrite 2 at some point in the future, since its ok to read them from edge web browser but I would prefer to be able to read them in my kindle itself if its even possible to remove its new DRM formatting at all.
May 20, 2018 at 11:14 am
I am getting the following message for all three lin.
This page is no longer available. It has either expired, been removed by its creator, or removed by one of the Pastebin staff.
August 31, 2018 at 2:29 am
otis elevators history
New Blog | Darkreverser’s Weblog
December 28, 2018 at 10:08 pm
london locksmith
New Blog | Darkreverser’s Weblog
October 1, 2020 at 11:04 pm
Hello, I wish for to subscribe for this blog to get latest updates, thus where can i do it please assist.
October 29, 2020 at 5:51 am
dorree
New Blog | Darkreverser’s Weblog
March 4, 2021 at 10:46 pm
carrie
New Blog | Darkreverser’s Weblog
March 11, 2023 at 11:11 am
dedicated server dst
New Blog | Darkreverser’s Weblog