Dekai Manga Archive [Fall 2020]

Category:
Literature - English-translated
Date:
2020-11-19 17:31 UTC
Submitter:
dmarchive
Seeders:
17
Information:
No information.
Leechers:
39
File size:
4.9 TiB
Completed:
250
Info hash:
7a72489cbccdf8257f7ffbaa61d73d912a64ab7d
Download Torrent or Magnet
### Dekai Manga Archive - Fall 2020 This is the first release of the Dekai Manga Archive [DMA] containing over 29k mangas. It includes every (as of 23/09/2020) manga on MangaDex.org that has been translated into english + some mangas from other sources, such as bato.to, mangareader.net and others. Please support this project by seeding as much and as long as you can. Keep in mind you do not need to download the complete archive in order to seed. Partially seeding only the mangas that you've downloaded is already a big help. Please also support your favourite manga websites so that they can keep the lights on. !! Make sure to buy the official mangas if/when they become available to support the authors and the manga industry !! ------------------------ **Announcement for the next DMA release (2022):** Because of various technical issues in the way I've setup this first version of the archive as well as the drastic changes on the MangaDex side (new API, new manga & chapter IDs, retroactive image compression, e.t.c.) I have decided to redo the archive for the next release. This means that the next release will not include a incremental update. It will therefore be necessary to re-download everything. All mangas will be carried over to the new release and most (>99%) chapters will also be included in the new release. The next version will make it much easier to do incremental updates in such a way that anyone could create a new diff update that could be merged with the "base" version. It will also include extensive meta data that makes it easier to write scripts to automate various things. However, new updates are dependent on MangaDex not changing their DB and API fundamentally (again), and of course, MangaDex and it's API being online. ------------------------------------------------------------------- **Statistics** + Mangas_____: 29'143 + Chapters___: 491'557 + Images_____: 11'209'077 + Size_______: 5.1TiB / 5.5TB + Size compr.: 4.9TiB / 5.4TB + Language___: English only + Last scrape: 23/09/2020 to 31/10/2020 + Last update: 02/11/2020 to 05/11/2020* * partial update of latest chapters & mangas ------------------------------------------------------------------- **Info:** - If you want to check the contents of the archive, just download the "--- README ---" folder. The following files are included: * Index.html - Website containing a list of all mangas (includes search & sort function) * Index.json - Same info as in the html file but formatted as json for easy parsing via scripts * File & Hashlist.txt - A list of every image, including MD5 hashes. Don't open this file if you don't now what you're doing, unless you want to set your PC on fire. I recommend to use the HTML website, as it is very userfriendly. Opening the HTML site will take 3-5 minutes because of it's size, during loading you may not see the sorting & search bar. Once loaded, you'll see a list of all manga titles, folders, chapters, last update timestamp, disk space and amount of files, and you can sort by any of them. (though sorting by size is a bit bugged) There is also a search bar for convenient searching. You can search by manga title, MangaDex ID or group name. - If you are looking for specific chapters of a manga and you want to check whether they are included in the archive, then just download the Info.txt file of the desired manga before downloading the zip archive. It contains a list of every chapter and every image included in the compressed zip file. - This torrent may not work with all torrent clients. We do not recommend to use rtorrent because it can crash. You may want to use a separate client just for this torrent because it may impact the performance of other torrents. ------------------------------------------------------------------- **Scripts:** - Dekai Manga Sorter (https://github.com/Painketsu/DekaiMangaSorter) - Script to organize manga into tidy folders per-volume since it's separated into chapters by default (created by Painketsu) [compatible with "Fall 2020" (DMA-v1) release] - Mangadex-DL (https://github.com/frozenpandaman/mangadex-dl) - Script to download mangas from MangaDex directly so you can create your own archive (created by frozenpandaman et al.) ------------------------------------------------------------------- **Caveats:** - Mangas with unnecessary long titles (which are more of a sentence than a title...) have been shortened to something not ridiculous. Keep that in mind when searching for a manga. - Characters in manga titles or group names that are invalid in either Windows, macOS or Linux, have been replaced with other characters. " was replaced with apostrophe ' and the characters <, >, :, \, /, |, ? and * have been replaced with an underscore _ or hyphen - . Leading dots have been encased in square brackets, i.e. [.]. So, ".hack//G.U.+" has become "[.]hack--G.U.+". This looks awkward but was necessary to ensure that folder names are displayed correctly. - Occasionally, some chapters may have a volume identifier (i.e. "v1 c001") while others have not (i.e. "c002"). This makes sorting a bit weird. This stems from inconsistent information in the source material. [The sorting issue will be fixed in the next release, but will require you either re-download everything or use a script to re-name chapters] ------------------------------------------------------------------- **Known Issues:** - A list of known issues can be found in the README folder "--- README ---/Known Issues.txt" - Report issues here: [email protected]. - Solo Leveling: Unfortunately there are some duplicate chapters. It seems there was an issue with the renaming script. You can delete chapter folders that end with the number 2. ------------------------------------------------------------------- **Some additional Infos:** + 11 scrapers worked 24/7 for ~43 days to scrape all the mangas from MangaDex using their API + Heavily customized versions of the mangadex-dl (https://github.com/frozenpandaman/mangadex-dl) python script from frozenpandaman was used to scrape MD + 270 Mangadex@Home servers were queried during scraping. ** I plan on seeding this torrent until this archive is superseeded by a new version. Please help keeping this torrent alive by seeding as much and for as long as you can.

File list

Comments - 71

Tober

User
2020-11-19 17:40 UTC (edited)
New size record

kaisama13

User
2020-11-19 17:49 UTC
Damn, just open this page take several seconds.

shellshock

User
2020-11-19 18:11 UTC
Im guessing this is Fanscans/Expired Licenses only

Silverita

User
2020-11-19 18:31 UTC
Wow, I'm still shocked because of the size ? Any rare gem you've found that you recommend me?

Ingenioussubs

User
2020-11-19 21:36 UTC
Holy fuck this is huge! Finally someone's surpassed 4.6TB. Hopefully this one will seed.

Fuken_Weeb

User
2020-11-19 21:46 UTC
It's nice and all, but I don't think this is the most practical way to share this.

areen

User
2020-11-19 22:58 UTC
![Kek](https://cdn.discordapp.com/attachments/691160465342070805/779118241598537758/751482104583422063.png)

oxyghene

User
2020-11-20 00:14 UTC
so i downloaded a very very small chunk of this. quite amazing indeed. it is basically an immense backup from mangadex. most of what I checked checks out but theres a bunch stuff that got deleted from there

Spc_away

User
2020-11-20 01:07 UTC
You need several lifes to read all these.

Marv

User
2020-11-20 01:58 UTC (edited)
> `- Occasionally, some chapters may have a volume identifier (i.e. "v1 c001") while others have not (i.e. "c002"). This makes sorting a bit weird. This stems from inconsistent information in the source material.` Why not use the naming scheme (which is intended for archiving) to prevent these issues? Also, why combine the whole of each manga series into a single zip? Would add a lot of unnecessary work to updating the torrent if it ever is updated, I don't see why each manga couldn't be multiple volumes/chapters.

Pajeet

User
2020-11-20 02:05 UTC
The torrent client running on my potato PC simply cannot handle the sheer **Chad Factor** of this torrent. You Sir/Ma'am, are an absolute legend!

dmarchive (uploader)

User
2020-11-20 02:57 UTC (edited)
@oxyghene: most of the material is indeed from mangadex, but nothing was deleted, at least not on purpose. It is actually the opposite, there are various mangas & chapters in this archive that are not on MangaDex (anymore). If something is missing from the archive but is on mangadex (since before 23/09/2020), then that is an error and it would be great if you could report it here: dekaimangaarchive+report [at] protonmail.com. @Marv: If you have an idea for a better naming scheme then please let me know and send me an email with your idea. But there are many reasons why I've decide on the current naming scheme. Unfortunately over the 29k mangas there are many inconsistencies, both on mangadex, bato.to and other sites. For example, there are mangas where the chapters don't increment continuously, but reset with every volume. So there is a Vol 1 Chapter 1, Vol 2 Chapter 1, and so on. But then when a new group comes around and adds new chapters they suddenly decide to increment the chapters continuously. In that case, you can have two identical chapters with a different name: c050 and v8 c004 - different name, different group, same chapter... Mangadex would've needed to enforce a naming scheme from the beginning to prevent this. Fixing it now is not feasible because it cannot be reliably automated. The second problem is that I want to be able to merge new chapters with the archive and need to be able to reference chapters that are already in the archive with the chapters on mangadex. I am currently working on an alternative solution for this based on hashes. So in the future I would be able to identify the chapters based on the image hashes instead of the chapter names. And the reason for having a single zip file per manga is because of limitations with the torrent protocol. There are almost 500k chapters in this archive but I couldn't get a torrent with 500k files to work. I personally would've preferred offering the archive uncompressed without any zip files at all.

OmegaKaiser

User
2020-11-20 03:54 UTC
The ultimate torrent of ultimate destiny: A thought-provoking downloading and/or seeding experience! ![alt text](https://i.imgur.com/v5n5UIR.jpg) @dmarchive I ask of thee: Art thou our master...?

coolcheesewhiz

User
2020-11-20 05:03 UTC
This is the backlog to end all backlogs.

Persona5RoyalOnSwitch

User
2020-11-20 05:37 UTC
[@Silverita](https://nyaa.si/user/Silverita) If you want something super underground, my first choice would be Noru (also called Roe Deer: The Couriers). It's a sci-fi post-apocalyptic webtoon that's really short (18 chapters) and surprisingly good. Has some seinen feels to it.

Sweenzz

User
2020-11-20 05:45 UTC
yoooo thanks you man

oxyghene

User
2020-11-20 14:14 UTC
@dmarchive yeah, no, i meant stuff that was removed from dex, like for example chapters 1 to 10 of Oshi no Ko. this is so complete, its amazing. thanks so much for this. by the way, i noticed (for me at least) some files are zipped totally out of order (Sousou no Frieren, for instance) and some are ´weird´ in format (like Shuumatsu no Valkyrie, to give an example) if you open the zip file straight away with a comic or manga reader (like Comic Zeal on the iPad)

dmarchive (uploader)

User
2020-11-20 15:06 UTC (edited)
@oxyghene yes that is a known problem (see "caveats" in the description). The problem is that volume numbers are not always provided on mangadex. If the chapters have not yet been released in a volume, then it is not allowed to provide a volume number on mangadex and sometimes people just forget or are too lazy to check in which volume the chapter was released and omit it. If you go to the Sousou no frieren page on mangadex you'll see that the latest chapters have no volume number. In the next version I'll probably reverse the order (i.e. "c001 v1" instead of "v1 c001"). That will break sorting in some mangas with non-consecutive numbering, but it will fix more than it breaks. Alternatively in most mangas you can just order by the modify date to get the correct order - which is what I normally do and that is also why I've not realized the impact of this issue until now. Regarding the "weird" format, I could not find any issues with the zip file you mentioned. I can unzip it without issues on my PCs. You can send a screenshot of the issue to my mail, and I'll take a look at it.

XRA9

User
2020-11-20 18:56 UTC
https://files.catbox.moe/q70p27.pdf ^ This is "the" naming scheme that was talked about earlier, it was formulated for the exact situation you found yourself in (wildly inconsistent naming even within series). It can be pants-on-head retarded at times, but sorts perfectly.

Marv

User
2020-11-20 20:38 UTC
Yes, as XRA linked, the madokami naming scheme. And as for torrent limitations, I believe it would be better to upload multiple partial torrents (based on alphabetical order, manga id, or some other metric) with expanded files instead of merging each manga together.

dmarchive (uploader)

User
2020-11-21 03:06 UTC (edited)
@XRA9 thanks for the link. I'll try to update the archive to align with the madokami naming scheme in the next release. @Marv I understand why you prefer that, but I personally do not like torrents that are split into multiple parts. And in this case it would need many parts if each chapter was zipped seperately. I'll think about it and consider it for the next version. Thank you both for your input, please let me know if you have other ideas.

ATiVerse

User
2020-11-21 23:21 UTC (edited)
Deduplicating this against existing collections is going to be impossible, even the titles are inconsistently handled. We're going to need a programmatic approach to recompiling this entire dataset into something sane assuming it actually gets seeded fully, then likley a multi part rerelease. As expected for a data dump really, but stil l frustrating. File sort order is also not consistent between operating systems, despite their efforts.

dmarchive (uploader)

User
2020-11-22 10:53 UTC
@ATiVerse: Please do all that. I'd much rather just download a torrent that someone else made and complain in the comments than doing the work myself.

doodooface

User
2020-11-25 06:32 UTC
@dmarchive Amazing!! I'll leave this on my seedbox for a while.

Blanchimont

User
2020-11-25 16:35 UTC (edited)
Wonder if it's pushing it while on mobile home plan? Haven't had any complaints before, so... Thanks for the archive!

RinaChan

User
2020-11-27 14:03 UTC
This is the bloated nyaa history recorded so far that I know **O_O** Thanks!

AnimePL

User
2020-11-27 19:42 UTC
What? This is 5TB! I've not enough hard drive, my all disks have 2TB

Shakil

User
2020-12-10 14:30 UTC
Give me powerful internet and 6TB Drive, I will seed my entire life...

Blanchimont

User
2020-12-10 23:04 UTC
Too bad this snapshot wasn't taken at a time when ThePaulBunyanTrophy releases were still on mangadex. A lot of good series missing there, have to rely on aggregators for those...

NotBot

User
2020-12-12 21:38 UTC (edited)
![Sugoi Dekai](https://i.imgur.com/4a9MVlL.jpg)

DARK_A

User
2020-12-20 19:32 UTC
WOW JUST WOOOOOOW THIS IS SUPER MANGA SIZE I HOPE SOMEDAY I CAN DOWNLAD IT THANK SOOOOO MUCH

yvsei

User
2020-12-24 08:11 UTC
FAT fucking torrent

Blanchimont

User
2020-12-24 21:32 UTC (edited)
Done downloading. Now I wonder if I'm able to seed it fully before the next update... Wonder if there's any change the uploader might consider adding archives of scanlators who've had 'disagreements' with mangadex, like the case with the above-mentioned ThePaulBunyanTrophy? That's if the stuff can be found of course, on aggregators or elsewhere...

DARK_A

User
2021-01-07 06:46 UTC
I AM LIKE YOU I DONT LIKE TO SPLIT FILES I LIKE THEM ALL TOGETHER WHEN YOU ARE GOING TO UPDATE THE FILES ARE YOU GOING TO PUT NEW TORRENT OR JUST REHASH ? AND WHEN ARE YOU GOING TO IM NOT REQUISTING IT JUST ASKING PS: I LOVE WRITING IN CAPS BECAUSE IT EASY ON THE EYES

dmarchive (uploader)

User
2021-01-09 00:16 UTC (edited)
@Blanchimont - I was not aware of that drama. I have it added the list of 50 mangas I could find to the to do list. @DARK_A: The next torrent will, most likely, be distributed as an incremental as well as a new complete torrent, replacing this one. Though It will probably be best to re-download the whole archive as I will be changing the folder structure as suggested by previous comments. When the time comes, I'll need to figure out some technical issues so I cannot yet guarantee anything. Also, most of the mangas rely on Mangadex and they are as of late completely unstable because of overwhelming traffic. So as long as MD is unstable, I cannot do anything and do not wish to add additional load onto their servers. So if you wish to help please throw some BTC at MD. Regarding the timeframe: I'll update when it is technically possible and there is a meaningful change. Right now I'm thinking that the new torrent should be over 6TB in size. Since about 70-80GB are added to MD monthly, that means it will probably be a spring 2022 release.

splintof

User
2021-01-10 00:18 UTC
thank you so much for the work

DARK_A

User
2021-01-12 15:20 UTC
SO NEXT YEAR GOD WILLING THANK YOU AND TAKE YOUR TIME WE ARE SUPER GREATFULL TO YOU FOR THIS MY ADVICE WAIT MORE YEARS BECAUSE THIS ONE IS VERY GOOD BY THE WAY ARE YOU USING hakuneko ? IF NOT USE IT WILL MAKE YOUR LIFE WAY EASIER AND AGAIN THANK YOU SO MUCH TAKE CARE OF YOURSELF AND YOUR FAMILY BEST REGARDS

DefNotRok

User
2021-01-25 22:46 UTC
@Blanchimont Blanchimont How the fuck do you have enough space for this wtf and how long did you take to download

SupahXYT

User
2021-02-01 04:02 UTC
My "puny" 1TB VPS definitely won't be able to store this, I think I'm gonna have to download it on my own machine using RAID 0 or something... Not that I have the money for that.

Blanchimont

User
2021-02-01 16:00 UTC (edited)
@DefNotRok Download finished at Christmas, but seeding will take a couple of months still. Space is not a problem, I have just short of 60TB on my main machine, half of it free even after this. Connection is wireless internet. I do have my old dsl connection as backup, but I hardly ever use it as the wireless one is a lot faster...

Gradius

User
2021-02-04 00:52 UTC
Holy SH*T! Over 5TB.

qtpi

User
2021-02-17 18:10 UTC
It's beautiful. Thank you.

rarenight

User
2021-02-27 11:06 UTC (edited)
Holy shit. This is absolutely incredible and a dream come true. This project is the single most comprehensive archive of fan translated Manga available on the Internet today, and provides you with a literal lifetime's worth of manga to read. Traditionally, trying to archive fan translations has been esoteric and profoundly difficult to achieve, but the Dekai Manga Archive project blows past that and achieves full preservation effortlessly. Every team uploads all their new projects to Mangadex...so by archiving Mangadex, you archive the entire industry. It's genius and I can't believe you actually did it. AND you added checksums to each file which is much appreciated. Sure, other private projects like Madokami or Animebytes might have more content in aggregate (like licensed scans), but unlike those sources, this torrent allows easy and unlimited access to everything all at once, which is what true preservation is about. Effortless and freely available for anyone to access, fully offline with no concerns of censorship, removal or bans, and with no bullshit restrictions attached. The Internet collectively thanks you, OP. I hope you can continue making updates for the DMA far into the future because I will definitely seed and support this for as long as I can. This torrent is worth the bandwidth.

Kurdistan0001

User
2021-03-23 11:06 UTC
I wish I could download it and keep seeding it for you guys , thank you this is so beautiful

itamar1324

User
2021-03-26 12:51 UTC
i need to buy a new hard drive just for this holy shit this is huge

WhenMarkers

User
2021-04-03 11:23 UTC
Wait a minute... WHY?! My drive!!! I gotta buy new one... LOL

DaisukiAnime

User
2021-04-30 08:03 UTC
Killed my storage, but you are doing gods work So beautiful i can cry :')

Painketsu

User
2021-05-20 01:12 UTC
Made a small script to merge and sort all chapters into just volumes for easier reading if anyone's interested https://github.com/Painketsu/DekaiMangaSorter

dmarchive (uploader)

User
2021-05-20 09:27 UTC (edited)
@Painketsu: Nice script! I added it to the description. @All: I've posted an update regarding the next DMA release in the description.

Blanchimont

User
2021-06-05 17:52 UTC
@dmarchive Just to confirm, when/if the next iteration is posted in 2022, will it include only the then current snapshot of MD, or will it include the stuff in this that won't be in it? As every once in a while stuff like fan-translations of something officially licensed gets removed at request(like Nisekoi, or when MangaPlus chapter links replace the fan-translated ones), or when scanlators themselves remove their stuff, leading to the fact that while each new snapshot would have plenty of new material, there would always also be some things not carried over from the previous one. With differential updates that isn't likely be an issue as you just add the new one over the existing one. But as the next one will be a full archive snapshot, will it preserve any missing old stuff from this?

dmarchive (uploader)

User
2021-06-05 21:57 UTC
Yes and No, all mangas that are in the archive right now, will also be in the new updated archive, even if they have been removed from MD in the mean time. The same goes for big "purges" where the manga still exists on MD, but most of the chapters have been removed. However, I cannot guarantee that all chapters will make it into the new archive. But I expect the number of lost chapters to be very small. The technical reason for why that is would take a while to explain, but I have contacted MD staff and they have said that there may be a way to fix this issue. Going forward with the new version and it's improvements, this same issue will not happen anymore.

FateXBlood

User
2021-06-28 11:55 UTC
> It includes every manga on MangaDex.org that has been translated into English Oh, how sad my heart beat on seeing no non-English stuff.

shellshock

User
2021-07-01 17:06 UTC
This might be a little late but I would like to ask this is 100% the original quality correct? Like when transferring files you didn't lower the quality or save jpg at 90% though there isn't that big of a difference Also I'd recommend simply just doing completed series. Not this is a big deal or anything but doing completed wouldn't need you to constantly update or add new chapters But again you know your choice. Is nice to know your still checking the comments here

dmarchive (uploader)

User
2021-07-01 21:25 UTC (edited)
All images are taken from the source without any changes. No compression or other alterations are made. The only exceptions are corrupted images when the source image is also corrupt. In these cases, I may attempt to recover, and therefore change the file. If I do that, I'll add it in the "Known Issues" list in the README folder. And you are absolutely right regarding the completed series. Only including completed series would be much easier, at least in theory. Unfortunately it does not work in practice, because there is - as far as I know - no way to check whether all chapters of a series for a specific language are on MD. The status "completed" or " ongoing" refers to the manga itself, not the scanlation. So in practice there is no easy way to check for that. But since I am going to do an incremental setup for myself anyway it is not much additional work creating an incremental torrent, at least once everything is setup for the new API.

shellshock

User
2021-07-02 05:08 UTC (edited)
Cross check with mangaupdates might help https://www.mangaupdates.com/series.html?orderby=rating&display=list&filter=scanlated - Though tbh it will also say Completely scanlated if the officials are available illegally online. But at that point that doesn't matter. This is usually where i search to find fully translated manga myself Also though not now. But didn't Mangadex have a END at the last chapter?

dmarchive (uploader)

User
2021-07-02 10:23 UTC (edited)
It is possible to crossreference with other DBs, but since I'll do an incremental anyway I do not plan on implementing a check like that. But if someone does do a crossreference then it would be great if that person could send me the list of completed series :D I'll gladly include it in the metadata of the next release. And yes you can also check for the "END" chapter, but unfortunately not every group remembers to actually set the last chapter as "END". And that does not mean that all chapters are actually on MD, it just means the last chapter is. So you would need to add a consistency check and crossreference all chapters on MD with a list of all originally released chapters. And I did originally think of doing that, but to be honest it is too much of a hassle for me in addition to assemble and quality-check the archive. And besides, one of the primary goals for me is to have a copy of MD (and later also other sources such as bato.to) that is as complete as possible, for when MD goes down (again), and someday MD will go down permanently. I think we have learned that such websites do normally have a very limited lifetime unfortunately. So I want to include as much as possible in the archive so that when that happens, as little as possible is lost. And with the next release, anyone could theoretically create their own MD (including API) by merging the base release with the incrementals.

jnx_r

User
2021-10-05 19:10 UTC (edited)
Okay, my PC is ready. I've brought a new hard disk for the upcoming 2022 version. Can't wait for it now. Please deliver OP^. EDIT: Oh my, where are my manners, forgot to thank you for sharing, thank you OP! You are just awesome, I shall always remember this favor.

hustle

User
2021-12-19 02:05 UTC
(rolling up sleeves I will take on the challenge.

seriolover

User
2022-02-20 14:17 UTC
Something like this but with raw japanese manga would be awesome . I guess you used your own scraper script or maybe a combination of wvget or hhtrack, I consider?

dmarchive (uploader)

User
2022-05-18 18:34 UTC
@seriolover I used a python script called mangadex-dl (see description for more info) which uses the request library. I heavily modified it for my use-case but it worked surprisingly well, given that it probably was never intended for this. For the next release (I'm working on it) I'll use self-written python scripts, probably also based on the requests library + Postgres for the DB. P.S: I waited a bit with starting work on the next release because it took some time for the scanlation groups to upload all the chapters which were released while MD was offline.

jnx_r

User
2022-05-19 15:17 UTC
@OP, hmm, that's alright, so long as the plan is still on my friend. Tbh I'm pretty elated. I've prepared around 40tb for this. You don't really need to go through the trouble of compressing everything if you don't have to, especially if it has an impact on the quality.

iqfqrzlu

User
2022-07-06 21:49 UTC
I wonder how long it would take to read all of it... probably a couple decades.

Mr_Magus

User
2022-08-04 15:52 UTC
Holy fuck, this is going to take me close to a year to download with my bandwidth and hd space limitations. I think I might get a 5tb eternal hard drive just for this torrent.

Noisyboy1040

User
2023-07-13 01:59 UTC
Is this project still alive?

dmarchive (uploader)

User
2024-03-01 04:43 UTC
@Noisyboy1040 Unfortunately I didn't have a lot of time to invest in the project and I encountered some tech issues with the new API & merging the new with the old archive. I'll eventually get around to it, but it will take some time. Maybe this winter.