We're done culling the archives! Yaaay!
You presently have
48 hours* until Friday, August 6* to get everything you want from the following list:
*or however long it takes me to turn off the server.
Anyway, the links on that page have all been updated to the old website, so you can download whatever you'd like to keep. Do not scrape the website or you will be banned. Art dumps and drawfag threads will be uploaded later to MegaUpload or something so don't bother with those, you'll get them in a tidy .rar at some point, or they're already on tgchan (I'll get the links for those and add them here). If anyone would like to officially or unofficially start transcribing writefag threads to 1d4chan please go ahead.
Remember, this is a general 48-hour window, but I'm not going to delete the threads for a long while (if ever), so if you miss it don't fret. Just ask and I'll be happy to send you a copy of whatever you'd like. Again, don't scrape the site because you feel like everything deserves to be saved; it really, really fucking doesn't, and if you haven't seen the culled list yet, you'll quickly figure out why.
After the window I'll start moving ~5GB of the surviving threads to the Xen a day until it's up to speed, cull July, and we should be all set. By next week or so we should be up to speed, unless of course I get sucked into Minecraft to the exclusion of all else... again...
Edity: Oh yeah, you'll notice the "saved" thread still show up in the culled list. I can't be fucked to bother removing them from the list, but if you asked me to save something (like those mislabeled quest threads and the like) I'll be restoring them during the move.
Edity 2: I've decided to leave the graveyard up until at least Friday, if not beyond that; the bandwidth usage from people snagging "killed" threads is way less than I thought it was going to be. Remember, the graveyard is here. I'm going to start transferring tonight.
Everything's proceeding quite a bit slower than I'd planned, but it's still proceeding. I'm almost ready to begin uploading; I figure I'll move things in an orderly fashion, something like 5GB a day, until the archives are back up to normal. As you've likely noticed, the request interface and new threads still work, so it's not like we've been losing new content during this downtime. (And Comcast hasn't exactly been stable lately, so I'm going to pick a good time and start uploading.)
As before, add any requests to save material to the post below this one (do not ask for threads to be saved anywhere else or they may be missed!).
Okay, so here's the deal. I've
roped some poor suckers politely asked a few members of the sup/tg/ IRC to help me cull the horrible, horrible threads in the archive so whittle the filesize down to a manageable level. Their guidelines are:
As mentioned previously, the list of culled threads is here. You'll notice more subentries for individual months, and the pastebins where the aforementioned
chumps beautiful, beautiful people handed me their lists, along with their commentary or reasoning. If you've got a problem with something being removed, well, first, think long and fucking hard before you ask, and if you still decide it's really worthy of remaining, let me know in the IRC.
Once we're done culling, which may take a couple days (the archives are still accepting new threads), I will restore everything at the old location for 48 hours. This will give everyone enough time to retrieve content from the graveyard (surviving threads will probably not be restored yet at that stage). After that, I'm going to completely wipe the graveyard, upload the content back in stages, and finally decommission the old server for good.
Special thanks to
schlubs goddamned Internet Heroes Aeros, PurpleXVI, Phaethon and Symmetry for mucking out the e-stables for me. Five months' worth of threads, anything under -5 and all threads tagged 'image dump' have dropped 8.75GB, and we've got many months to go.
Edit: If anyone has an answer for where to put the drawfag content, please let me know. I can't keep it for bandwidth/space reasons and certain elements inform me that 1d4chan isn't the place for them (apparently it's not the place for fucking anything). I don't want to start administering a drawfag gallery, which is why 1d4chan would have been ideal, so if someone has a solid suggestion on where I can upload everything just so people can get copies of it or something, let me know in the comments.
Edit 2: 2010 is done and most of 2009; only three months graveyarded so far but we're up to 10.3GB culled! Yaaay progress.
Edit 3: 2009 is done, total of 15.4GB removed. 2008 to go and then we start the migration.
Edit 4: 2008 is done, up to 17.5GB removed. Going to hit 2007 and start uploading piecemeal.
So it looks like Comcast has finally started actually enforcing their bandwidth caps, which are presently 250GB per month; I've used 450-475GB for the last three, and it looks like at least 300+ of that has been from the site. While I'd normally be kinda proud about that, they've threatened to terminate my service if I don't cut it out...
Over the next couple days, I'm going to be transferring sup/tg/ to the Xen server. This does have a couple benefits, such as being able to throttle bandwidth limits (to prevent scraping) and provide a faster site experience in general. The downside, of course, is that this is going to take me a lot of time and effort to move everything and get it running the way it was before.
To help make this easier on myself, I'm also going to be doing a vicious purge of the archives, since there's just a lot of shit I've left there and figured "eh, why not?" Now I need to transfer 40GB of excrement flecked with nuggets of gold over to the new location, so it's time I started mucking out the stables a bit.
It's going to be pretty bumpy, but I don't foresee any major issues, other than the usual weird complaints about culling and a few things that ran in a Windows environment that I'll need to reconfigure under Debian (like the PDF creator for the Dark Heresy chargen). Who knows, maybe all this newness will help kick me into actually completing any of those projects I've been putting off, like updating the DH chargen or creating an EP chargen.
The current list of graveyarded threads is here, though you won't be able to view them yet. As I said, this is a very strict culling, so I may kill an item that deserves saving (particularly pieces of quest threads that may or may not need to be kept for completion purposes). Point out anything I miss in the comments here or in the IRC.
As noted, many parts of the site won't function until this is done, and a few pieces are going away entirely (the art archives for example). Don't bother reporting things unless they're still screwed up a week from now.
Addendum: I've put people to work slaving in the horrormines to start culling shit. Be prepared for lengthy lists. I've also moved Maid Quest and Frost Giantess Quest under the Collections section (which may not show up until I'm done moving shit around).
It's been a quiet month, site-wise. I finally fixed the Battlefield Calixis bug in the chargen and deleted characters created before January 1, 2010, removing 16,497 entries. I also parsed and compressed another 9GB of logs from the same day 'til present (goddamn do I get a fair amount of traffic).
Nothing else much to report, other than we've got a guild on Guild Wars now. Jump into #guildwars if you're interested. Writing the Eclipse Phase character generator and adding characters from Ascension and The Radical's Handbook to the Dark Heresy chargen have been on my mind, but I've been pretty busy with work and other things. I do fully intend to get these things done one day, you'll see!
I can't think of much else to report. I also need to remember to pageinate the archive, at least when you're filtering it (so it doesn't show hundreds of entries all at once); instead of showing the last month, maybe it should just show the last 50 entries. I'm not sure how I'd like to reorganize it, but it needs organization. (And yes, this will include deciding if I want to finally put Quests in another section, or maybe just have a checkbox that lets you filter out quests specifically.)
Okay, well, apparently I missed a bunch of thumbnails between December 29 and January 10, because nearly 160 threads were full of 4chan leech images for no readily apparent reason. I manually fixed the lot of them (well, using a script, but still had to do it one-by-one to double-check myself). If you notice anything clearly broken, please let me know so I don't spend a couple hours fixing shit.
Heeey I'm finally going to start cleaning out the archive! Happy day! It's only been... most of a year!
As a result of the really... really long while it's been since I've purged the unwanted content from the archives, I'm going to try to delete as much shit as possible as quickly as possible. This means no more funny comments, unfortunately (except for really egregious threads), and I'm going to cull pretty much everything in the red.
I made myself some new tools to make it easier to cull shit, so hopefully I'll get rid of more, faster, and more accurately (e.g. easily removing all threads tagged "troll"). As usual, I'll put everything deleted into a graveyard first to allow people to save copies of anything they think deserves saving.
And now, the current list: