The server move finally has a hard deadline: July 1, 2011. I have to move sup/tg/ to a new provider by then or everything simply disappears into the ether. So that means it's time for a culling!
Like the last time, the archive is going to be purged of unwanted threads by
whatever poor assholes I can rope into helping me a team of qualified, highly-trained content experts. What are the criteria, you ask? Wait, you didn't ask? WELL FUCK YOU I'M TELLING YOU ANYWAY:
Like before, the deleted threads will be moved to a graveyard where anyone who really wants that aforementioned horrible thread can still download it. (Image threads will not be given this treatment this time around. You knew better when you added it.) I will update this post or a future post with the growing list of graveyarded threads and links to download them.
When the culling is done, I will begin uploading threads by month to Dropbox, Megaupload, or whatever other service I can find as a failsafe. If I get hit by a bus, I don't want the archive to just disappear and lose the last four years' worth of amazing /tg/. I'll likely be moving to Varsbox unless someone else can suggest a good VPS provider that will fit the following criteria:
I'm going to get started on getting people to cull the archives. For those of you who give no shits either way, I've updated the archive page so that you can hide quest and draw threads now, so there's that.
Edit: For those of you helping me, I'd like you to go through a full month at a time (helped now by the handy dandy new archive page), and give me a list like the following:
Thread Number - Title - Score - Reason
You don't have to get fancy with the reason, just "quest thread" or something so I know why it should be removed. Remember, these are just recommendations; I may end up keeping a thread after all, though most of the time I'm sure the thread deserves to be eliminated. Please also include a separate list of just the thread numbers so I can plug them more easily into my tools. Extra lists of questionable items or threads to make into collections are also good.
Edit 2: Of the 6805 threads currently archived, 465 threads contain the tag 'draw' (drawfag, drawthread) and 2090 contain the tag 'quest'. That's right, a full 30% of the archive is composed of quest threads. ಠ_ಠ
Edit 3: 392 draw threads at a total of 3.8GB removed on the first pass. (Culled List) Also I guess I should clarify: I'm going to leave the "deleted" threads online until Friday or something, then they're permanently being deleted. The graveyard I'm going to leave for as long as I still have this server. I could leave Deleted and Graveyard together I guess but this gives me a directory to dump so I can get more disk space as I archive and transfer batches of threads.
Edit 4: Jesus Christ there were fucking 335 threads in May alone, 137 of which have "quest" in the tags. I need to look about changing the way threads get archived, or be extremely aggressive in future culls. Maybe something like deleting threads below 5 or 10 or something, or (ugh) requiring multiple votes to add a thread ala 4chanarchive.
Ultimate question: Should I ban quest threads, or just aggressively cull them?
Edit 5: Anon raised a good point that made me wonder: how long does Easymodo store threads? Do they have a purging policy or any of that? How far back does it go? Hell, it'd make my life massively easier if I just hosted a listing service for /tg/ that linked to archived threads on Easymodo. Would Eksopl mind? What does he use for a hosting provider, anyway? QUESTIONS, MAN.
If anyone could put me in touch with Eksopl so I can ask some questions about how he runs things over there, I'd be much obliged. Either I could just set the archive to link to Easymodo, or buy hosting from the same company.
Happy April Fool's Day everyone! I hope you're enjoying the shit all over the internet today! It's also fucking 7:30 in the morning christ I need sleep oh god why am I awake.
Only three pages today, I'll likely add one or two more later, but sleep now, oh god why. Old pages:
BEAR CHEF SAYS
We're done culling the archives! Yaaay!
You presently have
48 hours* until Friday, August 6* to get everything you want from the following list:
*or however long it takes me to turn off the server.
Anyway, the links on that page have all been updated to the old website, so you can download whatever you'd like to keep. Do not scrape the website or you will be banned. Art dumps and drawfag threads will be uploaded later to MegaUpload or something so don't bother with those, you'll get them in a tidy .rar at some point, or they're already on tgchan (I'll get the links for those and add them here). If anyone would like to officially or unofficially start transcribing writefag threads to 1d4chan please go ahead.
Remember, this is a general 48-hour window, but I'm not going to delete the threads for a long while (if ever), so if you miss it don't fret. Just ask and I'll be happy to send you a copy of whatever you'd like. Again, don't scrape the site because you feel like everything deserves to be saved; it really, really fucking doesn't, and if you haven't seen the culled list yet, you'll quickly figure out why.
After the window I'll start moving ~5GB of the surviving threads to the Xen a day until it's up to speed, cull July, and we should be all set. By next week or so we should be up to speed, unless of course I get sucked into Minecraft to the exclusion of all else... again...
Edity: Oh yeah, you'll notice the "saved" thread still show up in the culled list. I can't be fucked to bother removing them from the list, but if you asked me to save something (like those mislabeled quest threads and the like) I'll be restoring them during the move.
Edity 2: I've decided to leave the graveyard up until at least Friday, if not beyond that; the bandwidth usage from people snagging "killed" threads is way less than I thought it was going to be. Remember, the graveyard is here. I'm going to start transferring tonight.
Everything's proceeding quite a bit slower than I'd planned, but it's still proceeding. I'm almost ready to begin uploading; I figure I'll move things in an orderly fashion, something like 5GB a day, until the archives are back up to normal. As you've likely noticed, the request interface and new threads still work, so it's not like we've been losing new content during this downtime. (And Comcast hasn't exactly been stable lately, so I'm going to pick a good time and start uploading.)
As before, add any requests to save material to the post below this one (do not ask for threads to be saved anywhere else or they may be missed!).
Okay, so here's the deal. I've
roped some poor suckers politely asked a few members of the sup/tg/ IRC to help me cull the horrible, horrible threads in the archive so whittle the filesize down to a manageable level. Their guidelines are:
As mentioned previously, the list of culled threads is here. You'll notice more subentries for individual months, and the pastebins where the aforementioned
chumps beautiful, beautiful people handed me their lists, along with their commentary or reasoning. If you've got a problem with something being removed, well, first, think long and fucking hard before you ask, and if you still decide it's really worthy of remaining, let me know in the IRC.
Once we're done culling, which may take a couple days (the archives are still accepting new threads), I will restore everything at the old location for 48 hours. This will give everyone enough time to retrieve content from the graveyard (surviving threads will probably not be restored yet at that stage). After that, I'm going to completely wipe the graveyard, upload the content back in stages, and finally decommission the old server for good.
Special thanks to
schlubs goddamned Internet Heroes Aeros, PurpleXVI, Phaethon and Symmetry for mucking out the e-stables for me. Five months' worth of threads, anything under -5 and all threads tagged 'image dump' have dropped 8.75GB, and we've got many months to go.
Edit: If anyone has an answer for where to put the drawfag content, please let me know. I can't keep it for bandwidth/space reasons and certain elements inform me that 1d4chan isn't the place for them (apparently it's not the place for fucking anything). I don't want to start administering a drawfag gallery, which is why 1d4chan would have been ideal, so if someone has a solid suggestion on where I can upload everything just so people can get copies of it or something, let me know in the comments.
Edit 2: 2010 is done and most of 2009; only three months graveyarded so far but we're up to 10.3GB culled! Yaaay progress.
Edit 3: 2009 is done, total of 15.4GB removed. 2008 to go and then we start the migration.
Edit 4: 2008 is done, up to 17.5GB removed. Going to hit 2007 and start uploading piecemeal.
So it looks like Comcast has finally started actually enforcing their bandwidth caps, which are presently 250GB per month; I've used 450-475GB for the last three, and it looks like at least 300+ of that has been from the site. While I'd normally be kinda proud about that, they've threatened to terminate my service if I don't cut it out...
Over the next couple days, I'm going to be transferring sup/tg/ to the Xen server. This does have a couple benefits, such as being able to throttle bandwidth limits (to prevent scraping) and provide a faster site experience in general. The downside, of course, is that this is going to take me a lot of time and effort to move everything and get it running the way it was before.
To help make this easier on myself, I'm also going to be doing a vicious purge of the archives, since there's just a lot of shit I've left there and figured "eh, why not?" Now I need to transfer 40GB of excrement flecked with nuggets of gold over to the new location, so it's time I started mucking out the stables a bit.
It's going to be pretty bumpy, but I don't foresee any major issues, other than the usual weird complaints about culling and a few things that ran in a Windows environment that I'll need to reconfigure under Debian (like the PDF creator for the Dark Heresy chargen). Who knows, maybe all this newness will help kick me into actually completing any of those projects I've been putting off, like updating the DH chargen or creating an EP chargen.
The current list of graveyarded threads is here, though you won't be able to view them yet. As I said, this is a very strict culling, so I may kill an item that deserves saving (particularly pieces of quest threads that may or may not need to be kept for completion purposes). Point out anything I miss in the comments here or in the IRC.
As noted, many parts of the site won't function until this is done, and a few pieces are going away entirely (the art archives for example). Don't bother reporting things unless they're still screwed up a week from now.
Addendum: I've put people to work slaving in the horrormines to start culling shit. Be prepared for lengthy lists. I've also moved Maid Quest and Frost Giantess Quest under the Collections section (which may not show up until I'm done moving shit around).
It's been a quiet month, site-wise. I finally fixed the Battlefield Calixis bug in the chargen and deleted characters created before January 1, 2010, removing 16,497 entries. I also parsed and compressed another 9GB of logs from the same day 'til present (goddamn do I get a fair amount of traffic).
Nothing else much to report, other than we've got a guild on Guild Wars now. Jump into #guildwars if you're interested. Writing the Eclipse Phase character generator and adding characters from Ascension and The Radical's Handbook to the Dark Heresy chargen have been on my mind, but I've been pretty busy with work and other things. I do fully intend to get these things done one day, you'll see!
I can't think of much else to report. I also need to remember to pageinate the archive, at least when you're filtering it (so it doesn't show hundreds of entries all at once); instead of showing the last month, maybe it should just show the last 50 entries. I'm not sure how I'd like to reorganize it, but it needs organization. (And yes, this will include deciding if I want to finally put Quests in another section, or maybe just have a checkbox that lets you filter out quests specifically.)