Discovered in the Bleak Midwinter - January 2

[kiwisue.livejournal.com]

Happy New Year to all. May 2018 bring us good things, and "may the world be kind to you", and your loved ones, particularly if 2017 was tough going.

Over the last few months I've been party to conversations about fic: finding fic, saving it, knowing where it all is, and so on. The Automated Hatstand (AH) ceased to exist in August, but the older Hatstand archive is still operating. Hagsrus has run the Proslib mailing list consistently for the past umpty-thump years (more than a baker's dozen?). However the Circuit Archive (CA) has not uploaded any new stories since 2010 (with a sharp falling off for a year or two before that), and there are stability problems with the ageing software used by the site.


There are resources that help identify stories, and zines, such as Palely Loitering and prosfinder, and these can help put readers in touch with other people who have a copy of a story. Some writers/publishers still have personal websites:
Georgina Kirrin
Sebastian
Oblique Publications Archive

Of course there are other sites that have gone away forever, and I think fandom is more conscious than ever that there is very little that is permanent on the internet, especially personal sites, and there are only so many pages that have been captured by the Wayback machine. People want to know where the fic is, and if it is safe there, not for the writers alone, but for the readers too.

So where's this going?

Just before the AH closed its doors in August, I made a list of authors and titles. I then started to cross-match that data with the other main archives, with the aim of figuring out:
- What stories were hosted on the AH?
- Where else are the stories archived now?
- What stories do not appear to be archived anywhere now the AH has been deleted?

I got a list of 728 stories (698 when duplicates are removed). Then I downloaded the story lists from the Circuit Archive, Hatstand proper, Proslib and AO3, and crossreferenced. The upshot: out of 698 individual stories, 150 are not archived anywhere else*.

Because the Circuit Archive has been flaky lately I have done much the same thing for the 1954 files (including art and vids) on that archive. The only difference is that I've run out of time so I have only crossmatched with Proslib and AO3 at this stage. It looks as though there are about 180 stories that aren't archived anywhere other than the CA**. That's a lot of fic potentially gone one day. Anyway...

A crossmatched listing of the fic that was on-line at the Automated Hatstand, together with another list of the fanworks currently on-line at the Circuit Archive as of December 2017, is now available as a shareable googlesheet for your downloading pleasure: https://drive.google.com/open?id=1b8KCbIm1qQszQPq3YDXsZvE9RevRI1QFqmd2Bzz8gV4

The idea is that this becomes a resource that people can use to find out what stories they might be missing, and to decide on the ones they want to find.

I've removed formulae and other columns that might make the individual archive sheets too confusing. I'll progress the other archives when time permits and I intend to keep up to date with both AO3 and Proslib. Ideally it would be better to design a relational database and solidify some protocols to deal with the many slight variances between archives, but I have limited dB access now I'm not working, and most people find it easier to deal with a basic spreadsheet or googledoc.

There is a modified copy of the current Proslib index at http://hatstand.slashcity.net/proslib/dvdindex.html

*I think quite a few people downloaded the pdf's before the archive closed, so if you are after a story you should be able to find it by asking the usual sources.
**Of the "missing" stories, some may be in the queue to be sent out to proslib subscribers, of course, and some may be waiting for authors to tinker with them before uploading to AO3. There is also a very small number locked to members of AO3 only, but that isn't a problem for fans with an AO3 login. I have listed these as "no" against AO3 just in case.

Comments

[byslantedlight.livejournal.com]

What an excellent resource - and epic work, as well! Useful and interesting to see what was posted where (some of which I suspect is connected to which authors were active in fandom when, for instance) - and that three of my fics aren't saved to any other archive (when I specifically thought that one was, in fact).

Of course some of them are saved elsewhere - any that were posted to lj challenges or communities, for instance, are probably still there.Though the trick is finding them, of course. I've tried to index discoveredinalj and ci5hq under authors and titles, but I strongly suspect there are gaps...

[kiwisue.livejournal.com]

The initial work was in making the lists. CA was easy (run a blank search all); proslib already has an index; Hatstand wasn't too hard, the hard work for AutoHatstand was downloading the files which I wanted to do anyway, giving them useful names, I could then do a command prompt listing of the save directory.

AO3 was the bugger. 2600+ fics, 20 fics per page, clean up each and every page. I asked them if there was a report that could be run - apparently not. I got quite quick at it thankfully, but the numbers are relatively low now. That's how I twigged there was a problem with Angel in the Dark - there was one less fic than I thought there should be, given additions that I saw in the fandom search and RSS feed, and went searching for info...

I did think about the lj comms, there is probably a mass of fic there that hasn't been uploaded - of course some of it would have been edited and then uploaded too - are the gaps because you aren't sure every entry was tagged?

[byslantedlight.livejournal.com]

Yeah, various people just never sent their stuff to archives, or said "later" in the archiving header, and never did (I suspect I've been guilty of the latter myself - but then I do tend to archive myself too).

The suspected gaps in my comm indexes are just because I suspect I got so far with tagging, either backwards or forwards, and then paused for a break and might not always have got back to finish. I need to go back through the posts from scratch and check the tags - and I will when I have time... *g*

[kiwisue.livejournal.com]

Oh, I definitely know some of mine are in that lot. Time to clean up my act!

DIALJ's probably got the most actual fic. Then there's the journal that you and Josey used to give prompts like the geography challenge, with the results on individual lj's. At least I can say 'Sleeping on the Beach' and some of the others are archived.

[byslantedlight.livejournal.com]

Josey and I alternated journals for a while when we were doing the National Curriculum challenges - it was what prompted us to start discoveredinalj, in fact! *g*

[merentha13.livejournal.com]

Thank you for this! A great resource that must have taken a lot of work. This is really appreciated!

[kiwisue.livejournal.com]

I'm lucky that I have a reasonable amount of Excel experience. So I can turn text into columns easily (I can see even more ways to clean that up now). I can run look up tables of one listing against another. Apart from keeping up with AO3, the thing that takes time is manually checking the "no" responses. Extra/end spaces in titles, "The" and "A" as first or last word, ellipses of 3 or 4 dots, different pseuds, & for and - the list is rather long *g*. I'm sure what I've done is not perfect, but I will keep working on it.

Thanks!

[unbelievable2.livejournal.com]

Hard work! Thank you!
I must admit I have had huge trouble with Circuit Archive recently - so much so I thought it was my kit and stopped even bothering to try. I didnt realise there were known  problems. :(

[kiwisue.livejournal.com]

There were comments about the down time at prosfinder and a couple of other journals. As the maintainer has been incommunicado for years it is impossible to know exactly what is going on. However my experience is that the archive is up most of the time, and extended down time is likely to coincide with weekends and holidays. This fits in with the scenario of an archive based on a good but outdated open source system that the maintainer keeps working on to fix bugs. Some elements such as author contact details haven't worked for ages.

Over five years old now, so bear that in mind while reading:
http://the-safehouse.livejournal.com/1272533.html

[msmoat.livejournal.com]

Wow! What a tremendous service you've done for the Pros community. Thank you!

[kiwisue.livejournal.com]

Well I think it should be useful in tandem with other efforts. Perhaps we can reignite an awareness that saving and sharing fic is for the community, as well as one's individual library.

[shooting2kill.livejournal.com]

Talk about a (very useful) labour of love! Well done and many thanks.

[kiwisue.livejournal.com]

I hope it's useful, thanks!

[macklingirl.livejournal.com]

Thank you for the list.

I'm happy that I saved the whole CA and all stories from AH. And I downloaded most of the stories at AO3. I'm working now on the rest and The Hatstand and the stories from various LJ-Journals. But it is hard work and as I'm working with a pc the whole day, sometimes I don't like to look into the monitor at the evening.

But I'm willing to try to save all I could reach. :-)

Edit: I'm thinking of making copying the ProsLib CD onto my harddrive and then add all stories from AH and AO3 into the authors folders. But I think that would be a work for two or more years. *g*

Did I ever mention that fan comes from fanatic? Sometimes it suites me. ;-)

[kiwisue.livejournal.com]

I have a copy of the Proslib CD on my hard drive. It makes the folders more accessible. The auto hatstand stories are in another folder though, because I like to keep straight in my head where things come from. Also some stories on the CD are mailing list only, or they are the same story under a different name. It's quite confusing. That's why I started the spreadsheet, to try to match up the information between the folders.

I think there are a few people who have a mission to find and save the fic *g*.

[macklingirl.livejournal.com]

Yes, you're right. My idea really doesn't make sense with different stories with the same titel. Today I found some stories from people I never heard of via the wayback machine. :-)

[kiwisue.livejournal.com]

"Coming Home" is popular - a lot of authors have written that one!

[cim3745.livejournal.com]

Many thanks for that time-consuming work.

[kiwisue.livejournal.com]

Thank you! I hope it's useful.

[sc-fossil.livejournal.com]

Thanks for doing this!

[kiwisue.livejournal.com]

I hope it's useful.

[sw33n3y.livejournal.com]

Thanks for all the work you've put into this! It's good to know that historic Pros works are being catalogued.

[kiwisue.livejournal.com]

Thanks! The archives themselves, and proslib, are the biggest preservation effort out there. I guess what some of us are trying to do by recording their work is a kind of back-up, especially as things change around us and in fandom.

[freetraveller15.livejournal.com]

Thanks so much for your hard work! 😊

[kiwisue.livejournal.com]

Thanks!

[loxleyprince.livejournal.com]

Wow! What a fabulous resource! Thank you for this Pros-y Christmas present!

[cyanne]

This is an amazing resource, thanks for putting all the work in and for sharing it with us. I hate the thought of any history getting lost and it's so hard with everything being so fragments, and you did a wonderful job pulling it all together.