Ficly

Data Integrity

Warning: Geeky Content Ahead

I learned a lot when ficlets got shut down. It was just the latest in a series of web products that I loved that died untimely deaths. Thankfully, in ficlets’ case, we had some warning and we were able to save the stories. Even though Jason and I built ficlets, we didn’t have time to add a real data retrieval API into it or any way for users to back up their own stories – we had to crawl the site and hope we could get decent data out of it. So, when we started talking about building Ficly, I wanted to make sure that no matter what happened to the site or the hardware it runs on, the stories would be safe.

Here’s what Ficly has so far:

  • The database that holds everything is backed up and uploaded to Amazon’s Simple Storage Service every 12 hours. Your avatars are also uploaded there.
  • Every feed has built-in pagination support (At least, I think they all do – I know the important ones do). If you want to back up their stories, all you need to do is grab and save your feed changing http://ficly.com/authors/kevin-lawver/stories.atom?page=1 to the next number until no more stories show up (change kevin-lawver to your URL name).
  • All the code behind Ficly is stored in version control using Beanstalk, down to the configuration files for all the major processes we run. Jason and I also have local copies, of course.

Here’s what we plan on doing as we find time:

  • Generate a nightly backup of all the published stories on the site and make it available to whoever wants it as an XML or JSON file. This will eventually include the data to accurately rebuild chains of sequels and prequels (something you can’t yet do with the story feeds).
  • Generate a nightly backup of every author’s stories so you could download them periodically for a personal backup.

I can’t think of anything else right now, but I don’t like repeating mistakes, and I want to make sure that Ficly’s data, better yet, your stories are safe, no matter what happens. Ficly’s young, and we’re running very lean (on a single server that hosts everything). If something happens to that server, I want to make sure that everything is recoverable so we can bring everything back up where it belongs.

What else can we do? I’m open to ideas. I can’t say we’ll implement all of them or when, but we’ll consider everything.

Comments

  • g²LaPianistaIrlandesa

    Thanks for keeping us in the loop, Don Jefe… even though I must say I understood absolutely none of that.

  • The Fantastic Mister Fish

    I think making a “Delete Ficlet” button would be nice. I heard that you guys had created it on the help site but I can’t seem to find out how to do it. For some reason, a few of my fic-things have been posted twice.
    Oh and I think you should officially announce what the “fic-things” are called.

  • Jason Garber

    @Mr. Fish Calling them “stories” seems to be working out alright.

  • The Last Squeek

    Yeah, stories is fine for right now, but it would be nice to have a name for them. It`s good to know that what happened to ficlets will not happen here, even though I hope Ficly`s will never be in that situation.

  • blusparrow (LoA)

    thanks Kevin! It’s always good to feel like things are safe =).

  • ernmander

    Thank Kevin. I love the fact that you do these updates to let us know what is going on. Wonder if Twitter could learn a thing or two :)

  • Construct of a Mind

    Speaking of Twitter, I was hoping we could get some auto-notifications in here. For instance, when someone comments on a story, or if someone writes a pre/sequel, I’d LOVE to get an email or a tweet or an sms or something. This site is my creative muse and has been since I found Ficlets last year. Having a constant “Streamly” would be great.

  • Kevin Lawver

    Blake, we’re thinking about the best way to do something with Twitter… we’re just not sure what it is yet – but those are good ideas.

  • Construct of a Mind

    Also, Mr. Lawver, I was hoping for some auto-tweeting from my account. Like, when I publish a new story, it generates a Tweet for me, alerting my followers. Like my blog does now.
    Example:
    “Hey, I just published a new Ficly! Goto http://tinyurl.com/someurlhere #ficly”
    Or whatever… :D

  • dkscully

    Hey @blake… why not add your ideas here:
    http://getsatisfaction.com/ficly/topics/twitter_integration23

  • Ana Cristina

    Thanks for posting this, Kevin. It’s nice to know our accounts (not to mention our ficlies!!) are safe. :)

  • Construct of a Mind

    @dkscully , I’ll do that, thanks! But what I was talking about in the first post wasn’t Twitter-specific. I meant any form of notification would be awesome.

  • StudMuffin (LoA)

    I think calling the stories “Ficlets” might be perfect. It’s an homage to the original site,
    and it would work for those who don’t know about it anyway.

  • Trann

    My right-brane thanks you so much for returning. I need this place some days… Random thoughts:
    - love the look; very clean and fast
    - calling them “stories” works but I don’t think I can help but call them ficlets :)
    - privacy option when publishing each ficlet? perhaps someone wants to publish anonymously; internally, you know who they are (for auditing, abuse issues) but the public sees Anon? This could be good for certain contests, too, as popularity isn’t an issue.

  • Kevin Lawver

    Ficly is all about persona, which lends itself to anonymity. We don’t make you give us your e-mail address or any really personally identifiable data other than an OpenID URL. You could create a separate “anonymous” account for posting to challenges if you wanted to, but I think a lot of people on Ficly (including our very own main blogger) use a persona on the site instead of their “real” identity. I’ve been planning a blog post on personas, but haven’t gotten around to it yet… maybe now’s the time.

  • Pyropunk 51 (PPP LoA)

    Blessed are the pessimists for they have backups. As a “server dies” backup and recovery solution may I suggest an external drive with a “ghost” of the database and config files. In that case you should be up in running in a matter of minutes should something happen and you need build a new system.

    I call them ficlyts (pronounced like ficlets).

  • Pyropunk 51 (PPP LoA)

    I would like to be able to use ficlets in the memorial as inspiration for ficlyts on ficly.
    In that way some of the stories can have continuity even though the domain changes/d.
    Do you think that’s possible, Kevin?

  • Trann

    Kevin: of course, you’re right. Case in point: me. 8)
    Keep up the good work, mate!

  • Kevin Lawver

    @pyropunk – It’s in the to-do list. Not sure when we’ll get to it, but we have a whole bunch of stuff planned for the re-launch of the Inspiration page. Stay tuned!

  • scratch'n'scrawl

    Kevin, I’ve not long known about this site, but it is a bit of a godsend to me. And I appreciate all the hard work you’ve put in.

    In relation to Blake’s post, which I know you answered re: Twitter, please make allowances for those of us that do not like or use Twitter. I know little about programming, but surely email alerts, should one choose to have them, wouldn’t be too hard to implement?

  • Kevin Lawver

    scratch’n’scrawl – it’s on the list. We’ll get to it, I promise.

  • THX 0477

    What a fun discussion thread! And most of it wasn’t all that geeky, as I actually understood it. As for persona’s, yes we all create and use them here, though I do see Trann’s point. Sometimes the persona takes on a life of its own.

  • Trann

    One quickie:

    When writing a pre-/sequel, could the tags be carried over into the text field? I’m betting that, most times, the tags remain the same or are modified slightly. Cut and paste doesn’t work as it introduces numbering: 1. 2. 3. 4. …