Yeah, that sucked. This morning, I saw on twitter from a couple people that they couldn’t get to the site. We were packing up to drive home after visiting my parents (a ten hour drive down I-95). I spent a frantic 45 minutes before leaving trying to find a solution before we absolutely had to get on the road. Obviously, I didn’t figure it out.

The problem started because we somehow ran out of disk space last night, which caused the database to absolutely freak out. It refused to restart, even though I fixed some of the disk space problems.

We got home a little over an hour ago. As a last-ditch effort to fix the problem, I rebooted the server, and well, things work again. We have tons of disk space now and should be OK.

I think part of the problem is the new search stuff since the server was working just fine before we turned it on. You can expect search to go through some changes this week (I have a possible solution already). Update: That was quick. Apparently, the disk space issues caused the search stuff to go completely crazy, so I’ve removed it for now. I’ll get the new solution in as soon as I can this week, but it’s not going to happen tonight. I need some sleep!

Sorry for the downtime. I wish we had the resources to throw more hardware at the problem or to have someone dedicated to keeping us up and running – but we don’t. We’ll just have to get by.

Oh, and yes, the pledge drive is still going – we could use some more RAM and a bigger hard drive, so anything over the $800 will go towards making the server a little beefier.

Comments

• Mighty-Joe Young (A.K.A Strong Coffee)(LoA)

  you are so awesome kev

  • Posted over 10 years ago.

• Eloquent Mess {(LoA)}

  Glad to see Ficly back on its feet. I was starting to feel withdrawals…

  • Posted over 10 years ago.

• Alexa Reed (LoA)

  I was very worried, but I’m quite relieved now. Too bad it’s too late to start writing.

  • Posted over 10 years ago.

• g²LaPianistaIrlandesa

  I’m just glad it wasn’t like the weeklong blackout last year. Thanks for getting right on it, we all appreciate it!

  • Posted over 10 years ago.

• ALRO613 (LoA)

  Great stuff Kev…..

  • Posted over 10 years ago.

• ElshaHawk (LoA)

  Kev, you know you’re awesome! thank goodness for twitter and alro too!

  • Posted over 10 years ago.

• THX 0477

  Of course, that would happen at precisely that moment when it would be most inconvenient.

  • Posted over 10 years ago.

• TOAST

  TOAST’s One and Only Law of Server Maintenance: “S#!t happens and I’ll fix it when I damn well feel like it so shut up or I’ll make your computer sing Mariah Carey songs at you 24/7 you ungrateful sunnova hamster.”

  Thanks for all the work guys – we really do appreciate it.

  • Posted over 10 years ago.

• Kevin Lawver

  Thanks, everybody! And yes, our experience launching and maintaining Ficly has been a lot of $hit happens and coping – mostly around the search stuff. Now that I’ve shut down search, the server’s extremely happy using 1% of CPU, 7% of memory and about 8% of disk instead of 100% of all three.

  • Posted over 10 years ago.

• Mighty-Joe Young (A.K.A Strong Coffee)(LoA)

  cool the search wasnt crucial anyway

  • Posted over 10 years ago.