Downtime

Briansol

Admins
Admin
VIP
Sorry about that, all. Hope it wasn't too long. Not sure when it first started. Got an email about it around noon for it and I've been MIA for a few days (maybe weeks?) now... been moving and not really online.

the db had some table errors... mismatched indexes and stuff. should be semi back to normal now and fully repaired in a few hours (the repair script takes a long time)
 
Last edited:

BrutalB83

Brutal Moderator
Moderator
VIP
LOL, it's all good. Tried to visit a few times over the past couple days and was like, "welp, I guess that's it...HondaSwap is dead."
 

Briansol

Admins
Admin
VIP
Ok everyone,

Finally think I figured out the root cause of all these problems.

Tldr: a few weeks back, a server software upgrade changed a backup retention setting.
Hard drive filled up.
Errors logged but couldn't write due to disk space. This sacked several db tables with mis matched indexes.
The nightly purge cron failed repeatedly and even after the backups were cleaned up (they are supposed to export to s3) the search db table was so big that it timed out. This also made the repair so timely as it was 32gb in size. For comparison.... the posts table is 400mb.

So... I've purged the old searches. And the db should be right sized again and the nightly clean up should run correctly once again.

I'm hoping tonight we have a successful backup and post off to s3. Right now we are about a month out of a good backup. (11/1) is last known good backup..


I'm still planning a board software upgrade and mysql to mariadb conversion later this week.

Searches may be slow as the indexes are being rebuilt as we speak.

Thanks for all the reports. I.hope we're stable going forward.
 

Briansol

Admins
Admin
VIP
Had some issues again today.... i think i caused them while tweaking stuff (was upgrading to php 7.3 and our software didn't like it). Back to 7.2 for now... i need to upgrade the board software asap as we are very out of date.
 
Top