1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Downtime

Discussion in 'Site News, Updates, etc' started by Briansol, Nov 4, 2019.

  1. Briansol

    Briansol Admins Admin VIP

    Messages:
    22,069
    Likes Received:
    2,510
    Joined:
    Nov 18, 2007
    Location:
    CT
    Sorry about that, all. Hope it wasn't too long. Not sure when it first started. Got an email about it around noon for it and I've been MIA for a few days (maybe weeks?) now... been moving and not really online.

    the db had some table errors... mismatched indexes and stuff. should be semi back to normal now and fully repaired in a few hours (the repair script takes a long time)
     
    Last edited: Nov 5, 2019
    TurboMirage likes this.
  2. phunky.buddha

    phunky.buddha Admin with a big stick Admin VIP

    Messages:
    28,632
    Likes Received:
    299
    Joined:
    Sep 30, 2002
    Location:
    Dallas / Fort Worth, TX
  3. BrutalB83

    BrutalB83 Brutal Moderator Moderator VIP

    Messages:
    11,928
    Likes Received:
    462
    Joined:
    Jun 24, 2006
    Location:
    Rochester Hills, MI
    LOL, it's all good. Tried to visit a few times over the past couple days and was like, "welp, I guess that's it...HondaSwap is dead."
     
  4. Briansol

    Briansol Admins Admin VIP

    Messages:
    22,069
    Likes Received:
    2,510
    Joined:
    Nov 18, 2007
    Location:
    CT
    No one called or Facebooked me... :(
     
  5. Matts96HB

    Matts96HB . Moderator VIP

    Messages:
    6,854
    Likes Received:
    716
    Joined:
    Apr 16, 2006
    I was about to go crazy ex girlfriend and write you on LinkedIn
     
  6. Briansol

    Briansol Admins Admin VIP

    Messages:
    22,069
    Likes Received:
    2,510
    Joined:
    Nov 18, 2007
    Location:
    CT
    Ok everyone,

    Finally think I figured out the root cause of all these problems.

    Tldr: a few weeks back, a server software upgrade changed a backup retention setting.
    Hard drive filled up.
    Errors logged but couldn't write due to disk space. This sacked several db tables with mis matched indexes.
    The nightly purge cron failed repeatedly and even after the backups were cleaned up (they are supposed to export to s3) the search db table was so big that it timed out. This also made the repair so timely as it was 32gb in size. For comparison.... the posts table is 400mb.

    So... I've purged the old searches. And the db should be right sized again and the nightly clean up should run correctly once again.

    I'm hoping tonight we have a successful backup and post off to s3. Right now we are about a month out of a good backup. (11/1) is last known good backup..


    I'm still planning a board software upgrade and mysql to mariadb conversion later this week.

    Searches may be slow as the indexes are being rebuilt as we speak.

    Thanks for all the reports. I.hope we're stable going forward.
     
    |Chaz| and TurboMirage like this.
  7. awptickes

    awptickes unimpressed by you VIP

    Messages:
    9,595
    Likes Received:
    788
    Joined:
    Aug 29, 2007
    Location:
    MD
    I did, lol, just FB not linked in.

    I would have texted him, but I figured it could wait. lol.
     
  8. Briansol

    Briansol Admins Admin VIP

    Messages:
    22,069
    Likes Received:
    2,510
    Joined:
    Nov 18, 2007
    Location:
    CT
    Had some issues again today.... i think i caused them while tweaking stuff (was upgrading to php 7.3 and our software didn't like it). Back to 7.2 for now... i need to upgrade the board software asap as we are very out of date.
     
    TurboMirage likes this.

Share This Page