WTF Happened on February 3rd, 2021 (Forum Outage)

[extrememarine]

WTF Happened on February 3rd, 2021 (Forum Outage)



Great question – I’ll do my best to describe the events that occurred and caused the outage on February 3rd, 2021.

The site was hacked with malware on January 30th; that infection also impacted the forum software folder on Wednesday – February 3rd along with every other associated website on the account (FJREOM.com, the Shop e-commerce site, etc.) We decided the best course of action was to use the host’s recommended malware removal partner to have the malware removed and websites restored. Contrary to popular belief, I had not pulled the plug on the forum or put any individual member or members in time out; the ongoing discuss regarding the thread I closed was not connected to the outage in any way.

This event has been a learning experience regarding what services are provided by the host. We’ve made some adjustments to the host plan regarding storage and backup strategy that provides a path to a much quicker recovery should we experience a similar event in the future. The silver lining in this is 2 fold – first being that the SQL database is on a daily backup schedule – the database is the part of the forum that contains the posts – the content. This kept the lights flickering instead of being completely dark. The second part being that we have malware protection for the next year in place.

I can't express how much I appreciate Designflaw's continued assistance and patience with me and the site. He has been a tremendous help in managing the site and resources along with the customizations that are in place.

You may notice some features have not been restored fully yet (tapatalk access, post love extension, available smilies / emoji, etc.), but we are working on those detail items - I'll update status here as we work through them. One casulty of the reboot is the individual avatar imagines; everyone will need to upload new avatars - you can do that in the profile portion of the user control panel.

You are now free to move about the cabin – please enjoy the rest of your flight…

Wayne

[DesignFlaw06]

I'll add to Wayne's post a bit. Most of the downtime was due to issues with the host and their actions regarding the attack. We needed to wait for them to resolve their end. Wayne and I have now figured out what the best way to get support from the host for a quicker turn around. There was a bit of a perfect storm that occurred. A lot of "That shouldn't happen" from all sides. We've learned from it and in the grand scheme, I think the forum survived relatively unscathed. We're implementing some off-site backups for added redundancy and not be completely reliant on the host for those services.

Some known issues:

• Avatar Images - I was able to restore images from a 2018 backup. You'll have to re-upload if yours is missing.
  
• Post Love (Like Button) - Most of the custom code is back. There is an error when you click on the heart, but it still works. A refresh of the page will show you loved the post.
  
• Smilies - A few of the smilies are missing. Digging through old backups to try and find them

Wayne deserves the credit here. He has been talking with support, with me, dealing with the barrage of emails and texts, troubleshooting, etc. since the forums went down. He devoted more time and money than he should have to just so the lights could stay on. If there is any good to come from the last week, I hope that the users here have learned what a treasure this site has become and have a greater appreciation for Wayne's efforts keeping things going.

The faucet is on now, I just don't know where the leaks are. Thanks for your patience as we get back to 100%.

[Abercrombie FJR]

Well if there is any wonder if we appreciate Wayne's and your effort, I apologize. No amount of money or time should go unreimbursed and I will be first in line to throw money at the problem even if threadshitter Wheatonfjr or Cav47 broke the place.

Thanks for everything you guys did to bring this place back!

[wheatonFJR]

Thanks for the explanation guys...that helps to have an understanding moving forward. Im sure Wayne is glad not to have texts from me anymore on an hourly basis. Im with that idiot hose-dragger from North Dakota, let us know what can be done to avoid having ads for Michigan's finest beer being floated on this site.

Now I need to make up for the downtime...

[LKLD]

Thank you, for all the efforts put forth in restoring the Forum.

[Road Runner]

Thanks for all your efforts and fixing this for all of us. You guys are Awesome.
What really scares me is Mark saying he has to make up for lost time , so can you guys just block him for a while longer? For the greater good.

[FJRoss]

Great job and thank you!

[Festus]

Been there, done that. It's not fun. Hats off to you guys for having good backups and sticking with it.

[danh600]

Front page still says forums are closed. So many might not know they can get in depending...

Thanks guys for all the hard work.

This place being down reminds me I like it here more than any other social media place.

[extrememarine]

Dan - thanks, the front page has been updated. I need to find a new graphic to replace the flickering lights...

Front page still says forums are closed. So many might not know they can get in depending...

Thanks guys for all the hard work.

This place being down reminds me I like it here more than any other social media place.

[Intech]

Thanks for all the effort Wayne and Tim Dog!

[raYzerman]

Thanks for all the efforts guys! I had no doubts you'd get it done eventually, but I hope I get recognition for not sending one email or text during any of it. I'm with the Mayor from ND, and I'm right behind him in making a contribution to keep the lights from flickering. Don't be shy about telling us what expenses need to be covered, we can clean that up!!

[CollingsBob]

Thankyou...

[BkerChuck]

Awesome news and thanks to you guys for all of your hours of hard work.

Now who's going to explain to my boss the sudden burst of productivity that took place while this site was down.......

[Geezer]

I appreciate the amount of frustration and work involved to restore the forums and thank you very much.

It is totally incomprehensible to me that there are people out there who think it's fun to take down other people's web sites.

[extrememarine]

You were warned about this exact issues in one of my daily updates posted to the front page...

Now who's going to explain to my boss the sudden burst of productivity that took place while this site was down.......

[BikerGeek]

Thanks for your efforts, guys.

Outta curiosity, can ya override Wheatey's avatar and make it that flickering light thing? You know - dim bulb and all.

Asking for a friend.

[DesignFlaw06]

Thanks for your efforts, guys.

Outta curiosity, can ya override Wheatey's avatar and make it that flickering light thing? You know - dim bulb and all.

Asking for a friend.

Due to my epilepsy, I can't handle that many flickering lights on a page.

[Blister_1]

Wayne and Tim,

I'm with the others.... thank you for diligently fixing the issues. I came back from Mexico to the fiasco, and was WTF? I missed this place, and needed my fix.

[wheatonFJR]

Thanks for your efforts, guys.

Outta curiosity, can ya override Wheatey's avatar and make it that flickering light thing? You know - dim bulb and all.

Asking for a friend.

Why don't you take your giant corn cob and cram it up your








































































ass.