You live and learn . . .
Some time ago we had a little mishap we want you to tell about.
As time goes by, you are bound to break your own rules in your unbridled enthusiasm to make things better, or try stuff out. At some point I found it nice to use a hostname instead of the IP address of the server. Tested this out on the test server, and it looked like a simple basic thing to do. However, when performing the same thing on the live server, after a successful test on the test server, the result was that the server would not start completely anymore, and was stuck in maintenance mode . . .
We tried several things without any success. We never found what was wrong, but it was probably a combination of earlier thingies we did on the servers os. To get a good clean start, we decided to do a complete clean install. Our biggest fear here was that we had both a logical volume with all our media data and a soft raid with more important stuff. Loosing all our media would be bad, but not a real problem, and the data on the soft raid is regularly copied off-site. Also we use persistent volumes which are located on a separate drive from the os.
So after taking a few deep breaths, we started the clean installation, which was simple enough. Then we copied back the needed lines from the stab file (we saved before hand), and could see that Both the logical volume and the soft raid were available again. Big relief. And as said, because we had the persistent volumes + the docker compose files in place, we had the server up and running within an hour.
What is the lesson learned?
First of all make sure you use persistent volumes on another drive than your os, and make sure you have backups of at least the most important ones. Keep a backup of your docker-compose files, the latest greatest of course . . .
And do not play around on the os too much. You probably eventually will mess up things, but if you have your bases covered you can at least heb up and running in a very short time.