When mysql won’t start due to innodb errors

Over the years innodb has caused me more trouble than anything else in the LAMP stack. MyISAM tables seem to work and repair without issue. Innodb seem to be endless trouble. Alas, clients need them so I have to make them work.

Today a mysql server crashed and wouldn’t restart. A few of the errors:

File operation call: 'read' returned OS error 71.
InnoDB: Operating system error number 2 in a file operation

among others. I won’t get into the causes – I’m sure 99% is due to poor shutdowns and other issues that are not really the fault of mysql (other than the lack of graceful failure whichh myisam seems to be so good at).

Here is a quick fix when under pressure – this won’t fix your problems if you have a single innodb table you need but if you have a mixed set of databases and tables this will get at least something working – which in my case was better than nothing.

Note that everything I’m saying to do is very bad form and practice. But if you have critical databases that need to be up it may help you out. As always, back everything up first just in case. You’ll have a back-up of a broken database but perhaps it will be a bit less broke than the version you have if the following goes wrong.

First find and move all databases with innodb tables to a temp directory :

mkdir /var/lib/tmp_disable
find /var/lib/mysql -name "*.ibd" -print
# you'll see various files inding in .ibd - look at the directory they are in (the database name) and move that dir out of the mysql data directory
mv <database-dir> /var/lib/tmp_disable

Finally move the global .ibd and logs to the tmp directory: 

mv /var/lib/mysql/ibdata1 /var/lib/mysql/ib_logfile* /var/lib/tmp_disable

mysql won’t like this and it’s not really a good idea. However your mysql server should at least start now. A new (empty) ibdata1 will be created. This will break a lot of things if your data is stored in this global datafile. However if you have each innodb table writing to its own ibd (by setting innodb_file_per_table in /etc/my.cnf) you might be ok.

Next, move each database back into /var/lib/mysql on-by-one, restarting mysql each time to see if it will still start. What to do with the database that is causing the problem? No easy answer to that unfortunately. I usually build a temporary mysql server in a VPS to work on it. At least your overall server will run while you are able to troubleshoot the problem database table(s).

Fixing promox lockups

My proxmox 6 test install locked up every day or two. Here is the fix. This in particular seems to apply to common hardware on Hetzner servers.

In /etc/default/grub, update this line :

GRUB_CMDLINE_LINUX_DEFAULT="consoleblank=0 intel_idle.max_cstate=1"

Reboot and you should find your server stable.

Read more on the proxmox forum at https://forum.proxmox.com/threads/random-crashes-reboots-mit-proxmox-ve-6-1-auf-ex62-nvme-hetzner.63597/page-2

Does the Yamaha DX100 send program change commands?

The manual for the DX100 says it sends program changes yet mine refused to do so. Here is how to make the DX100 send program changes.

To send program changes you must have “ch info” set to on (as the manual says). What the manual does not mention is that “sys info” must be set to off. Importantly, after you set “sys info” to off you must also first press the Internal Bank and select any program in that bank to start program change transmission. If you press other banks program changes are not sent until you first press bank 1 (the internal bank). After this program change is always transmitted until you turn off “ch info” or turn on “sys info”

Not sure if this is a bug or not but it will help to think of it as one – turning off “sys info” does not turn on program changing until you first select program 1 (internal bank, patch one).

By the way, when “sys info” is on, the DX100 transmits the sysex for the patch on each patch change.

Today’s lesson: Make use of package managers

I needed to distribute, backup and install some files to a large number of servers today. I had wrote most of a small script when I was reminded of a maxim that I often forget: Make use of package managers.

Everything my script did a package manager would do much better. The quickest way to create a simple package (an rpm in my case) is to simply tar up the files that need distributed (with all ownership and permissions set) then use a tool like tar2rpm.sh to create an rpm. You can use similar tools for the package manager of your choice.

Before I was reminded to use a package manager I was quite content writing my little script, refining it and admiring my handiwork. I reflected some on the writing of scripts – my thoughts :

Writing scripts – ideally elegant ones – is a good bit of art and creativity to inject into the technical world of system administration and development. It is nearly alway worth the time.

I was also reminded to stop judging other people’s scripts – I don’t really judge per se, but I do tend at times to slightly frown upon someone else’s method of writing something. The cause of course is the very thing that can make a script elegant – readability versus cleverness. Too much cleverness can be annoying. No cleverness – the same. I need to give other people’s cleverness a bit more time before marking it as unnecessarily obtuse. The same goes for scripts that are a bit too verbose (a more rare issue).

Fixing Apple Mail (mail.app / mac mail) “cannot verify username or password”

Every version of mac mail seems to have new ways to make the setup and configuration of pop3 mail not work. Today I ran into a new problem and solution.

This particular fix is for accounts that are set up and working but will not save changes made to the configuration after the initial setup. In my particular case I had set up a pop3/smtp combo without secure authentication (for a test) and now needed to go into the configuration and enable secure authentication. I checked the boxes for secure authentication but when saving I continually received the “cannot verify username and password” message.

I fixed this by going to the first tab and un-checking “Take this account online”. I was then able to save the setup. In the primary mail window I was able to take the account online and send and receive mail (in fact the account was already online despite having un-ticked “Take this account online).

Frustrating but at least it does work for those cases in which you need a POP3 mail account setup.

If this doesn’t work for for you, check these other more common issues:

  • Ensure all password fields are filled in. In many versions of mac mail, the passwords will be set blank after each attempt.
  • Likewise for the account username (which should normally be the full email address)
  • If all fields are set correctly, you many times have to repeatedly go through the process of “save/continue/etc”, verify, failure until finally it “magically” works (even though you changed nothing). I have cases where I’ve clicked 5 to 10 times to finally get it to take. I’ve not been able to find the cause (even though in most cases I controlled the server and could watch all server-side logs and connections).

Install Docker on a clean Centos 8 vps

Note: none of this actually works – just notes for now.

dnf install 'dnf-command(config-manager)'
dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
dnf install --nobest docker-ce


# ideally you would install the latest but you'll find missing dependancies
# find which docker versions are available
dnf list docker-ce --showduplicates | sort

# copy the name of the latest version
# will look like
# docker-ce.x86_64            3:19.03.8-3.el7                     docker-ce-stable
#
# removed the architecture portion (.x86_64) and add the version separated by a dash
dnf install docker-ce-3:19.03.8-3.el7

systemctl disable firewalld
systemctl enable --now docker

source: https://linuxconfig.org/how-to-install-docker-in-rhel-8

source : https://www.techrepublic.com/article/how-to-install-docker-ce-on-centos-8/

Enable on proxmox :

on host node

cat /sys/module/kvm_intel/parameters/nested   # should show N
echo "options kvm-intel nested=Y" > /etc/modprobe.d/kvm-intel.conf
modprobe -r kvm_intel
modprobe kvm_intel
cat /sys/module/kvm_intel/parameters/nested  # should show Y

# restart container
pct stop <container-id>
pct start <container-id>

# check inside container
egrep '(vmx|svm)' --color=always /proc/cpuinfo

Re-enable Firefox TLS 1.3 warnings

Firefox now refuses to load sites offering less than TLS 1.3 – at times you may need to allow this for a particular site to load/testing/etc.

However how do you re-enable for the warnings? Type :

about:config

into the address bar and then search for “TLS”

You’ll find this entry :

security.tls.version.enable-deprecated

Change it from true to false (in the version used at the time of this post you could click the icon on the right side of the screen to flip the value from true to false).

Localized Nihilism

Do your personal sufferings have meaning? I think yes – but that meaning is likely unknowable in the short term. Perhaps a type of micro-nihilism is appropriate. Give time and thought to to understanding the world at large – God, emotion, meaning. Your own problems? Mark them as meaningless.

Self pity (both deserved and contrived) will come but don’t dwell on it. Save nihilist tendencies for your own troubles – don’t let them invade the world at large. It is a mistake to allow you own suffering to lead you to believe all of life is meaningless. Too much self-reflection is rightly deemed prideful.

Everything is under God’s control. Don’t believe in God? Ok. The universe has order. Don’t fall too hard for the idea of chaos. Chaos Theory is valid but it breaks down when you try to define actual randomness. Proving the existence of randomness is hard. It’s not a popular belief but I think our universe is very likely deterministic – one big chain of events of which even the chaotic events can be explained if one could follow the chain at that quantum level. True randomness and chaos require extra-dimensional input. They lead us back to either the supernatural or some form of multi-verse commingling (do parallel universes bump up against each other and leak into each other?)

A deterministic universe doesn’t bestow meaning, however perhaps it does give inevitability that you should find a form of strange comfort in. To be clear, my own believe is in a God that is in control (even if you choose to see him as the simulation master in a holographic fake universe). God has a righteous reason for everything that happens.

 

Rsync with Hetzner storage boxes

In all examples, replace u2xxxxx-sub5 and u2xxxxx.your-storagebox.de with the names of your user and storage box.

First, make sure “external reachability” and “ssh” are enabled.

You can make the base directory easily by ftp’ing in first and creating the directory. Next, open the web interface and set the base directory for the sub-account to the one you just created.

To use rsync a directory to the storage box:

rsync -aze 'ssh -p 23' directory-name-here u2xxxxx-sub5@u2xxxxx.your-storagebox.de:

To use rsync a directory from the storage box into the current directory (signified by a single dot)

rsync -aze 'ssh -p 23' u2xxxxx-sub5@u2xxxxx.your-storagebox.de: .

You can also connect via ftp.

Problems:

Because rsync is not running as root on the remote server you cannot preserve all file ownership and symlinks (?), etc. Add the option :

--fake-super

to store user and permission values.

You can also use borg backup over rsync to solve this problem. I use it for some projects but it is not ideal for all uses.