Slow login and su on Centos 7 (not an ssh or dns problem)

I had a particular Centos 8 box that had very slow logins but over ssh but also using su. The issue was not with dns and ssh. I finally found the problem – or at least the fix (related to systemd, sigh):

systemctl restart systemd-logind

The server was running inside of openvz so perhaps there were resource issues at some point. I did not have time to determine why it broke but the above did fix it.

Reset Centos 8 root password

reboot the server

at grub menu, press up/down to pause auto booting

select top menu item, press e (to edit)

in editing menu, go to end of line starting with “linux” and add “rd.break”

press control-x to boot using this edited grub command line

when the server starts you should not have to login

at the prompt, type :

mount -o remount,rw /sysroot/
chroot /sysroot
passwd root
shutdown -r now

Arch Linux virtualbox install on OSX

If your Virtualbox screen is too small, go to preferences and adjust screen scaling to 300% (or your choice). You’ll need to restart any running virtual machine to see the effect of this.

fdisk /dev/sda

# inside fdisk
n
p
enter
enter
w

# you should now out of fdisk and back in the shell
mkfs.ext4 /dev/sda1

wifi-menu
ping kernel.org

pacman -Syy

pacman -S reflector
reflector -c "US" -f 12 -l 10 -n 12 --save /etc/pacman.d/mirrorlist

mount /dev/sda1 /mnt
pacstrap /mnt base linux linux-firmware vim nano
genfstab -U /mnt >> /mnt/etc/fstab
arch-chroot /mnt
timedatectl list-timezones
timedatectl set-timezone <your-time-zone-here>


# i skipped this
locale-gen
echo LANG=en_GB.UTF-8 > /etc/locale.conf
export LANG=en_GB.UTF-8

echo "arch.metajack.org" > /etc/hostname

largely from : https://itsfoss.com/install-arch-linux/

When mysql won’t start due to innodb errors

Over the years innodb has caused me more trouble than anything else in the LAMP stack. MyISAM tables seem to work and repair without issue. Innodb seem to be endless trouble. Alas, clients need them so I have to make them work.

Today a mysql server crashed and wouldn’t restart. A few of the errors:

File operation call: 'read' returned OS error 71.
InnoDB: Operating system error number 2 in a file operation

among others. I won’t get into the causes – I’m sure 99% is due to poor shutdowns and other issues that are not really the fault of mysql (other than the lack of graceful failure whichh myisam seems to be so good at).

Here is a quick fix when under pressure – this won’t fix your problems if you have a single innodb table you need but if you have a mixed set of databases and tables this will get at least something working – which in my case was better than nothing.

Note that everything I’m saying to do is very bad form and practice. But if you have critical databases that need to be up it may help you out. As always, back everything up first just in case. You’ll have a back-up of a broken database but perhaps it will be a bit less broke than the version you have if the following goes wrong.

First find and move all databases with innodb tables to a temp directory :

mkdir /var/lib/tmp_disable
find /var/lib/mysql -name "*.ibd" -print
# you'll see various files inding in .ibd - look at the directory they are in (the database name) and move that dir out of the mysql data directory
mv <database-dir> /var/lib/tmp_disable

Finally move the global .ibd and logs to the tmp directory: 

mv /var/lib/mysql/ibdata1 /var/lib/mysql/ib_logfile* /var/lib/tmp_disable

mysql won’t like this and it’s not really a good idea. However your mysql server should at least start now. A new (empty) ibdata1 will be created. This will break a lot of things if your data is stored in this global datafile. However if you have each innodb table writing to its own ibd (by setting innodb_file_per_table in /etc/my.cnf) you might be ok.

Next, move each database back into /var/lib/mysql on-by-one, restarting mysql each time to see if it will still start. What to do with the database that is causing the problem? No easy answer to that unfortunately. I usually build a temporary mysql server in a VPS to work on it. At least your overall server will run while you are able to troubleshoot the problem database table(s).

Fixing promox lockups

My proxmox 6 test install locked up every day or two. Here is the fix. This in particular seems to apply to common hardware on Hetzner servers.

In /etc/default/grub, update this line :

GRUB_CMDLINE_LINUX_DEFAULT="consoleblank=0 intel_idle.max_cstate=1"

Reboot and you should find your server stable.

Read more on the proxmox forum at https://forum.proxmox.com/threads/random-crashes-reboots-mit-proxmox-ve-6-1-auf-ex62-nvme-hetzner.63597/page-2

Does the Yamaha DX100 send program change commands?

The manual for the DX100 says it sends program changes yet mine refused to do so. Here is how to make the DX100 send program changes.

To send program changes you must have “ch info” set to on (as the manual says). What the manual does not mention is that “sys info” must be set to off. Importantly, after you set “sys info” to off you must also first press the Internal Bank and select any program in that bank to start program change transmission. If you press other banks program changes are not sent until you first press bank 1 (the internal bank). After this program change is always transmitted until you turn off “ch info” or turn on “sys info”

Not sure if this is a bug or not but it will help to think of it as one – turning off “sys info” does not turn on program changing until you first select program 1 (internal bank, patch one).

By the way, when “sys info” is on, the DX100 transmits the sysex for the patch on each patch change.

Today’s lesson: Make use of package managers

I needed to distribute, backup and install some files to a large number of servers today. I had wrote most of a small script when I was reminded of a maxim that I often forget: Make use of package managers.

Everything my script did a package manager would do much better. The quickest way to create a simple package (an rpm in my case) is to simply tar up the files that need distributed (with all ownership and permissions set) then use a tool like tar2rpm.sh to create an rpm. You can use similar tools for the package manager of your choice.

Before I was reminded to use a package manager I was quite content writing my little script, refining it and admiring my handiwork. I reflected some on the writing of scripts – my thoughts :

Writing scripts – ideally elegant ones – is a good bit of art and creativity to inject into the technical world of system administration and development. It is nearly alway worth the time.

I was also reminded to stop judging other people’s scripts – I don’t really judge per se, but I do tend at times to slightly frown upon someone else’s method of writing something. The cause of course is the very thing that can make a script elegant – readability versus cleverness. Too much cleverness can be annoying. No cleverness – the same. I need to give other people’s cleverness a bit more time before marking it as unnecessarily obtuse. The same goes for scripts that are a bit too verbose (a more rare issue).