Modifying and rebuilding an RPM from source

This is a topic covered over and over on the internet and yet I can never find the quick guide I need. Here for my notes are how to modify and rebuild an RPM from source.

First, download the SRPM and install it:

wget http://repo.joomhosting.eu/centos/7/SRPMS/httpd-2.4.41-1.el7.src.rpm
rpm -Uvh httpd-2.4.41-1.el7.src.rpm

You may see some errors about groups, you can ignore these.

You should now have a /root/rpmbuild/ dir – if not, find where the .spec file was put (you’ll find the other needed files near it):

find / -name "*.spec" -print

Before you change anything, try to just rebuild the rpm unmodified. You’ll likely find things you need that are missing. Fix all this first.

rpmbuild -ba SPECS/httpd.spec

When you have the default rebuild working, you can start changing things. If you only need to change a configure option you can edit the .spec file in /SPECS. If you need to patch code, you’ll need to (ideally and properly) use patch files added to the sources directory. If this is not possible, you will have to copy the .tgz for the source package, unarchive it, modify it, then tar it back up and put your modified copy in /SOURCES

Finding an expired certificate

I had a strange issue today regarding a particular expired certificate on a webserver. The problem was that the server in question was an nginx reverse proxy that proxied many domains and contained many hundreds of certificates. For various, um, “technical reasons”, I could not locate the exact cert that was bad. I only knew that one of them was. Normally the default .crt (which was causing the issue in this case) should be the first loaded nginx host. For one reason or another I still could not find it. I did know that date (it expired in September). Here is how I found it:

find / -type f -name "*.crt" -print -exec openssl x509 -enddate -in {} -noout \; | grep 'Sep'

I shouldn’t have had to search / – the entire server and all attached volumes – but in this case I was a bit desperate. Note that the command line above actually hides (due to the grep for the date) the filename in question. Once I knew I had found a match I removed the pipe into grep, viewed all the results with less and found the problem.

Related tip that may have brought you here: nagios will not check sni for ssl domains unless you add “–sni domain.com” to the check_http line

Creating a simple, quick Borg backup

Here is how to create a simple borg backup. Something more along the lines of a zip file than a careful backup system. I’m using this method for long term storage. The primary benefits are de-duplication and compression.

Again, to clarify – these are not backups meant to be updated each day/week/month. These are simply long term storage that I normally would have made by rsyncing or making a tgz.

To create, run the following (see below for encrypted backup option):

# create an empty archive called test1
borg init -e none test1 

# add files to the archive
borg create test1::first /path/to/files

# verify/check the archive
borg info test1

My test case gave 50% savings:

                       Original size      Compressed size    Deduplicated size
All archives:               50.42 GB             35.15 GB             25.58 GB

                       Unique chunks         Total chunks
Chunk index:                  215930               520240

Don’t forget that you can mount borg archives to browse files. 

borg mount test1::first /mnt/borg
borg unmount /mnt/borg

Test restore on another host:

# bundle up the directory
tar czpvf test1.borg.tgz test1

On another host:

# retrieve from original host
wget http://yourdomain.com/test1.borg.tgz

# unpack the archive
tar xzpvf test1.borg.tgz

# check/verify the archive
/usr/local/sbin/borg info test1
Warning: Attempting to access a previously unknown unencrypted repository!
Do you want to continue? [yN] y

# create a directory to restore to
mkdir restore

# mount the archive
borg mount test1 restore/

# work with the files in the archive
# ls restore

# umount the archive
borg umount restore

# extract all files from the archive
# first, list the snapshots
borg list test1

# extract a snapshot
cd restore/
borg extract ../test1::first

Creating an encrypted backup

Adding encryption is easy :

# create an empty, encrypted archive called test1
# enter the passphrase when prompted
borg init -encryption=repokey test1 

# export the key
borg key export test1 test1.key

# create the archive
# enter passphrase when prompted
borg create test1::first /path/to/files

History:

2020/11/18 at 3:56 pm

Using a private key file with sftp over lftp

I ran into a few issues trying to use a non-default private key file with lftp. Here is how I got it to work:

inside ~/.lftprc put 

set sftp:connect-program "ssh -p 23 -a -x -i /home/my_user/.ssh/key_name_here"

The important items are what is passed to the -i option – this should be the private key you want to use. In my example I’m also using a non-standard port (-p 23 for port 23)

To connect, run:

lftp -u your_user_name,blank sftp://your-server.org

For this to work you need to:

  • specify sftp:// as the server type
  • set the password to a junk value, I use “blank”

Slow login and su on Centos 7 (not an ssh or dns problem)

I had a particular Centos 8 box that had very slow logins but over ssh but also using su. The issue was not with dns and ssh. I finally found the problem – or at least the fix (related to systemd, sigh):

systemctl restart systemd-logind

The server was running inside of openvz so perhaps there were resource issues at some point. I did not have time to determine why it broke but the above did fix it.

Reset Centos 8 root password

reboot the server

at grub menu, press up/down to pause auto booting

select top menu item, press e (to edit)

in editing menu, go to end of line starting with “linux” and add “rd.break”

press control-x to boot using this edited grub command line

when the server starts you should not have to login

at the prompt, type :

mount -o remount,rw /sysroot/
chroot /sysroot
passwd root
shutdown -r now

Arch Linux virtualbox install on OSX

If your Virtualbox screen is too small, go to preferences and adjust screen scaling to 300% (or your choice). You’ll need to restart any running virtual machine to see the effect of this.

fdisk /dev/sda

# inside fdisk
n
p
enter
enter
w

# you should now out of fdisk and back in the shell
mkfs.ext4 /dev/sda1

wifi-menu
ping kernel.org

pacman -Syy

pacman -S reflector
reflector -c "US" -f 12 -l 10 -n 12 --save /etc/pacman.d/mirrorlist

mount /dev/sda1 /mnt
pacstrap /mnt base linux linux-firmware vim nano
genfstab -U /mnt >> /mnt/etc/fstab
arch-chroot /mnt
timedatectl list-timezones
timedatectl set-timezone <your-time-zone-here>


# i skipped this
locale-gen
echo LANG=en_GB.UTF-8 > /etc/locale.conf
export LANG=en_GB.UTF-8

echo "arch.metajack.org" > /etc/hostname

largely from : https://itsfoss.com/install-arch-linux/

When mysql won’t start due to innodb errors

Over the years innodb has caused me more trouble than anything else in the LAMP stack. MyISAM tables seem to work and repair without issue. Innodb seem to be endless trouble. Alas, clients need them so I have to make them work.

Today a mysql server crashed and wouldn’t restart. A few of the errors:

File operation call: 'read' returned OS error 71.
InnoDB: Operating system error number 2 in a file operation

among others. I won’t get into the causes – I’m sure 99% is due to poor shutdowns and other issues that are not really the fault of mysql (other than the lack of graceful failure whichh myisam seems to be so good at).

Here is a quick fix when under pressure – this won’t fix your problems if you have a single innodb table you need but if you have a mixed set of databases and tables this will get at least something working – which in my case was better than nothing.

Note that everything I’m saying to do is very bad form and practice. But if you have critical databases that need to be up it may help you out. As always, back everything up first just in case. You’ll have a back-up of a broken database but perhaps it will be a bit less broke than the version you have if the following goes wrong.

First find and move all databases with innodb tables to a temp directory :

mkdir /var/lib/tmp_disable
find /var/lib/mysql -name "*.ibd" -print
# you'll see various files inding in .ibd - look at the directory they are in (the database name) and move that dir out of the mysql data directory
mv <database-dir> /var/lib/tmp_disable

Finally move the global .ibd and logs to the tmp directory: 

mv /var/lib/mysql/ibdata1 /var/lib/mysql/ib_logfile* /var/lib/tmp_disable

mysql won’t like this and it’s not really a good idea. However your mysql server should at least start now. A new (empty) ibdata1 will be created. This will break a lot of things if your data is stored in this global datafile. However if you have each innodb table writing to its own ibd (by setting innodb_file_per_table in /etc/my.cnf) you might be ok.

Next, move each database back into /var/lib/mysql on-by-one, restarting mysql each time to see if it will still start. What to do with the database that is causing the problem? No easy answer to that unfortunately. I usually build a temporary mysql server in a VPS to work on it. At least your overall server will run while you are able to troubleshoot the problem database table(s).