I have a two node cluster I setup for Home Assistant, setup following this guide: Home Assistant High Availability - LINBIT
It’s been working just fine for a few months now. Once a month I’ll shell onto the standby node, use apt to pull and install all new updates, pull a new docker image for HomeAssistant, and then force a failover from the active node to the node I’ve just updated. I then update the node that is now the standby node, and leave it in standby until the next month.
A few days ago I ran the usual apt update and apt -y full-upgrade. Everything went well until it hit the new drbd package. This is the error I get now:
Setting up drbd-dkms (9.2.9~rc.1-1ppa1~jammy1) ...
Removing old drbd-9.2.9~rc.1-1ppa1~jammy1 DKMS files...
Deleting module drbd-9.2.9~rc.1-1ppa1~jammy1 completely from the DKMS tree.
Loading new drbd-9.2.9~rc.1-1ppa1~jammy1 DKMS files...
Building for 5.15.0-1053-raspi
Building initial module for 5.15.0-1053-raspi
ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/drbd-dkms.0.crash'
Error! Build of drbd.ko failed for: 5.15.0-1053-raspi (aarch64)
Make sure the name of the generated module is correct and at the root of the
build directory, or consult make.log in the build directory
/var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/ for more information.
dpkg: error processing package drbd-dkms (--configure):
installed drbd-dkms package post-installation script subprocess returned error exit status 7
Setting up ruby3.0 (3.0.2-7ubuntu2.5) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.7) ...
Errors were encountered while processing:
drbd-dkms
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)
Removing /var/crash/drbd-dkms.0.crash only has the effect of it no longer complaining about it existing and just creates a new one.
On the node that is still working, the currently installed drbd package is 9.2.8-1ppa1~jammy1. I tried apt-get install drbd-dkms=9.2.8-1ppa1~jammy1
and got
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package drbd-dkms is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
drbd-module-source
E: Version '9.2.8-1ppa1~jammy1' for 'drbd-dkms' was not found
I’m not entirely sure what my options are now, and was hoping someone could point me at a way to roll back to 9.2.8-1ppa1~jammy1?
Any advice would be greatly appreciated.
Contents of /var/crash/drbd-dkms.0.crash:
ProblemType: Package
DKMSBuildLog:
DKMS make.log for drbd-9.2.9~rc.1-1ppa1~jammy1 for kernel 5.15.0-1053-raspi (aarch64)
Tue Apr 30 11:57:57 EDT 2024
make: Entering directory '/var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd'
Calling toplevel makefile of kernel source tree, which I believe is in
KDIR=/lib/modules/5.15.0-1053-raspi/build
make -C /lib/modules/5.15.0-1053-raspi/build "PRE_CFLAGS=" M=/var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd obj-m=dummy-for-compat.o dummy-for-compat-h.o
COMPAT have_sane_test_environment
COMPAT have_sane_test_environment_always_fail
LN build-current -> build-5.15.0-1053-raspi/
UPD /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/.drbd_kernelrelease
LN compat.h -> build-5.15.0-1053-raspi/compat.h
LN .compat_test -> build-5.15.0-1053-raspi/.compat_test
LN compat.5.15.0-1053-raspi.h -> build-5.15.0-1053-raspi/compat.h
LN .compat_test.5.15.0-1053-raspi -> build-5.15.0-1053-raspi/.compat_test
COMPAT __vmalloc_has_2_params
COMPAT add_disk_returns_int
COMPAT before_4_13_kernel_read
COMPAT bio_alloc_has_4_params
COMPAT blkdev_get_by_path_has_holder_ops
COMPAT blkdev_issue_discard_takes_flags
COMPAT blkdev_issue_zeroout_discard
COMPAT blkdev_put_has_holder
COMPAT block_device_operations_open_takes_gendisk
COMPAT block_device_operations_release_takes_single_argument
COMPAT can_include_vermagic_h
COMPAT dax_direct_access_takes_mode
COMPAT fs_dax_get_by_bdev_takes_start_off
COMPAT fs_dax_get_by_bdev_takes_start_off_and_holder
COMPAT genl_policy_in_ops
COMPAT have_BIO_MAX_VECS
COMPAT have_CRYPTO_TFM_NEED_KEY
COMPAT have_GENHD_FL_NO_PART
COMPAT have_SHASH_DESC_ON_STACK
COMPAT have_WB_congested_enum
COMPAT have___bio_add_page
COMPAT have_allow_kernel_signal
COMPAT have_bdev_discard_granularity
COMPAT have_bdev_max_discard_sectors
COMPAT have_bdev_nr_sectors
COMPAT have_bdev_open_by_path
COMPAT have_bdevname
COMPAT have_bdgrab
COMPAT have_bdi_congested
COMPAT have_bdi_congested_fn
COMPAT have_bio_advance_iter_single
COMPAT have_bio_alloc_clone
COMPAT have_bio_bi_bdev
COMPAT have_bio_bi_error
COMPAT have_bio_bi_opf
COMPAT have_bio_bi_status
COMPAT have_bio_clone_fast
COMPAT have_bio_op_shift
COMPAT have_bio_set_dev
COMPAT have_bio_set_op_attrs
COMPAT have_bio_split_to_limits
COMPAT have_bio_start_io_acct
COMPAT have_bioset_init
COMPAT have_bioset_need_bvecs
COMPAT have_blk_alloc_disk
COMPAT have_blk_alloc_queue_rh
COMPAT have_blk_check_plugged
COMPAT have_blk_cleanup_disk
COMPAT have_blk_mode_t
COMPAT have_blk_opf_t
COMPAT have_blk_qc_t_make_request
COMPAT have_blk_qc_t_submit_bio
COMPAT have_blk_queue_flag_set
COMPAT have_blk_queue_make_request
COMPAT have_blk_queue_max_write_same_sectors
COMPAT have_blk_queue_merge_bvec
COMPAT have_blk_queue_split_bio
COMPAT have_blk_queue_split_q_bio
COMPAT have_blk_queue_split_q_bio_bioset
COMPAT have_blk_queue_update_readahead
COMPAT have_blk_queue_write_cache
COMPAT have_bvec_kmap_local
COMPAT have_d_inode
COMPAT have_disk_update_readahead
COMPAT have_enum_req_op
COMPAT have_fallthrough
COMPAT have_fs_dax_get_by_bdev
COMPAT have_generic_start_io_acct_q_rw_sect_part
COMPAT have_generic_start_io_acct_rw_sect_part
COMPAT have_genl_info_userhdr
COMPAT have_get_random_u32
COMPAT have_get_random_u32_below
COMPAT have_hd_struct
COMPAT have_ib_cq_init_attr
COMPAT have_ib_get_dma_mr
COMPAT have_idr_is_empty
COMPAT have_inode_lock
COMPAT have_kmap_local_page
COMPAT have_ktime_to_timespec64
COMPAT have_kvfree
COMPAT have_kvfree_rcu
COMPAT have_kvfree_rcu_mightsleep
COMPAT have_list_is_first
COMPAT have_list_next_entry
COMPAT have_lookup_user_key
COMPAT have_max_send_recv_sge
COMPAT have_nla_nest_start_noflag
COMPAT have_nla_parse_deprecated
COMPAT have_nla_put_64bit
COMPAT have_nla_strscpy
COMPAT have_part_stat_h
COMPAT have_part_stat_read_accum
COMPAT have_pointer_backing_dev_info
COMPAT have_proc_create_single
COMPAT have_queue_flag_discard
COMPAT have_queue_flag_stable_writes
COMPAT have_rb_declare_callbacks_max
COMPAT have_refcount_inc
COMPAT have_req_hardbarrier
COMPAT have_req_noidle
COMPAT have_req_nounmap
COMPAT have_req_op_write
COMPAT have_req_op_write_zeroes
COMPAT have_req_write
COMPAT have_revalidate_disk_size
COMPAT have_sched_set_fifo
COMPAT have_sched_signal_h
COMPAT have_security_netlink_recv
COMPAT have_sendpage
COMPAT have_sendpage_ok
COMPAT have_set_capacity_and_notify
COMPAT have_shash_desc_zero
COMPAT have_simple_positive
COMPAT have_sk_use_task_frag
COMPAT have_sock_set_keepalive
COMPAT have_strscpy
COMPAT have_struct_bvec_iter
COMPAT have_struct_size
COMPAT have_submit_bio_noacct
COMPAT have_tasklet_setup
COMPAT have_tcp_sock_set_cork
COMPAT have_tcp_sock_set_keepcnt
COMPAT have_tcp_sock_set_keepidle
COMPAT have_tcp_sock_set_nodelay
COMPAT have_tcp_sock_set_quickack
COMPAT have_time64_to_tm
COMPAT have_timer_setup
COMPAT have_timer_shutdown
COMPAT have_tls_get_record_type
COMPAT have_tls_tx_rx
COMPAT have_void_make_request
COMPAT have_void_submit_bio
COMPAT ib_alloc_pd_has_2_params
COMPAT ib_device_has_ops
COMPAT ib_post_send_const_params
COMPAT ib_query_device_has_3_params
COMPAT need_drbd_wrappers
COMPAT need_make_request_recursion
COMPAT need_skb_abort_seq_read
COMPAT part_stat_read_takes_block_device
COMPAT queue_limits_has_discard_zeroes_data
COMPAT rdma_create_id_has_net_ns
COMPAT rdma_reject_has_reason_arg
COMPAT sk_data_ready_has_1_param
COMPAT sock_create_kern_has_netns_parameter
COMPAT sock_ops_returns_addr_len
COMPAT struct_gendisk_has_backing_dev_info
UPD /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/compat.h
make -C /lib/modules/5.15.0-1053-raspi/build "PRE_CFLAGS=" M=/var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd obj-m=dummy-for-prep.o dummy-for-patch.o
GEN /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_buildtag.c
GEN /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/.kernel.config.gz
COPY /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/Kbuild
make -C /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd -f Makefile.spatch /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/compat.patch
COPY /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-kernel-compat/handshake/Kbuild
CHK build-5.15.0-1053-raspi/compat.patch
LN /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/compat.patch -> ../drbd-kernel-compat/cocci_cache/e5b4538b952d02aa28acb205221181bc/compat.patch
set -- /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/compat.patch /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/compat.h; \
test $1 -nt $2 || touch -r $1 $2
SPLIT /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/compat.patch
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-headers/compat.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-headers/drbd_meta_data.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-headers/drbd_protocol.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-headers/drbd_strings.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-headers/drbd_strings.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-headers/drbd_transport.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-headers/linux/drbd.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-headers/linux/drbd_genl.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-headers/linux/drbd_genl_api.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-headers/linux/drbd_limits.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-headers/linux/genl_magic_func-genl_register_family_with_ops_groups.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-headers/linux/genl_magic_func-genl_register_mc_group.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-headers/linux/genl_magic_func.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-headers/linux/genl_magic_struct.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-headers/windrbd/windrbd_ioctl.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-kernel-compat/cocci_macros.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-kernel-compat/drbd_wrappers.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-kernel-compat/drbd_wrappers.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-kernel-compat/gen_patch_names.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-kernel-compat/handshake/genl.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-kernel-compat/handshake/genl.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-kernel-compat/handshake/handshake.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-kernel-compat/handshake/netlink.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-kernel-compat/handshake/request.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-kernel-compat/handshake/tlshd.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-kernel-compat/linux/libnvdimm.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-kernel-compat/linux/lru_cache.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-kernel-compat/linux/overflow.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-kernel-compat/lru_cache.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-kernel-compat/net/handshake.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-kernel-compat/nsecs_to_jiffies.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-kernel-compat/uapi/linux/handshake.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd-kernel-compat/uapi/linux/sched/types.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_actlog.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_bitmap.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_dax_pmem.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_dax_pmem.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_debugfs.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_debugfs.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_int.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_interval.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_interval.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_kref_debug.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_kref_debug.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_main.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_nl.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_nla.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_nla.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_polymorph_printk.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_proc.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_receiver.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_req.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_req.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_sender.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_state.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_state.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_state_change.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_transport_lb-tcp.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_transport.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_transport_rdma.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_transport_tcp.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_transport_template.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/drbd_vli.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/kref_debug.c
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/kref_debug.h
PATCH /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/linux/drbd_config.h
test -e build-current && \
make -C /lib/modules/5.15.0-1053-raspi/build "PRE_CFLAGS=" M=/var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current modules
+ grep -qe ^.define COMPAT_HAVE_TLS_TX_RX$ /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/compat.h
+ echo m
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_buildtag.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_bitmap.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_proc.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd-kernel-compat/handshake/genl.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd-kernel-compat/handshake/netlink.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd-kernel-compat/handshake/request.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_sender.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_receiver.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd-kernel-compat/handshake/tlshd.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_req.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_actlog.o
LD [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd-kernel-compat/handshake/handshake.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_main.o
COPY /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_strings.c
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_nl.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_interval.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_state.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_nla.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_transport.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd-kernel-compat/drbd_wrappers.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_debugfs.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_transport_tcp.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_transport_lb-tcp.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_transport_rdma.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_strings.o
LD [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd.o
+ grep -qe ^.define COMPAT_HAVE_TLS_TX_RX$ /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/compat.h
+ echo m
MODPOST /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/Module.symvers
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd-kernel-compat/handshake/handshake.mod.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd.mod.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_transport_lb-tcp.mod.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_transport_rdma.mod.o
CC [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_transport_tcp.mod.o
LD [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd-kernel-compat/handshake/handshake.ko
LD [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd.ko
LD [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_transport_lb-tcp.ko
LD [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_transport_rdma.ko
LD [M] /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-current/drbd_transport_tcp.ko
.======================================================================.
# KDIR=/lib/modules/5.15.0-1053-raspi/build #
# KERNELVERSION=5.15.148 #
# KERNELRELEASE=5.15.0-1053-raspi #
# #
# /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd/build-5.15.0-1053-raspi/ #
# drbd-kernel-compat/handshake/handshake.ko #
# drbd.ko #
# drbd_transport_lb-tcp.ko #
# drbd_transport_rdma.ko #
# drbd_transport_tcp.ko #
'======================================================================'
make: Leaving directory '/var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~jammy1/build/src/drbd'
DKMSKernelVersion: 5.15.0-1053-raspi
Date: Tue Apr 30 12:00:11 2024
Package: drbd-dkms 9.2.9~rc.1-1ppa1~jammy1
PackageVersion: 9.2.9~rc.1-1ppa1~jammy1
SourcePackage: drbd
Title: drbd-dkms 9.2.9~rc.1-1ppa1~jammy1: drbd kernel module failed to build
Ryan
May 1, 2024, 12:29am
3
The latest release candidate (RC) version of DRBD ( 9.2.9~rc.1-1) is currently broken within the PPA and there should be a fix soon. The LINBIT PPA gets RC versions pushed to it but unfortunately, only the latest package versions are available, even if it is an RC.
Until there’s an updated package for 9.2.9, I’d suggest:
If you haven’t already done so, I’d put the upgraded node into standby (Pacemaker) by running crm node standby.
Downgrade the upgraded node’s kernel to match the still working node. Run uname -a after rebooting to verify the kernel versions.
Temporarily uninstall drbd-dkms from the upgraded node.
Running modinfo drbd from the still working node will tell you where the compiled DRBD kernel modules are located. You should see something similar to:
root@linstor-sat-1:/home/vagrant# modinfo drbd
filename: /lib/modules/6.1.0-20-amd64/updates/dkms/drbd.ko
softdep: post: handshake
alias: block-major-147-*
license: GPL
version: 9.2.8
[...]
Taking a look inside the /lib/modules/6.1.0-20-amd64/updates/dkms/ directory:
root@linstor-sat-1:~# ls -lah /lib/modules/6.1.0-20-amd64/updates/dkms/
total 1.8M
drwxr-xr-x 2 root root 4.0K Apr 25 09:57 .
drwxr-xr-x 3 root root 4.0K Apr 25 09:57 ..
-rw-r--r-- 1 root root 1.5M Apr 25 09:57 drbd.ko
-rw-r--r-- 1 root root 70K Apr 25 09:57 drbd_transport_lb-tcp.ko
-rw-r--r-- 1 root root 108K Apr 25 09:57 drbd_transport_rdma.ko
-rw-r--r-- 1 root root 58K Apr 25 09:57 drbd_transport_tcp.ko
-rw-r--r-- 1 root root 44K Apr 25 09:57 handshake.ko
Copy the kernel modules similar to the above to the same location on the upgraded node via scp, rsync, etc.
At this point the DRBD kernel module should be able to load successfully on the upgraded node by running drbdadm up <resource_name> or drbdadm up all works as well.
Verify DRBD is synchronizing with drbadm status.
Take the upgraded node out of standby (Pacemaker) with crm node online.
Alternatively, you could try building and installing DRBD from source on the upgraded node instead of the steps above.
In the future just be a little cautious with kernel upgrades and DRBD kernel module upgrades. For example, avoiding RC releases can be done by running a quick apt search drbd-dkms to see which DRBD version is currently available. Just remember to also skip kernel upgrades when you’re skipping a DRBD release candidate upgrade.
2 Likes
You might also want to consider using “apt-mark hold” command to prevent specific packages from auto updating and having a lab environment where you test packages before deploying them in production.
1 Like
Thanks, Ryan! That was useful guidance, and I now understand a lot more about how the boot config on a RPI works /boot get ignored, and it seems everything is driven from /boot/firmware. There is a config.txt file in there that points to the vmlinuz and initrd.img to be used: it was simple enough to copy over the older kernel’s vmlinuz and initrd.img from /boot to /boot/firmware and change the directives in config.txt to point to them, and reboot.
I initially forgot to copy over the matching System.map to /boot/firmware as well, which, meant the kernel modules I’d copied over couldn’t be found. Once I’d figured it out, everything worked just fine.
I even went and build DRBD from source, just for fun
Cluster is back up again: thank you.
Hey!
Ryan
May 2, 2024, 5:52pm
7
Anytime! A recovery/workaround is easy enough when you have at least one working system to copy from. Glad we could help.
