Q:

Kern.log and Syslog Growing Uncontrollably (with details)

Similar to other questions about uncontrollable bloat in kern.log and syslog, but in my case I can’t figure out what in the kernel is causing it because I don’t recognize the process and my google searches about it haven’t helped me much. 
Two separate raspberry pi 4’s running Ubuntu 20.04 for several weeks. All was well. Last night, I ran a standard apt upgrade and then my kern.log and syslog started growing uncontrollably at a rate of about 0.5-1 Gb per hour, and now I’m stuck truncating them over and over again while I try to figure out what this is that’s filling the kernel log. Here’s the output from tailing the kern.log file. It’s just this repeating over and over again, 100s of times per second.
May 23 07:43:03 raspi2 kernel: [12298.982475] WARNING: CPU: 0 PID: 4409 at drivers/mmc/host/sdhci.c:1101 sdhci_prepare_data+0x3dc/0x7b0
May 23 07:43:03 raspi2 kernel: [12298.982478] Modules linked in: binfmt_misc iptable_filter bpfilter dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua btsdio bluetooth ecdh_generic ecc brcmfmac bcm2835_v4l2(CE) brcmutil bcm2835_mmal_vchiq(CE) videobuf2_vmalloc videobuf2_memops snd_bcm2835(CE) videobuf2_v4l2 cfg80211 videobuf2_common snd_pcm videodev snd_timer mc snd raspberrypi_hwmon vc_sm_cma(CE) rpivid_mem uio_pdrv_genirq uio sch_fq_codel drm ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_ce spidev phy_generic ses enclosure scsi_transport_sas uas usb_storage aes_neon_bs aes_neon_blk crypto_simd cryptd
May 23 07:43:03 raspi2 kernel: [12298.982532] CPU: 0 PID: 4409 Comm: kworker/0:0H Tainted: G        WC  E     5.4.0-1011-raspi #11-Ubuntu
May 23 07:43:03 raspi2 kernel: [12298.982533] Hardware name: Raspberry Pi 4 Model B Rev 1.2 (DT)
May 23 07:43:03 raspi2 kernel: [12298.982545] Workqueue: kblockd blk_mq_run_work_fn
May 23 07:43:03 raspi2 kernel: [12298.982551] pstate: a0400085 (NzCv daIf +PAN -UAO)
May 23 07:43:03 raspi2 kernel: [12298.982553] pc : sdhci_prepare_data+0x3dc/0x7b0
May 23 07:43:03 raspi2 kernel: [12298.982556] lr : sdhci_prepare_data+0x2cc/0x7b0
May 23 07:43:03 raspi2 kernel: [12298.982557] sp : ffff80001156b9b0
May 23 07:43:03 raspi2 kernel: [12298.982558] x29: ffff80001156b9b0 x28: ffff0000eb62ec00 
May 23 07:43:03 raspi2 kernel: [12298.982560] x27: 0000000000000002 x26: 0000000000000000 
May 23 07:43:03 raspi2 kernel: [12298.982562] x25: ffff0000eb5e7000 x24: ffff0000eb5e7580 
May 23 07:43:03 raspi2 kernel: [12298.982564] x23: 0000000000418958 x22: ffff0000eb06a158 
May 23 07:43:03 raspi2 kernel: [12298.982566] x21: ffff0000277d0000 x20: ffff0000eb06a1d8 
May 23 07:43:03 raspi2 kernel: [12298.982568] x19: ffff0000eb5e7580 x18: 0000000000000000 
May 23 07:43:03 raspi2 kernel: [12298.982570] x17: 0000000000000000 x16: 0000000000000000 
May 23 07:43:03 raspi2 kernel: [12298.982572] x15: 0000000000000000 x14: 622061756c615f68 
May 23 07:43:03 raspi2 kernel: [12298.982574] x13: 645f697363732063 x12: 6d655f68645f6973 
May 23 07:43:03 raspi2 kernel: [12298.982576] x11: 637320636164725f x10: 68645f6973637320 
May 23 07:43:03 raspi2 kernel: [12298.982577] x9 : 6874617069746c75 x8 : 6d5f6d6420726574 
May 23 07:43:03 raspi2 kernel: [12298.982579] x7 : 6c69667062207265 x6 : ffffcb8fd5209a44 
May 23 07:43:03 raspi2 kernel: [12298.982581] x5 : ffffffffffffffff x4 : 0000000000000020 
May 23 07:43:03 raspi2 kernel: [12298.982583] x3 : 0000000000000001 x2 : fffffe0003581300 
May 23 07:43:03 raspi2 kernel: [12298.982585] x1 : fffffe0003581300 x0 : 00000000ffffffe4 
May 23 07:43:03 raspi2 kernel: [12298.982588] Call trace:
May 23 07:43:03 raspi2 kernel: [12298.982591]  sdhci_prepare_data+0x3dc/0x7b0
May 23 07:43:03 raspi2 kernel: [12298.982593]  sdhci_send_command+0xe0/0x5f0
May 23 07:43:03 raspi2 kernel: [12298.982595]  sdhci_request+0x110/0x150
May 23 07:43:03 raspi2 kernel: [12298.982599]  __mmc_start_request+0x88/0x1a8
May 23 07:43:03 raspi2 kernel: [12298.982601]  mmc_start_request+0x98/0xc0
May 23 07:43:03 raspi2 kernel: [12298.982603]  mmc_blk_mq_issue_rq+0x30c/0x778
May 23 07:43:03 raspi2 kernel: [12298.982605]  mmc_mq_queue_rq+0x14c/0x320
May 23 07:43:03 raspi2 kernel: [12298.982608]  blk_mq_dispatch_rq_list+0xa4/0x5f8
May 23 07:43:03 raspi2 kernel: [12298.982612]  blk_mq_do_dispatch_sched+0x68/0x108
May 23 07:43:03 raspi2 kernel: [12298.982614]  blk_mq_sched_dispatch_requests+0x164/0x1c0
May 23 07:43:03 raspi2 kernel: [12298.982617]  __blk_mq_run_hw_queue+0xfc/0x158
May 23 07:43:03 raspi2 kernel: [12298.982619]  blk_mq_run_work_fn+0x28/0x38
May 23 07:43:03 raspi2 kernel: [12298.982622]  process_one_work+0x1d0/0x430
May 23 07:43:03 raspi2 kernel: [12298.982624]  worker_thread+0x54/0x4a0
May 23 07:43:03 raspi2 kernel: [12298.982628]  kthread+0xfc/0x128
May 23 07:43:03 raspi2 kernel: [12298.982631]  ret_from_fork+0x10/0x1c
May 23 07:43:03 raspi2 kernel: [12298.982634] ---[ end trace a7bb7a8fcc54873a ]---

Over and over again until the logs are filling the disk. Happening exactly the same on both pi’s, only change I made to them was upgrading via apt upgrade, as described.
Looks like something with sdhci, but I can’t figure out: a) what part of upgrading caused this to start [why wasn’t this happening before now] and b) how to resolve the underlying issue. 
Since operations appear normal for my systems, do I just need to tell rsyslog to ignore messages from sdhci? I actually want to fix this, not just ignore if practical. 
Thanks!

A:

I have had the same problem today (kern.log grows fast; about 2GB) and downgraded the kernel to the previous version with the following commands:
Remove the 1001 version from the system:
apt remove linux-image-5.4.0-1011-raspi 

Reinstall the previous version of kernel, modules and headers:
apt-get install --reinstall linux-image-5.4.0-1008-raspi
apt-get install --reinstall linux-headers-5.4.0-1008-raspi
apt-get install --reinstall  linux-modules-5.4.0-1008-raspi

I rebooted successfully the PI4 with the previous kernel version. The next step is to exclude the kernel from updates until the problem is fixed:
apt-mark hold linux-image-5.4.0-1008-raspi
apt-mark hold linux-headers-5.4.0-1008-raspi
apt-mark hold linux-modules-5.4.0-1008-raspi

Afterwards you can remove the unused kernel version:
apt remove linux-raspi-headers-5.4.0-1011

# after a reboot:
pi04:~/.ssh# uname -a
Linux pi04 5.4.0-1008-raspi #8-Ubuntu SMP Wed Apr 8 11:13:06 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

# to clean up the system the 1011 headers and modules can be removed

apt remove linux-headers-5.4.0-1011-raspi
apt remove linux-modules-5.4.0-1011-raspi

from your system.
My installation history for the system: 
Filename: /var/log/apt/history.log

Start-Date: 2020-05-22  09:51:29
Commandline: apt upgrade
Install: linux-headers-5.4.0-1011-raspi:arm64 (5.4.0-1011.11, automatic), linux-modules-5.4.0-1011-raspi:arm64 (5.4.0-1011.11, automatic), linux-image-5.4.0-1011-raspi:arm64 (5.4.0-1011.11, automatic), linux-raspi-headers-5.4.0-1011:arm64 (5.4.0-1011.11, automatic)
Upgrade: linux-raspi:arm64 (5.4.0.1008.8, 5.4.0.1011.11), linux-image-raspi:arm64 (5.4.0.1008.8, 5.4.0.1011.11), linux-headers-raspi:arm64 (5.4.0.1008.8, 5.4.0.1011.11)
End-Date: 2020-05-22  09:53:15

Start-Date: 2020-05-23  22:14:40
Commandline: apt remove linux-image-5.4.0-1011-raspi
Remove: linux-raspi:arm64 (5.4.0.1011.11), linux-image-raspi:arm64 (5.4.0.1011.11), linux-image-5.4.0-1011-raspi:arm64 (5.4.0-1011.11)
End-Date: 2020-05-23  22:14:59

Start-Date: 2020-05-23  22:15:38
Commandline: apt autoremove
Remove: linux-headers-raspi:arm64 (5.4.0.1011.11)
End-Date: 2020-05-23  22:15:38

Start-Date: 2020-05-23  22:23:43
Commandline: apt-get install --reinstall linux-image-5.4.0-1008-raspi
Reinstall: linux-image-5.4.0-1008-raspi:arm64 (5.4.0-1008.8)
End-Date: 2020-05-23  22:25:02

Start-Date: 2020-05-23  22:27:46
Commandline: apt-get install --reinstall linux-headers-5.4.0-1008-raspi
Reinstall: linux-headers-5.4.0-1008-raspi:arm64 (5.4.0-1008.8)
End-Date: 2020-05-23  22:27:57

Start-Date: 2020-05-23  22:28:35
Commandline: apt-get install --reinstall linux-modules-5.4.0-1008-raspi
Reinstall: linux-modules-5.4.0-1008-raspi:arm64 (5.4.0-1008.8)
End-Date: 2020-05-23  22:30:03

Start-Date: 2020-05-23  22:59:16
Commandline: apt remove linux-raspi-headers-5.4.0-1011
Remove: linux-headers-5.4.0-1011-raspi:arm64 (5.4.0-1011.11), linux-raspi-headers-5.4.0-1011:arm64 (5.4.0-1011.11)
End-Date: 2020-05-23  22:59:25

