NVidia-smi shows API mismatch

Sometime when updated NVidia driver and CUDA on Rocky Linux systems, running nvidia-smi shows that kernel driver version mismatch. If you run

dmseg

It will show:

NVRM: API mismatch: the client has the version aaa.bbb, but
NVRM: this kernel module has the version ccc.ddd. Please
NVRM: make sure that this kernel module and all NVIDIA driver
NVRM: components have the same version.

And aaa.bbb is not the same with ccc.ddd.

This happens that the corresponding nvidia driver was not properly registered by dkms.

Other solution suggested to reboot the server, reinstall drivers, recreate initramfs, and rmmod of corresponding nvidia mods. These methods sometimes works. When they are all not working, you can try

dkms install -m nvidia -v 570.144

where replacing 570.144 to your most recent installed nvidia driver version. Then reboot the server. This should work.