Since this applies only to UEFI systems, you want to check if you have a UEFI system. On any Linux live USB you are running to install, run this:
sudo dmesg | grep -i efivars
You should get an output something like this:
[ 0.301784] Registered efivars operations
If that's the case, then you can proceed. First we will remove Kubuntu from boot order. Run this to see UEFI boot order:
sudo efibootmgr
This is potentially the output you could get.
As you can see, there's Ubuntu in the boot order in my case. Find the bootorder number beside it (0006 in my case) and then run this command:
sudo efibootmgr -B -b {bootorder}
Replace the {bootorder} with your bootorder you found earlier. In my case, it would be sudo efibootmgr -B -b 0006
This should remove Ubuntu (which is what Kubuntu installs itself as) from the UEFI boot order.
Now, for removing its files from the ESP depends on your distro. For KDE based distros, open the KDE Partition assistant. Identify the ESP partition. If it already has a mount point, use that. In my case, the live USB didn't mount it.
In that case, identify the location of partition (mine is /dev/sda1), run these commands in the terminal/Konsole (and replace /dev/sda1 with whatever your partition path is):
mkdir ~/Desktop/efi
sudo mount /dev/sda1 ~/Desktop/efi
Now you can browse the ESP. Open the efi folder on your Desktop. Inside, you will find another folder called EFI. Open that, and you will find the Ubuntu folder. Delete that folder (and empty the trash). If it doesn't let you, right-click and click Open in Terminal or Open Terminal here and run this command:
sudo rm -r ./ubuntu
That's it!
Now, if you are using a GNOME based distro, this is slightly different. Open Disks and identify the ESP partition.
As you can see, it lists the partition as EFI system. You can click the triangle to mount it if it isn't mounted already and click the mount location. Find the EFI/ubuntu folder and delete it and if it doesn't work, run the command after opening terminal in that folder.