Close Rhythmbox and delete the Rhythmbox database in ~/.local/share/rhythmbox/rhythmdb.xml.
If you also want Rhythmbox to forget your library paths (the directories where it automatically looks for music), then you need to delete them from dconf:
gsettings set org.gnome.rhythmbox.rhythmdb locations "['file:///dev/null']"
You can then set new paths in the Rhythmbox GUI, under Edit -> Preferences -> Music.
This will fix issues where Rhythmbox knows about lots of different directories that may have duplicate files.