If you can follow, then of course you can do unfollow / disfollow. But I am not confirmed about -ve marking for it.
All members are correct for saying that YES you can do so. But, I think, there is no any -ve charges of points for doing so. Confirm it by giving it a try…. !