Fish was always a bit of a luxury for us, so we only ever ate it on occasion. Except for "Fish and Chips" which is a bit of a NZ staple, where the fish is deep fried in batter. I did really like it though. But had no problems giving it up, considering we didn't really eat too much of it.
But the fish stocks are getting so low, from both recreational and commercial overfishing these days, that lots of whales etc are starving to death in the oceans and beaching themselves because of it.
Like others have said its just not needed in ones diet, and us humans have been taking far too much for too long. "Why add to the problem?" is pretty much my position.