I think the main thing that bugged me with GoT was the glorification of graphic violence and the effect that it had on society in general (inuring many to that). The best example I've got is the Rape of Sansa: imagine if that had been S1E1, the public outcry you would have heard over the graphic, on-screen depiction of the rape of an underage girl. The show would have been yanked from the air. By the time the series got around to it, however, they had already pushed society's tolerance for that well past that point....and I find that depressing.
As for the books themselves, I found them to be too heavy (and depressing) -- it was a long read that I was glad to be finished with at the time.