The late philosopher Sidney Hook has some interesting things to say regarding the authority of the Supreme Court in his book Paradoxes of Freedom, published in 1987. Essentially, he claims that the authority exercised by the Supremes in determining adopted legislation to be unconstitutional is (1) not granted by the Constitution, and (2) fundamentally undemocratic (perhaps anti-democratic is more appropriate).
Hook seems very fond of Thomas Jefferson. I'm not. I think he was very much a hypocrite, in various ways, and particularly regarding political power. His actions when he had power were quite contrary to and went far beyond the limits he claimed so frequently should be placed on those who held power when he did not. There's no doubting his significance in American history, though, or his brilliance. In any case, his problems with the judiciary when he presided over the United States were many and serious, and I don't know that he was quite the objective analyst of judicial power Hook seems to think him to be.
Hook does have a point, though. The Constitution doesn't appear to say anything regarding the power to hold legislation unconstitutional which was assumed by the Court long ago and is and has been exercised so avidly by it and lesser courts ever since. And, there is something not very democratic about the Supremes, and the judiciary generally, having the last word regarding the validity and enforceability of legislation adopted by the duly elected representatives of the people. The Supremes, of course, are appointed for life, and there is no significant check on them once they ascend, as it were, to the bench of the nation's highest court. They may die or grow incapable, or may be subject to impeachment for high crimes and misdemeanors, but that is all.
Hook argues, rather persuasively I think, that the (legendary in some respects) Founders never intended this result.
I think there are some controls in place. I have rather more confidence in the force of precedent than Hook does, and think the judiciary is, generally in any case, naturally conservative in the sense that judges don't feel comfortable in actively tinkering with the law by the time they reach the higher courts. The Supremes seem to have had the ability to control themselves in most cases (I know there are some who would disagree with me on this point). But it's interesting to consider what would happen if a group of 5 or so determined "lawmakers" somehow found their way onto the Court, and began actively to strike down legislation in pursuit of a particular agenda which could not be justified in the manner legal decisions normally (but not in all cases) are supported at least on their face, i.e., by established law of some kind.
Just what could be done in that case? Such activity doesn't seem to be in the nature of a high crime or misdemeanor. I think all kinds of extra-legal pressures would be employed, assuming the agenda turns out to be unpopular. But it would seem that the only legally effective recourse would be a constitutional amendment expressly restricting judicial authority. There are of course other possibilities depending on how just extreme matters become, such as lower courts refusing to follow Supreme Court decisions, popular uprisings, dissolution of the Union, but I like to think that these are unlikely, even in our remarkable time.