Of course judges are political actors, that's not a bad thing
Every time there's a controversial judicial decision from the Supreme Court, we get an outpouring of aggrieved commentators declaring that the decision was political and therefore illegitimate, and a corresponding set of responses that courts are not political actors but neutral interpreters of law, usually from people happy with the decision.
Both of these groups are wrong. Courts are political actors, and there's absolutely nothing wrong with that.
The idea that judges aren't political actors strikes me as plain bizarre. These are people empowered under law with the discretion to make decisions that have public policy outcomes.1 No different than the elected officials, the appointed officials, or the civil servants. If the legal discretion to determine the length of time a human has to remain in prison isn't a political decision, then you have a different definition of political than me.2 To say nothing of the ability to strike down a piece of statutory law as unconstitutional.
One primary moral value of a political judge is mercy, and its corresponding authority is flexibilty. The law cannot evaluate all circumstances, nor can all cases be covered by simple statutory guidelines. The very nature of judging is to consider the novelty of circumstance within the law. This is both inherently political and incredibly valuable.
What is true is that judges have different political constraints on them than other actors in the system. Most notably, of course, under the constitution federal judges served during good behavior, essentially lifetime appointments. That intends (and succeeds) to largely insulate them from public opinion, electoral considerations, partisan or interest-group pressure, or the political directives of the other branches. It also can sometimes put them badly out of step with contemporary political dynamics.
Supreme Court Justices are also constrained by their (relative) inability to enforce their own decisions. This leads to a system of self-restraint that more or less natural creates the emergence of adherence to precedent, rational explanation and justification of decisions, and a public presentation of self that emphasizes a non-partisan and apolitical approach to decision-making. Anything else would lead to a loss of prestige and/or trust from the public and other political actors, which in turn would make it more difficult for the Court to gain legitimacy (and thus enforcement) of its decisions.
Consequently, the Court itself is highly invested in not only a non-partisan public image, but also an apolitical one, whatever the underlying reality. Chief Justice Roberts routinely states the Court is not a political actor. He loves to say they are just "calling balls and strikes," conjuring up the image of a neutral strike zone with well-defined boundaries. And the Justices tend to distance themselves from both electoral and party politics. All of this is necessary to the maintenance of the political authority of the Court.
When Roberts says "we are not part of the political process," he is in one sense obviously correct. The Court does not get to write congressional statutes or make administrative decisions in the executive branch. Their role is (typically) not to be policy entrepreneurs, and they cannot act from scratch without a real case before them. But that's just a narrow slice of politics, and to say that the ability to decide the scope of congressional power under the Constitution in novel situations is outside the political process is plainly wrong; whether the Missouri Compromise line is a valid exercise of congressional authority to regulate the territories---as the Court took up as part of the Dred Scott case---simply cannot be understood as a technocratic question. It stands and falls on political judgements about the Constitution and the Act in question.
None of this is bad! Our separation of powers system benefits from the political actors having multiple overlapping sets of constraints on them. Part of the trick is aligning the constraints with the function. And the Constitution does it pretty well. To put constitutional interpretation in the hands of unelected judges with lifetime appointments sounds fishy when a populist rails at the Court, but it's actually a perfectly sensible arrangement if you think a constitutional court makes sense. Where else are we going to lodge constitutional review of congressional statutes?3
And the Court is responsive to public opinion, just not in the way the other political actors are. First, the Court is appointed and confirmed by actors in the elected branches. Second, the need to maintain its public legitimacy constrains the Court to not hew too far outside the boundaries of popular logic. To do otherwise would be to politicize the Court in a manner that sapped its own power. So while the Court can---and does---get out of step with the short-term dynamics of the political system because it is on a slow time-scale of change, it contains a self-corrective.4
In my view, the Court could benefit from being somewhat more overtly political, in the sense that I think more politicians should be appointed as Justices. Right now, the Court is largely people who came up through the prestigious law schools and spent time either on lower courts or in the Justice Department; the types of people a lot of Americans don't think of as political actors. That tends to create a Court that is overly-fixated on formal legalism. Justices with more political backgrounds (Hugo Black, Earl Warren, Sandra Day O'Connor, etc.), whatever other flaws they may have, tend to be more institutional realists about politics.
In this sense, they are formal political actors. Plenty of political actors are not empowered under law: interest groups, political parties, lobbyists, etc.↩
The most classic definition of politics is probably the simple who gets what in public decision-making.↩
This is one concern I have about term-limiting the Court. It gives Justices a time horizon beyond the Court, which could create unwanted political incentives in the decision-making and/or a loss of public prestige for the court because of the perception of such incentives. You can see this problem in some state courts that have renewable terms and/or elections.↩
And if the Court pushed things to far and got too out of line with the public and or Congress, it might suffer an existential threat from the legislature in the form of Court-packing, jurisdiction stripping, or impeachment.↩