The term "balance of terror" is one which was used during the Cold War, referring to nuclear weapon stockpiling by both sides. The knowledge that by using these terrible devices, one side may win the war, but both sides will effectively be destroyed, kept a tense sort of peace. This seems to be the state of affairs at the beginning of the episode: after the conflict of a century before, the Federation and the Romulans kept to their own sides of the Neutral Zone. The Romulans breach the peace in an effort to determine the strength of Star Fleet at this time. If proved to be relatively weak, the Romulans will feel empowered to attempt a conquest of Federation territory. This is why, as Spock points out, they can't afford to appear irresolute or unready. Star Fleet Command seems curiously slow to comprehend this, their desire to maintain the status quo so great that they are ready to write off the destroyed outposts and even the Enterprise rather than risk renewed conflict. This supposes, however, that the Romulan aggressors would be willing to back off if not challenged, a rather naive assumption at best, and a craven abandonment of those whom the Romulans would target at worst. This attitude is echoed in the episodes of T.N.G. and D.S.9 dealing with the demilitarized zone established with the Cardassians, and Star Fleet's willingness to overlook clear violations of it in order to keep the peace, regardless of the consequences to their citizens being victimized by them. On a purely practical note, such wavering generally only delays the inevitable, as Churchill famously pointed out: "You were given the choice between war and dishonour. You chose dishonour and you will have war." Fortunately, Kirk does not make the same decision, and Star Fleet Command eventually gets around to approving use of force if he deems it necessary, though too late to be of any practical use- typical bureaucracy.
We also see that both of them have moments of self doubt in which they question their decisions and their duties. Kirk confides to McCoy his desire to be somewhere- anywhere- else, and his fear of doing the wrong thing. The Romulan commander muses bitterly on the prospect of yet another war, questioning the necessity for it, telling his trusted centurion that he would almost prefer death. At the end of the day, though, whatever their personal misgivings, both are determined to do what is necessary to stop the other. And because of their equal skill, it could go either way. I think what decides the outcome in the end is the differences in the two crews. The crew of the Enterprise is completely loyal to Kirk. He knows that, even when he makes a controversial decision which may put him at odds with Star Fleet, that they will follow him and carry out his orders. The Romulan Commander has no such advantage. In a system where personal advancement in rank is as fiercely sought as military advancement, any sign of weakness will be pounced on and used by his underlings. This is why he orders the final attack on the Enterprise against his better judgement, which ends so disastrously.
* Mark Lenard, who plays the Romulan commander, really made the rounds as a Star Trek alien: he also plays Spock's Vulcan father in a later episode of T.O.S., and in the first Star Trek movie has a role as a Klingon.