Let him drag the halter a couple of days. Working in a small pen, periodically go in with him and pick op the lead, and then, very gradually, teach him to give to pressure - light tugs - from his left side. When I say give I mean at first when he even just shifts his weight toward pressure, then give him slack and reward him with a few back scratches. Be patient, calm, and speak to him in a soft voice. Gradually expect more "give" progressing from a lean, to a step, to several steps, etc. Always reward with slack and if he seems to like it scratching on his back. As he becomes more calm you can move toward scratching his neck. Go slow and have patience. If he begins to move on his own just "walk" him in whichever direction he chooses to go - do not fight against him. As you get him to take "steps", then begin giving the pressure from the normal lead position. Remember to reward him often, stay calm, and keep your movements smooth and steady. When he will walk a distance with pressure, then you may tie him as well.
Do this often and consistently and (given a normal temperament calf) within a few days you will have him broken. Not to say you won't have your moments, but the problem you described should be significantly reduced.