Attention History Geeks!

Earlier today, I stated that President Lincoln was the only sitting U.S. President to come under enemy fire, as he was the subject of sharpshooter attacks at Ft. Stevens in 1864 during Jubal Early's raid on Washington. (In fact, a medical officer standing a few feet away from Lincoln was hit by a Confederate bullet.)

However, an alert reader left a comment to the effect that he though James Madison had been involved in combat when the British descended on Washington in 1814. This site states that Madison and a number of his political associates were present during the opening stages of the disastrous Battle of Bladensburg, MD, and that Madison actually appeared armed for the battle.

Does anybody know anything more about Madison's participation at Bladensburg? Would it be accurate to say that he "came under enemy fire"? Or did he retire from the field before the British guns could reach him?

UPDATE: From Madison's Memorandum on the Battle of Bladensburg -

After a short turn to the Marine barracks whither the Secretary of the Navy had gone, I mentioned to Mr. Rush who was with me my purpose of going to Bladensburg and my object in so doing. He readily accompanied me. On approaching the Town, we learned from William Simmons, that Winder was not there, and that the enemy were entering it. We rode up to him instantly. The Secretaries of State and War were with him. I asked the latter whether he had spoken with Genl. Winder on the subject of his arrangements and views. He said he had not. I remarked that tho' there was so little time for it, it was possible he might offer some advice or suggestion that might not be too late, to be turned to account; on which he rode up to the General as I did myself. The un-ruliness of my horse prevented me from joining in the short conversation that took place. When it was over, I asked Genl. Armstrong whether he had seen occasion to suggest any improvement in any part of the arrangements. He said that he had not; that from his view of them they appeared to be as good as circumstances admitted.

When the Battle had decidedly commenced, I observed to the Secretary of War and Secretary of State that it would be proper to withdraw to a position in the rear, where we could act according to circumstances; leaving military movements now to the military functionaries who were responsible for them. This we did, Mr. Rush soon joining us. When it became manifest that the battle was lost; Mr. Rush accompanying me, I fell down into the road leading to the city and returned to it.

Well, it's unclear, but it seems Madison cleared out as soon as the battle started.