Fort at Number 4

In 1744, during King George's War, many of the area's outlying farms and buildings were burned by the French and their Native allies. Some settlers, along with some Native warriors, were killed in ambushes and small skirmishes. Other settlers were taken prisoner, to be ransomed back in Canada. The settler families would all but abandon the fort in the fall of 1746; a small contingent of men stayed on at the fort until February of 1747. The fortification was later reoccupied by Capt. Phineas Stevens and 30 militia men in late March of 1747. On April 7, 11 days after Capt. Stevens and his men arrived, the fort was besieged by a force combining French militia and Abenaki warriors under the command of Ensign Joseph Boucher de Niverville of the French Marines.

The siege lasted three days, until the French and Natives decided to head back to Canada rather than risk a direct attack on the fort, thus preventing further raids on settlements to the south and east. Reports of the incident claimed the sieging force was more than 500 strong, with numbers growing to over 700 as the story was repeated. French accounts of the siege put the number of Natives and accompanying French closer to 50 individuals. Commodore Charles Knowles, later 1st Baronet of the Royal Navy, whilst Governor of Louisburg visiting Boston, was so impressed that he presented Stevens with “as costly and elegant a sword as could be procured in Boston”. Afterwards, the township was named Charlestown in honour of Sir Charles Knowles,[4] who later became Rear-Admiral of Great Britain.

One Native raid made into the town in August 1754, immediately prior to the French and Indian War, led to the capture of Susanna Willard Johnson and her family, most of whom were eventually sold into slavery. Following Johnson's release several decades later, she wrote a popular captivity narrative of her ordeal.

General John Stark's statue at the Bennington Battle Monument

During the last of the four French and Indian Wars, many soldiers were stationed in the Fort at Number 4 to protect the frontier. They included Colonel Nathan Whiting's Regiment of Connecticut, and Colonel John Goffe's New Hampshire Provincial Regiment. Returning from a raid on St. Francis, Quebec, Robert Rogers in 1759 sought help here for his hungry Rangers at Fort Wentworth far up the Connecticut River. Also at that time, General Jeffery Amherst ordered a road to be built between the fort and another fort newly captured at Crown Point, located on the shores of Lake Champlain in New York. Consequently, Capt. John Stark and a company of Rangers, together with Col. Goffe's Regiment, built the Crown Point Military Road. It was 77.5 miles (124.7 km) long, with many blockhouses along its route to protect supplies and travelers through the wilderness that would later become Vermont. With the defeat of the French in 1761, and the Treaty of Paris in 1763, the need for the fort decreased.

While traveling to the Battle of Bennington in 1777, John Stark (then a brigadier general) gathered the New Hampshire Militia regiments, numbering about 1,500 militiamen, at the site.[5][6] The fort fell into disrepair after the American Revolutionary War.[5]