On the Prospect of Planting Arts and Learning in America (written in 1726), reported in Bartlett's Familiar Quotations, 10th ed. (1919). Compare: "Westward the star of empire takes its way", Epigraph to Bancroft's History of the United States; "What worlds in the yet unformed Occident / May come refin'd with th' accents that are ours?", Samuel Daniel, Musophilus (1599), Stanza 163.
According to W. Cleon Skousen, the first four empires are the Neo-Babylonian Empire, the Persian Empire, the Macedonian Empire, and the (Western, Eastern, and Holy) Roman Empire (Gospel Diamond Dust, Volume Two, Verity Publishing, 1998).