Doom is like the Borg in Star Trek. When it suits the story, one Borg can nearly take out the entire Federation. But other times, one Federation vessel handles a fleet of Borg ships.
Doom can't beat the FF, but I can't remember a time that the Avengers have beaten Doom himself. Yes they beat his schemes, but not so much the man.
Shoot, Doom conquers a war torn Counter-earth, beats the Beyonder (who had previously beaten Galactus) and yet he (or a Doombot) was beaten by Squirrel Girl and the Shroud.
He is probably most inconsistantly written character in comics.