No.
Removing Castle as a fort increases towering competitiveness as a whole.
Benefits far outweigh the negatives of not having it as a fort anymore