A castle is a token that can be placed on a small city (one made out of two city 'end' pieces) to enable the owner of the castle to score points for the next feature completed in the castle's 6-tile fief.
If you finish a 2-tile mini city you can either score the four points immediately or you can turn the city into a castle by placing one of your three castles (two in 5-6 player game) on the city and keeping the follower on it. The follower is now a lord and cannot be affected by other events to do with cities and knights.
The owner of the castle will score for the next feature completed in the castle's fief. The fief comprises of the two tiles the castle is on and the 2 tiles either side which makes 6 tiles in total. The owner cannot choose which feature is scored, it must be the next one. The castle is worth 0 points at the end of the game but it gives any adjoining farms 4 points instead of the usual 3.