Remember the castle is there to support the force (to leeward) on the top of the centreboard case. If you decrease the distance between the top and bottom of the centreboard the force on each increases.
So, if you remove the castle, you will need to replace that support with something else. Potentially more than was already there.
Some of the modern boats do this by laminating a pad of foam onto the deck at the case.