The way I see it the cache is fine - X is not.
IMHO de-serialization of single instances should not be an issue of the cache. It is a task for the according class. The main issue here is that this class is changing frequently. I suggest to separate the concern of caching instances and the concern of de-serializing the object. The latter has to be improved so that X can de-serialize older formats as well. This might be very tricky and expensive. If it is too expensive, you must ask yourself do you really need to load old versions as long as X is changing frequently?
BTW a version identifier seems mandatory. Without further knowledge of the structure of X I can only make some guesses, but the structure of X seems to be logically modular (e.g. you spoke of traits). If so, maybe it would help to make this structure explicit.