I think you hit the nail on the head with one word: fertile.

Men who get a late start, and see success in say their 40s, still want to have a family etc. And only a younger woman will be able to provide that.

One of the most successful marriages I've seen had an 18 year gap.

Not to mention, gay partnerships very commonly have a 20+ age difference. And same with outcasts or other "outside the mainstream" people come together.