Generic Dictionary with multiple keys

I am building a caching system that holds among others a list of Users. 

Sometimes we access the Users by userId (the primary key).
Sometimes we access the Users by name (a string)

Performance is critical, therefore we currently use two generic dictionaries:
One is Dictionary<int?, User>, the other is Dictionary<string, User>

The issue I am trying to solve is the double maintenance of the User objects.
So, if a User changes, we have to remember to update two dictionaries.

Is there another way to have a "duplicate key" Dictionary without duplicating the values (i.e. Users)?

I found some articles about Tuples<> (also called ClassKeys) but this will not work as it requires all key values when accessing the dictionary.  I only have one at a time (userId or Name).

Of course, we could use a simple LINQ statement against the User list (i.e. .Where( u => u.Name == name)), but the list is very large and performance will not be optimal (or is there a way to build an in-memory index?).

Thanks.