The fix I found is to reload the xmodmap changes every second.
I can only call this a dirty hack and would like to stress that something like this is never a good idea, will probably cause secondary problems and should generally be avoided. Having said that, it seems to work pretty well.
system("xmodmap - < /home/wagner/.xmodmap")
once a second in a loop. This does not seem to create any noticeable system load and so far I have not noticed any problems with keyboard input or mouse movements. If your set-up can switch faster than 1 second (my KVM + monitor combination cannot), you may want to run this more frequently, e.g. every 500ms.
Obviously other calls to xmodmap can be used as well.
at now < echo name_of_script
but that is probably an abuse of at.
Just use the search engine of your choice on
"demon shell_you_use" to find various ways of doing this.