X provides some security when run over a network, but the scope of it is limited and exploits can easily be devised. At the very least, you should make sure that the xauth protocol is used as opposed to the vastly inferior xhost protocol. A better solution is provided by routing all X connections through a secure shell session. This does not only provide more reliable authentication, but it also encrypts all data.