If you are getting this below error in while opening SAPINST GUI in linux machine then run below command.

May 8, 2015 6:38:04 AM [Info]: GUI started.
May 8, 2015 6:38:04 AM [Info]: >> No protocol specified
May 8, 2015 6:38:04 AM [Info]: >> Closed input stream
May 8, 2015 6:38:04 AM [Error]: GUI terminated with return code 4
May 8, 2015 6:38:05 AM [Info]: >> Invocation target exception: Can’t connect to X11 window server using ‘hostname:2’ as the value of the DISPLAY variable; you may consider checking your DISPLAY environment variable.

xhost +

on base machine where we start vnc.

then ssh the host and start GUI.