Currently, if you have enable "partialrefresh" as "true" then ideally it should refresh only the panel for which this property is enabled.
If you enable this property for view panel where sorting is applied on view column, then on click og column header whole Xpage is refreshed.
As such, data (computed fields etc) on other panel also get refreshed.
This is working as designed.
So, need a feature to allow view column sorting to refresh only the view panel without refreshing the entire XPage.