If you are running NetScaler 10.5 and you try to take a trace from the GUI and view it in WireShark you will likely get this error message. You will also notice that the trace is a lot smaller than it should be
Citrix eDocs states that you must be running WireShark version 1.11.3 to which they provide a link, which is broken.
The latest stable release of WireShark is 1.10.8 and the Development release is 1.12.0-rc2. Neither of which are able to open the trace files. There is also the fact that the trace files taken via the GUI are much much smaller than they should be.
I have tried a lot of combinations of settings in the GUI, none of which made any difference.
To get round this issue you need to run the trace using the CLI using the commands below
start nstrace size 0
And then use WinSCP to download the trace file from /var/nstrace