Running native console apps causes their stdout to be written in the Host

[daviwil]

We've just discovered an issue where running native console apps causes the app's stdout to appear in the output of the Host process. This breaks the connection with the editor client because unexpected characters start to appear in the message stream. We need to find a reasonable way to handle native console app output so that it gets routed through the standard host output mechanism.

One workaround for now is that the user can run their application piped through Out-String, for example:

ipconfig.exe | Out-String

[adamdriscoll]

We might have a solution to this in PoshTools. I'm away from keyboard at the moment or I would check.

Sent from my iPhone

On Feb 8, 2016, at 6:56 PM, David Wilson notifications@github.com wrote:

We've just discovered an issue where running native console apps causes the app's stdout to appear in the output of the Host process. This breaks the connection with the editor client because unexpected characters start to appear in the message stream. We need to find a reasonable way to handle native console app output so that it gets routed through the standard host output mechanism.

One workaround for now is that the user can run their application piped through Out-String, for example:

ipconfig.exe | Out-String
—
Reply to this email directly or view it on GitHub.

[daviwil]

I've been scanning through the code and commits and can't tell what exactly causes PoshTools' REPL to get the stdout correctly. Almost seems like the service host process facilitates it somehow but I don't see a direct correlation.

[rkeithhill]

With very little testing, this seems to work:

        public PowerShellContext()
        {
            this.psHost = new ConsoleServicePSHost();
            this.initialSessionState = InitialSessionState.CreateDefault2();

            Runspace runspace = RunspaceFactory.CreateRunspace(psHost, this.initialSessionState);
            runspace.ApartmentState = ApartmentState.STA;
            runspace.ThreadOptions = PSThreadOptions.ReuseThread;
            runspace.Open();

            this.ownsInitialRunspace = true;

            this.Initialize(runspace);

            // Use reflection to execute ConsoleVisibility.AlwaysCaptureApplicationIO = true;
            Type conVisibilityType = Type.GetType("System.Management.Automation.ConsoleVisibility, System.Management.Automation, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35");
            PropertyInfo propInfo =
                conVisibilityType.GetProperty(
                    "AlwaysCaptureApplicationIO", BindingFlags.Static | BindingFlags.Public);

            propInfo.SetValue(null, true);
        }

Might be good to ask someone on the core PS team if this is the right approach. There are some other properties on S.M.A.NativeCommandProcessor that might also fit (IsServerSide).

[daviwil]

Wow, I've never seen this before! If it works well, I'm all for it. From what I understand, the "right" way to do this long term is like what ConEmu does, installing a hook into each launched application and intercepting all the output. That takes a lot of code and is hard work, so if your current solution works in most cases I'd prefer it for now.

[rkeithhill]

Can you check the ISE codebase and see what it uses to capture console output? Does it call this AlwaysCaptureApplicationIO property or some other internal member of NativeCommandProcessor?

[rkeithhill]

BTW the clue was in NativeCommandProcessor's CalculateIORedirection method:

if (NativeCommandProcessor.IsServerSide)
{
  redirectInput = true;
  redirectOutput = true;
  redirectError = true;
}
else if (NativeCommandProcessor.IsConsoleApplication(this.Path))
{
  ConsoleVisibility.AllocateHiddenConsole();
  if (ConsoleVisibility.AlwaysCaptureApplicationIO)
  {
    redirectOutput = true;
    redirectError = true;
  }
}

[daviwil]

I don't find any references to NativeCommandProcessor or AlwaysCaptureApplicationIO in the ISE codebase. Interestingly, ISE's PSHost implementation has nothing in the NotifyBeginApplication/NotifyEndApplication methods.

When the ISE runs commands while not stopped at a breakpoint it passes them through Out-Default (or Out-String if stopped at breakpoint). I have been searching for various different ways of referring to stdout or output redirection and haven't found anything concrete yet.

One clue I see in the ISE is that the first time I run a native app, I briefly see a console window appear and then disappear. Not sure who/what is spawning that and why I don't see it on subsequent executions.

So far your approach seems to be the most promising, though I wonder why I don't see any other host using it :) I'll ask someone tomorrow to see what pitfalls there may be.

[daviwil]

I dug around the PowerShell codebase a bit, it looks like AlwaysCaptureApplicationIO has been there at least as long as PowerShell v3 so we shouldn't have to worry about it not working on older versions. I've sent an e-mail to Jason to ask if there are any more options. Maybe we can use this option as a short-term solution and implement something more robust for the next release? I don't see any harm in using it just for now.

[daviwil]

Talked to Jason about this, he says that redirection of native apps might cause weird behavior in some cases and suggests not using stdin/stdout as our host message channel so that we can use it to grab the output of native apps that get run in process. Long term I think that's a better thing to do but it'd require adding WebSocket channel support to VS Code so that we can get rid of our stdin/stdout channel. Honestly, that may solve some other problems we have around encrypted message transmission, so it may not be bad to make the switch sooner rather than later.

I might go ahead and try integrating the fix you found so that we can move forward with the release today. Since we aren't advertising a full console experience yet, I think it's better to make progress on that front even if there are some rough spots in the short term.

[rkeithhill]

Sounds good.