Adding iTerm2 and Kitty protocol via `timg`, wants idea

[Moelf]

timg_jll is in the pipeline and should land soon, for the moment being:

import timg_jll
function timgshow(io::IO, img_path::AbstractString; 
        is_kitty = !isnothing(get(ENV, "KITTY_WINDOW_ID", nothing)),
        is_iTerm = !isnothing(get(ENV, "ITERM_PROFILE", nothing))
    )
    @info img_path
    s = if is_kitty
        read(`$(timg_jll.timg()) -p k -W --grid=5 $img_path`, String)
    elseif is_iTerm
        read(`$(timg_jll.timg()) -p i -W --grid=5 $img_path`, String)
    else
        read(`$(timg_jll.timg()) -p q -W --grid=5 $img_path`, String)
    end
    print(io, s)
end

yields:

on Linux, left is Alacritty and right is Kitty. The quarter-pixel result is okay but whatever. (sadly Alacritty doesn't support Sixel either), but the point is we could have very convincing result for Kitty and iTerm2.

I'd like to know if people are interested in having this and if so, what's the best way to integrate into the existing interface/dispatch routes.

(video is also possible if we really want it:

Peek.2021-05-23.01-53.mp4

[johnnychen94]

Wow, great to know this! I'd be very happy to have this in ImageInTerminal.

I do have a rewrite plan (#52), my current vision about ImageInTerminal is to:

1. mimic a virtual Display, i.e., TerminalGraphicDisplay for a terminal environment (WIP: add TerminalGraphicDisplay for image/png #49)
2. rewrite ImageInTerminal into an image encoding package with various encoding backends.
3. offload image display-related codes to ImageShow via TerminalGraphicDisplay

I've moved #43 to ImageShow, so ImageShow currently has two/three advanced video display functions: play/explore/gif. By introducing a virtual Display in ImageInTerminal, we can reuse these codes.

---

For backward compatibility to Julia 1.0, advanced image encoders (Sixel, timg) are designed as plugins. The current imshow logic will be rewrite into a so-called BasicEncoder with encode([io=stdout], img, ::BasicEncoder) API.

Hence if you can provide a small wrapper on timg_jll and also provide a similar encode([io=stdout], img, ::TimgEncoder) thing, it would be very easy to add timg encoding here in the future.

I've played with this design in my Sixel.jl so you might want to take a look for inspiration and example.

---

Except for the play/explore feature, I don't plan to have very good video support in the near future.

timgshow(stdout, ".../file.mp4") looks good, but I might not use this API. The reason is: we can directly depend on VideoIO, and add more interactive functionality on the Julia side. The play/explore is one such example.

---

Does img_jll supports non-stdout io types, e.g., IOBuffer, IOStream?

[johnnychen94]

Because ImageInTerminal is designed to directly process AbstractArray{<:Colorant} data, to properly have timg used in ImageInTerminal, one key challenge is to not use the highest level API for files, otherwise, the overhead is just way too big (by writing to a temporary file and reading it back).

We have to look at their source codes and find out how to call the encoder for a Julia array. For example, this is how I encode a Julia array with libsixel's low-level API: https://github.com/johnnychen94/Sixel.jl/blob/master/src/backend/libsixel/encoder.jl#L21-L38

When that's done, timg_jll dependencies to FFMPEG_jll/JpegTurbo_jll and others might not be needed at all.

[johnnychen94]

I haven't look at the timg implementations, but if it's just dispatching the encoding to iTerm2/kitty API, something like https://github.com/Keno/TerminalExtensions.jl is perhaps a better idea as it's written in pure Julia.

[Moelf]

it's just dispatching the encoding to iTerm2/kitty API, something like

actually, that package does seem like a better place once we have re-written the encoding in pure-Julia. Basically iTerm2 and Kitty has their own protocol that works like ~ [escapes][base64 encoded image][escape], the issue however is that Kitty's version is not as polished and you'd have to handle cursor and positioning and clearing yourself. timg makes it easier.

But the core is (supposedly) very simple and can easily be done in Julia. Except Kitty doesn't have very clear documentation unlike iTerm2: https://iterm2.com/documentation-images.html

[Moelf]

https://github.com/simonschoelly/KittyTerminalImages.jl

aaaaaand I just found out this

[johnnychen94]

@Moelf Or do you have any specific plan to add iTerm2/Kitty support to ImageInTerminal? If not, I guess this issue can be closed.