Storage testing

[hendrikvanantwerpen]

This introduces a CLI test which runs indexing and querying on some source files. These should help catch issues like #299 earlier.

As part of this PR:

• A new test-cli workflow job that runs index, status, and query operations on a local source directory. This fails at the moment, because the current code is broken, but the check is not required to merge.

• Instead of cloning a repo to test on, two archives (in LFS) with curated test cases are added to the repo. One holds several TS files in various sizes for the performance benchmark. The other holds a minimal project for the CLI test. The advnatages:

  • The tests become stand-alone, no repo cloning necessary.
  • The test size is reduced, resulting in quicker CI times, and less resource wasting. For the performance test, we are mostly interested in changes in the profile. Analyzing multiple files of similar size, which happened with the previous repo, doesn't add meaningful data.

It might be nice to test the index/query flow in a proper Rust test eventually. But calling the CLI in CI is the most expedient way to do some basic CLI verification.

[github-actions]

Performance Summary

Comparing base bb7457a with head 876b244 on TypeScript-4.9.5 benchmark. For details see workflow artifacts. Note that performance is tested on the last commits with changes in stack-graphs, not on every commit.

Before

--------------------------------------------------------------------------------
Command:            base/target/release/tree-sitter-stack-graphs-typescript index -D base.sqlite --max-file-time=30 --hide-error-details -- TypeScript-4.9.5/src/compiler
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 base-perf-results/perf.out
--------------------------------------------------------------------------------


    MB
864.7^                               #                                        
     |            @                  #                                        
     |            @                  #                                        
     |            @                  #                                  :     
     |            @          :       #                               :  :     
     |            @       :  :       #                               :  :     
     |            @       :  :      :#               :               :  :     
     |            @       :  :      :#             : :               : ::     
     |      :    :@       :  :      :#::           : :    ::        :: ::     
     |      ::   :@   :  ::  :  :   :#:            :::    :         :: ::     
     |      ::   :@   :  ::  :  ::: :#:   ::  :: :::::  :::    ::  ::: :::    
     |      :::  :@@  :  ::  :::::  :#:   : ::: :: :::: : : ::::   ::: ::: @::
   0 +----------------------------------------------------------------------->Gi
     0                                                                   166.5

After

--------------------------------------------------------------------------------
Command:            head/target/release/tree-sitter-stack-graphs-typescript index -D head.sqlite --max-file-time=30 --hide-error-details -- TypeScript-4.9.5/src/compiler
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 head-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
1.120^            #                                                           
     |            #                                                           
     |            #                                                           
     |            #                                                           
     |            #                  :                                   :    
     |            #          :       :                                  ::    
     |           @#          :       :                                : ::    
     |           @#      :   :       :    :                          :: ::    
     |       :   @#      :  ::      :: : ::           :              :: ::    
     |     : :  :@#  :  ::  ::      :: : ::         :::    :         :@ ::    
     |     :::  :@#: :: ::: ::      :: ::::  @ :@   @::  : @    :@   :@::: :  
     |     :::  :@#::::::::::: :::  :: ::::::@ :@ ::@::::::@ ::::@::::@:::::: 
   0 +----------------------------------------------------------------------->Gi
     0                                                                   186.3

[github-actions]

Performance Summary

Comparing base bb7457a with head d66dbfd on typescript_benchmark benchmark. For details see workflow artifacts. Note that performance is tested on the last commits with changes in stack-graphs, not on every commit.

Before

--------------------------------------------------------------------------------
Command:            base/target/release/tree-sitter-stack-graphs-typescript index -D base.sqlite --max-file-time=30 --hide-error-details -- typescript_benchmark/src/compiler/utilities.ts
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 base-perf-results/perf.out
--------------------------------------------------------------------------------


    MB
850.3^                                                                     #  
     |                                                                    @#: 
     |                                                              @@ ::@@#: 
     |                                                              @ :::@@#::
     |                                                             @@ :::@@#::
     |                                                            @@@ :::@@#::
     |                                                         @@@@@@ :::@@#::
     |                                                        @@@ @@@ :::@@#::
     |                                                      @@@@@ @@@ :::@@#::
     |                                                    @@@ @@@ @@@ :::@@#::
     |                                                   @@@@ @@@ @@@ :::@@#::
     |                                                  @@@@@ @@@ @@@ :::@@#::
   0 +----------------------------------------------------------------------->Gi
     0                                                                   17.85

After

--------------------------------------------------------------------------------
Command:            head/target/release/tree-sitter-stack-graphs-typescript index -D head.sqlite --max-file-time=30 --hide-error-details -- typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 head-perf-results/perf.out
--------------------------------------------------------------------------------


    MB
715.8^                                                                   #    
     |                                               @                   #    
     |                                               @                   #    
     |                                              :@    :            ::#    
     |                                            :::@    :            ::#   :
     |                   @@    :@                 : :@    :            ::#   :
     |                  :@ : :::@                :: :@ ::@:            ::#  ::
     |                  :@ ::: :@        ::      :: :@:: @:            ::# :::
     |                  :@ ::: :@       ::       :: :@:: @:          @@::#::::
     |                 @:@ ::: :@  :  ::::       :: :@:: @:          @ ::#::::
     |                 @:@ ::: :@  :::: :: :    ::: :@:: @:       :: @ ::#::::
     |                @@:@ ::: :@  :: : :: :::  ::: :@:: @: ::::::: :@ ::#::::
   0 +----------------------------------------------------------------------->Gi
     0                                                                   56.00

[github-actions]

Performance Summary

Comparing base bb7457a with head 342d987 on typescript_benchmark benchmark. For details see workflow artifacts. Note that performance is tested on the last commits with changes in stack-graphs, not on every commit.

Before

--------------------------------------------------------------------------------
Command:            base/target/release/tree-sitter-stack-graphs-typescript index -D base.sqlite --max-file-time=30 --hide-error-details -- base/data/typescript_benchmark/src/compiler/utilities.ts
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 base-perf-results/perf.out
--------------------------------------------------------------------------------


    MB
850.3^                                                              #         
     |                                                             @#:        
     |                                                        @@ :@@#:        
     |                                                        @ ::@@#::      :
     |                                                       @@ ::@@#::     ::
     |                                                      @@@ ::@@#::   @:::
     |                                                   @@@@@@ ::@@#:::::@:::
     |                                                  @@@ @@@ ::@@#:::::@:::
     |                                                 @@@@ @@@ ::@@#:::::@:::
     |                                               @@@@@@ @@@ ::@@#:::::@:::
     |                                              @@@@@@@ @@@ ::@@#:::::@:::
     |                                             @@@@@@@@ @@@ ::@@#:::::@:::
   0 +----------------------------------------------------------------------->Gi
     0                                                                   19.82

After

--------------------------------------------------------------------------------
Command:            head/target/release/tree-sitter-stack-graphs-typescript index -D head.sqlite --max-file-time=30 --hide-error-details -- head/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 head-perf-results/perf.out
--------------------------------------------------------------------------------


    MB
715.8^                                                                      # 
     |                                                    :                 # 
     |                                                   @:                 # 
     |                                                 ::@:               ::# 
     |                                                 : @:               : # 
     |                     :     :@@                  :: @:             ::: # 
     |                    ::::  ::@                   :: @:             : : # 
     |                    ::: ::::@         ::        :: @:             : : #:
     |                    ::: : ::@        ::        @:: @:            :: : #:
     |                   @::: : ::@   :  ::::        @:: @:           @:: : #:
     |                   @::: : ::@   :::: ::  ::   :@:: @:      :::  @:: : #:
     |                   @::: : ::@   :: : :: :: :  :@:: @:::::  : :  @:: : #:
   0 +----------------------------------------------------------------------->Gi
     0                                                                   51.26

[rewinfrey]

👍 to verifying the various CLI use cases in an Actions workflow. I left a question about implications of using LFS, but otherwise the changes look good.

Thank you also for the concurrency control group updates to these workflows!

[rewinfrey]

I'm still not clear on when LFS is required vs not required in local dev. Does this require interacting with LFS for local dev in anyway?

[hendrikvanantwerpen]

Good question 🤔 I think when you change things matching the patterns, you'll have to. But they're currently only used in CI, so in most cases I don't think so. But I'll try this out and otherwise open a follow-up PR updating docs around this. (An external contributor wants to address the serialization bug, so merging this will help set them up easier.)