major grammar updates

[pdawyndt]

Design principles

A new version of the Scratch-LN grammar should follow these design principles:

• the grammar can only make a distinction between certain block shapes
  • stack blocks (surrounded by optional square brackets; see later that we no longer need the square brackets for options) (no distinction made between blocks with two connectors, hat blocks, cap blocks and C-blocks; this is deferred to the visitor)
  • reporter blocks (surrounded by round brackets)
  • boolean blocks (surrounded by angle brackets)
• the grammar uses a modifier to make a distinction between
  • atomic reporter blocks (default)
  • list reporter blocks (modifier required)
• the grammar can only make a distinction between certain literal types
  • text (surrounded by single or double quotes and optional curly braces; selected options should be treated as text, which frees the use of square brackets for other use)
  • numbers (surrounded by curly braces)
  • colors (starts with a hash and optionally surrounded by curly braces)
• the grammar makes a distinction between
  • block delimiters (separate connected blocks)
  • stack delimiters (separate distinct stacks: both top-level stacks and substacks)
• the grammar makes a distinction between top-level stacks and substacks
  • a top-level stack can either be a standalone reporter block, a standalone boolean block or a sequence of connected stack blocks (this is what the grammar calls a stack)
  • a substack can only be a sequence of connected stack blocks
• the grammar can only assume that
  • all blocks may have arguments (surrounded by curly braces; either make them mandatory or recognize parts-of-tags as specific literal types; the latter requires escaping more characters that are used to recognize literals, otherwise only the curly braces need escaping)
  • all stack blocks may have two connectors (reporter blocks and boolean blocks have no connectors; they are always standalone)
  • all stack blocks may have substacks that are only composed of stack blocks (this is supported using Python-style indentation)
• grammar uses Python-style indentation to reflect the shape of C-blocks; this would remove the end keyword that is not visible on the Scratch blocks anyway and would allow the grammar to recognize substacks
• visitor uses block specification from Scratch Extensions to
  • make a distinction between built-in blocks (stack blocks, reporter blocks and boolean blocks) and user-defined blocks; warnings should be issued when certain block shapes can only be built-in (no user-defined reporter blocks or boolean blocks for now in Scratch 3.0)
  • check restrictions on arguments of built-in blocks; in that respect the grammar should not recognize options but treat them as strings with the visitor checking if a selected is valid; a warning should be issued if arguments are invalid (wrong type for literals, wrong shape for blocks)
  • make a distinction between different shapes of stack blocks (cap blocks and hat blocks that only have one connector); a warning should be issued if a hat block is not on top of a stack or if a cap block does not close a stack (as a recovery, these blocks could start a new stack, but may be problematic for substacks)
  • check which built-in blocks can have substacks; having the C-style blocks supported by indentation and the checking being done in the visitor would actually make the grammar keyword-free; issue a warning if a stack block does not support substacks (only built-in blocks may have substacks); this would drop all C-blocks from the grammar and allow the visitor to recognize them dynamically based on the extensions that have been loaded and the language selected for Scratch-LN

[pdawyndt]

Advantages

• cleaner grammar
• better support of built-in blocks by the use of Scratch Extensions standard for built-in block specification; actually Scratch 3.0 should itself use the Scratch Extensions standard as a specification of the built-in blocks, which would allow Scratch-LN to guarantee that all built-in blocks are supported
• more informative warnings issued by visitor (generate errors/warnings in visitor #61)
• support for Scratch Extensions (support Scratch 3.0 extensions #63)
• localization of Scratch-LN

[pdawyndt]

Other ideas

• make block delimiters optional between stack blocks that are surrounded by square brackets to support both horizontal and vertical blocks layouts (support blocks with horizontal layout #90)

  • horizontal layout

    [block][block]...[block]  // first stack
    
    [block][block]...[block]  // second stack
    

    investigate if we could even drop the extra newline between two horizontal stacks

    substacking uses nesting in the horizontal layout

    [block]...[block][block [block][block]...[block]][block]...[block] // block with substack
    

  • vertical layout

    [block]  // first stack
    [block]
    ...
    [block]
    
    [block]  // second stack
    [block]
    ...
    [block]
    

    or a shorthand notation that drops the square brackets around stack blocks in vertical layout

    block    // first stack
    block
    ...
    block
    
    block    // second stack
    block
    ...
    block
    

    substacking uses indentation in the vertical layout

    block        // top-level stack
    ....
    block
        block    // substack
        ...
        block
    block        // continuation of top-level stack
    ....
    block
    

  investigate if and when, square brackets and block delimiters can be mixed in the same environment (where they can be dropped) or if it is needed to let the user specify beforehand whether she want to use horizontal (mandatory square brackets around stack blocks) or vertical (optional square brackets around stack blocks) layout

• allow grammar to make distinction between built-in/extension blocks and user-defined blocks by dynamically generating the grammar from the block specification in Scratch Extensions format; this would allow syntax highlighting (ACE editor for live preview #1) and code completion (code completion for dropdowns #38) of Scratch-LN; the grammar could also enforce shapes and types on built-in/extension blocks and their arguments that prohibits certain connections and nestings

• the definition of a user-defined stack block could be treated the same way as an extension, which could be supported by extending the list of built-in/extension blocks used by the visitor or by injecting new grammar rules while parsing a user-defined stack block; as a result, the parser could make distinction between user-defined blocks that have and haven't been defined in the Scratch-LN script and issue a warning if a closed world model is followed

• the notation

  move (10) steps
  

  is actually closer to the visual representation than

  move {10} steps
  

  so we might consider to make the following changes:

  • square brackets ([]) represent stack blocks or menu parameters; the former are always optional; ; there are different types of stack blocks: connector blocks (two connectors), hat blocks (no top connector), cap blocks (no bottom connector) and menu blocks (no connectors but menu selection), but these cannot be discriminated when the block definitions are available (in the visitor or when dynamically injected in the grammar)
  • round brackets (()) represent reporter blocks (including variables) or reporter parameters (the latter are optional when an actual argument is passed)
  • angle brackets (<>) represent boolean blocks (including variables) or boolean parameters (the latter are optional when an actual argument is passed)
  • double quotes ("") represent shadow blocks (that can be considered as string literals and can not stand on their own); shadow blocks are a special kind of reporter blocks (they also yield a value)

  the full version of the move %1 steps and when %1 key pressed blocks would then become

  [move () steps]               # (1) reporter parameter without an argument (no shadow)
  [move ("") steps]             # (2) reporter parameter filled with an empty shadow block (2)
  [move ("10") steps]           # (3) reporter parameter filled with a shadow block filled with a string
  [move ((count)) steps]        # (4) reporter parameter filled with a variable
  [when [] key pressed]         # (5) menu parameter without an argument
  [when [[space]] key pressed]  # (6) menu parameter filled with a stack block (representing a menu)
  

  as syntactic sugar we could then have the following rules

  1. drop square brackets around stack blocks (also in case where they are passed as an argument to a menu parameter)
  2. drop round brackets around reporter parameters when an actual value is passed (not when there is no shadow (string) or reporter block (including reporter variables))
  3. drop angle brackets around boolean parameters when an actual value is passed (not when there is no boolean block (including reporter variables))
  4. allow menu blocks to be written as shadow blocks, so that can both be delimited by square brackets ([]) or double quotes (""); they actually yield a value
  5. drop square brackets around menu parameters when an actual value is passed (not when there is no stack block or shadow block passed)

  this would allow the following shorthands for the above examples

  move () steps                 # (1) applying rules (i) (rule 2 can not be applied because of the empty shadow)
  move "" steps                 # (2) applying rules (i) and (ii)
  move "10" steps               # (3) applying rules (i) and (ii)
  move (count) steps            # (4) applying rules (i) and (ii)
  when [] key pressed           # (5) applying rule (i) only once (second pair of brackets can not be dropped because no argument is passed)
  when [space] key pressed      # (6) applying rule (i) twice
  when "space" key pressed      # (6) apply rule (iv) to write [space] as "space" and then apply rules (i) and (v)
  

  this would remove the curly braces ({}) from the syntax, as they are replaced by stack ({}), reporter (()) and boolean (<>) parameters; grammar could check if argument matches the parameter if the characters are used to denote the parameters; the characters used to denote the parameters could be dropped if an actual argument is passed and the visitor could then infer the parameter type from the argument type; the visitor could also check if both the parameter type and the argument value match the block specification, otherwise a warning is issued and an "error" extension block is injected in Scratch as a visual placeholder that indicates where the error has happened

  this would also have a natural extension to using substacks using nesting

  [if <<mouse down ?>> then [[move ("10") steps][turn cw ("15") degrees]] else [[move ("20") steps][turn cw ("25") degrees]]]
  

  is then the longer version of (horizontal layout)

  if <mouse down> [move "10" steps][turn cw "15" degrees] else [move "20" steps][turn cw "25" degrees]
  

  or (vertical layout)

  if <mouse down>
      move "10" steps
      turn cw "15" degrees
  else
      move "20" steps
      turn cw "25" degrees
  

• instead of trying to using a custom block in combination with a warning that an invalid argument was used, we could better inject a new block as an "error" extension to Scratch 3.0, using the extension mechanism to report errors, and then add the warning issued as a comment to this block

• support of the Scratchblocks modifiers for ::color or a specific category could also be supported by dynamically adding block specifications / grammar rules; same for the Scratchblocks support for custom C-blocks

• we might try to represent block shapes such as in (here with horizontal layout in mind)

  block shape	notation
  connector	>block>
  hat	|block>
  cap	>block|
  connector	|block|

  a stack could then be written as

  |hat> >connector> >connector> >connector> >cap |
  

  or in short

  | hat > connector > connector > connector > cap |
  

  where both the hat and cap blocks are optional

  > connector > connector > connector >
  

• RESEARCH QUESTION: if the grammar only focuses on the visual aspects of the language (connecting blocks, nesting blocks and substacking), we might even move into the direction of having a grammar that supports all feature that are in Google Blockly, instead of having something Scratch-specific

• terminology:

  • connecting blocks (stacking) snaps blocks together into a stack
  • nesting blocks (arguments) includes one block into another block
  • substacking (clauses) includes one stack into another block

[pdawyndt]

alternative indentation

block
| block
| block
block
block
| block
| block
block

alternative vertical stacking

--------------------
hat
---___--------------
connector
---___--------------
connector
---___--------------
connector
---___--------------
cap
--------------------

as a shorthand for

--------------------
hat
---___--------------
---___--------------
connector
---___--------------
---___--------------
connector
---___--------------
---___--------------
connector
---___--------------
---___--------------
cap
--------------------

this could also be used to represent C-blocks

|---___-----------------
| if <mouse down> then
| |---___---------------
| |---___---------------
| | move "10" steps 
| |---___---------------
| |---___---------------
| else
| |---___---------------
| |---___---------------
| | move "20" steps 
| |---___---------------
| |---___---------------
|---___-----------------