Skip to content
This repository was archived by the owner on Apr 10, 2025. It is now read-only.

Design Doc: Fully Shared PSOL System Tests

Jump to bottom
Jeff Kaufman edited this page Jan 9, 2017 · 1 revision

Fully Shared PSOL System Tests

Jeff Kaufman, 2013-03-04

System tests that run on mod_pagespeed, ngx_pagespeed, and any other ports we may do.

When you write a new feature you write unit tests. Those tests work everywhere. This is good. You also write system tests. Unless those system tests are simple enough to work in system_test.sh, which most aren’t, you have to write different tests for automatic and mod_pagespeed. Which is already annoying, but add ngx_pagespeed and it becomes impractical.

It’s also frustrating working with the current test environment. Not only do you have to write bash but tests all run sequentially in shared state. When a test fails you can’t re-run just that test, and crafting a reproducible stand-alone test case is a pain. Most of our tests could run in parallel, but the shell script doesn’t make that practical. And if a global variable, for example http_proxy, gets set wrong somewhere it can be a lot of work to track down what happened.

The logic for a test is also currently split up over many places. There’s the test code in apache_system_test.sh but then there’s pagespeed configuration, origin server configuration, and raw files on the origin server.

Let’s have system tests where the code for a test is all in one place, they work across all of the servers we support, and we don’t have to write in shell.

Example A

A test would look like this:

  class UrlValuedAttributeTest(psoltest.SystemTest):
    """Verify dynamically defined url-valued attributes are respected.
   
    Test to make sure dynamically defined url-valued attributes are
    rewritten by rewrite_domains.  See
    mod_pagespeed_test/rewrite_domains.html: in addition to having one
    <img> URL, one <form> URL, and one <a> url it also has one
    <span src=...> URL, one <hr imgsrc=...> URL, and one <hr src=...>
    URL, all referencing src.example.com.  The first three should be
    rewritten because of hardcoded rules, the span.src and hr.imgsrc
    should be rewritten because of ModPagespeedUrlValuedAttribute
    directives, and the hr.src should be left unmodified.  The rewritten
    ones should all be rewritten to dst.example.com.
    """

    def Configure(self):
      self.server = self.NewServer(
        "url_attribute.example.com",
        pagespeed="""
          # Don't actually try to rewrite any resources; the ones in                     
          # rewrite_domains.html don't actually exist.
          RewriteLevel PassThrough

          # This is used for testing dynamically defined url-valued                      
          # attributes
          UrlValuedAttribute span src Hyperlink
          UrlValuedAttribute hr imgsrc Image
          DomainRewriteHyperlinks on
          MapRewriteDomain http://dst.example.com http://src.example.com
          EnableFilters rewrite_domains
          UrlValuedAttribute custom a Image
          UrlValuedAttribute custom b otherResource
          UrlValuedAttribute custom c hyperlink
    """)
   
    def TestDomainsRewritten(self):
      def DomainsRewritten(response):
        return response.body.count("http://dst.example.com")) == 5
      self.server.FetchUntil("rewrite_domains.html", DomainsRewritten)

    def TestPagespeedResourcesCreated(self):
      # There are five resources that should be optimized
      def ResourcesRewritten(response):
        return response.body.count(".pagespeed.") == 5

      # Make sure <custom d=...> isn't modified at all, but that everything
      # else is recognized as a url and rewritten from ../foo to /foo.  This
      # means that only one reference to ../mod_pagespeed should remain,
      # <custom d=...>.
      def CustomDNotModified(response):
        return len(re.findall("d=.[.][.]/mod_pa", response.body)) == 1

      def OnlyOneRelativeUrlLeft(response):
        return response.body.count("../mod_pa") == 1

      # There are five images that should be optimized.
      def AllImagesOptimized(response):
        return response.body.count(".pagespeed.ic") == 5

      self.server.FetchUntil("url_valued_attribute_extend_cache.html"
  	                     "?ModPagespeedFilters=core,+left_trim_urls",
                              ResourcesRewritten,
                              CustomDNotModified,
                              OnlyOneRelativeUrlLeft,
                              AllImagesOptimized)

This replaces:

debug.conf.template:
  ...
  <VirtualHost localhost:@@APACHE_SECONDARY_PORT@@>
    ServerName url_attribute.example.com

    DocumentRoot "@@APACHE_DOC_ROOT@@"
    ModPagespeedFileCachePath "@@MOD_PAGESPEED_CACHE@@/"

    # Don't actually try to rewrite any resources; the ones in                     
    # rewrite_domains.html don't actually exist.                                   
    ModPagespeedRewriteLevel PassThrough

    # This is used for testing dynamically defined url-valued                      
    # attributes                                                                   
    ModPagespeedUrlValuedAttribute span src Hyperlink
    ModPagespeedUrlValuedAttribute hr imgsrc Image
    ModPagespeedDomainRewriteHyperlinks on
    ModPagespeedMapRewriteDomain http://dst.example.com http://src.example.com
    ModPagespeedEnableFilters rewrite_domains
    ModPagespeedUrlValuedAttribute custom a Image
    ModPagespeedUrlValuedAttribute custom b otherResource
    ModPagespeedUrlValuedAttribute custom c hyperlink
  </VirtualHost>
  ...

apache_system_test.sh:
  ...
  # Test to make sure dynamically defined url-valued attributes are rewritten
  # by rewrite_domains.  See mod_pagespeed_test/rewrite_domains.html: in
  # addition to having one <img> URL, one <form> URL, and one <a> url it also
  # has one <span src=...> URL, one <hr imgsrc=...> URL, and one <hr src=...>
  # URL, all referencing src.example.com.  The first three should be rewritten
  # because of hardcoded rules, the span.src and hr.imgsrc should be rewritten
  # because of ModPagespeedUrlValuedAttribute directives, and the hr.src
  # should be left unmodified.  The rewritten ones should all be rewritten to

  # dst.example.com.  
  HOST_NAME="http://url_attribute.example.com"
  TEST="$HOST_NAME/mod_pagespeed_test"
  REWRITE_DOMAINS="$TEST/rewrite_domains.html"
  UVA_EXTEND_CACHE="$TEST/url_valued_attribute_extend_cache.html"
  UVA_EXTEND_CACHE+="?ModPagespeedFilters=core,+left_trim_urls"

  start_test Rewrite domains in dynamically defined url-valued attributes.

  RESPONSE_OUT=$(http_proxy=$SECONDARY_HOSTNAME $WGET_DUMP $REWRITE_DOMAINS)
  MATCHES=$(echo "$RESPONSE_OUT" | fgrep -c http://dst.example.com)
  check [ $MATCHES -eq 5 ]
  MATCHES=$(echo "$RESPONSE_OUT" | \
      fgrep -c '<hr src=http://src.example.com/hr-image>')
  check [ $MATCHES -eq 1 ]

  start_test Additional url-valued attributes are fully respected.

  # There are five resources that should be optimized                            
  http_proxy=$SECONDARY_HOSTNAME \
      FetchUntil $UVA_EXTEND_CACHE 'fgrep -c .pagespeed.' 5

  # Make sure <custom d=...> isn't modified at all, but that everything else
  # is recognized as a url and rewritten from ../foo to /foo.  This means that
  # only one reference to ../mod_pagespeed should remain, <custom d=...>.             
  http_proxy=$SECONDARY_HOSTNAME \
      FetchUntil $UVA_EXTEND_CACHE 'grep -c d=.[.][.]/mod_pa' 1
  http_proxy=$SECONDARY_HOSTNAME \
      FetchUntil $UVA_EXTEND_CACHE 'fgrep -c ../mod_pa' 1

  # There are five images that should be optimized.                              
  http_proxy=$SECONDARY_HOSTNAME \
      FetchUntil $UVA_EXTEND_CACHE 'fgrep -c .pagespeed.ic' 5

And generates the following configurations:

apache:
  <VirtualHost localhost:8083>
    ServerName url_attribute.example.com
    DocumentRoot "/path/to/htdocs"
    ModPagespeedFileCachePath "/path/to/pagespeed_cache/"

    # Don't actually try to rewrite any resources; the ones in
    # rewrite_domains.html don't actually exist.
    ModPagespeedRewriteLevel PassThrough

    # This is used for testing dynamically defined url-valued                      
    # attributes                                                                   
    ModPagespeedUrlValuedAttribute span src Hyperlink
    ModPagespeedUrlValuedAttribute hr imgsrc Image
    ModPagespeedDomainRewriteHyperlinks on
    ModPagespeedMapRewriteDomain http://dst.example.com http://src.example.com
    ModPagespeedEnableFilters rewrite_domains
    ModPagespeedUrlValuedAttribute custom a Image
    ModPagespeedUrlValuedAttribute custom b otherResource
    ModPagespeedUrlValuedAttribute custom c hyperlink
  </VirtualHost>

nginx:
  server {
    server_name url_attribute.example.com;
    root /path/to/htdocs;

    listen 8083;

    pagespeed FileCachePath /path/to/pagespeed_cache/;

    # Don't actually try to rewrite any resources; the ones in                     
    # rewrite_domains.html don't actually exist.                             
    pagespeed RewriteLevel PassThrough;

    # This is used for testing dynamically defined url-valued                      
    # attributes                                                                   
    pagespeed UrlValuedAttribute span src Hyperlink;
    pagespeed UrlValuedAttribute hr imgsrc Image;
    pagespeed DomainRewriteHyperlinks on;
    pagespeed MapRewriteDomain http://dst.example.com http://src.example.com;
    pagespeed EnableFilters rewrite_domains;
    pagespeed UrlValuedAttribute custom a Image;
    pagespeed UrlValuedAttribute custom b otherResource;
    pagespeed UrlValuedAttribute custom c hyperlink;
  }

Example B

  class AvoidRenamingIntrospectiveJavascriptTest(psoltest.SystemTest):
    """Test the AvoidRenamingIntrospectiveJavascript feature (ARIS).

    ARIS should keep introspective scripts like introspection.js from being
    inlined, combined, or otherwise having their urls modified.
    """

    def Configure(self):
      self.aris_on = self.NewServer(
        "aris_on.example.com",
        pagespeed="""
          AvoidRenamingIntrospectiveJavascript on
          InPlaceResourceOptimization on
          """)

      self.aris_off = self.NewServer(
        "aris_off.example.com",
        pagespeed="""
          AvoidRenamingIntrospectiveJavascript off
          InPlaceResourceOptimization on
          """)

    def TestCombining(self):
      url = "avoid_renaming_introspective_javascript.html"

      # aris disables js combining for introspective js and only i-js
      combine_javascript_url = (
        "url" + "?ModPagespeedFilters=combine_javascript")

      def OnlyNormalScriptsCombined(response):
        return response.body.count("src=") == 2
      self.aris_on.FetchUntil(combine_javascript_url,
                              OnlyNormalScriptsCombined)

      # aris disables js combining only when enabled
      def AllScriptsCombined(response):
        return response.body.count("src=") == 1
      self.aris_off.FetchUntil(combine_javascript_url, AllScriptsCombined)

    def TestInlining(self):
      # aris disables js inlining for introspective js and only i-js
      inline_javascript_url = "url" + “?ModPagespeedFilters=inline_javascript”

      def IntrospectiveScriptNotInlined(response):
        return response.body.count("src=") == 1
      self.aris_on.FetchUntil(inline_javascript_url,
                              IntrospectiveScriptNotInlined)

      # aris disables js inlining only when enabled
      def AllScriptsInlined(response):
        return "src=" not in response.body
      self.aris_off.FetchUntil(inline_javascript_url, AllScriptsInlined)

    def TestCacheExtensionAndAllCoreFilters(self):
      # aris disables js cache extension for i-js and only i-js
      rewrite_javascript_url = (
        "url" + “?ModPagespeedFilters=rewrite_javascript”)
      def NormalScriptsRewritten(response):
        return ‘src="../normal.js"’ not in response.body
      def IntrospectiveScriptNotRewritten(response):
        return ‘src="../introspection.js" in response.body
      self.aris_on.FetchUntil(rewrite_javascript_url,
                              NormalScriptsRewritten,
                              IntrospectiveScriptNotRewritten)

      # aris disables js cache extension only when enabled
      def IntrospectiveScriptRewritten(response):
        return ‘src="../introspection.js"’ not in response.body
      self.aris_off.FetchUntil(rewrite_javascript_url,
                               NormalScriptsRewritten,
                               IntrospectiveScriptRewritten)

      core_testing_url = "url" + “?ModPagespeedFilters=core,testing”
      self.aris_on.FetchUntil(core_testing_url,
                              NormalScriptsRewritten,
                              IntrospectiveScriptNotRewritten)

      self.aris_off.FetchUntil(core_testing_url,
                               NormalScriptsRewritten,
                               IntrospectiveScriptRewritten)

    def TestInPlaceRewriting(self):
      # rewrite introspective javascript in the in-place flow, even with aris
      def CommentRemoved(response):
        return "introspective" not in response.body
      self.aris_on.FetchUntil(“introspection.js”, CommentRemoved)

This replaces:

debug.conf.template:

  ModPagespeedAvoidRenamingIntrospectiveJavascript off

mod_pagespeed_test/avoid_renaming_introspective_javascript_on/.htaccess:

  ModPagespeedAvoidRenamingIntrospectiveJavascript on

apache_system_test.sh:

  start_test aris disables js combining for introspective js and only i-js
  URL="$TEST_ROOT/avoid_renaming_introspective_javascript__on/?\                   
  ModPagespeedFilters=combine_javascript"
  FetchUntil $URL 'grep -c src=' 2

  start_test aris disables js combining only when enabled
  URL="$TEST_ROOT/avoid_renaming_introspective_javascript__off.html?\              
  ModPagespeedFilters=combine_javascript"
  FetchUntil $URL 'grep -c src=' 1

  test_filter inline_javascript inlines a small JS file
  start_test aris disables js inlining for introspective js and only i-js
  URL="$TEST_ROOT/avoid_renaming_introspective_javascript__on/?\                   
  ModPagespeedFilters=inline_javascript"
  FetchUntil $URL 'grep -c src=' 1

  start_test aris disables js inlining only when enabled
  URL="$TEST_ROOT/avoid_renaming_introspective_javascript__off.html?\              
  ModPagespeedFilters=inline_javascript"
  FetchUntil $URL 'grep -c src=' 0

  test_filter rewrite_javascript minifies JavaScript and saves bytes.
  start_test aris disables js cache extension for i-js and only i-js
  URL="$TEST_ROOT/avoid_renaming_introspective_javascript__on/?\                   
  ModPagespeedFilters=rewrite_javascript"
  # first check something that should get rewritten to know we're done with        
  # rewriting                                                                      
  FetchUntil -save $URL 'grep -c "src=\"../normal.js\""' 0
  check [ $(grep -c "src=\"../introspection.js\"" $FETCH_FILE) = 1 ]

  start_test aris disables js cache extension only when enabled
  URL="$TEST_ROOT/avoid_renaming_introspective_javascript__off.html?\              
  ModPagespeedFilters=rewrite_javascript"
  FetchUntil -save $URL 'grep -c src=\"normal.js\"' 0
  check [ $(grep -c src=\"introspection.js\" $FETCH_FILE) = 0 ]

  # Check that no filter changes urls for introspective javascript if              
  # avoid_renaming_introspective_javascript is on                                  
  start_test aris disables url modification for introspective js
  URL="$TEST_ROOT/avoid_renaming_introspective_javascript__on/?\                   
  ModPagespeedFilters=testing,core"
  # first check something that should get rewritten to know we're done with        
  # rewriting                                                                      
  FetchUntil -save $URL 'grep -c src=\"../normal.js\"' 0
  check [ $(grep -c src=\"../introspection.js\" $FETCH_FILE) = 1 ]

  start_test aris disables url modification only when enabled
  URL="$TEST_ROOT/avoid_renaming_introspective_javascript__off.html?\              
  ModPagespeedFilters=testing,core"
  FetchUntil -save $URL 'grep -c src=\"normal.js\"' 0
  check [ $(grep -c src=\"introspection.js\" $FETCH_FILE) = 0 ]

And generates the following configurations:

apache:
  <VirtualHost localhost:8083>
    ServerName aris_on.example.com
    DocumentRoot "/path/to/htdocs"
    ModPagespeedFileCachePath "/path/to/pagespeed_cache/"

    ModPagespeedAvoidRenamingIntrospectiveJavascript on
    ModPagespeedInPlaceResourceOptimization on

  </VirtualHost>
  <VirtualHost localhost:8083>
    ServerName aris_off.example.com
    DocumentRoot "/path/to/htdocs"
    ModPagespeedFileCachePath "/path/to/pagespeed_cache/"

    ModPagespeedAvoidRenamingIntrospectiveJavascript off

    ModPagespeedInPlaceResourceOptimization on

  </VirtualHost>

nginx:
  server {
    server_name aris_on.example.com;
    root /path/to/htdocs;

    listen 8083;

    pagespeed FileCachePath /path/to/pagespeed_cache/;

    pagespeed AvoidRenamingIntrospectiveJavascript on;
    pagespeed InPlaceResourceOptimization on;

  }

  server {
    server_name aris_off.example.com;
    root /path/to/htdocs;

    listen 8083;

    pagespeed FileCachePath /path/to/pagespeed_cache/;

    pagespeed AvoidRenamingIntrospectiveJavascript off;
    pagespeed InPlaceResourceOptimization on;

  }

Example C

  class FuriousTest(psoltest.SystemTest):
    """Test the experiment framework (Furious)."””
    def Configure(self):
      self.furious_ga = self.NewServer(
        "furious.example.com",
        pagespeed="""
          RunExperiment on
          AnalyticsID 123-45-6734
          ExperimentVariable 2
          ExperimentSpec id=7;enable=recompress_images;disa\
                         ble=convert_jpeg_to_progressive;percent=50
          ExperimentSpec id=2;enable=recompress_images;percent=50
          """)

      self.furious_no_ga = self.NewServer(
        "furious_no_ga.example.com",
        pagespeed="""
          RunExperiment on
          ExperimentVariable 2
          ExperimentSpec id=7;enable=recompress_images;disa\
                         ble=convert_jpeg_to_progressive;percent=50
          ExperimentSpec id=2;enable=recompress_images;percent=50
          """)

    def GaAgnosticFetch(self, *args, **kwargs):
      self.furious_ga.Fetch(*args, **kwargs)
      self.furiuos_no_ga.Fetch(*args, **kwargs)

    def GaAgnosticFetchUntil(self, *args, **kwargs):
      self.furious_ga.FetchUntil(*args, **kwargs)
      self.furiuos_no_ga.FetchUntil(*args, **kwargs)

    def TestSetCookie(self):
      # The apache-specific tests that verify that experiments work with
      # a .htaccess would go elsewhere. 

      # Check that we set the _GFURIOUS cookie on cookieless requests.
      def CookieIsSet(response):
        return response.HasHeader("_GFURIOUS")
      self.GaAgnosticFetch("extend_cache.html", CookieIsSet)

      # ModPagespeedFilters query param should disable experiments.
      def CookieIsNotSet(response):
        return not CookieIsSet(response)
      self.GaAgnosticFetch("extend_cache.html?ModPagespeed=on&"
                           "ModPagespeedFilters=rewrite_css", CookieIsNotSet)

      # If the user is already assigned, no need to assign them again.
      self.GaAgnosticFetch("extend_cache.html"
                           CookieIsNotSet,
                           headers=["Cookie: _GFURIOUS=2"])

      def TestBeaconIncludesExperimentId(self):
        def BeaconIncludesExperimentId(experiment_id):
          def helper(response):
            return re.match("pagespeed.addInstrumentationInit("
                            "'/mod_pagespeed_beacon', 'load', ''," '', "
                            "'%s', '[^']*/extend_cache.html'" % experiment_id,
                            response.body)
          return helper
        self.GaAgnosticFetch("extend_cache.html"
                             BeaconIncludesExperimentId(2),
                             headers=["Cookie: _GFURIOUS=2"])

        self.GaAgnosticFetch("extend_cache.html"
                             BeaconIncludesExperimentId(7),
                             headers=["Cookie: _GFURIOUS=7"])

        # no-experiment group beacon should not include an experiment id.
        self.GaAgnosticFetch("extend_cache.html"
                             BeaconIncludesExperimentId(“”),
	                     headers=["Cookie: _GFURIOUS=0"])

      def TestResourceUrlsIncludeExperimentIndexes(self):
        # We expect id=7 to be index=a and id=2 to be index=b because that's the         
        # order they're defined above.
        def ResourceTaggedA(response):
          return response.body.count(".pagespeed.a.ic.") == 1
        self.GaAgnosticFetch("extend_cache.html", ResourceTaggedA,
                             headers=["Cookie: _GFURIOUS=7"])

        def ResourceTaggedB(response):
          return response.body.count(".pagespeed.b.ic.") == 1
        self.GaAgnosticFetch("extend_cache.html", ResourceTaggedB,
                             headers=["Cookie: _GFURIOUS=2"])

      def TestImagesDifferWhenUrlSpecifiesDifferentExperiments(self):
        # While the images are the same, image B should be smaller because in
        # the config file we enable convert_jpeg_to_progressive only for id=2
        # (side B). Ideally we would check that it was actually progressive, by
        # checking whether "identify -verbose filename" produced "Interlace:
        # JPEG" or "Interlace: None", but that would introduce a dependency on
        # imagemagick. This is just as accurate, but more brittle (because
        # changes to our compression code would change the computed file sizes).
        def ByteCount(expected_bytes):
          def helper(response):
            return len(response.body) == expected_bytes
          return helper                                                          
        self.GaAgnosticFetchUntil(
          "/images/xPuzzle.jpg.pagespeed.a.ic.fakehash.jpg", ByteCount(231192))
        self.GaAgnosticFetchUntil(
          "/images/xPuzzle.jpg.pagespeed.b.ic.fakehash.jpg", ByteCount(216942))

      def TestAnalyticsJavascriptAddedOnlyWhenGaIsOn(self)
        def AnalyticsJs(response):
          return "Experiment:" in response.body
        def NoAnalyticsJs(response):
          return not AnalyticsJs(response)

        # Analytics is not added for any group when GA is off.
        self.furious_no_ga.Fetch("extend_cache.html", NoAnalyticsJs,
                                 headers=["Cookie: _GFURIOUS=2"])
        self.furious_no_ga.Fetch("extend_cache.html", NoAnalyticsJs,
                                 headers=["Cookie: _GFURIOUS=7"])
        self.furious_no_ga.Fetch("extend_cache.html", NoAnalyticsJs,
                                 headers=["Cookie: _GFURIOUS=0"])

        # Analytics is added for the experimental group when GA is on.
        self.furious_ga.Fetch("extend_cache.html", AnalyticsJs,
                              headers=["Cookie: _GFURIOUS=2"])
        self.furious_ga.Fetch("extend_cache.html", AnalyticsJs,
                              headers=["Cookie: _GFURIOUS=7"])

        # Analytics is not added for the no-experiment group even when GA is on.
        self.furious_ga.Fetch("extend_cache.html", NoAnalyticsJs,
                              headers=["Cookie: _GFURIOUS=0"])

This replaces:

debug.conf.template:
  #FURIOUS_GA # This is used for testing the Furious experiment framework.         
  #FURIOUS_GA ModPagespeedRunExperiment on                                         
  #FURIOUS_GA ModPagespeedAnalyticsID "123-45-6734"                                
  #FURIOUS_GA ModPagespeedExperimentVariable 2                                     
  #FURIOUS_GA ModPagespeedExperimentSpec "id=7;enable=recompress_images;disa\

              ble=convert_jpeg_to_progressive;percent=50"                                            
  #FURIOUS_GA ModPagespeedExperimentSpec \

              "id=2;enable=recompress_images;percent=50"                                                                                

  #FURIOUS_NO_GA # This is used for testing the Furious experiment framework
  #FURIOUS_NO_GA # still works when no analytics ID is specified.  It should
  #FURIOUS_NO_GA # assign users to experiments and use appropriate 
  #FURIOUS_NO_GA # experimental options, but not report back to Google
  #FURIOUS_NO_GA # Analytics.  The instrumentation beacon, however, will still

  #FURIOUS_NO_GA # contain the experiment id.          
  #FURIOUS_NO_GA ModPagespeedRunExperiment on                                      
  #FURIOUS_NO_GA ModPagespeedExperimentVariable 2                                  
  #FURIOUS_NO_GA ModPagespeedExperimentSpec "id=7;enable=recompress_images;di

                 sable=convert_jpeg_to_progressive;percent=50"                                         
  #FURIOUS_NO_GA ModPagespeedExperimentSpec \

                 "id=2;enable=recompress_images;percent=50"

apache_furious_test_ga.sh:

  start_test Analytics javascript is added for the experimental group.
  OUT=$($WGET_DUMP --header='Cookie: _GFURIOUS=2' $EXTEND_CACHE)
  check_from "$OUT" fgrep -q 'Experiment: 2'
  OUT=$($WGET_DUMP --header='Cookie: _GFURIOUS=7' $EXTEND_CACHE)
  check_from "$OUT" fgrep -q 'Experiment: 7'

  start_test Analytics javascript is not added for the no-experiment group.
  OUT=$($WGET_DUMP --header='Cookie: _GFURIOUS=0' $EXTEND_CACHE)
  check_not_from "$OUT" fgrep -q 'Experiment:'

apache_furious_test_no_ga.sh:

  start_test Analytics javascript is not added for any group.
  OUT=$($WGET_DUMP --header='Cookie: _GFURIOUS=2' $EXTEND_CACHE)
  check_not_from "$OUT" fgrep -q 'Experiment:'
  OUT=$($WGET_DUMP --header='Cookie: _GFURIOUS=7' $EXTEND_CACHE)
  check_not_from "$OUT" fgrep -q 'Experiment:'
  OUT=$($WGET_DUMP --header='Cookie: _GFURIOUS=0' $EXTEND_CACHE)
  check_not_from "$OUT" fgrep -q 'Experiment:'

apache_furious_test.sh:

  echo Testing whether or not Furious is working.
  start_test mod_pagespeed_example must have a .htaccess file.
  check test -f $EXAMPLE_FILE_DIR/.htaccess

  start_test _GFURIOUS cookie is set.
  OUT=$($WGET_DUMP $EXTEND_CACHE)
  check_from "$OUT" fgrep "_GFURIOUS="

  start_test mod_pagespeed_test must not have a .htaccess file.
  check_not test -f $TEST_ROOT_FILE_DIR/.htaccess

  start_test ModPagespeedFilters query param should disable experiments.
  OUT=$($WGET_DUMP '$EXTEND_CACHE?ModPagespeed=on&ModPagespeedFilte\

                   rs=rewrite_css')
  check_not_from "$OUT" fgrep '_GFURIOUS='

  start_test If the user is already assigned, no need to assign them again.
  OUT=$($WGET_DUMP --header='Cookie: _GFURIOUS=2' $EXTEND_CACHE)
  check_not_from "$OUT" fgrep '_GFURIOUS='

  start_test The beacon should include the experiment id.
  OUT=$($WGET_DUMP --header='Cookie: _GFURIOUS=2' $EXTEND_CACHE)
  check_from "$OUT" grep "pagespeed.addInstrumentationInit('/mod_pagespeed_be\

                          acon', 'load', '', '', '2', 'http://localhost\

                                                    [:0-9]*/mod_pagespeed_example/extend_cache.html');"
  OUT=$($WGET_DUMP --header='Cookie: _GFURIOUS=7' $EXTEND_CACHE)
  check_from "$OUT" grep "pagespeed.addInstrumentationInit('/mod_pagespeed_be\

                    acon', 'load', '', '', '7', 'http://localhost\

                    [:0-9]*/mod_pagespeed_example/extend_cache.html');"

  start_test no-experiment group beacon should not include an experiment id.
  OUT=$($WGET_DUMP --header='Cookie: _GFURIOUS=0' $EXTEND_CACHE)
  check_not_from "$OUT" grep 'mod_pagespeed_beacon.*exptid'

  # We expect id=7 to be index=a and id=2 to be index=b because that's the         
  # order they're defined in the config file.                                      
  start_test Resource urls are rewritten to include experiment indexes.
  WGET_ARGS="--header 'Cookie:_GFURIOUS=7'" FetchUntil $EXTEND_CACHE \
    "fgrep -c .pagespeed.a.ic." 1
  WGET_ARGS="--header 'Cookie:_GFURIOUS=2'" FetchUntil $EXTEND_CACHE \
    "fgrep -c .pagespeed.b.ic." 1
  OUT=$($WGET_DUMP --header='Cookie: _GFURIOUS=7' $EXTEND_CACHE)
  check_from "$OUT" fgrep ".pagespeed.a.ic."
  OUT=$($WGET_DUMP --header='Cookie: _GFURIOUS=2' $EXTEND_CACHE)
  check_from "$OUT" fgrep ".pagespeed.b.ic."

  start_test Images are different when url specifies different experiments.
  # While the images are the same, image B should be smaller because in the
  # config file we enable convert_jpeg_to_progressive only for id=2 (side B).
  # Ideally we would check that it was actually progressive, by checking
  # whether "identify -verbose filename" produced "Interlace: JPEG" or
  # "Interlace: None", but that would introduce a dependency on imagemagick.
  # This is just as accurate, but more brittle (because changes to our
  # compression code would change the computed file sizes).                                                          
  IMG_A="$EXAMPLE/images/xPuzzle.jpg.pagespeed.a.ic.fakehash.jpg"
  IMG_B="$EXAMPLE/images/xPuzzle.jpg.pagespeed.b.ic.fakehash.jpg"
  FetchUntil $IMG_A 'wc -c' 231192
  FetchUntil $IMG_B 'wc -c' 216942

And generates the following configurations:

apache:

  <VirtualHost localhost:8083>
    ServerName furious_ga.example.com
    DocumentRoot "/path/to/htdocs"
    ModPagespeedFileCachePath "/path/to/pagespeed_cache/"

    ModPagespeedRunExperiment on                                         
    ModPagespeedAnalyticsID "123-45-6734"                                
    ModPagespeedExperimentVariable 2                                     
    ModPagespeedExperimentSpec "id=7;enable=recompress_images;disable=conv\

    ert_jpeg_to_progressive;percent=50"                                            
    ModPagespeedExperimentSpec "id=2;enable=recompress_images;percent=50"   

  </VirtualHost>
  <VirtualHost localhost:8083>
    ServerName furious_no_ga.example.com
    DocumentRoot "/path/to/htdocs"
    ModPagespeedFileCachePath "/path/to/pagespeed_cache/"

    ModPagespeedRunExperiment on                                
    ModPagespeedExperimentVariable 2                                     
    ModPagespeedExperimentSpec "id=7;enable=recompress_images;disable=conv\

    ert_jpeg_to_progressive;percent=50"                                            
    ModPagespeedExperimentSpec "id=2;enable=recompress_images;percent=50" 

  </VirtualHost>

nginx:

  server {
    server_name furious_ga.example.com;
    root /path/to/htdocs;

    listen 8083;

    pagespeed FileCachePath /path/to/pagespeed_cache/;

    pagespeed RunExperiment on;                                         
    pagespeed AnalyticsID "123-45-6734";                                
    pagespeed ExperimentVariable 2;                                     
    pagespeed ExperimentSpec "id=7;enable=recompress_images;disable=conv\

    ert_jpeg_to_progressive;percent=50";                                            
    pagespeed ExperimentSpec "id=2;enable=recompress_images;percent=50"; 

  }

  server {
    server_name furious_no_ga.example.com;
    root /path/to/htdocs;

    listen 8083;

    pagespeed FileCachePath /path/to/pagespeed_cache/;

    pagespeed RunExperiment on;

    pagespeed ExperimentVariable 2;                                     
    pagespeed ExperimentSpec "id=7;enable=recompress_images;disable=conv\

    ert_jpeg_to_progressive;percent=50";                                            
    pagespeed ExperimentSpec "id=2;enable=recompress_images;percent=50"; 

  }

Implementation

Test classes (AvoidRenamingIntrospectiveJavascriptTest, UrlValuedAttributeTest, …) all get their Configure() methods called serially when we start up testing. Then the test runner stands up a server supporting all relevant configurations (or multiple servers). Then we can run all Test*() methods on all the tests, and this stage can be as parallel as we like.

Clone this wiki locally