Separate test installation directories and environments #915

hamiltont · 2014-07-07T09:22:13Z

This PR introduces the potential to install items into per-test directories using the new flag --install-strategy (unified | pertest). Adding this necessitated a consistent way to reference the environment, so this PR also creates an environment variable FWROOT that is propagated to all subprocesses/subshells (a number of setup.py files already create a similar variable, this just formalizes it into the framework). A similar variable IROOT tells each subshell or subprocess where its installs directory is.

Propagating these core variables fwroot/iroot allows each test to declare it's own environment relative to these variables. For now, I've placed environment declaring code in <test-dir>/bash_profile.sh, but there's likely a better name. See go/bash_profile.sh for an example of the go test declaring it's own environment using these variables.

This is entirely backwards compatible - unless a test overrides the default environment with it's own bash_profile.sh, the default environment is what is used. This also removes the need to copy over top of ~/.profile, ~/.bash_profile, ~/.bashrc, as this loads the default environment itself and don't rely on bash to load it automatically.

This is all based on #900, which can be a bit confusing. To see a clean comparison of what this adds, please visit this comparison. I based it on #900 because it was a lot easier

This PR resulted from discussion on #899. It should also help to close #448 if merged

This is not fully tested, just putting it out for comment before I spend time testing. I've run enough tests to know it works generally, but it likely needs some cleanup. Also, I've dropped the --install-software flag because it was redundant with --install

msmith-techempower · 2014-07-07T15:03:28Z

Question/thought: could we set up a trivial proxy on localhost and route requests through it? This would improve our test purposes for new-users who are going to go through every test on the first go where there is overlapped downloading. It could be part of the environment setup.

msmith-techempower · 2014-07-07T15:09:52Z

Testing question: must my tester(s) first pull #900 and then merge this PR, or is this a separate PR that has all the changes from #900 already (read: can simply pull this PR into a new branch off master and test)?

hamiltont · 2014-07-07T16:24:24Z

The latter, this already includes all the commits from 900. The proxy idea
is nice

This commit is my proof-of concept, only the 'go' test works right now. You can look at go/ to see the changes required on a per-framework basis - essentially I've modified go/setup.py to move the declaration of the environment into go/bash_profile.sh Each framework now gets a personal <framework>/installs directory, and can declare any needed environment variables in <framework>/bash_profile.sh, such as modifications to the PATH. The FwBm code now provides an FWROOT environment variable to *all* shell scripts (instead of ~/Frameworks being assumed). Also, an IROOT environment variable is provided to any framework-specific files (e.g. install.sh and bash_profile.sh) for the location of the install directory for that framework. See go/bash_profile.sh to see me using both IROOT and FWROOT. By using IROOT, the strategy for installation is still controlled by python code and the frameworks just reference IROOT and FWROOT in their scripts. These variables are a nice step towards solving TechEmpower#448 The new function replace_environ in setup_util.py goes a long way towards addressing TechEmpower#899 - as you can see I've removed all the changes to the ~/.bash_profile from prerequisites.sh (and this commit serves as proof that everything still works) and most of the changes from config/benchmark_profile

- adds an '--install-strategy' option - framework_test properly sets the install location before calling setup.py - removes '--install-software' option (checks for '--install' instead) Still proof-of-concept, only 'go' works!

…test-name>

hamiltont · 2014-07-09T19:31:25Z

I have tested this now with a few frameworks, success. I have not done a 100% test of everything.

I've moved the pertest installation root to FWROOT/installs/pertest/<test-name> so that it's underneath the normal installs/ directory. It was annoying to chase down all those different installs folders underneath each individual test...

Just to be clear, this PR adds

separating install directories
the core features needed to setup a test-specific environment via <test_name>/bash_profile.sh.

This PR does not provide a bash_profile.sh for each test. If a test does not have a bash_profile, then it gets the environment that is setup in config/benchmark_config, and as far as that test is concerned nothing has changed. If a test wants to ensure it has it's own isolated environment, it can use the "new" method of bash_profile.sh

…hmarks into environ

hamiltont · 2014-07-09T21:20:47Z

This wasn't fully rebased onto master yet, now it is

methane · 2014-07-10T18:08:34Z

I'm +1 on separating environment variables. But -1 on install all softwares per tests.
For example, virtualenv is commonly used for lightweight separating Python environment.
Buiding nginx many times just for reverse proxy is bad idea.
php have many frameworks and libraries separated by default.

Can I use pertest environment variable without pertest installation?

methane · 2014-07-10T18:10:20Z

I'm sorry. I misunderstood.
<test>/bash_profile.sh is under <test>, not under installs/<test>.

methane · 2014-07-10T18:14:10Z

toolset/setup/linux/setup_util.py

+def replace_environ(config=None, root=None, print_result=False, command='true'):
+    # Clean up our current environment, preserving some important items
+    mini_environ = os.environ.copy()
+    mini_environ.clear()


This 2 lines means mini_environ = {} ?

Ha ha, yea....

I'll clean it up when I get a chance. I guess I didn't know at the time that env was just a dict, thought it might be some fancy data structure that I needed to copy and clear :-P

hamiltont · 2014-07-10T18:45:34Z

Can I use pertest environment variable without pertest installation?

Just to make it explicit for anyone listening in, yes you can

Based on advice from https://github.com/methane

hamiltont · 2014-07-11T02:56:12Z

@methane just applied those fixes, thanks!

hamiltont · 2014-07-11T03:49:57Z

Could someone confirm that the following is expected:

When I run a test, I get two folders under results/, one called latest and one named for the current time. Both of these folders contain a results.json file, but only latest/results.json contains the raw data. The timestamp\results.json appears to have no raw data. Is this normal!? If not, we need to examine this PR closely to see what is causing this

methane · 2014-07-11T04:15:56Z

@hamiltont Could you try #893 ?

hamiltont · 2014-07-11T06:29:37Z

@methane tested, #893 fixes the unexpected operation, so it's not a problem with this PR. Thanks!

msmith-techempower · 2014-07-11T14:38:02Z

@hamiltont I am confirming that what you see is expected (though, probably not optimal).

latest contains the last benchmark run data. If you ran a test against gemini and then a test against undertow, you should only see the undertow data in latest/results.json.

In order to get that [timestamp]/results.json, you have to execute the parse directive. This is how we come up with the entire "Round X" results file.

idlewan · 2014-07-24T10:39:41Z

Looking at the per-test install directory (https://travis-ci.org/hamiltont/FrameworkBenchmarks/jobs/30719688), something feels not right:

Searching for mongrel2
Installing webserver: mongrel2 in $FWROOT/installs/pertest/nawak/nawak/nawak/nimrod
Searching for zeromq
Installing webserver: zeromq in $FWROOT/installs/pertest/nawak/nawak/nawak/nimrod

It seems that the scripts don't go back to $FWROOT/installs/pertest/thenameofthetestedframework between installing the dependencies. For example, with nawak you have:

# nawak/install.sh
fw_depends nawak nimrod mongrel2

which probably means that nawak installs first, then nimrod, then mongrel2.

# toolset/setup/linux/frameworks/nawak.sh
[...]
git clone git://github.com/idlewan/nawak.git nawak/nawak
cd nawak/nawak
[...]

(there is no need for a subdirectory now, so I should change nawak/nawak to nawak anyway)
And then with nimrod you've got:

# toolset/setup/linux/languages/nimrod.sh
[...]
cd nimrod
[...]

which results in nimrod installed under the nawak git directory, and mongrel2 + zeromq installed under the nimrod directory. We might need to cd back up somewhere, to change

$FWROOT/installs/pertest/nawak/nawak/nawak/nimrod/mongrel2

into:

$FWROOT/installs/pertest/nawak/mongrel2

hamiltont · 2014-07-24T17:56:42Z

@idlewan Good catch. The fix is simple, I'll put in a pr in a bit. If you have the experience, perhaps you could modify the mongrel2.sh and zeromq.sh files so they are installed into prefix directories instead of onto the host system?

idlewan · 2014-07-24T19:28:54Z

Thanks for the fix.

I'm not sure it's strictly necessary to enforce local installation for mongrel2 and zeromq, at least not until some other framework use them too. We already install globally lots of things with apt-get, is there a need not to install anything globally?

hamiltont · 2014-07-24T19:33:29Z

Local install is 1) to be polite for users running TFB on their main
computer and 2) to avoid future conflicts. I know a lot is currently
installed globally, but the more we can remove this the better IMO

idlewan · 2014-07-24T19:47:55Z

I see.
I guess that would mean fiddling with prefix=..., C_INCLUDE_PATH and LD_LIBRARY_PATH, but it should be doable. It would still need the global dependencies currently installed with apt, though (libsqlite3-dev, libsqlite3, uuid-runtime).

hamiltont · 2014-07-24T19:51:41Z

The global stuff is probably never going to be changed from normal apt-get,
it's a pretty small set of reasonably generic software

hamiltont mentioned this pull request Jul 7, 2014

Environment variables for installs/ include external dependencies #899

Closed

hamiltont added 10 commits July 8, 2014 16:04

Create FWROOT and ensure consistent ENV

d9af0b8

Generalize framework-specific install directories

bdebad6

- adds an '--install-strategy' option - framework_test properly sets the install location before calling setup.py - removes '--install-software' option (checks for '--install' instead) Still proof-of-concept, only 'go' works!

Use FWROOT in bash_functions for cleaner output

26916ef

Minor improvements

3d3fba6

Make IROOT available during test execution

dac437e

Use FWROOT to improve bash ERR printing and installation files

7020664

Setup environment for activeweb test

3273af9

Prettify prerequisite installation

99f169b

Move pertest installation directories under FWROOT/installs/pertest/<…

1c5d54a

…test-name>

Merge branch 'master' of https://github.com/TechEmpower/FrameworkBenc…

c1a7b53

…hmarks into environ

hamiltont added 3 commits July 9, 2014 18:51

Ensuring global profile is used if test-specific profile is not present

cbb6220

Adding logging to framework_test

808c515

Finish fixing bash_profile loading

200ce85

methane reviewed Jul 10, 2014
View reviewed changes

Cleaned up code

24138df

Based on advice from https://github.com/methane

hamiltont mentioned this pull request Jul 11, 2014

Consider using HOSTS file for machine roles rather than configuration rewriting #545

Closed

hamiltont mentioned this pull request Jul 11, 2014

New benchmark: ULib framework #921

Closed

jyentechempower merged commit 24138df into TechEmpower:master Jul 14, 2014

This was referenced Jul 14, 2014

Directory structure re-organization #448

Closed

Preserve TFB_ variables when replacing environment #929

Merged

aschneider-techempower mentioned this pull request Jul 18, 2014

Split Play tests / upgrade non-DB tests to Play 2.3 #883

Merged

idlewan mentioned this pull request Jul 24, 2014

Setup continuous verification using travis-ci #939

Merged

This was referenced Aug 10, 2014

add zoot-activate framework #880

Closed

Add Zend framework 1 #874

Merged

ruby 2.1.1 and rails 4.1.0 #847

Closed

Adding support for Martini & Goji frameworks - Go #837

Closed

Separate test installation directories and environments #915

Separate test installation directories and environments #915

Uh oh!

Conversation

hamiltont commented Jul 7, 2014

Uh oh!

msmith-techempower commented Jul 7, 2014

Uh oh!

msmith-techempower commented Jul 7, 2014

Uh oh!

hamiltont commented Jul 7, 2014

Uh oh!

hamiltont commented Jul 9, 2014

Uh oh!

hamiltont commented Jul 9, 2014

Uh oh!

methane commented Jul 10, 2014

Uh oh!

methane commented Jul 10, 2014

Uh oh!

methane Jul 10, 2014

Choose a reason for hiding this comment

Uh oh!

hamiltont Jul 10, 2014

Choose a reason for hiding this comment

Uh oh!

hamiltont commented Jul 10, 2014

Uh oh!

hamiltont commented Jul 11, 2014

Uh oh!

hamiltont commented Jul 11, 2014

Uh oh!

methane commented Jul 11, 2014

Uh oh!

hamiltont commented Jul 11, 2014

Uh oh!

msmith-techempower commented Jul 11, 2014

Uh oh!

idlewan commented Jul 24, 2014

Uh oh!

hamiltont commented Jul 24, 2014

Uh oh!

idlewan commented Jul 24, 2014

Uh oh!

hamiltont commented Jul 24, 2014

Uh oh!

idlewan commented Jul 24, 2014

Uh oh!

hamiltont commented Jul 24, 2014

Uh oh!

Uh oh!