Home

Welcome to the wiki of RES replicators!

Common FAQs

To what extent are we supposed to modify the code to make it run?

In principle the code should run as is. Of course your job is to test this and figure out what could be changed to make it run. In particular, small modifications like fixing a wrong path should always be done. It's very important that you keep track of the changes you make to the code base and that you mention those in your report. Obviously git is the right tool for that.

Examples:

Matlab: Author puts load tables\Tables_AR_MSE_Size.mat. You change and report load 'tables/Tables_AR_MSE_Size.mat' or load(fullfile(pwd, 'tables', 'Tables_AR_MSE_Size.mat'))

What if tables and figures generated by the replication package differ only slightly from results in the paper?

In your report, you are supposed to inform the data editor about any difference in both numbers, regardless of how small the difference might be. If the displayed numbers are not identical, it needs to be reported with a reference to the exact table/figure numbers.

Can the data replicator contact directly the authors to ask clarifications about the package?

No. This is above all to protect the identity of the data replicator. If something is not clear or not working, always get in touch with the Data Editor.

Nuvolos

I have to run a script that takes several hours to run on Nuvolos. When I log in again, I find that Nuvolos has shut down my app. How to prevent Nuvolos from shutting down apps for inactivity?

Nuvolos has a 6-hour standard timeout at the moment of writing (October 2024). You can change the settings of your apps to increase the timeout due to inactivity up to a max of 24 hours. This can be achieved by clicking on the "Actions" options of your app in the "Applications" view, selecting "Configure" and moving the "Inactivity limit" slider to the right.

My Nuvolos apps keep crashing during (computationally) heavy jobs which require large amounts of RAM. How to prevent this from happening?

If your app consumes more memory than what is available, the app will be killed. You will not get an error message in most cases, but your app will just freeze or shut down. You can increase the size (CPU + RAM) by clicking on the cogwheel next to the apps start button in the “Applications” view. For more info, refer to the Tips and Tricks for Replicators).

Stata

Which libraries are required?

tl;dr : use nuvolos, which by default has no libraries installed. Therefore, if the authors did not specify library xyz in their readme and we run it, there will be an error.

Here is some background discussion on statalist.

There is no way to get a list of required libraries out of a stata project by just looking at the code. This is because there is no notion of loading a library in stata. Built-in commands and user contributed commands look the same. So it’s basically impossible to know from the outset which packages need to be installed, if the author does not tell you explicitly. Before you try to run anything, you should clear out your stata searchpath for the current session (closing and opening will reset everything to normal, so don’t worry). So, by default stata looks in a few locations on your computer for installed packages. you can see those by typing adopath:

. adopath
  [1]  (BASE)      "/Applications/Stata 18.0/ado/base/"
  [2]  (SITE)      "/Applications/Stata 18.0/ado/site/"
  [3]              "."
  [4]  (PERSONAL)  "/Users/floswald/Documents/Stata/ado/personal/"
  [5]  (PLUS)      "/Users/floswald/Library/Application Support/Stata/ado/plus/"
  [6]  (OLDPLACE)  "~/ado/"

You see, if you had installed, say, the package estout some time ago, and the author uses estout in their code, this will just run, even if they did not explicitly tell us that one needs the estout library for this. So: you need to clear out the above list, so that it does not look in your system locations. Here’s one way. just copy and paste:

tokenize `"$S_ADO"', parse(";")
while `"`1'"' != "" {
  if `"`1'"'!="BASE" cap adopath - `"`1'"'
  macro shift
}

you can also manually change this list by typing adopath - "/Users/floswald/Documents/Stata/ado/personal/" for instance. (the - means to remove, the + means to add...)

do adopath again to check that the list is now reduced to base stata locations (you need those). Further, type

which ado

in stata to check that there is no libraries “installed” right now (stata can’t see those installed).

next, you tell stata to look for (and install) any libraries in a dedicated folder inside the replication package. you create a new folder inside their replication package. call it something like adofiles or so. Once you are done, navigate to the replication package with stata:

cd 3-replication-package
adopath ++ "./adofiles/"

This would be enough if the author provides all libraries within their package which is often not the case. Often they tell us to ssc install this and that package. That goes to another location. It’s listed here:

. sysdir list
   STATA:  /Applications/Stata 18.0/
    BASE:  /Applications/Stata 18.0/ado/base/
    SITE:  /Applications/Stata 18.0/ado/site/
    PLUS:  /Users/floswald/Library/Application Support/Stata/ado/plus/
PERSONAL:  /Users/floswald/Documents/Stata/ado/personal/
OLDPLACE:  ~/ado/

we need to swap the entry for the PLUS field. that’s where ssc install will put stuff. So (I created an extra folder statalibs here, you can use the same folder adofiles from above if you want…)

sysdir set PLUS /Users/floswald/statalibs

Then make sure the following is all true. i’m first trying to execute a few commands which I knew were installed in my system (reghdfe) and now don’t work any more. That’s great. now you start to run the code of the authors. It will throw an error each time they are using a command which is not base stata, you add it to your report, you install it with ssc install or whichever way. Finally to illustrate what installing packages does, observe the following:

. sysdir
   STATA:  /Applications/Stata 18.0/
    BASE:  /Applications/Stata 18.0/ado/base/
    SITE:  /Applications/Stata 18.0/ado/site/
    PLUS:  /Users/floswald/statalibs/
PERSONAL:  /Users/floswald/Documents/Stata/ado/personal/
OLDPLACE:  ~/ado/

. reghdfe
command reghdfe is unrecognized
r(199);

. ado

. adopath
  [1]              "statalibs"
  [2]  (BASE)      "/Applications/Stata 18.0/ado/base/"

. ssc install binscatter
checking binscatter consistency and verifying not already installed...
installing into /Users/floswald/statalibs/...
installation complete.

. which binscatter
statalibs/b/binscatter.ado
*! version 7.02  24nov2013  Michael Stepner, [email protected]

. ssc install reghdfe
checking reghdfe consistency and verifying not already installed...
installing into /Users/floswald/statalibs/...
installation complete.

. which reghdfe
statalibs/r/reghdfe.ado
*! version 6.12.3 08aug2023

Matlab

Author code contains parpool(12) (which breaks as soon as the machine does not have at least 12 cores) - can I change it to parpool() which takes as default the number of cores of the machine.

Yes. Add to report.

FAQ about IRB and experimental Documentation

is the IRB approval document the same thing as an ethical clearance certificate provided by the author's institution? I'm looking at the ethical clearance certificate provided by XXXX for EJ but there is no writing about it being from the IRB.

It depends if we can identify the person who signed as equivalent to the IRB officer.

Proprietary Data Formats

is .RData non proprietary data format?

Yes. It's the R internal storage format (similar to .Rds and Rda). In general, stata .dta, matlab .mat are the most common culprits, which should be supplied as .csv or other formats (.Rds and Rda would work!)

If the authors conduct their own experiment to generate data, they don't have to cite that data, correct?

Correct. They have to cite all external data (including data generated in their or other researchers' previous experiments).

Where can I find the uploaded files in nuvolos? in the /files folder.