expfactory package

Submodules

expfactory.battery module

battery.py: part of expfactory package Functions to generate batteries

expfactory.battery.add_custom_variables(custom_variables, template)[source]

add_custom_variables takes a list of tuples and a template, where each tuple is a (“[TAG]”,”substitution”) and the template is an open file with the tag. :param custom_variables: a list of tuples (see description above) :param template: an open file to replace “tag” with “substitute”

expfactory.battery.generate(battery_dest=None, battery_repo=None, experiment_repo=None, experiments=None, config=None, make_config=True, warning=True, time=30)[source]

will create a battery from a template and list of experiments :param battery_dest: is the output folder for your battery. This folder MUST NOT EXIST. If not specified, a temp folder is created :param battery_repo: location of psiturk-battery repo to use as a template. If not specified, will be downloaded to a temporary directory :param experiment_repo: location of a expfactory-experiments repo to check for valid experiments. If not specified, will be downloaded to a temporary directory :param experiments: a list of experiments, meaning the “exp_id” variable in the config.json, to include. This variable also conincides with the experiment folder name. :param config: A dictionary with keys that coincide with parameters in the config.txt file for a expfactory experiment. If not provided, a dummy config will be generated. :param make_config: A boolean (default True) to control generation of the config. If there is a config generated before calling this function, this should be set to False. :param warning: Show config.json warnings when validating experiments. Default is True :param time: maximum amount of time for battery to endure (default 30 minutes) to select experiments

expfactory.battery.generate_base(battery_dest, tasks=None, experiment_repo=None, survey_repo=None, game_repo=None, add_experiments=True, add_surveys=True, add_games=True, battery_repo=None, warning=True)[source]

generate_base returns a folder with downloaded experiments, surveys, and battery, either specified by the user or a temporary directory, to be used by generate_local and generate (for psiturk) :param battery_dest: [required] is the output folder for your battery. This folder MUST NOT EXIST. :param battery_repo: location of psiturk-battery repo to use as a template. If not specified, will be downloaded to a temporary directory :param experiment_repo: location of a expfactory-experiments repo to check for valid experiments. If not specified, will be downloaded to a temporary directory :param survey_repo: location of a expfactory-surveys repo to check for valid surveys. If not specified, will be downloaded to a temporary directory :param tasks: a list of experiments and surveys, meaning the “exp_id” variable in the config.json, to include. This variable also conincides with the tasks folder name. :param warning: show warnings when validating experiments (default True)

expfactory.battery.generate_config(battery_dest, fields)[source]

takes a dictionary, and for matching fields, substitues and prints to “config.txt” in a specified battery directory :param battery_dest: should be the copied, skeleton battery folder in generation :param fields: should be a dictionary with fields that match those in the config non matching fields will be ignored.

expfactory.battery.generate_local(battery_dest=None, subject_id=None, battery_repo=None, experiment_repo=None, experiments=None, warning=True, time=30)[source]

generate_local deploys a local battery will create a battery from a template and list of experiments :param battery_dest: is the output folder for your battery. This folder MUST NOT EXIST. If not specified, a temp directory will be used :param battery_repo: location of psiturk-battery repo to use as a template. If not specified, will be downloaded to a temporary directory :param experiment_repo: location of a expfactory-experiments repo to check for valid experiments. If not specified, will be downloaded to a temporary directory :param experiments: a list of experiments, meaning the “exp_id” variable in the config.json, to include. This variable also conincides with the experiment folder name. :param subject_id: The subject id to embed in the experiment, and the name of the results file. If none is provided, a unique ID will be generated. :param time: Maximum amount of time for battery to endure, to select experiments

expfactory.battery.get_concat_js(valid_experiments)[source]

Return javascript concat section for valid experiments, based on psiturk.json :param valid_experiments: full paths to valid experiments to include

..note:

         case "simple-rt":
                 experiments = experiments.concat(simple-rt_experiment)
                 break;
         case "choice-rt":
                 experiments = experiments.concat(choice-rt_experiment)
                 break;

Format for experiment variables is [exp_id]_experiment
expfactory.battery.get_config()[source]

load in a dummy config file from expfactory

expfactory.battery.get_experiment_run(valid_experiments, deployment='local')[source]

returns a dictionary of experiment run code (right now just jspsych init objects) :param valid_experiments: full path to valid experiments folders, OR a loaded config.json (dict)

expfactory.battery.get_load_js(valid_experiments, url_prefix='')[source]

Return javascript to load list of valid experiments, based on psiturk.json :param valid_experiments: a list of full paths to valid experiments to include

..note::
Format is:
{
case “simple_rt”:
loadjscssfile(“static/css/experiments/simple_rt.css”,”css”) loadjscssfile(“static/js/experiments/simple_rt.js”,”js”) break;
case “choice_rt”:
loadjscssfile(“static/css/experiments/choice_rt.css”,”css”) loadjscssfile(“static/js/experiments/choice_rt.js”,”js”) break;

... }

expfactory.battery.get_load_static(valid_experiments, url_prefix='', unique=True)[source]

return the scripts and styles as <link> and <script> to embed in a page directly :param unique: return only unique scripts [default=True]

expfactory.battery.get_timing_js(valid_experiments)[source]

Produce string (json / dictionary) of experiment timings :param valid_experiments: a list of full paths to valid experiments to include

..note:

Produces the following format for each experiment

{name:"simple_rt", time: 3.5}, {name:"choice_rt", time: 4}, ...
expfactory.battery.move_experiments(valid_experiments, battery_dest, repo_type='experiments')[source]

Moves valid experiments into the experiments folder in battery repo :param valid_experiments: a list of full paths to valid experiments :param battery_dest: full path to battery destination folder :param repo_type: the kind of task to move (default is experiments)

expfactory.battery.template_experiments(battery_dest, battery_repo, valid_experiments, template_load=None, template_exp=None, template_exp_output=None, custom_variables=None)[source]

template_experiments: For each valid experiment, copies the entire folder into the battery destination directory, and generates templates with appropriate paths to run them :param battery_dest: full path to destination folder of battery :param battery_repo: full path to psiturk-battery repo template :param valid_experiments: a list of full paths to experiment folders to include :param template_load: the load_experiments.js template file. If not specified, the file from the battery repo is used. :param template_exp: the exp.html template file that runs load_experiment.js. If not specified, the psiturk file from the battery repo is used. :param template_exp_output: The output file for template_exp. if not specified, the default psiturk templates/exp.html is used :param custom_variables: A dictionary of custom variables to add to templates. Keys should either be “exp” or “load”, and values should be tuples with the first index the thing to sub (eg, [SUB_THIS_SUB]) and the second the substitition to make.

expfactory.experiment module

experiment.py: part of expfactory package Functions to work with javascript experiments

expfactory.experiment.check_acceptable_variables(experiment_name, field_dict, template, field_dict_key)[source]

check_acceptable_variables takes a field (eg, meta[0][field]) that has a dictionary, and some template key (eg, jspsych) and makes sure the keys of the dictionary are within the allowable for the template type (the key). :param experiment_name: the name of the experiment :param field_dict: the field value from the config.json, a dictionary :param field_dict_key: a key to look up in the field_dict, which should contain a dictionary of {“key”:”value”} variables :param template: the key name, for looking up acceptable values using get_acceptable_values

expfactory.experiment.check_boolean(experiment_name, value, variable_name)[source]

check_boolean checks if a value is boolean :param experiment_name: the name of the experiment :param value: the value to check :param variable_name: the name of the variable (the key being indexed in the dictionary)

expfactory.experiment.dowarning(reason)[source]
expfactory.experiment.find_changed(new_repo, comparison_repo, return_experiments=True, repo_type='experiments')[source]

find_changed returns a list of changed files or experiments between two repos :param new_repo: the updated repo - any new files, or changed files, will be returned :param comparison_repo: the old repo to compare against. A file changed or missing in this repo in the new_repo indicates it should be tested :param return_experiments: return experiment folders. Default is True. If False, will return complete file list

expfactory.experiment.get_acceptable_values(package_name)[source]
expfactory.experiment.get_experiments(experiment_repo, load=False, warning=True, repo_type='experiments')[source]

return loaded json for all valid experiments from an experiment folder :param experiment_repo: full path to the experiments repo :param load: if True, returns a list of loaded config.json objects. If False (default) returns the paths to the experiments :param repo_type: tells the user what kind of task is being parsed, default is “experiments,” but can also be “surveys” when called by get_surveys

expfactory.experiment.get_valid_templates()[source]
expfactory.experiment.get_validation_fields()[source]

Returns a list of tuples (each a field)

..note:

specifies fields required for a valid json
(field,value,type)
field: the field name
value: indicates minimum required entires
       0: not required, no warning
       1: required, not valid
       2: not required, warning      
type: indicates the variable type
expfactory.experiment.load_experiment(experiment_folder)[source]

load_experiment: reads in the config.json for an :param experiment folder: full path to experiment folder

expfactory.experiment.load_experiments(experiment_folders)[source]

a wrapper for load_experiment to read multiple experiments :param experiment_folders: a list of experiment folders to load, full paths

expfactory.experiment.make_lookup(experiment_list, key_field)[source]

returns dict object to quickly look up query experiment on exp_id :param experiment_list: a list of query (dict objects) :param key_field: the key in the dictionary to base the lookup key (str) :returns lookup: dict (json) with key as “key_field” from query_list

expfactory.experiment.notvalid(reason)[source]
expfactory.experiment.validate(experiment_folder=None, warning=True)[source]
Parameters:
  • experiment_folder – full path to experiment folder with config.json
  • warning – issue a warning for empty fields with level 2 (warning)

..note:

takes an experiment folder, and looks for validation based on:

- config.json
- files existing specified in config.json

All fields should be defined, but for now we just care about run scripts

expfactory.interface module

class expfactory.interface.EFServer(*args, **kwargs)[source]

Bases: flask.app.Flask

expfactory.interface.allowed_file(filename)[source]
class expfactory.interface.apiExperimentSingle[source]

Bases: flask_restful.Resource

return complete meta data for specific experiment :param exp_id: exp_id for experiment to preview

endpoint = 'apiexperimentsingle'
get(exp_id)[source]
mediatypes(resource_cls)
methods = ['GET']
class expfactory.interface.apiExperiments[source]

Bases: flask_restful.Resource

Main view for REST API to display all available experiments

endpoint = 'apiexperiments'
get()[source]
mediatypes(resource_cls)
methods = ['GET']
expfactory.interface.battery()[source]
expfactory.interface.clean_up(dirpath)[source]
expfactory.interface.get_field(request, fields, value)[source]

Get value from a form field

expfactory.interface.home()[source]
expfactory.interface.select()[source]
expfactory.interface.start(port=8088)[source]
expfactory.interface.validate()[source]

expfactory.utils module

utils.py: part of expfactory package

expfactory.utils.clean_fields(mydict)[source]

Ensure utf-8

expfactory.utils.copy_directory(src, dest)[source]

Copy an entire directory recursively

expfactory.utils.find_directories(root, fullpath=True)[source]

Return directories at one level specified by user (not recursive)

expfactory.utils.find_subdirectories(basepath)[source]

Return directories (and sub) starting from a base

expfactory.utils.get_installdir()[source]
expfactory.utils.get_template(template_file)[source]

get_template: read in and return a template file

expfactory.utils.get_url(url)[source]

return a url as string

expfactory.utils.is_type(var, types=[<type 'int'>, <type 'float'>, <type 'list'>])[source]

Check type

expfactory.utils.remove_unicode_dict(input_dict)[source]

remove unicode keys and values from dict, encoding in utf8

expfactory.utils.save_pretty_json(outfile, myjson)[source]
expfactory.utils.save_template(output_file, html_snippet)[source]
expfactory.utils.sub_template(template, template_tag, substitution)[source]

make a substitution for a template_tag in a template

expfactory.scripts module

script.py: part of expfactory api Runtime executable

expfactory.scripts.main()[source]

expfactory.views module

views.py

part of the experiment factory package functions for developing experiments and batteries, viewing and testing things

expfactory.views.choice(a, size=None, replace=True, p=None)

Generates a random sample from a given 1-D array

New in version 1.7.0.

a : 1-D array-like or int
If an ndarray, a random sample is generated from its elements. If an int, the random sample is generated as if a was np.arange(n)
size : int or tuple of ints, optional
Output shape. Default is None, in which case a single value is returned.
replace : boolean, optional
Whether the sample is with or without replacement
p : 1-D array-like, optional
The probabilities associated with each entry in a. If not given the sample assumes a uniform distribtion over all entries in a.
samples : 1-D ndarray, shape (size,)
The generated random samples
ValueError
If a is an int and less than zero, if a or p are not 1-dimensional, if a is an array-like of size 0, if p is not a vector of probabilities, if a and p have different lengths, or if replace=False and the sample size is greater than the population size

randint, shuffle, permutation

Generate a uniform random sample from np.arange(5) of size 3:

>>> np.random.choice(5, 3)
array([0, 3, 4])
>>> #This is equivalent to np.random.randint(0,5,3)

Generate a non-uniform random sample from np.arange(5) of size 3:

>>> np.random.choice(5, 3, p=[0.1, 0, 0.3, 0.6, 0])
array([3, 3, 0])

Generate a uniform random sample from np.arange(5) of size 3 without replacement:

>>> np.random.choice(5, 3, replace=False)
array([3,1,0])
>>> #This is equivalent to np.random.shuffle(np.arange(5))[:3]

Generate a non-uniform random sample from np.arange(5) of size 3 without replacement:

>>> np.random.choice(5, 3, replace=False, p=[0.1, 0, 0.3, 0.6, 0])
array([2, 3, 0])

Any of the above can be repeated with an arbitrary array-like instead of just integers. For instance:

>>> aa_milne_arr = ['pooh', 'rabbit', 'piglet', 'Christopher']
>>> np.random.choice(aa_milne_arr, 5, p=[0.5, 0.1, 0.1, 0.3])
array(['pooh', 'pooh', 'pooh', 'Christopher', 'piglet'],
      dtype='|S11')
expfactory.views.embed_experiment(folder, url_prefix='')[source]

return an html snippet for embedding into an application. This assumes the same directory structure, that all jspsych files can be found in static/js/jspych, and experiments under static/experiments/[folder] :param folder: full path to experiment folder with config.json

expfactory.views.generate_experiment_web(output_dir, experiment_folder=None, survey_folder=None, games_folder=None, make_table=True, make_index=True, make_experiments=True, make_data=True, make_surveys=True, make_games=True)[source]

get_experiment_table Generate a table with links to preview all experiments :param experiment_folder: folder with experiments inside :param survey_folder: folder with surveys inside :param games_folder: folder with games inside :param output_dir: output folder for experiment and table html, and battery files :param make_table: generate table.html :param make_index: generate d3 visualization index :param make_experiments: generate experiment preview files (linked from table and index) :param make_data: generate json/tsv data to download :param make_surveys: generate static files for surveys repos :param make_games: generate static files for games repos

expfactory.views.get_cognitiveatlas_hierarchy(experiment_tags=None, get_html=False)[source]

return :param experiment_tags: a list of experiment exp_id tags to include. If None provided, all valid experiments will be used. :param get_html: if True, returns rendered HTML template with hierarchy. False returns json data structure.

expfactory.views.get_experiment_html(experiment, experiment_folder, url_prefix='', deployment='local')[source]

return the html template to test a single experiment :param experiment: the loaded config.json for an experiment (json) :param experiment_folder: the experiment folder, needed for reading in a survey :param url_prefix: prefix to put before paths, in case of custom deployment :param deployment: deployment environment, one of docker, docker-preview, or local [default]

expfactory.views.preview_experiment(folder=None, battery_folder=None, port=None)[source]

preview an experiment locally with the –preview tag (for development) :param folder: full path to experiment folder to preview. If none specified, PWD is used :param battery_folder: full path to battery folder to use as a template. If none specified, the expfactory-battery repo will be used. :param port: the port number, default will be randomly generated between 8000 and 9999 :param robot: if True, a web server is started as a separate process for a robot to run

expfactory.views.run_battery(destination=None, experiments=None, experiment_folder=None, subject_id=None, battery_folder=None, port=None, time=30)[source]

run_battery runs or previews an entire battery locally with the –run tag. If no experiments are provided, all in the folder will be used. :param destination: destination folder for battery. If none provided, tmp directory is used :param experiments: list of experiment tags to add to battery :param experiment_folder: the folder of experiments to deploy the battery from. :param subject_id: subject id to embed into battery. If none, will be randomly generated :param battery_folder: full path to battery folder to use as a template. If none specified, the expfactory-battery repo will be used. :param port: the port number, default will be randomly generated between 8000 and 9999 :param time: total number of minutes for experiments to add to battery

expfactory.views.run_single(exp_id, repo_type, destination=None, source_repo=None, battery_repo=None, port=None, subject_id=None)[source]

run_survey runs or previews an entire battery locally with the –run tag. If no experiments are provided, all in the folder will be used. :param destination: destination folder for battery. If none provided, tmp directory is used :param exp_id: exp_id for survey, experiment, or game to run (unique ID) :param repo_type: must be within experiments,surveys, or games :param source_repo: a source repo for the experiment, survey, or game :param subject_id: subject id to embed into result. If none, will be randomly generated :param battery_folder: full path to battery folder to use as a template. If none specified, the expfactory-battery repo will be used. :param port: the port number, default will be randomly generated between 8000 and 9999

expfactory.views.tmp_experiment(folder=None, battery_folder=None)[source]

generate temporary directory with experiment :param folder: full path to experiment folder to preview (experiment, survey, or game). If none specified, PWD is used :param battery_folder: full path to battery folder to use as a template. If none specified, the expfactory-battery repo will be used.

expfactory.vm module

vm.py: part of expfactory package Functions to generate virtual machines to run expfactory batteries

Add a custom logo to the vm battery :param battery_repo: the full path to the battery repo base, assumed to have an “img” folder :param logo: the full path to the logo to copy. Should ideally be png.

expfactory.vm.custom_battery_download(tmpdir=None, repos=['experiments', 'battery', 'surveys', 'games'])[source]

Download battery and experiment repos to a temporary folder to build a custom battery, return the path to the tmp folders :param tmpdir: The directory to download to. If none, a temporary directory will be made :param repos: The repositories to download, valid choices are “experiments” “battery” and “vm”

expfactory.vm.download_repo(repo_type, destination)[source]

Download a expfactory infrastructure repo “repo_type” to a “destination” :param repo_type: can be one of “experiments” “battery” “vm” :param destination: the full path to the destination for the repo

expfactory.vm.generate_database_url(dbtype=None, username=None, password=None, host=None, table=None, template=None)[source]

Generate a database url from input parameters, or get a template :param dbtype: the type of database, must be one of “mysql” or “postgresql” :param username: username to connect to the database :param password: password to connect to the database :param host: database host :para table: table in the database :param template: can be one of “mysql” “sqlite3” or “postgresql” If specified, all other parameters are ignored, and a default database URL is produced to work in a Vagrant VM produced by expfactory

expfactory.vm.get_jspsych_init(experiment, deployment='local', finished_message=None)[source]

return entire jspsych init structure :param experiment: the loaded config.json for the experiment :param deployment: specify to deploy local (default), or docker-mturk,docker-local (expfactory-docker). :param finished_message: custom message to show at the end of the experiment with Redo and Next Experiment Buttons

expfactory.vm.get_stylejs(experiment, url_prefix='')[source]

return string for js and css scripts to insert into a page based on battery path structure

expfactory.vm.prepare_vm(battery_dest, fields=None, vm_repo=None, vm_type='vagrant')[source]

Prepare virtual machine to run local with vagrant, or with vagrant-aws :param battery_dest: the battery destination folder :param fields: if specified, will be added to the config.txt :param vm_repo: the expfactory-vm repo to use for templates. If not provided, one will be downloaded to a temporary directory for use. :param vm_type: one of “vagrant” or “aws” Default is “vagrant” meaning a localvirtual machine with vagrant.

expfactory.vm.specify_experiments(battery_dest, experiments)[source]

Specify experiments for a Vagrantfile in an output folder :param battery_dest: destination folder for battery :param experiments: a list of experiment tags to include

Module contents