Create User Account¶

1. In the top right corner of each page, click on the Sign Up link.
2. Fill out all the fields, then click the Create Account button at the end. Please note that the Username field does not support email addresses but will allow the following characters: a-Z, 0-9, _ (underscores), - (hyphens), and . (periods). Congrats you now have a Dataverse account!

Edit Your Account¶

1. To edit your account after you have logged in, click on your account name in the header on the right hand side and click on Account Information.
2. On the top right of your account page, click on the “Edit Account” button and from there you can select to edit either your Account Information or your Account Password.
3. Select “Save Changes” when you are done.

Generate Your API Token¶

1. To generate your API token, click on your name in the hearder on right hand side and then click on Account Information.
2. On the top right of your account page, click on the “Edit Account” button and click on API Token in the list.
3. Your API Token is located on that page.

My Data¶

The My Data section of your account page displays a listing of all the dataverses, datasets, and files you have either created, uploaded or that you have access to edit. You are able to filter through all the dataverses, datasets, and files listed there using the filter box or use the facets on the left side to only view a specific Publication Status or Role.

Notifications: Setup & Maintainance¶

Notifications appear in the notifications tab on your account page and are also displayed as a number next to your account name.

You will receive a notification when:

• You’ve created your account
• You’ve created a dataverse or added a dataset
• Another Dataverse user has requested access to a restricted file in one of your datasets

Dataverse will email your unread notifications once a day. Notifications will only be emailed one time even if you haven’t read the notification on the Dataverse site.

Reset Your Account Password¶

If you cannot remember the password for your Dataverse account, click on Log In in the top right corner of any page. Once on that page, click on the Forgot Password? link below where you would enter your username and password. Enter your email address and click Submit Password Request to receive an email with a link to reset your password.

*Note: if you have forgotten your username, you can do this same process to receive your username in an email.

Finding Data¶

Without logging in to Dataverse, users can browse Dataverse, search for dataverses, datasets, and files, view dataset descriptions and files for published datasets, and subset, analyze and visualize data for published (restricted & not restricted) data files. To view a unpublished dataverse, dataset, or file, a user will need to be given permission from that dataverse’s administrator to access it.

A user can search the dataverses, datasets, and files within a particular dataverse by using the search bar found on a dataverse page. For example, if you are on the Murray Research Archive Dataverse page, you can search that specific dataverse’s contents by using the search bar and/or facets displayed on the page.

Using Data¶

Create a Dataverse (within the “root” Dataverse)¶

Creating a dataverse is easy but first you must be a registered user (see Create Account).

1. Once you are logged in click on the “Add Data” button and in the dropdown menu select “New Dataverse”.

2. Once on the “New Dataverse” page fill in the following fields:

   • Name: Enter the name of your dataverse.
   • Identifier: This is an abbreviation, usually lower-case, that becomes part of the URL for the new dataverse. Special characters (~,`, !, @, #, $, %, ^, &, and *) and spaces are not allowed. Note: if you change the Dataverse URL field, the URL for your Dataverse changes (http//.../’url’), which affects links to this page.
   • Email: This is the email address that will be used as the contact for this particular dataverse. You can have more than one contact email address for your dataverse.
   • Affiliation: Add any Affiliation that can be associated to this particular dataverse (e.g., project name, institute name, department name, journal name, etc). This is automatically filled out if you have added an affiliation for your user account.
   • Description: Provide a description of this dataverse. This will display on the home page of your dataverse and in the search result list. The description field supports certain HTML tags (<a>, <b>, <blockquote>, <br>, <code>, <del>, <dd>, <dl>, <dt>, <em>, <hr>, <h1>-<h3>, <i>, <img>, <kbd>, <li>, <ol>, <p>, <pre>, <s>, <sup>, <sub>, <strong>, <strike>, <ul>).
   • Category: Select a category that best describes the type of dataverse this will be. For example, if this is a dataverse for an individual researcher’s datasets, select Researcher. If this is a dataverse for an institution, select Organization & Institution.
   • Choose the sets of Metadata Elements for datasets in this dataverse: by default the metadata elements will be from the host dataverse that this new dataverse is created in. Dataverse offers metadata standards for multiple domains. To learn more about the metadata standards in Dataverse please check out the appendix (insert link here)
   • Select facets for this dataverse: by default the facets that will appear on your dataverse landing page will be from the host dataverse that this new dataverse was created in. The facets are simply metadata fields that can be used to help others easily find dataverses and datasets within this dataverse. You can select as many facets as you would like.

3. Selected metadata elements are also used to pick which metadata fields you would like to use for creating templates for your datasets. Metadata fields can be hidden, or selected as required or optional. Once you have selected all the fields you would like to use, you can create your template(s) after you finish creating your dataverse.

4. Click “Create Dataverse” button and you’re done!

*Required fields are denoted by a red asterisk.

Edit Dataverse¶

To edit your dataverse, navigate to your dataverse homepage and select the “Edit Dataverse” button, where you will be presented with the following editing options:

• General Information : edit name, identifier, category, contact email, affiliation, description, Metadata Elements, and facets for your dataverse.
• Theme + Widgets : upload a logo for your dataverse, add a link to your department or personal website, and select colors for your dataverse in order to brand it. Also, you can get code to add to your website to have your dataverse display on it.
• Permissions : give Dataverse users permissions to your dataverse, i.e.-can edit datasets, and see which users already have which permissions for your dataverse
• Dataset Templates : these are useful when you have several datasets that have the same information in multiple metadata fields that you would prefer not to have to keep manually typing in
• Dataset Guestbooks : allows you to collect data about who is downloading the files from your datasets
• Featured Dataverses : if you have one or more dataverses, you can use this option to show them at the top of your dataverse page to help others easily find interesting or important dataverses
• Delete Dataverse: you are able to delete your dataverse as long as it is not published and does not have any draft datasets

General Information¶

The General Information page is how you edit the information you filled in while creating your dataverse. If you need to change or add a contact email address, this is the place to do it. Additionally, you can update the metadata elements used for datasets within the dataverse, change which metadafields are hidden, required, or optional, and update the facets you would like displayed for browsing the dataverse. If you plan on using templates, you need to select the metadata fields on the General Information page.

Tip: The metadata fields you select as required, will appear on the Create Dataset form when someone goes to add a dataset to the dataverse.

Theme + Widgets¶

The Theme + Widgets feature provides you with a way to customize the look of your dataverse as well as provide code for you to put on your personal website to have your dataverse appear there. (For adding your dataverse to an OpenScholar site, follow these instructions.)

For Theme, you can decide either to use the customization from the dataverse above yours or upload your own image file. Supported image types are jpeg, tiff, or png and should be no larger than 500kb. The maximum display for an image file in a dataverse’s theme is 940 piels wide by 120 pixels high. Additionally, you can select the colors for the header of your dataverse and the text that appears in your dataverse. You can also add a link to your personal website, the website for your organization or institution, your department, journal, etc.

There are two options for Widgets, a Dataverse Search box widget and a Dataverse Listing widget. The Dataverse Search Box will add a search box to your website that when someone enters a search term in and clicks Find, will bring them to Dataverse to see the results. The Dataverse Listing widget will provide a listing of all your dataverses and datasets. When someone clicks on a dataverse or dataset in the widget, it will bring them to your dataverse to see the actual data. Within the Widgets page, you can copy and paste the code for the widget you would like to have on your website.

Adding Widgets to an OpenScholar Website¶

1. Log in to your OpenScholar website
2. Either build a new page or navigate to the page you would like to use to show the Dataverse widgets.
3. Click on the Settings Cog and select Layout
4. At the top right, select Add New Widget and under Misc. you will see the Dataverse Search Box and the Dataverse Listing widgets. Click on the widget you would like to add (we recommend using both), fill out the form, and then drag it to where you would like it to display in the page.

Permissions¶

When you access a dataverse’s permissions page, you will see there are three sections: Permissions, Users/Groups, and Roles.

Clicking on Permissions will bring you to this page: By clicking on the Edit Access button, you are able to change the settings allowing no one or anyone to add either dataverses or datasets to a dataverse. The Edit Access pop up allows you to also select if someone adding a dataset to this dataverse should be allowed to publish it (Curator role) or if the dataset will be submitted to the administrator of this dataverse to be reviewed then published (Contributor role). These Access settings can be changed at any time.

Dataset Templates¶

Templates are useful when you have several datasets that have the same information in multiple metadata fields that you would prefer not to have to keep manually typing in or want to use a custom set of Terms of Use and Access for multiple datasets in a dataverse. In Dataverse 4.0, templates are created at the dataverse level, can be deleted (so it does not show for future datasets), set to default (not required), or can be copied so you do not have to start over when creating a new template with similiar metadata from another template. When a template is deleted, it does not impact the datasets that have used the template already.

How do you create a template?

1. Navigate to your dataverse, click on the Edit Dataverse button and select Dataset Templates.
2. Once you have clicked on Dataset Templates, you will be brought to the Dataset Templates page. On this page, you can 1) decide to use the dataset templates from your parent dataverse 2) create a new dataset template or 3) do both.
3. Click on the Create Dataset Template to get started. You will see that the template is the same as the create dataset page with an additional field at the top of the page to add a name for the template.
4. After adding information into the metadata fields you have information for and clicking Save and Add Terms, you will be brought to the page where you can add custom Terms of Use and Access. If you do not need custom Terms of Use and Access, click the Save Dataset Template, and only the metadata fields will be saved.
5. After clicking Save Dataset Template, you will be brought back to the Manage Dataset Templates page and should see your template listed there now with the make default, edit, view, or delete options.
6. A dataverse does not have to have a default template and users can select which template they would like to use while on the Create Dataset page.
7. You can also click on the View button on the Manage Dataset Templates page to see what metadata fields have information filled in.

* Please note that the ability to choose which metadata fields are hidden, required, or optional is done on the General Information page for the dataverse.

Dataset Guestbooks¶

Guestbooks allow you to collect data about who is downloading the files from your datasets. You can decide to collect account information (username, given name & last name, affiliation, etc.) as well as create custom questions (e.g., What do you plan to use this data for?). You are also able to download the data collected from the enabled guestbooks as Excel files to store and use outside of Dataverse.

How do you create a guestbook?

1. After creating a dataverse, click on the Edit Dataverse button and select Dataset Guestbook
2. By default, guestbooks created in the dataverse your dataverse is in, will appear. If you do not want to use or see those guestbooks, uncheck the checkbox that says Include Guestbooks from Root Dataverse.
3. To create a new guestbook, click the Create Dataset Guestbook button on the rightside of the page.
4. Name the guestbook, determine the account information that you would like to be required (all account information fields show when someone downloads a file), and then add Custom Questions (can be required or not required).
5. Hit the Create Dataset Guestbook button once you have finished.

What can you do with a guestbook? After creating a guestbook, you will notice there are several options for a guestbook and appear in the list of guestbooks.

• If you want to use a guestbook you have created, you will first need to click the button in the Action column that says Enable. Once a guestbook has been enabled, you can go to the License + Terms for a dataset and select a guestbook for it.
• There are also options to view, copy, edit, or delete a guestbook.
• Once someone has downloaded a file in a dataset where a guestbook has been assigned, an option to download collected data will appear.

Featured Dataverses¶

Featured Dataverses is a way to display sub dataverses in your dataverse that you want to feature for people to easily see when they visit your dataverse.

Click on Featured Dataverses and a pop up will appear. Select which sub dataverses you would like to have appear.

Note: Featured Dataverses can only be used with published dataverses.

Linked Dataverses + Linked Datasets¶

Currently, the ability to link a dataverse to another dataverse or a dataset to a dataverse is a super user only feature.

If you need to have a dataverse or dataset linked in the Harvard Dataverse installation, please contact support@dataverse.org.

Publish Your Dataverse¶

Once your dataverse is ready to go public, go to your dataverse page, click on the “Publish” button on the right hand side of the page. A pop-up will appear to confirm that you are ready to actually Publish, since once a dataverse is made public, it can no longer be unpublished.

Supported Metadata¶

A dataset contains three levels of metadata:

1. Citation Metadata: any metadata that would be needed for generating a data citation and other general metadata that could be applied to any dataset;
2. Domain specific Metadata: with specific support currently for Social Science, Life Science, Geospatial, and Astronomy datasets; and
3. File-level Metadata: varies depending on the type of data file - see File Handling and Uploading section below for more details).

For more details about what Citation and Domain specific metadata is supported please see our Appendix.

File Handling + Uploading¶

All file formats are supported, up to 2GB per file for the Harvard Dataverse. Please contact support@dataverse.org if you need to upload a file that is larger than 2GB. You can also add descriptions and categorize each of them by adding tags.

The file types listed below are supported by additional functionality, which can include downloading in different formats, subsets, file-level metadata preservation, file-level data citation; and exploration through data visualization and analysis.

Adding a New Dataset¶

1. Navigate to the dataverse in which you want to add a dataset.
2. Click on the “Add Data” button and select “New Dataset” in the dropdown menu.
3. To quickly get started, enter at minimum all the required fields with an asterisk to get a Data Citation with a DOI (e.g., the Dataset Title, Author, Description, Contact Email and Subject).
4. Scroll down to the “Files” section and click on “Select Files to Add” to add all the relevant files to your Dataset. You can also upload your files directly from your Dropbox. Tip: You can drag and drop or select multiple files at a time from your desktop, directly into the upload widget. Your files will appear below the “Select Files to Add” button where you can add a description and tags (via the “Edit Tag” button) for each file. Additionally, an MD5 checksum will be added for each file. If you upload a tabular file a Universal Numerical Fingerprint (UNF) will be added to this file.
5. Click the “Save Dataset” button when you are done. Your unpublished dataset is now created.

Note 1: You can add additional metadata once you have completed the initial dataset creation by going to Edit Dataset > Metadata.

Edit Dataset¶

Go to the dataset you would like to edit where you will see the listing of files. Select the files you would like to edit by using either the Select All checkbox or individually selecting files. Next, click on the Edit button above the files and select if you would like to:

• Delete the selected files
• Edit the file metadata (file name, description) for the selected files
• Restrict the selected files
• Unrestrict the selected files (only if the selected files are restricted)
• Add tags to the selected files

All of these actions, besides editing file metadata, will happen within this page and not bring you to another page. If you restrict files, you will also be asked to fill out the Terms of Access for the files. If Terms of Access already exist, you will be asked to confirm them.

Upload New Files¶

To upload new files to a dataset, go the dataset you want to update and click on the Upload Files Button in the files tab. From there you will be brought to the Upload page for the dataset. Once you have uploaded files, you will be able to edit the file metadata, restrict, add tags, or delete them before saving.

Terms¶

In the Terms tab, which can also be found by clicking on the Edit dropdown button of a Dataset, you can setup how users can use your data once they have downloaded it (CC0 waiver or custom Terms of Use), how they can access your data if you have files that are restricted (terms of access), and enable a Guestbook for your dataset so that you can track who is using your data and for what purposes. These are explained in further detail below:

Permissions¶

Publish Dataset¶

When you publish a dataset (available to an Admin, Curator, or any custom role which has this level of permission assigned), you make it available to the public so that other users can browse or search for it. Once your dataset is ready to go public, go to your dataset page and click on the “Publish” button on the right hand side of the page. A pop-up will appear to confirm that you are ready to actually Publish since once a dataset is made public it can no longer be unpublished.

Whenever you edit your dataset, you are able to publish a new version of the dataset. The publish dataset button will reappear whenever you edit the metadata of the dataset or add a file.

Note: Prior to publishing your dataset the Data Citation will indicate that this is a draft but the “DRAFT VERSION” text will be removed as soon as you Publish.

Submit for Review¶

If you have a Contributor role (can edit metadata, upload files, and edit files, edit Terms, Guestbook, and Submit datasets for review) in a Dataverse you can submit your dataset for review when you have finished uploading your files and filling in all of the relevant metadata fields. To Submit for Review, go to your dataset and click on the “Submit for Review” button, which is located next to the “Edit” button on the upper-right. Once Submitted for Review: the Admin or Curator for this Dataverse will be notified to review this dataset before they decide to either “Publish” the dataset or “Return to Author”. If the dataset is published the contributor will be notified that it is now published. If the dataset is returned to the author, the contributor of this dataset will be notified that they need to make modifications before it can be submitted for review again.

Dataset Versioning¶

Versioning is important for long term-research data management where metadata and/or files are updated over time.

Once you have published a dataset, any metadata or file changes (e.g, by uploading a new file, changing file metadata, adding or editing metadata) will be tracked in our versioning feature. For example if you were at version 1 of your dataset, and you edit your dataset a new draft version of this dataset will be created. To get to the already published version 1 of your dataset, click on the “View Dataset Versions” button on the top left section of your dataset. To go back to the unpublished version click on the same button. Once you are ready to publish this new version of your dataset, select the “Publish Dataset” button on the top right side of the page. If you were at version 1 of your dataset, and depending on the types of changes you have made, you will be asked to select to publish your draft as either version 1.1 or version 2.0 (important note: if you add a file, your dataset will automatically be bumped up to a major version (example: if you were at 1.0 you will go to 2.0).

Dataset Versions Tab

To view what has exactly changed starting from the originally published version to any subsequent published versions: click on the Versions tab on the dataset page to see all versions and changes made for that particular dataset. Once you have more than one version (can be version 1 and a draft), you can click the Show Details link in the Versions tab to learn more about the metadata fields and files that were either added or edited.

If you have more than two versions of a dataset, you can select any two versions to compare the differences between them. After selecting two versions, click on the “Show Differences” button to see the version differences details.

Deaccession Your Dataset [not recommended]¶

Deaccessioning a dataset or a version of a dataset is a very serious action that should only occur if there is a legal or valid reason for the dataset to no longer be accessible to the public. If you absolutely must deaccession, you can deaccession a version of a dataset or an entire dataset. To deaccession, go to a dataset you’ve already published (or add a new one and publish it), click on Edit Dataset, then Deaccession Dataset. If you have multiple versions of a dataset, you can select here which versions you want to deaccession or choose to deaccession the entire dataset. You must also include a reason as to why this dataset was deaccessioned from a dropdown list of options. There is also a free-text box to add more details as to why this was deaccessioned. If the dataset has moved to a different repository or site you are encouraged to include a URL (preferably persistent) for users to continue to be able to access this dataset in the future.

If you deaccession the most recently published version of the dataset but not all versions of the dataset, you are able to go in and create a new draft for the dataset. For example, you have a version 1 and version 2 of a dataset, both published, and deaccession version 2. You are then able to edit version 1 of the dataset and a new draft will be created.

Important Note: A tombstone landing page with the basic citation metadata will always be accessible to the public if they use the persistent URL (Handle or DOI) provided in the citation for that dataset. Users will not be able to see any of the files or additional metadata that were previously available prior to deaccession.

Metadata References¶

Dataverse is committed to using standard-compliant metadata to ensure that Dataverse metadata can be mapped easily to standard metadata schemas and be exported into JSON format (XML for tabular file metadata) for preservation and interoperability.

Detailed below are what metadata schemas we support for Citation and Domain Specific Metadata in Dataverse:

• Citation Metadata: compliant with DDI Lite, DDI 2.5 Codebook, DataCite 3.1, and Dublin Core’s DCMI Metadata Terms (see .tsv version). Language field uses ISO 639-2 controlled vocabulary.
• Geospatial Metadata: compliant with DDI Lite, DDI 2.5 Codebook, DataCite, and Dublin Core (see .tsv version). Country / Nation field uses ISO 3166-1 controlled vocabulary.
• Social Science & Humanities Metadata: compliant with DDI Lite, DDI 2.5 Codebook, and Dublin Core (see .tsv version).
• Astronomy and Astrophysics Metadata : These metadata elements can be mapped/exported to the International Virtual Observatory Alliance’s (IVOA) VOResource Schema format and is based on Virtual Observatory (VO) Discovery and Provenance Metadata (see .tsv version).
• Life Sciences Metadata: based on ISA-Tab Specification, along with controlled vocabulary from subsets of the OBI Ontology and the NCBI Taxonomy for Organisms (see .tsv version).

Java¶

Oracle JDK or OpenJDK 1.7.x. Use the latest available. MacOS X comes with the JDK (but make sure you keep it updated). On a RedHat and similar Linux distributions, install it with something like

$ yum install java-1.7.0-openjdk-devel

Glassfish¶

Glassfish Version 4.1 is required.

Important: once Glassfish is installed, a new version of the WELD library (v2.2.10.SP1) must be downloaded and installed. This fixes a serious issue in the library supplied with Glassfish 4.1.

• Download and install Glassfish (installed in /usr/local/glassfish4 in the example commands below):

  $ wget http://dlc-cdn.sun.com/glassfish/4.1/release/glassfish-4.1.zip
  $ unzip glassfish-4.1.zip
  $ mv glassfish4 /usr/local
  

• Remove the stock WELD jar; download WELD v2.2.10.SP1 and install it in the modules folder:

  $ cd /usr/local/glassfish4/glassfish/modules
  $ /bin/rm weld-osgi-bundle.jar
  $ wget http://central.maven.org/maven2/org/jboss/weld/weld-osgi-bundle/2.2.10.SP1/weld-osgi-bundle-2.2.10.SP1-glassfish4.jar
  $ /usr/local/glassfish4/bin/asadmin start-domain domain1
  

• Verify Weld version:

  $ /usr/local/glassfish4/bin/asadmin osgi lb | grep 'Weld OSGi Bundle'
  

PostgreSQL¶

$ wget http://yum.postgresql.org/9.3/redhat/rhel-6-x86_64/pgdg-centos93-9.3-1.noarch.rpm
rpm -ivh pgdg-centos93-9.3-1.noarch.rpm

$ yum install postgresql93-server.x86_64
$ chkconfig postgresql-9.3 on
$ service postgresql-9.3 initdb
$ service postgresql-9.3 start

Solr¶

• Download and Install Solr::

  $ wget https://archive.apache.org/dist/lucene/solr/4.6.0/solr-4.6.0.tgz $ tar xvzf solr-4.6.0.tgz $ rsync -auv solr-4.6.0 /usr/local/ $ cd /usr/local/solr-4.6.0/example/solr/collection1/conf/ $ mv schema.xml schema.xml.backup $ wget -q –no-check-certificate https://github.com/IQSS/dataverse/raw/master/conf/solr/4.6.0/schema.xml

  In order to start Solr, you will need a customized schema file that is supplied in the Dataverse distribution bundle.

Start Up Scripts¶

• Example of Glassfish Startup file:

  set -e
  ASADMIN=/usr/local/glassfish4/bin/asadmin
  case "$1" in
  start)
          echo -n "Starting GlassFish server: glassfish"
          # Increase file descriptor limit:
          ulimit -n 32768
          # Allow "memory overcommit":
          # (basically, this allows to run exec() calls from inside the
          # app, without the Unix fork() call physically hogging 2X
          # the amount of memory glassfish is already using)
          echo 1 > /proc/sys/vm/overcommit_memory
  
          # Set UTF8 as the default encoding:
          LANG=en_US.UTF-8; export LANG
          $ASADMIN start-domain domain1
          echo "."
          ;;
            stop)
          echo -n "Stopping GlassFish server: glassfish"
  
          $ASADMIN stop-domain domain1
          echo "."
          ;;
  
            *)
          echo "Usage: /etc/init.d/glassfish {start|stop}"
          exit 1
          esac
  exit 0
  

Download and run the installer¶

Download the installer package (dvnstall.zip). Unpack the zip file - this will create the directory dvinstall. Execuite the installer script (install):

cd dvinstall

./install

The script will prompt you for some configuration values. If this is a test/evaluation installation, it should be safe to accept nthe defaults for most of the settiongs. For for a developer’s installation we recommend that you choose localhost for the host name.

The script is to a large degree a derivative of the old installer from DVN 3.x. It is written in Perl.

All the Glassfish configuration tasks performed by the installer are isolated in the shell script scripts/install/glassfish-setup.sh (as asadmin commands).

Dataverse Admin Account¶

Now that you’ve run the application installer and have your own Dataverse instance, you need to configure the Dataverse Administrator user. By default installer pre-sets the Admin credentials as follows:

First Name: Dataverse
Last Name:  Admin
Affiliation: Dataverse.org
Position: Admin
Email: dataverse@mailinator.com

Log in as the user dataverseAdmin with the password “admin” and change these values to suit your installation.

(Alteratively, you can modify the file dvinstall/data/user-admin.json in the installer bundle before you run the installer. The password is in dvinstall/setup-all.sh, which references this JSON file.)

Solr Configuration¶

Dataverse requires a specific Solr schema file called schema.xml that can be found in the Dataverse distribution. It should replace the default example/solr/collection1/conf/schema.xml file that ships with Solr.

If WARN  org.eclipse.jetty.http.HttpParser  – HttpParser Full for /127.0.0.1:8983 appears in the Solr log, adding <Set name="requestHeaderSize">8192</Set> (or a higher number of bytes) to Solr’s jetty.xml in the section matching the XPath expression //Call[@name='addConnector']/Arg/New[@class='org.eclipse.jetty.server.bio.SocketConnector'] may resolve the issue. See also https://support.lucidworks.com/hc/en-us/articles/201424796-Error-when-submitting-large-query-strings-

Settings¶

JVM Options¶

Dropbox Configuration¶

• Add JVM option in the domain.xml:

asadmin create-jvm-options "-Ddataverse.dropbox.key=<Enter your dropbox key here>"

The guide is intended for anyone who needs to install the Dataverse app.

If you encounter any problems during installation, please contact the development team at support@thedata.org or our Dataverse Users Community.

0. PREREQUISITS¶

yum install libapreq2
rpm -ivh http://mirror.hmdc.harvard.edu/HMDC-Public/RedHat-6/rapache-1.2.6-rpm0.x86_64.rpm

1. Set Up R¶

R is used both by the Dataverse application, directly, and the TwoRavens companion app.

Two distinct interfaces are used to access R: Dataverse uses Rserve; and TwoRavens sends jobs to R running under rApache using Rook interface.

We provide a shell script (conf/R/r-setup.sh in the Dataverse source tree; you will need the other 3 files in that directory as well - https://github.com/IQSS/dataverse/conf/R/) that will attempt to install the required 3rd party packages; it will also configure Rserve and rserve user. rApache configuration will be addressed in its own section.

The script will attempt to download the packages from CRAN (or a mirror) and GitHub, so the system must have access to the internet. On a server fully firewalled from the world, packages can be installed from downloaded sources. This is left as an exercise for the reader. Consult the script for insight.

See the Appendix for the information on the specific packages, and versions that the script will attempt to install.

2. Install the TwoRavens Application¶

cd /var/www/html/dataexplore
chmod +x install.pl
./install.pl

Appendix¶

Explained below are the steps needed to manually install and configure the required R packages, and to configure TwoRavens to run under rApache (these are performed by the r-setup.sh and install.pl scripts above). Provided for reference.

r-setup.sh script:¶

TwoRavens requires the following R packages and versions to be installed:

Note that some of these packages have their own dependencies, and additional installations are likely necessary. TwoRavens is not compatible with older versions of these R packages.

install.pl script:¶

RSourceOnStartup "/var/www/html/dataexplore/rook/rooksource.R"
<Location /custom/zeligapp>
   SetHandler r-handler
   RFileEval /var/www/html/dataexplore/rook/rookzelig.R:Rook::Server$call(zelig.app)
</Location>
<Location /custom/subsetapp>
   SetHandler r-handler
   RFileEval /var/www/html/dataexplore/rook/rooksubset.R:Rook::Server$call(subset.app)
</Location>
<Location /custom/transformapp>
   SetHandler r-handler
   RFileEval /var/www/html/dataexplore/rook/rooktransform.R:Rook::Server$call(transform.app)
</Location>
<Location /custom/dataapp>
   SetHandler r-handler
   RFileEval /var/www/html/dataexplore/rook/rookdata.R:Rook::Server$call(data.app)
</Location>

mkdir --parents /var/www/html/custom/pic_dir

mkdir --parents /var/www/html/custom/preprocess_dir

mkdir --parents /var/www/html/custom/log_dir

chown -R apache.apache /var/www/html/custom

Status: Experimental¶

Shibboleth support in Dataverse should be considered experimental until the following issue is closed: https://github.com/IQSS/dataverse/issues/2117

In Dataverse 4.0, Shibboleth support requires fronting Glassfish with Apache as described below, but this configuration has not been heavily tested in a production environment and is not recommended until this issue is closed: https://github.com/IQSS/dataverse/issues/2180

System Requirements¶

Only Red Hat Enterprise Linux (RHEL) 6 and derivatives such as CentOS have been tested and only on x86_64. RHEL 7 and Centos 7 should work but you’ll need to adjust the yum repo config accordingly.

Debian and Ubuntu are not recommended until this issue is closed: https://github.com/IQSS/dataverse/issues/1059

Installation¶

Configuration¶

<VirtualHost *:80>

ServerName shibtest.dataverse.org

# From https://wiki.apache.org/httpd/RewriteHTTPToHTTPS

RewriteEngine On
# This will enable the Rewrite capabilities

RewriteCond %{HTTPS} !=on
# This checks to make sure the connection is not already HTTPS

RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]
# This rule will redirect users from their original location, to the same location but using HTTPS.
# i.e.  http://www.example.com/foo/ to https://www.example.com/foo/
# The leading slash is made optional so that this will work either in httpd.conf
# or .htaccess context

</VirtualHost>

# don't pass paths used by rApache and TwoRavens to Glassfish
ProxyPassMatch ^/RApacheInfo$ !
ProxyPassMatch ^/custom !
ProxyPassMatch ^/dataexplore !
# don't pass paths used by Shibboleth to Glassfish
ProxyPassMatch ^/Shibboleth.sso !
ProxyPassMatch ^/shibboleth-ds !
# pass everything else to Glassfish
ProxyPass / ajp://localhost:8009/

<Location /shib.xhtml>
  AuthType shibboleth
  ShibRequestSetting requireSession 1
  require valid-user
</Location>

<!--
This is an example shibboleth2.xml generated originally by http://testshib.org
and tweaked for Dataverse.  See also:

- attribute-map.xml
- dataverse-idp-metadata.xml

https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPConfiguration
-->

<SPConfig xmlns="urn:mace:shibboleth:2.0:native:sp:config" xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata"
    clockSkew="1800">

    <!-- FIXME: change the entityID to your hostname. -->
    <ApplicationDefaults entityID="https://shibtest.dataverse.org/sp"
        REMOTE_USER="eppn" attributePrefix="AJP_">

        <!-- You should use secure cookies if at all possible.  See cookieProps in this Wiki article. -->
        <!-- https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPSessions -->
        <Sessions lifetime="28800" timeout="3600" checkAddress="false" relayState="ss:mem" handlerSSL="false">

	    <SSO>
	      SAML2 SAML1
	    </SSO>

            <!-- SAML and local-only logout. -->
            <!-- https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPServiceLogout -->
            <Logout>SAML2 Local</Logout>

            <!--
                Handlers allow you to interact with the SP and gather more information.  Try them out!
                Attribute values received by the SP through SAML will be visible at:
                http://shibtest.dataverse.org/Shibboleth.sso/Session
            -->

            <!-- Extension service that generates "approximate" metadata based on SP configuration. -->
            <Handler type="MetadataGenerator" Location="/Metadata" signing="false"/>

            <!-- Status reporting service. -->
            <Handler type="Status" Location="/Status" acl="127.0.0.1"/>

            <!-- Session diagnostic service. -->
            <Handler type="Session" Location="/Session" showAttributeValues="true"/>

            <!-- JSON feed of discovery information. -->
            <Handler type="DiscoveryFeed" Location="/DiscoFeed"/>

        </Sessions>

        <!-- Error pages to display to yourself if something goes horribly wrong. -->
        <Errors supportContact="root@localhost" logoLocation="/shibboleth-sp/logo.jpg"
                styleSheet="/shibboleth-sp/main.css"/>

        <!-- Loads and trusts a metadata file that describes only the Testshib IdP and how to communicate with it. -->
        <!-- IdPs we want allow go in /etc/shibboleth/dataverse-idp-metadata.xml -->
        <MetadataProvider type="XML" file="dataverse-idp-metadata.xml" backingFilePath="local-idp-metadata.xml" legacyOrgNames="true" reloadInterval="7200"/>

        <!-- Attribute and trust options you shouldn't need to change. -->
        <AttributeExtractor type="XML" validate="true" path="attribute-map.xml"/>
        <AttributeResolver type="Query" subjectMatch="true"/>
        <AttributeFilter type="XML" validate="true" path="attribute-policy.xml"/>

        <!-- Your SP generated these credentials.  They're used to talk to IdP's. -->
        <CredentialResolver type="File" key="sp-key.pem" certificate="sp-cert.pem"/>

    </ApplicationDefaults>

    <!-- Security policies you shouldn't change unless you know what you're doing. -->
    <SecurityPolicyProvider type="XML" validate="true" path="security-policy.xml"/>

    <!-- Low-level configuration about protocols and bindings available for use. -->
    <ProtocolProvider type="XML" validate="true" reloadChanges="false" path="protocols.xml"/>

</SPConfig>

Shibboleth Attributes¶

The following attributes are required for a successful Shibboleth login:

• Shib-Identity-Provider
• eppn
• givenName
• sn
• email

See also https://www.incommon.org/federation/attributesummary.html and https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPAttributeAccess

For discussion of “eppn” vs. other attributes such as “eduPersonTargetedID” or “NameID”, please see https://github.com/IQSS/dataverse/issues/1422

Testing¶

http://testshib.org is a fantastic resource for testing Shibboleth configurations.

First, download your metadata like this (substituting your hostname in both places):

curl https://shibtest.dataverse.org/Shibboleth.sso/Metadata > shibtest.dataverse.org

Then upload it to http://testshib.org/register.html

Then try to log in.

Please note that when you try logging in via the TestShib IdP, it is expected that you’ll see errors about the “mail” attribute being null. (TestShib is aware of this but it isn’t a problem for testing.)

When you are done testing, you can delete the TestShib users you created like this:

curl -X DELETE http://localhost:8080/api/admin/authenticatedUsers/myself

Backups¶

It’s important to back up these files:

• /etc/shibboleth/sp-cert.pem
• /etc/shibboleth/sp-key.pem

Feedback¶

Given that Shibboleth support is experimental and new, feedback is very welcome at support@dataverse.org or via http://community.dataverse.org/community-groups/auth.html .

User Administration¶

Backward incompatible changes¶

For better security, usernames and passwords are no longer accepted. The use of an API token is required.

In addition, differences in Dataverse 4.0 have lead to a few minor backward incompatible changes in the Dataverse implementation of SWORD, which are listed below. Old v1 URLs should continue to work but the Service Document will contain a deprecation warning and responses will contain v1.1 URLs. See also Known issues.

• Newly required fields when creating/editing datasets for compliance with the Joint Declaration for Data Citation principles.
  • dcterms:creator (maps to authorName)
  • dcterms:description
• Deaccessioning is no longer supported. An alternative will be developed at https://github.com/IQSS/dataverse/issues/778
• The Service Document will show a single API Terms of Use rather than root level and dataverse level Deposit Terms of Use.

New features as of v1.1¶

• Dataverse 4.0 supports API tokens and they must be used rather that a username and password. In the curl examples below, you will see curl -u $API_TOKEN: showing that you should send your API token as the username and nothing as the password. For example, curl -u 54b143b5-d001-4254-afc0-a1c0f6a5b5a7:.
• Dataverses can be published via SWORD
• Datasets versions will only be increased to the next minor version (i.e. 1.1) rather than a major version (2.0) if possible. This depends on the nature of the change.
• “Author Affiliation” can now be populated with an XML attribute. For example: <dcterms:creator affiliation=”Coffee Bean State University”>Stumptown, Jane</dcterms:creator>
• “Contributor” can now be populated and the “Type” (Editor, Funder, Researcher, etc.) can be specified with an XML attribute. For example: <dcterms:contributor type=”Funder”>CaffeineForAll</dcterms:contributor>
• “License” can now be set with dcterms:license and the possible values are “CC0” and “NONE”. “License” interacts with “Terms of Use” (dcterms:rights) in that if you include dcterms:rights in the XML, the license will be set to “NONE”. If you don’t include dcterms:rights, the license will default to “CC0”. It is invalid to specify “CC0” as a license and also include dcterms:rights; an error will be returned. For backwards compatibility, dcterms:rights is allowed to be blank (i.e. <dcterms:rights></dcterms:rights>) but blank values will not be persisted to the database and the license will be set to “NONE”.
• “Contact E-mail” is automatically populated from dataset owners email.
• “Subject” uses our controlled vocabulary list of subjects. This list is in the Citation Metadata of our User Guide > Metadata References. Otherwise, if a term does not match our controlled vocabulary list, it will put any subject terms in “Keyword”. If Subject is empty it is automatically populated with “N/A”.
• Zero-length files are now allowed (but not necessarily encouraged).
• “Depositor” and “Deposit Date” are auto-populated.

curl examples¶

<?xml version="1.0"?>
<entry xmlns="http://www.w3.org/2005/Atom"
       xmlns:dcterms="http://purl.org/dc/terms/">
   <!-- some embedded metadata -->
   <dcterms:title>Roasting at Home</dcterms:title>
   <dcterms:creator>Peets, John</dcterms:creator>
   <dcterms:creator affiliation="Coffee Bean State University">Stumptown, Jane</dcterms:creator>
   <!-- Dataverse controlled vocabulary subject term -->
   <dcterms:subject>Chemistry</dcterms:subject>
   <!-- keywords -->
   <dcterms:subject>coffee</dcterms:subject>
   <dcterms:subject>beverage</dcterms:subject>
   <dcterms:subject>caffeine</dcterms:subject>
   <dcterms:description>Considerations before you start roasting your own coffee at home.</dcterms:description>
   <!-- Producer with financial or admin responsibility of the data -->
   <dcterms:publisher>Coffee Bean State University</dcterms:publisher>
   <dcterms:contributor type="Funder">CaffeineForAll</dcterms:contributor>
   <!-- production date -->
   <dcterms:date>2013-07-11</dcterms:date>
   <!-- kind of data -->
   <dcterms:type>aggregate data</dcterms:type>
   <!-- List of sources of the data collection-->
   <dcterms:source>Stumptown, Jane. 2011. Home Roasting. Coffeemill Press.</dcterms:source>
   <!-- related materials -->
   <dcterms:relation>Peets, John. 2010. Roasting Coffee at the Coffee Shop. Coffeemill Press</dcterms:relation>
   <!-- geographic coverage -->
   <dcterms:coverage>United States</dcterms:coverage>
   <dcterms:coverage>Canada</dcterms:coverage>
   <!-- license and restrictions -->
   <dcterms:license>NONE</dcterms:license>
   <dcterms:rights>Downloader will not use the Materials in any way prohibited by applicable laws.</dcterms:rights>
   <!-- related publications -->
   <dcterms:isReferencedBy holdingsURI="http://dx.doi.org/10.1038/dvn333" agency="DOI" IDNo="10.1038/dvn333">Peets, J., &amp; Stumptown, J. (2013). Roasting at Home. New England Journal of Coffee, 3(1), 22-34.</dcterms:isReferencedBy>
</entry>

Known issues¶

• Potential mismatch between the dataverses (“collections” from a SWORD perspective) the user can deposit data into in returned by the Service Document and which dataverses the user can actually deposit data into. This is due to an incomplete transition from the old DVN 3.x “admin-only” style permission checking to the new permissions system in Dataverse 4.0 ( https://github.com/IQSS/dataverse/issues/1070 ). The mismatch was reported at https://github.com/IQSS/dataverse/issues/1443
• Should see all the fields filled in for a dataset regardless of what the parent dataverse specifies: https://github.com/IQSS/dataverse/issues/756
• Inefficiency in constructing the Service Document: https://github.com/IQSS/dataverse/issues/784
• Inefficiency in constructing the list of datasets: https://github.com/IQSS/dataverse/issues/784

Roadmap¶

These are features we’d like to add in the future:

• Implement SWORD 2.0 Profile 6.4: https://github.com/IQSS/dataverse/issues/183
• Support deaccessioning via API: https://github.com/IQSS/dataverse/issues/778
• Let file metadata (i.e. description) be specified during zip upload: https://github.com/IQSS/dataverse/issues/723
• SWORD: Display of actual dcterms xml element for equivalent of required field not found: https://github.com/IQSS/dataverse/issues/1019

Bug fixes in v1.1¶

• Fix Abdera ArrayIndexOutOfBoundsException with non-existent atom-entry-study.xml in SWORD jar (upstream ideally) https://github.com/IQSS/dataverse/issues/893
• Sword API: Can’t create study when hidden characters are introduced in atom.xml https://github.com/IQSS/dataverse/issues/894

Client libraries¶

• Python: https://github.com/swordapp/python-client-sword2
• Java: https://github.com/swordapp/JavaClient2.0
• R: https://github.com/ropensci/dvn
• Ruby: https://github.com/swordapp/sword2ruby
• PHP: https://github.com/swordapp/swordappv2-php-library

About¶

The Search API supports the same searching, sorting, and faceting operations as the Dataverse web interface.

Unlike the web interface, this new API is limited to published data until issue 1299 is resolved.

The parameters and JSON response are partly inspired by the GitHub Search API.

Parameters¶

Basic Search Example¶

https://apitest.dataverse.org/api/search?q=trees

{
    "status":"OK",
    "data":{
        "q":"trees",
        "total_count":4,
        "start":0,
        "spelling_alternatives":{
            "trees":"[tree]"
        },
        "items":[
            {
                "name":"Trees",
                "type":"dataverse",
                "url":"https://apitest.dataverse.org/dataverse/trees",
                "image_url":"https://apitest.dataverse.org/api/access/dvCardImage/7",
                "identifier":"trees",
                "description":"A tree dataverse with some birds",
                "published_at":"2015-01-12T16:05:12Z"
            },
            {
                "name":"Chestnut Trees",
                "type":"dataverse",
                "url":"https://apitest.dataverse.org/dataverse/chestnuttrees",
                "image_url":"https://apitest.dataverse.org/api/access/dvCardImage/9",
                "identifier":"chestnuttrees",
                "description":"A dataverse with chestnut trees and an oriole",
                "published_at":"2015-01-12T18:02:32Z"
            },
            {
                "name":"trees.png",
                "type":"file",
                "url":"https://apitest.dataverse.org/api/access/datafile/12",
                "image_url":"https://apitest.dataverse.org/api/access/preview/12",
                "file_id":"12",
                "description":"",
                "published_at":"2015-01-12T16:05:44Z",
                "file_type":"PNG Image",
                "size_in_bytes":8361,
                "md5":"0386269a5acb2c57b4eade587ff4db64",
                "dataset_citation":"Spruce, Sabrina, 2015, \"Spruce Goose\", http://dx.doi.org/10.5072/FK2/Y6RGTQ,  Root Dataverse,  V1"
            },
            {
                "name":"Birds",
                "type":"dataverse",
                "url":"https://apitest.dataverse.org/dataverse/birds",
                "image_url":"https://apitest.dataverse.org/api/access/dvCardImage/2",
                "identifier":"birds",
                "description":"A bird dataverse with some trees",
                "published_at":"2015-01-12T18:01:51Z"
            }
        ],
        "count_in_response":4
    }
}

Advanced Search Example¶

https://apitest.dataverse.org/api/search?q=finch&show_relevance=true&show_facets=true&fq=publication_date_s:2015&subtree=birds

In this example, show_relevance=true matches per field are shown. Available facets are shown with show_facets=true and of the facets is being used with fq=publication_date_s:2015. The search is being narrowed to the dataverse with the identifier “birds” with the parameter subtree=birds.

{
    "status":"OK",
    "data":{
        "q":"finch",
        "total_count":2,
        "start":0,
        "spelling_alternatives":{
        },
        "items":[
            {
                "name":"Finches",
                "type":"dataverse",
                "url":"https://apitest.dataverse.org/dataverse/finches",
                "image_url":"https://apitest.dataverse.org/api/access/dvCardImage/3",
                "identifier":"finches",
                "description":"A dataverse with finches",
                "published_at":"2015-01-12T18:01:15Z",
                "matches":[
                    {
                        "description":{
                            "snippets":[
                                "A dataverse with <span class=\"search-term-match\">finches</span>"
                            ]
                        }
                    },
                    {
                        "name":{
                            "snippets":[
                                "<span class=\"search-term-match\">Finches</span>"
                            ]
                        }
                    }
                ]
            },
            {
                "name":"Darwin's Finches",
                "type":"dataset",
                "url":"http://dx.doi.org/10.5072/FK2/CE0052",
                "image_url":"https://apitest.dataverse.org/api/access/dsPreview/2",
                "global_id":"doi:10.5072/FK2/CE0052",
                "published_at":"2015-01-12T18:01:37Z",
                "citation":"Finch, Fiona, 2015, \"Darwin's Finches\", http://dx.doi.org/10.5072/FK2/CE0052,  Root Dataverse,  V1",
                "description": "Darwin's finches (also known as the Galápagos finches) are a group of about fifteen species of passerine birds.",
                "matches":[
                    {
                        "authorName":{
                            "snippets":[
                                "<span class=\"search-term-match\">Finch</span>, Fiona"
                            ]
                        }
                    },
                    {
                        "dsDescriptionValue":{
                            "snippets":[
                                "Darwin's <span class=\"search-term-match\">finches</span> (also known as the Galápagos <span class=\"search-term-match\">finches</span>) are a group of about fifteen species"
                            ]
                        }
                    },
                    {
                        "title":{
                            "snippets":[
                                "Darwin's <span class=\"search-term-match\">Finches</span>"
                            ]
                        }
                    }
                ],
                "authors":[
                    "Finch, Fiona"
                ]
            }
        ],
        "facets":[
            {
                "dvCategory_s":{
                    "friendly":"Dataverse Category",
                    "labels":[
                        {
                            "Uncategorized":1
                        }
                    ]
                },
                "affiliation_ss":{
                    "friendly":"Affiliation",
                    "labels":[
                        {
                            "Birds Inc.":1
                        }
                    ]
                },
                "publication_date_s":{
                    "friendly":"Publication Date",
                    "labels":[
                        {
                            "2015":2
                        }
                    ]
                }
            }
        ],
        "count_in_response":2
    }
}

Iteration¶

Be default, up to 10 results are returned with every request (though this can be increased with the per_page parameter). To iterate through many results, increase the start parameter on each iteration until you reach the total_count in the response. An example in Python is below.

#!/usr/bin/env python
import urllib2
import json
base = 'https://apitest.dataverse.org'
rows = 10
start = 0
page = 1
condition = True # emulate do-while
while (condition):
    url = base + '/api/search?q=*' + "&start=" + str(start)
    data = json.load(urllib2.urlopen(url))
    total = data['data']['total_count']
    print "=== Page", page, "==="
    print "start:", start, " total:", total
    for i in data['data']['items']:
        print "- ", i['name'], "(" + i['type'] + ")"
    start = start + rows
    page += 1
    condition = start < total

Output from iteration example

=== Page 1 ===
start: 0  total: 12
-  Spruce Goose (dataset)
-  trees.png (file)
-  Spruce (dataverse)
-  Trees (dataverse)
-  Darwin's Finches (dataset)
-  Finches (dataverse)
-  Birds (dataverse)
-  Rings of Conifers (dataset)
-  Chestnut Trees (dataverse)
-  Sparrows (dataverse)
=== Page 2 ===
start: 10  total: 12
-  Chestnut Sparrows (dataverse)
-  Wrens (dataverse)

Basic File Access¶

Basic acces URI:

/api/access/datafile/$id

Multiple File (“bundle”) download¶

/api/access/datafiles/$id1,$id2,...$idN

Returns the files listed, zipped.

“All Formats” bundled access for Tabular Files.¶

/api/access/datafile/bundle/$id

This is convenience packaging method is available for tabular data files. It returns a zipped bundle that contains the data in the following formats:

• Tab-delimited;
• “Saved Original”, the proprietary (SPSS, Stata, R, etc.) file from which the tabular data was ingested;
• Generated R Data frame (unless the “original” above was in R);
• Data (Variable) metadata record, in DDI XML;
• File citation, in Endnote and RIS formats.

Data Variable Metadata Access¶

These methods are only available for tabular data files. (i.e., data files with associated data table and variable objects).

/api/access/datafile/$id/metadata/ddi

In its basic form the verb above returns a DDI fragment that describes the file and the data variables in it. The DDI XML is the only format supported so far. In the future, support for formatting the output in JSON will (may?) be added.

The DDI returned will only have 2 top-level sections:

• a single fileDscr, with the basic file information plus the numbers of variables and observations and the UNF of the file.
• a single dataDscr section, with one var section for each variable.

Example:

http://localhost:8080/api/meta/datafile/6

<codeBook version="2.0">
   <fileDscr ID="f6">
      <fileTxt>
         <fileName>_73084.tab</fileName>
         <dimensns>
            <caseQnty>3</caseQnty>
            <varQnty>2</varQnty>
         </dimensns>
         <fileType>text/tab-separated-values</fileType>
      </fileTxt>
      <notes level="file" type="VDC:UNF" subject="Universal Numeric Fingerprint">UNF:6:zChnyI3fjwNP+6qW0VryVQ==</notes>
   </fileDscr>
   <dataDscr>
      <var ID="v1" name="id" intrvl="discrete">
         <location fileid="f6"/>
         <labl level="variable">Personen-ID</labl>
         <sumStat type="mean">2.0</sumStat>
         <sumStat type="mode">.</sumStat>
         <sumStat type="medn">2.0</sumStat>
         <sumStat type="stdev">1.0</sumStat>
         <sumStat type="min">1.0</sumStat>
         <sumStat type="vald">3.0</sumStat>
         <sumStat type="invd">0.0</sumStat>
         <sumStat type="max">3.0</sumStat>
         <varFormat type="numeric"/>
         <notes subject="Universal Numeric Fingerprint" level="variable" type="VDC:UNF">UNF:6:AvELPR5QTaBbnq6S22Msow==</notes>
      </var>
      <var ID="v3" name="sex" intrvl="discrete">
         <location fileid="f6"/>
         <labl level="variable">Geschlecht</labl>
         <sumStat type="mean">1.3333333333333333</sumStat>
         <sumStat type="max">2.0</sumStat>
         <sumStat type="vald">3.0</sumStat>
         <sumStat type="mode">.</sumStat>
         <sumStat type="stdev">0.5773502691896257</sumStat>
         <sumStat type="invd">0.0</sumStat>
         <sumStat type="medn">1.0</sumStat>
         <sumStat type="min">1.0</sumStat>
         <catgry>
            <catValu>1</catValu>
            <labl level="category">Mann</labl>
         </catgry>
         <catgry>
            <catValu>2</catValu>
            <labl level="category">Frau</labl>
         </catgry>
         <varFormat type="numeric"/>
         <notes subject="Universal Numeric Fingerprint" level="variable" type="VDC:UNF">UNF:6:XqQaMwOA63taX1YyBzTZYQ==</notes>
      </var>
   </dataDscr>
</codeBook>

More information on the DDI is available at (TODO).

Advanced options/Parameters:

It is possible to request only specific subsets of, rather than the full file-level DDI record. This can be a useful optimization, in cases such as when an application needs to look up a single variable; especially with data files with large numbers of variables.

Partial record parameters:

(TODO).

/api/access/datafile/$id/metadata/preprocessed

This method provides the “Pre-processed Data” - a summary record that describes the values of the data vectors in the tabular file, in JSON. These metadata values are used by TwoRavens, the companion data exploration utility of the Dataverse application.

Authentication and Authorization¶

Data Access API supports both session- and API key-based authentication.

If a session is available, and it is already associated with an authenticated user, it will be used for access authorization. If not, or if the user in question is not authorized to access the requested object, an attempt will be made to authorize based on an API key, if supplied. All of the API verbs above support the key parameter key=....

Endpoints¶

POST http://$SERVER/api/dataverses/$id?key=$apiKey

GET http://$SERVER/api/dataverses/$id

DELETE http://$SERVER/api/dataverses/$id?key=$apiKey

GET http://$SERVER/api/dataverses/$id/contents

GET http://$SERVER/api/dataverses/$id/roles?key=$apiKey

GET http://$SERVER/api/dataverses/$id/facets?key=$apiKey

POST http://$SERVER/api/dataverses/$id/roles?key=$apiKey

GET http://$SERVER/api/dataverses/$id/assignments?key=$apiKey

POST http://$SERVER/api/dataverses/$id/assignments?key=$apiKey

{
  "assignee": "@uma",
  "role": "curator"
}

DELETE http://$SERVER/api/dataverses/$id/assignments/$id?key=$apiKey

GET http://$SERVER/api/dataverses/$id/metadatablocks?key=$apiKey

POST http://$SERVER/api/dataverses/$id/metadatablocks?key=$apiKey

GET http://$SERVER/api/dataverses/$id/metadatablocks/:isRoot?key=$apiKey

POST http://$SERVER/api/dataverses/$id/metadatablocks/:isRoot?key=$apiKey

POST http://$SERVER/api/dataverses/$id/datasets/?key=$apiKey

POST http://$SERVER/api/dataverses/$identifier/actions/:publish?key=$apiKey

GET http://$SERVER/api/datasets/$id?key=$apiKey

DELETE http://$SERVER/api/datasets/$id?key=$apiKey

GET http://$SERVER/api/datasets/$id/versions?key=$apiKey

GET http://$SERVER/api/datasets/$id/versions/$versionNumber?key=$apiKey

GET http://$SERVER/api/datasets/$id/versions/$versionId/files?key=$apiKey

GET http://$SERVER/api/datasets/$id/versions/$versionId/metadata?key=$apiKey

GET http://$SERVER/api/datasets/$id/versions/$versionId/metadata/$blockname?key=$apiKey

PUT http://$SERVER/api/datasets/$id/versions/:draft?key=$apiKey

POST http://$SERVER/api/datasets/$id/actions/:publish?type=$type&key=$apiKey

DELETE http://$SERVER/api/datasets/$id/versions/:draft?key=$apiKey

POST http://$SERVER/api/builtin-users?password=$password&key=$key

GET http://$SERVER/api/builtin-users/$username/api-token?password=$password

POST http://$SERVER/api/roles?dvo=$dataverseIdtf&key=$apiKey

GET http://$SERVER/api/roles/$id

DELETE http://$SERVER/api/roles/$id

POST http://$server/api/dataverses/$id/groups

{
 "description":"Describe the group here",
 "displayName":"Close Collaborators",
 "aliasInOwner":"ccs"
}

GET http://$server/api/dataverses/$id/groups

GET http://$server/api/dataverses/$dv/groups/$groupAlias

PUT http://$server/api/dataverses/$dv/groups/$groupAlias

DELETE http://$server/api/dataverses/$dv/groups/$groupAlias

POST http://$server/api/dataverses/$dv/groups/$groupAlias/roleAssignees

PUT http://$server/api/dataverses/$dv/groups/$groupAlias/roleAssignees/$roleAssigneeIdentifier

DELETE http://$server/api/dataverses/$dv/groups/$groupAlias/roleAssignees/$roleAssigneeIdentifier

GET http://$SERVER/api/metadatablocks

GET http://$SERVER/api/metadatablocks/$identifier

GET http://$SERVER/api/admin/settings

PUT http://$SERVER/api/admin/settings/$name

GET http://$SERVER/api/admin/settings/$name

DELETE http://$SERVER/api/admin/settings/$name

GET http://$SERVER/api/admin/authenticationProviderFactories

GET http://$SERVER/api/admin/authenticationProviders

POST http://$SERVER/api/admin/authenticationProviders

GET http://$SERVER/api/admin/authenticationProviders/$id

POST http://$SERVER/api/admin/authenticationProviders/$id/:enabled

curl -H "Content-type: application/json"  -X POST -d"false" http://localhost:8080/api/admin/authenticationProviders/echo-dignified/:enabled

DELETE http://$SERVER/api/admin/authenticationProviders/$id/

GET http://$SERVER/api/admin/roles

POST http://$SERVER/api/admin/roles

POST http://$SERVER/api/admin/superuser/$identifier

GET http://$SERVER/api/admin/groups/ip

POST http://$SERVER/api/admin/groups/ip

GET http://$SERVER/api/admin/groups/ip/$groupIdtf

DELETE http://$SERVER/api/admin/groups/ip/$groupIdtf

GET http://$SERVER/api/admin/savedsearches/list

GET http://$SERVER/api/admin/savedsearches/$id

PUT http://$SERVER/api/admin/savedsearches/makelinks/$id?debug=true

PUT http://$SERVER/api/admin/savedsearches/makelinks/all?debug=true

Python¶

https://github.com/IQSS/dataverse-client-python is the offical Python package for Dataverse APIs.

Robert Liebowitz from the Center for Open Science heads its development and the library is used to integrate the Open Science Framework (OSF) with Dataverse via an add-on which itself is open source and listed on the Apps page.

R¶

https://github.com/IQSS/dataverse-client-r is the official R package for Dataverse APIs.

It was created by Thomas Leeper whose dataverse can be found at https://dataverse.harvard.edu/dataverse/leeper

Javascript¶

PHP¶

Python¶

Java¶

Intended Audience¶

This guide is intended primarily for developers who want to work on the main Dataverse code base at https://github.com/IQSS/dataverse

To get started, you’ll want to set up your Development Environment and make sure you understand the Branching Strategy. Thorough Testing is encouraged (and expected). Opinions about Coding Style are welcome!

If you have any questions at all, please reach out to other developers per https://github.com/IQSS/dataverse/blob/master/CONTRIBUTING.md

Roadmap¶

For the Dataverse development roadmap, please see https://github.com/IQSS/dataverse/milestones

The Contributing to Dataverse document in the root of the source tree provides guidance on:

• the use of labels to organize and prioritize issues
• making pull requests
• how to contact the development team

Related Guides¶

If you are a developer who wants to make use of Dataverse APIs, please see the API Guide.

If you are a sysadmin who likes to code, you may be interested in hacking on installation scripts mentioned in the Installation Guide. We validate the installation scripts with Tools such as Vagrant.

Related Projects¶

As a developer, you also may be interested in these projects related to Dataverse:

• Zelig (R) - run statistical models on files uploaded to Dataverse: https://github.com/IQSS/Zelig
• TwoRavens (Javascript) - a d3.js interface for exploring data and running Zelig models: https://github.com/IQSS/TwoRavens
• Universal Numerical Fingerprint (UNF) (Java) - a Universal Numerical Fingerprint: https://github.com/IQSS/UNF
• DataTags (Java and Scala) - tag datasets with privacy levels: https://github.com/IQSS/DataTags
• GeoConnect (Python) - create a map by uploading files to Dataverse: https://github.com/IQSS/geoconnect
• Dataverse API client libraries - use Dataverse APIs from various languages: Client Libraries
• Third party apps - make use of Dataverse APIs: Apps
• [Your project here] :)

Assumptions¶

This guide assumes you are using a Mac but we do have pages for Windows and Ubuntu.

Requirements¶

Recommendations¶

Setting up your dev environment¶

Shibboleth¶

If you are working on anything related to users, please keep in mind that your changes will likely affect Shibboleth users. Rather than setting up Shibboleth on your laptop, developers are advised to simply add a value to their database to enable Shibboleth “dev mode” like this:

curl http://localhost:8080/api/admin/settings/:DebugShibAccountType -X PUT -d RANDOM

For a list of possible values, please “find usages” on the settings key above and look at the enum.

Now when you go to http://localhost:8080/shib.xhtml you should be prompted to create a Shibboleth account.

Rebuilding your dev environment¶

If you have an old copy of the database and old Solr data and want to start fresh, here are the recommended steps:

• drop your old database
• clear out your existing Solr index: scripts/search/clear
• run the installer script above - it will create the db, deploy the app, populate the db with reference data and run all the scripts that create the domain metadata fields. You no longer need to perform these steps separately.
• confirm you are using the latest Dataverse-specific Solr schema.xml per the “Installing and Running Solr” section of this guide
• confirm http://localhost:8080 is up
• If you want to set some dataset-specific facets, go to the root dataverse (or any dataverse; the selections can be inherited) and click “General Information” and make choices under “Select Facets”. There is a ticket to automate this: https://github.com/IQSS/dataverse/issues/619

Goals¶

The goals of the Dataverse branching strategy are twofold:

• have developers “stay together” in the same release branch (i.e. 4.0.1), resolving conflicts early
• allow for concurrent development in multiple branches, for example:
  • hot fix branch created from 4.0 tag (i.e. 4.0-patch.1)
  • bug fixes in an unreleased 4.0.1 release branch found in QA
  • feature development in an upcoming 4.0.2 release branch

Release branches that match milestones and future version numbers (i.e. 4.0.1, 4.0.2) are used to achieve these goals. Think of release branches as trains. If you miss the 4.0.1 train, hopefully you’ll catch the 4.0.2! The goal is always to get the best code into each release.

Persistent Branches¶

The “master” branch is the only persistent branch. Commits should never be made directly to master. Commits are only made to release branches. Release branches are merged in “master” just before tagging per Making Releases.

Release Branches¶

When developers feel the code in a release branch (i.e. 4.0.1) is ready, it is passed to QA. At this point feature development in that release branch (i.e. 4.0.1) is frozen and a new release branch (i.e. 4.0.2) is created from that commit. The frozen release branch is sent to QA for testing.

Feature Development¶

Developers who have push access to https://github.com/IQSS/dataverse are welcome to push commits directly to the branch corresponding to the milestone the issue tracking the feature has been assigned. For example, if you are working on https://github.com/IQSS/dataverse/issues/2088 and the issue has been assigned to milestone 4.0.1, you are welcome to push directly to the 4.0.1 branch.

Developers who do not have push access should first read https://github.com/IQSS/dataverse/blob/master/CONTRIBUTING.md for general guidelines about contributing code and contacting developers who have the ability to (hopefully!) merge in your contribution. You will likely be advised to submit a pull request against the current release branch that is not yet frozen. For example, if 4.0.1 is frozen (i.e. in QA) the pull request should be made against 4.0.2.

Fixing Bugs in Unreleased Code¶

Bugs in the release branch currently in QA (i.e. 4.0.1) will be fixed in that branch referencing an issue number. Assuming the fix is good and passes QA, the release branch in QA (i.e. 4.0.1) will be merged into the release branch currently under feature development (i.e. 4.0.2) to get the bug fix.

Fixing Bugs in Released code¶

Bugs found in released code will (like features) be assigned to milestones visible at https://github.com/IQSS/dataverse/milestones

Unit Tests¶

Write them. JUnit is your friend.

Integration Tests¶

Write them. REST-assured and Selenium are recommended. A Jenkins environment is available for automated testing: https://build.hmdc.harvard.edu:8443

Quick Fix¶

If you find a typo or a small error in the documentation you can easily fix it using GitHub.

• Fork the repository
• Go to [your GitHub username]/dataverse/doc/sphinx-guides/source and access the file you would like to fix
• Switch to the branch that is currently under development
• Click the Edit button in the upper-right corner and fix the error
• Submit a pull request

Other Changes (Sphinx)¶

The documentation for Dataverse was written using Sphinx (http://sphinx-doc.org/). If you are interested in suggesting changes or updates we recommend that you create the html files using Sphinx locally and the submit a pull request through GitHub. Here are the instructions on how to proceed:

Logging¶

By default, Glassfish logs at the “INFO” level but logging can be increased to “FINE” on the fly with (for example) asadmin set-log-levels edu.harvard.iq.dataverse.api.Datasets=FINE. Running asadmin list-log-levels will show the current logging levels.

Bump Version Numbers¶

Before tagging, ensure the version number has been incremented in the following places:

• pom.xml (and scripts that reference the name of the war file)
• src/main/java/VersionNumber.properties
• doc/sphinx-guides/source/conf.py

Finalize Documentation¶

The source for user-facing documentation (including this very guide) is under https://github.com/IQSS/dataverse/tree/master/doc

Docs don’t write themselves. Please help out! Before a release is tagged documentation related to that release should be checked in to the release branch (i.e. 4.0.1), ultimately to be hosted under a version number at http://guides.dataverse.org

Write Release Notes¶

See http://keepachangelog.com

Merge Release Branch¶

The release branch (i.e. 4.0.1) should be merged into “master” before tagging.

Tag the Release¶

The tag will be the number of the milestone and release branch (i.e. 4.0.1).

Make Release Available for Download¶

Upload the following to https://github.com/IQSS/dataverse/releases

• installer (for new installs)
• war file
• database migration script

Netbeans Connector Chrome Extension¶

The Netbeans Connector extension for Chrome allows you to see changes you’ve made to HTML pages the moment you save the file without having to refresh your browser. See also http://wiki.netbeans.org/ChromeExtensionInstallation

Maven¶

With Maven installed you can run mvn package and mvn test from the command line. It can be downloaded from https://maven.apache.org

PageKite¶

PageKite is a fantastic service that can be used to share your local development environment over the Internet on a public IP address.

With PageKite running on your laptop, the world can access a URL such as http://pdurbin.pagekite.me to see what you see at http://localhost:8080

Sign up at https://pagekite.net and follow the installation instructions or simply download https://pagekite.net/pk/pagekite.py

The first time you run ./pagekite.py a file at ~/.pagekite.rc will be created. You can edit this file to configure PageKite to serve up port 8080 (the default GlassFish HTTP port) or the port of your choosing.

According to https://pagekite.net/support/free-for-foss/ PageKite (very generously!) offers free accounts to developers writing software the meets http://opensource.org/docs/definition.php such as Dataverse.

Vagrant¶

Vagrant allows you to spin up a virtual machine running Dataverse on your development workstation.

From the root of the git repo, run vagrant up and eventually you should be able to reach an installation of Dataverse at http://localhost:8888 (or whatever forwarded_port indicates in the Vagrantfile)

The Vagrant environment can also be used for Shibboleth testing in conjunction with PageKite configured like this:

service_on = http:@kitename : localhost:8888 : @kitesecret

service_on = https:@kitename : localhost:9999 : @kitesecret

Please note that before running vagrant up for the first time, you’ll need to ensure that required software (GlassFish, Solr, etc.) is available within Vagrant. If you type cd downloads and ./download.sh the software should be properly downloaded.

MSV¶

MSV (Multi Schema Validator) can be used from the command line to validate an XML document against a schema. Download the latest version from https://java.net/downloads/msv/releases/ (msv.20090415.zip as of this writing), extract it, and run it like this:

$ java -jar /tmp/msv-20090415/msv.jar Version2-0.xsd ddi.xml
start parsing a grammar.
validating ddi.xml
the document is valid.