Account Information¶

As a registered user, you can:

• Create your own dataverse, if permitted, and customize it.
• Add datasets to dataverses, if permitted.
• Contribute to existing datasets, if permitted.
• Request access to restricted files, if permitted.

Remote Authentication¶

Too many passwords? You can set up your Dataverse account to use log in credentials from one of the following remote authentication providers. This way, you can log in using your existing credentials from another service.

My Data¶

The My Data section of your account page displays a listing of all the dataverses, datasets, and files you have either created, uploaded or that you have access to edit. You are able to filter through all the dataverses, datasets, and files listed there using the filter box. You may also use the facets on the left side to only view a specific Publication Status or Role.

Notifications¶

Notifications appear in the notifications tab on your account page and are also displayed as a number next to your account name.

You will receive a notification when:

• You’ve created your account
• You’ve created a dataverse or added a dataset
• Another Dataverse user has requested access to a restricted file in one of your datasets

Notifications will only be emailed one time even if you haven’t read the notification on the Dataverse site.

API Token¶

1. To create your API token, click on your name in the header on the right hand side and then click on API Token.
2. In this tab, you can create your API Token for the first time as well as recreate it if you need a new API Token or if your API Token becomes compromised.

Finding Data¶

Without logging in to Dataverse, users can browse Dataverse, search for dataverses, datasets, and files, view dataset descriptions and files for published datasets, and subset, analyze, and visualize data for published (restricted & not restricted) data files. To view an unpublished dataverse, dataset, or file, a user will need to be given permission from that dataverse’s administrator to access it.

A user can search within a specific dataverse for the dataverses, datasets, and files it contains by using the search bar and facets displayed on that dataverse’s page.

Using Data¶

Create a Dataverse (Within the “Root” Dataverse)¶

Creating a dataverse is easy but first you must be a registered user (see Account Creation + Management).

1. Once you are logged in click on the “Add Data” button and in the dropdown menu select “New Dataverse”.

2. Once on the “New Dataverse” page fill in the following fields:

   • Name: Enter the name of your dataverse.
   • Identifier: This is an abbreviation, usually lower-case, that becomes part of the URL for the new dataverse. Special characters (~,`, !, @, #, $, %, ^, &, and *) and spaces are not allowed. Note: if you change the Dataverse URL field, the URL for your Dataverse changes (http//…/’url’), which affects links to this page.
   • Email: This is the email address that will be used as the contact for this particular dataverse. You can have more than one contact email address for your dataverse.
   • Affiliation: Add any Affiliation that can be associated to this particular dataverse (e.g., project name, institute name, department name, journal name, etc). This is automatically filled out if you have added an affiliation for your user account.
   • Description: Provide a description of this dataverse. This will display on the home page of your dataverse and in the search result list. The description field supports certain HTML tags (<a>, <b>, <blockquote>, <br>, <code>, <del>, <dd>, <dl>, <dt>, <em>, <hr>, <h1>-<h3>, <i>, <img>, <kbd>, <li>, <ol>, <p>, <pre>, <s>, <sup>, <sub>, <strong>, <strike>, <ul>).
   • Category: Select a category that best describes the type of dataverse this will be. For example, if this is a dataverse for an individual researcher’s datasets, select Researcher. If this is a dataverse for an institution, select Organization & Institution.
   • Choose the sets of Metadata Elements for datasets in this dataverse: By default the metadata elements will be from the host dataverse that this new dataverse is created in. Dataverse offers metadata standards for multiple domains. To learn more about the metadata standards in Dataverse please check out the Appendix.
   • Select facets for this dataverse: by default the facets that will appear on your dataverse landing page will be from the host dataverse that this new dataverse was created in. The facets are simply metadata fields that can be used to help others easily find dataverses and datasets within this dataverse. You can select as many facets as you would like.

3. Selected metadata elements are also used to pick which metadata fields you would like to use for creating templates for your datasets. Metadata fields can be hidden, or selected as required or optional. Once you have selected all the fields you would like to use, you can create your template(s) after you finish creating your dataverse.
4. Click “Create Dataverse” button and you’re done!

*Required fields are denoted by a red asterisk.

Edit Dataverse¶

To edit your dataverse, navigate to your dataverse homepage and select the “Edit Dataverse” button, where you will be presented with the following editing options:

• General Information: edit name, identifier, category, contact email, affiliation, description, Metadata Elements, and facets for your dataverse
• Theme: upload a logo for your dataverse, add a link to your department or personal website, and select colors for your dataverse in order to brand it
• Widgets: get code to add to your website to have your dataverse display on it
• Permissions: give Dataverse users permissions to your dataverse, i.e.-can edit datasets, and see which users already have which permissions for your dataverse
• Dataset Templates: these are useful when you have several datasets that have the same information in multiple metadata fields that you would prefer not to have to keep manually typing in
• Dataset Guestbooks: allows you to collect data about who is downloading the files from your datasets
• Featured Dataverses: if you have one or more dataverses, you can use this option to show them at the top of your dataverse page to help others easily find interesting or important dataverses
• Delete Dataverse: you are able to delete your dataverse as long as it is not published and does not have any draft datasets

General Information¶

The General Information page is how you edit the information you filled in while creating your dataverse. If you need to change or add a contact email address, this is the place to do it. Additionally, you can update the metadata elements used for datasets within the dataverse, change which metadata fields are hidden, required, or optional, and update the facets you would like displayed for browsing the dataverse. If you plan on using templates, you need to select the metadata fields on the General Information page.

Tip: The metadata fields you select as required will appear on the Create Dataset form when someone goes to add a dataset to the dataverse.

Theme¶

The Theme feature provides you with a way to customize the look of your dataverse. You can decide either to use the theme from the dataverse containing your dataverse (even up to the root dataverse, AKA the homepage), or upload your own image file. Supported image types are JPEG, TIFF, or PNG and should be no larger than 500 KB. The maximum display size for an image file in a dataverse’s theme is 940 pixels wide by 120 pixels high. Additionally, you can select the colors for the header of your dataverse and the text that appears in your dataverse. You can also add a link to your personal website, the website for your organization or institution, your department, journal, etc.

Widgets¶

The Widgets feature provides you with code for you to put on your personal website to have your dataverse displayed there. There are two types of Widgets for a dataverse, a Dataverse Search Box widget and a Dataverse Listing widget. From the Widgets tab on the Theme + Widgets page, you can copy and paste the code snippets for the widget you would like to add to your website. If you need to adjust the height of the widget on your website, you may do so by editing the heightPx=500 parameter in the code snippet.

Roles & Permissions¶

Dataverse user accounts can be granted roles that define which actions they are allowed to take on specific dataverses, datasets, and/or files. Each role comes with a set of permissions, which define the specific actions that users may take.

Roles and permissions may also be granted to groups. Groups can be defined as a collection of Dataverse user accounts, a collection of IP addresses (e.g. all users of a library’s computers), or a collection of all users who log in using a particular institutional login (e.g. everyone who logs in with a particular university’s account credentials).

Admins of a dataverse can assign roles and permissions to the users of that dataverse. If you are an admin on a dataverse, then you will find the link to the Permissions page under the Edit dropdown on the dataverse page.

Clicking on Permissions will bring you to this page:

When you access a dataverse’s permissions page, you will see three sections:

Permissions: Here you can decide the requirements that determine which types of users can add datasets and sub dataverses to your dataverse, and what permissions they’ll be granted when they do so.

Users/Groups: Here you can assign roles to specific users or groups, determining which actions they are permitted to take on your dataverse. You can also reference a list of all users who have roles assigned to them for your dataverse and remove their roles if you please.

Roles: Here you can reference a full list of roles that can be assigned to users of your dataverse. Each role lists the permissions that it offers.

Please note that even on a newly created dataverse, you may see user and groups have already been granted role(s) if your installation has :InheritParentRoleAssignments set. For more on this setting, see the Configuration section of the Installation Guide.

Dataset Templates¶

Templates are useful when you have several datasets that have the same information in multiple metadata fields that you would prefer not to have to keep manually typing in, or if you want to use a custom set of Terms of Use and Access for multiple datasets in a dataverse. In Dataverse 4.0, templates are created at the dataverse level, can be deleted (so it does not show for future datasets), set to default (not required), or can be copied so you do not have to start over when creating a new template with similar metadata from another template. When a template is deleted, it does not impact the datasets that have used the template already.

How do you create a template?

1. Navigate to your dataverse, click on the Edit Dataverse button and select Dataset Templates.
2. Once you have clicked on Dataset Templates, you will be brought to the Dataset Templates page. On this page, you can 1) decide to use the dataset templates from your parent dataverse 2) create a new dataset template or 3) do both.
3. Click on the Create Dataset Template to get started. You will see that the template is the same as the create dataset page with an additional field at the top of the page to add a name for the template.
4. After adding information into the metadata fields you have information for and clicking Save and Add Terms, you will be brought to the page where you can add custom Terms of Use and Access. If you do not need custom Terms of Use and Access, click the Save Dataset Template, and only the metadata fields will be saved.
5. After clicking Save Dataset Template, you will be brought back to the Manage Dataset Templates page and should see your template listed there now with the make default, edit, view, or delete options.
6. A dataverse does not have to have a default template and users can select which template they would like to use while on the Create Dataset page.
7. You can also click on the View button on the Manage Dataset Templates page to see what metadata fields have information filled in.

* Please note that the ability to choose which metadata fields are hidden, required, or optional is done on the General Information page for the dataverse.

Dataset Guestbooks¶

Guestbooks allow you to collect data about who is downloading the files from your datasets. You can decide to collect account information (username, given name & last name, affiliation, etc.) as well as create custom questions (e.g., What do you plan to use this data for?). You are also able to download the data collected from the enabled guestbooks as Excel files to store and use outside of Dataverse.

How do you create a guestbook?

1. After creating a dataverse, click on the Edit Dataverse button and select Dataset Guestbook
2. By default, guestbooks created in the dataverse your dataverse is in, will appear. If you do not want to use or see those guestbooks, uncheck the checkbox that says Include Guestbooks from Root Dataverse.
3. To create a new guestbook, click the Create Dataset Guestbook button on the right side of the page.
4. Name the guestbook, determine the account information that you would like to be required (all account information fields show when someone downloads a file), and then add Custom Questions (can be required or not required).
5. Hit the Create Dataset Guestbook button once you have finished.

What can you do with a guestbook? After creating a guestbook, you will notice there are several options for a guestbook and appear in the list of guestbooks.

• If you want to use a guestbook you have created, you will first need to click the button in the Action column that says Enable. Once a guestbook has been enabled, you can go to the License + Terms for a dataset and select a guestbook for it.
• There are also options to view, copy, edit, or delete a guestbook.
• Once someone has downloaded a file in a dataset where a guestbook has been assigned, an option to download collected data will appear.

Featured Dataverses¶

Featured Dataverses is a way to display sub dataverses in your dataverse that you want to feature for people to easily see when they visit your dataverse.

Click on Featured Dataverses and a pop up will appear. Select which sub dataverses you would like to have appear.

Note: Featured Dataverses can only be used with published dataverses.

Dataset Linking¶

Dataset linking allows a dataverse owner to “link” their dataverse to a dataset that exists outside of that dataverse, so it appears in the dataverse’s list of contents without actually being in that dataverse. You can link other users’ datasets to your dataverse, but that does not transfer editing or other special permissions to you. The linked dataset will still be under the original user’s control.

For example, researchers working on a collaborative study across institutions can each link their own individual institutional dataverses to the one collaborative dataset, making it easier for interested parties from each institution to find the study.

In order to link a dataset, you will need your account to have the “Add Dataset” permission on the Dataverse that is doing the linking. If you created the dataverse then you should have this permission already, but if not then you will need to ask the admin of that dataverse to assign that permission to your account. You do not need any special permissions on the dataset being linked.

To link a dataset to your dataverse, you must navigate to that dataset and click the white “Link” button in the upper-right corner of the dataset page. This will open up a window where you can type in the name of the dataverse that you would like to link the dataset to. Select your dataverse and click the save button. This will establish the link, and the dataset will now appear under your dataverse.

There is currently no way to remove established links in the UI. If you need to remove a link between a dataverse and a dataset, please contact the support team for the Dataverse installation you are using.

Dataverse Linking¶

Similarly to dataset linking, dataverse linking allows a dataverse owner to “link” their dataverse to another dataverse, so the dataverse being linked will appear in the linking dataverse’s list of contents without actually being in that dataverse. Currently, the ability to link a dataverse to another dataverse is a superuser only feature.

If you need to have a dataverse linked to your dataverse, please contact the support team for the Dataverse installation you are using.

Publish Your Dataverse¶

Once your dataverse is ready to go public, go to your dataverse page, click on the “Publish” button on the right hand side of the page. A pop-up will appear to confirm that you are ready to actually Publish, since once a dataverse is made public, it can no longer be unpublished.

Supported Metadata¶

A dataset contains three levels of metadata:

1. Citation Metadata: any metadata that would be needed for generating a data citation and other general metadata that could be applied to any dataset;
2. Domain Specific Metadata: with specific support currently for Social Science, Life Science, Geospatial, and Astronomy datasets; and
3. File-level Metadata: varies depending on the type of data file - see File Handling and Uploading section below for more details).

For more details about what Citation and Domain Specific Metadata is supported please see our Appendix.

Note that once a dataset has been published its metadata may be exported. A button on the dataset page’s metadata tab will allow a user to export the metadata of the most recently published version of the dataset. Currently supported export formats are DDI, Dublin Core, Schema.org JSON-LD, and Dataverse’s native JSON format.

Adding a New Dataset¶

1. Navigate to the dataverse in which you want to add a dataset.
2. Click on the “Add Data” button and select “New Dataset” in the dropdown menu.
3. To quickly get started, enter at minimum all the required fields with an asterisk (e.g., the Dataset Title, Author, Description, Contact Email and Subject) to get a Data Citation with a DOI.
4. Scroll down to the “Files” section and click on “Select Files to Add” to add all the relevant files to your Dataset. You can also upload your files directly from your Dropbox. Tip: You can drag and drop or select multiple files at a time from your desktop directly into the upload widget. Your files will appear below the “Select Files to Add” button where you can add a description and tags (via the “Edit Tag” button) for each file. Additionally, an MD5 checksum will be added for each file. If you upload a tabular file a Universal Numerical Fingerprint (UNF) will be added to this file.
5. Click the “Save Dataset” button when you are done. Your unpublished dataset is now created.

Note: You can add additional metadata once you have completed the initial dataset creation by going to Edit Dataset > Metadata.

File Upload¶

The Dataverse software offers multiple methods of uploading files to a dataset. These upload methods are configurable by the administrator of a Dataverse installation, so you might not see some of these options on the Dataverse site you’re using.

If there are multiple upload options available, then you must choose which one to use for your dataset. A dataset may only use one upload method. Once you upload a file using one of the available upload methods, that method is locked in for that dataset. If you need to switch upload methods for a dataset that already contains files, then please contact Support by clicking on the Support link at the top of the application.

You can upload files to a dataset while first creating that dataset. You can also upload files after creating a dataset by clicking the “Edit” button at the top of the dataset page and from the dropdown list selecting “Files (Upload)” or clicking the “Upload Files” button above the files table in the Files tab. From either option you will be brought to the Upload Files page for that dataset.

Certain file types in Dataverse are supported by additional functionality, which can include downloading in different formats, subsets, file-level metadata preservation, file-level data citation; and exploration through data visualization and analysis. See the File Handling section of this page for more information.

java -jar DVUploader-v1.0.0.jar -server=<Dataverse server URL> -did=<Dataset DOI> -key=<User's API Key> <file or directory list>

File Handling¶

Certain file types in Dataverse are supported by additional functionality, which can include downloading in different formats, subsets, file-level metadata preservation, file-level data citation; and exploration through data visualization and analysis. See the sections below for information about special functionality for specific file types.

Edit Files¶

Replace Files¶

In cases where you would like to revise an existing file rather than add a new one, you can do so using our Replace File feature. This will allow you to track the history of this file across versions of your dataset, both before and after replacing it. This could be useful for updating your data or fixing mistakes in your data. Because replacing a file creates an explicit link between the previous dataset version and the current version, the file replace feature is not available for unpublished dataset drafts. Also note that replacing a file will not automatically carry over that file’s metadata, but once the file is replaced then its original metadata can still be found by referencing the previous version of the file under the “Versions” tab of the file page.

To replace a file, go to the file page for that file, click on the “Edit” button, and from the dropdown list select “Replace”. This will bring you to the Replace File page, where you can see the metadata for the most recently published version of the file and you can upload your replacement file. Once you have uploaded the replacement file, you can edit its name, description, and tags. When you’re finished, click the “Save Changes” button.

After successfully replacing a file, a new dataset draft version will be created. A summary of your actions will be recorded in the “Versions” tab on on both the dataset page and file page. The Versions tab allows you to access all previous versions of the file across all previous versions of your dataset, including the old version of the file before you replaced it.

Terms¶

Dataset terms can be viewed and edited from the Terms tab of the dataset page, or under the Edit dropdown button of a Dataset. There, you can set up how users can use your data once they have downloaded it (CC0 waiver or custom Terms of Use), how they can access your data if you have files that are restricted (terms of access), and enable a Guestbook for your dataset so that you can track who is using your data and for what purposes. These are explained in further detail below:

Roles & Permissions¶

Dataverse user accounts can be granted roles that define which actions they are allowed to take on specific dataverses, datasets, and/or files. Each role comes with a set of permissions, which define the specific actions that users may take.

Roles and permissions may also be granted to groups. Groups can be defined as a collection of Dataverse user accounts, a collection of IP addresses (e.g. all users of a library’s computers), or a collection of all users who log in using a particular institutional login (e.g. everyone who logs in with a particular university’s account credentials).

Data Provenance¶

Data Provenance is a record of where your data came from and how it reached its current form. It describes the origin of a data file, any transformations that have been made to that file, and any persons or organizations associated with that file. A data file’s provenance can aid in reproducibility and compliance with legal regulations. Dataverse can help you keep track of your data’s provenance. Currently, Dataverse only makes provenance information available to those who have edit permissions on your dataset, but in the near future we plan to expand this feature to make provenance information available to the public. You can track our progress in this issue on the Dataverse GitHub repository.

Dataverse accepts provenance information in two forms: a Provenance File or a free-text Provenance Description. You can attach this provenance information to your data files in Dataverse as part of the file upload process, by clicking Edit -> Provenance:

This will open a window where you can add your Provenance File and/or Provenance Description:

A Provenance File is the preferred way of submitting provenance information to Dataverse because it provides a detailed and trustworthy record. Provenance files are typically generated during the process of data analysis, using provenance capture tools like provR, RDataTracker, NoWorkFlow, recordr, or CamFlow.

Once you upload a provenance file, Dataverse will need some additional information in order to accurately connect it to your data file. Once provenance file upload finishes, an input box labeled “Connect entity” will appear under the file. Provenance files contain a list of “entities”, which include your data file as well as any objects associated with it (e.g. a chart, a spellchecker, etc.). You will need to tell Dataverse which entity within the provenance file represents your data file. You may type the name of the entity into the box, or click the arrow next to the box and select the entity from a list of all entities in the provenance file.

For more information on entities and the contents of provenance files, see the W3C PROV Model Primer.

Once you’ve uploaded your Provenance File and connected the proper entity, you can hit the Preview button to view the raw JSON of the Provenance File. This can help you confirm that you’ve uploaded the right file. Be sure to double-check it, because the Provenance File will made permanent once it’s finalized. At that point you will not be able to replace, remove, or otherwise edit the Provenance File. This ensures that the Provenance File maintains a stable, immutable record of the data file’s history. This finalization of the Provenance File happens at different points depending on the status of your data file. If this is a brand new data file that has never been published before, then its associated Provenance File will be made permanent once you publish the dataset. If this data file has been published in a previous version of your dataset, then its associated Provenance File will be made permanent as soon as you upload the Provenance File and click “Save Changes” on the warning popup.

A Provenance Description allows you to add more provenance information in addition to or in place of a provenance file. This is a free-text field that allows you to enter any information you feel might be relevant to those interested in learning about the provenance of your data. This might be a good place to describe provenance factors like what operating system you used when working with the data file, what functions or libraries you used, how data was merged into the file, what version of the file you used, etc. The Provenance Description is not as useful or trustworthy as a provenance file, but it can still provide value. Unlike the Provenance File, the Provenance Description is never made permanent: you can always edit, remove, or replace it at any time.

You can return to attach provenance to your data file later on by clicking the “Add + Edit Metadata” button on the file page, and then clicking the “Edit -> Provenance” button.

Thumbnails + Widgets¶

Publish Dataset¶

When you publish a dataset (available to an Admin, Curator, or any custom role which has this level of permission assigned), you make it available to the public so that other users can browse or search for it. Once your dataset is ready to go public, go to your dataset page and click on the “Publish” button on the right hand side of the page. A pop-up will appear to confirm that you are ready to actually Publish since once a dataset is made public it can no longer be unpublished.

Whenever you edit your dataset, you are able to publish a new version of the dataset. The publish dataset button will reappear whenever you edit the metadata of the dataset or add a file.

Note: Prior to publishing your dataset the Data Citation will indicate that this is a draft but the “DRAFT VERSION” text will be removed as soon as you Publish.

Submit for Review¶

If you have a Contributor role (can edit metadata, upload files, and edit files, edit Terms, Guestbook, and Submit datasets for review) in a Dataverse you can submit your dataset for review when you have finished uploading your files and filling in all of the relevant metadata fields. To Submit for Review, go to your dataset and click on the “Submit for Review” button, which is located next to the “Edit” button on the upper-right. Once Submitted for Review: the Admin or Curator for this Dataverse will be notified to review this dataset before they decide to either “Publish” the dataset or “Return to Author”. If the dataset is published the contributor will be notified that it is now published. If the dataset is returned to the author, the contributor of this dataset will be notified that they need to make modifications before it can be submitted for review again.

Private URL to Review Unpublished Dataset¶

Creating a Private URL for your dataset allows you to share your dataset (for viewing and downloading of files) before it is published to a wide group of individuals who may not have a user account on Dataverse. Anyone you send the Private URL to will not have to log into Dataverse to view the dataset.

1. Go to your unpublished dataset
2. Select the “Edit” button
3. Select “Private URL” in the dropdown menu
4. In the pop-up select “Create Private URL”
5. Copy the Private URL which has been created for this dataset and it can now be shared with anyone you wish to have access to view or download files in your unpublished dataset.

To disable a Private URL and to revoke access, follow the same steps as above until step #3 when you return to the popup, click the “Disable Private URL” button.

Dataset Versions¶

Versioning is important for long-term research data management where metadata and/or files are updated over time. It is used to track any metadata or file changes (e.g., by uploading a new file, changing file metadata, adding or editing metadata) once you have published your dataset.

Once you edit your published dataset a new draft version of this dataset will be created. To publish this new version of your dataset, select the “Publish Dataset” button on the top right side of the page. If you were at version 1 of your dataset, depending on the types of changes you had made, you would be asked to publish your draft as either version 1.1 or version 2.0.

Important Note: If you add a file, your dataset will automatically be bumped up to a major version (e.g., if you were at 1.0 you will go to 2.0).

On the Versions tab of a dataset page, there is a versions table that displays the version history of the dataset. You can use the version number links in this table to navigate between the different versions of the dataset, including the unpublished draft version, if you have permission to access it.

There is also a Versions tab on the file page. The versions table for a file displays the same information as the dataset, but the summaries are filtered down to only show the actions related to that file. If a new dataset version were created without any changes to an individual file, that file’s version summary for that dataset version would read “No changes associated with this version”.

Cloud Storage + Computing¶

Dataverse installations can be configured to facilitate cloud-based storage and/or computing (this feature is considered experimental at this time, and some of the kinks are still being worked out). While the default configuration for Dataverse uses a local file system for storing data, a cloud-enabled Dataverse installation can use a Swift object storage database for its data. This allows users to perform computations on data using an integrated cloud computing environment.

Dataset Deaccession¶

To deaccession, go to your published dataset (or add a new one and publish it), click the “Edit” button, and from the dropdown menu select “Deaccession Dataset”. If you have multiple versions of a dataset, you can select here which versions you want to deaccession or choose to deaccession the entire dataset.

You must also include a reason as to why this dataset was deaccessioned. Select the most appropriate reason from the dropdown list of options. If you select “Other”, you must also provide additional information.

Add more information as to why this was deaccessioned in the free-text box. If the dataset has moved to a different repository or site you are encouraged to include a URL (preferably persistent) for users to continue to be able to access this dataset in the future.

If you deaccession the most recently published version of the dataset but not all versions of the dataset, you may then revisit an earlier version and create a new non-deaccessioned draft for the dataset. For example, imagine you have a version 1 and version 2 of a dataset, both published, and you deaccession version 2. You may then edit version 1 of the dataset and a new draft version will be created.

Important Note: A tombstone landing page with the basic citation metadata will always be accessible to the public if they use the persistent URL (Handle or DOI) provided in the citation for that dataset. Users will not be able to see any of the files or additional metadata that were previously available prior to deaccession.

Metadata References¶

Dataverse is committed to using standard-compliant metadata to ensure that Dataverse metadata can be mapped easily to standard metadata schemas and be exported into JSON format (XML for tabular file metadata) for preservation and interoperability.

Detailed below are what metadata schemas we support for Citation and Domain Specific Metadata in Dataverse:

• Citation Metadata: compliant with DDI Lite, DDI 2.5 Codebook, DataCite 3.1, and Dublin Core’s DCMI Metadata Terms (see .tsv version). Language field uses ISO 639-1 controlled vocabulary.
• Geospatial Metadata: compliant with DDI Lite, DDI 2.5 Codebook, DataCite, and Dublin Core (see .tsv version). Country / Nation field uses ISO 3166-1 controlled vocabulary.
• Social Science & Humanities Metadata: compliant with DDI Lite, DDI 2.5 Codebook, and Dublin Core (see .tsv version).
• Astronomy and Astrophysics Metadata : These metadata elements can be mapped/exported to the International Virtual Observatory Alliance’s (IVOA) VOResource Schema format and is based on Virtual Observatory (VO) Discovery and Provenance Metadata (see .tsv version).
• Life Sciences Metadata: based on ISA-Tab Specification, along with controlled vocabulary from subsets of the OBI Ontology and the NCBI Taxonomy for Organisms (see .tsv version).

See also the Dataverse 4.0 Metadata Crosswalk: DDI, DataCite, DC, DCTerms, VO, ISA-Tab document and the Metadata Customization section of the Admin Guide.

Harvesting¶

Metadata Export¶

This part of the Dashboard is simply a reminder message that metadata export happens through the Harvard Dataverse API. See the Metadata Export section and the Native API section of the API Guide for more details.

Users¶

This dashboard tool allows you to search a list of all users of your Dataverse installation. You can remove roles from user accounts and assign or remove superuser status. See the User Administration section for more details.

Your Dataverse as a Metadata Harvester¶

Harvesting is a process of exchanging metadata with other repositories. As a harvesting client, your Dataverse can gather metadata records from remote sources. These can be other Dataverse instances or other archives that support OAI-PMH, the standard harvesting protocol. Harvested metadata records will be indexed and made searchable by your users. Clicking on a harvested dataset in the search results takes the user to the original repository. Harvested datasets cannot be edited in your Dataverse installation.

Harvested records can be kept in sync with the original repository through scheduled incremental updates, daily or weekly. Alternatively, harvests can be run on demand, by the Admin.

Managing Harvesting Clients¶

To start harvesting metadata from a remote OAI repository, you first create and configure a Harvesting Client.

Clients are managed on the “Harvesting Clients” page accessible via the Dashboard. Click on the Add Client button to get started.

The process of creating a new, or editing an existing client, is largely self-explanatory. It is split into logical steps, in a way that allows the user to go back and correct the entries made earlier. The process is interactive and guidance text is provided. For example, the user is required to enter the URL of the remote OAI server. When they click Next, the application will try to establish a connection to the server in order to verify that it is working, and to obtain the information about the sets of metadata records and the metadata formats it supports. The choices offered to the user on the next page will be based on this extra information. If the application fails to establish a connection to the remote archive at the address specified, or if an invalid response is received, the user is given an opportunity to check and correct the URL they entered.

New in Dataverse 4, vs. DVN 3¶

• Note that when creating a client you will need to select an existing local dataverse to host the datasets harvested. In DVN 3, a dedicated “harvesting dataverse” would be created specifically for each remote harvesting source. In Dataverse 4, harvested content can be added to any dataverse. This means that a dataverse can now contain datasets harvested from multiple sources and/or a mix of local and harvested datasets.

• An extra “Archive Type” pull down menu is added to the Create and Edit dialogs. This setting, selected from the choices such as “Dataverse 4”, “DVN, v2-3”, “Generic OAI”, etc. is used to properly format the harvested metadata as they are shown in the search results. It is very important to select the type that best describes this remote server, as failure to do so can result in information missing from the search results, and, a failure to redirect the user to the archival source of the data!

  It is, however, very easy to correct a mistake like this. For example, let’s say you have created a client to harvest from the XYZ Institute and specified the archive type as “Dataverse 4”. You have been able to harvest content, the datasets appear in search result, but clicking on them results in a “Page Not Found” error on the remote site. At which point you realize that the XYZ Institute admins have not yet upgraded to Dataverse 4, still running DVN v3.1.2 instead. All you need to do is go back to the Harvesting Clients page, and change the setting to “DVN, v2-3”. This will fix the redirects without having to re-harvest the datasets.

• Another extra entry, “Archive Description”, is added to the Edit Harvesting Client dialog. This description appears at the bottom of each search result card for a harvested dataset or datafile. By default, this text reads “This Dataset is harvested from our partners. Clicking the link will take you directly to the archival source of the data.” Here it can be customized to be more descriptive, for example, “This Dataset is harvested from our partners at the XYZ Institute…”

Your Dataverse as an OAI server¶

As a harvesting server, your Dataverse can make some of the local dataset metadata available to remote harvesting clients. These can be other Dataverse instances, or any other clients that support OAI-PMH harvesting protocol. Note that the terms “Harvesting Server” and “OAI Server” are being used interchangeably throughout this guide and in the inline help text.

How does it work?¶

Only the published, unrestricted datasets in your Dataverse can be made harvestable. Remote clients normally keep their records in sync through scheduled incremental updates, daily or weekly, thus minimizing the load on your server. Note that it is only the metadata that are harvested. Remote harvesters will generally not attempt to download the data files associated with the harvested datasets.

Harvesting server can be enabled or disabled on the “Harvesting Server” page accessible via the Dashboard. Harvesting server is by default disabled on a brand new, “out of the box” Dataverse.

OAI Sets¶

Once the service is enabled, you define collections of local datasets that will be available to remote harvesters as OAI Sets. Once again, the terms “OAI Set” and “Harvesting Set” are used interchangeably. Sets are defined by search queries. Any such query that finds any number of published, local (non-harvested) datasets can be used to create an OAI set. Sets can overlap local dataverses, and can include as few or as many of your local datasets as you wish. A good way to master the Dataverse search query language is to experiment with the Advanced Search page. We also recommend that you consult the Search API section of the Dataverse User Guide.

Once you have entered the search query and clicked Next, the number of search results found will be shown on the next screen. This way, if you are seeing a number that’s different from what you expected, you can go back and try to re-define the query.

Some useful examples of search queries to define OAI sets:

• A good way to create a set that would include all your local, published datasets is to do so by the Unique Identifier authority registered to your Dataverse, for example:

  dsPersistentId:"doi:1234/"

  Note that double quotes must be used, since the search field value contains the colon symbol!

  Note also that the search terms limiting the results to published and local datasets are added to the query automatically, so you don’t need to worry about that.

• A query to create a set to include the datasets from a specific local dataverse:

  parentId:NNN

  where NNN is the database id of the dataverse object (consult the Dataverse table of the SQL database used by the application to verify the database id).

• A query to find all the dataset by a certain author:

  authorName:YYY

  where YYY is the name.

• Complex queries can be created with multiple logical AND and OR operators. For example,

  (authorName:YYY OR authorName:ZZZ) AND dsPublicationDate:NNNN

• Some further query examples:

  For specific datasets using a persistentID:

  (dsPersistentId:10.5000/ZZYYXX/ OR dsPersistentId:10.5000/XXYYZZ)

  For all datasets within a specific ID authority:

  dsPersistentId:10.5000/XXYYZZ

  For all dataverses with subjects of Astronomy and Astrophysics or Earth and Environmental Sciences:

  (dvSubject:"Astronomy and Astrophysics" OR dvSubject:"Earth and Environmental Sciences")

  For all datasets containing the keyword “censorship”:

  keywordValue:censorship

OAI Set updates¶

Every time a new harvesting set is created, or changes are made to an existing set, the contents of the set are automatically updated - the Dataverse application will find the datasets defined by the query, and attempt to run the metadata export on the ones that haven’t been exported yet. Only the datasets for which the export has completed successfully, and the results cached on the filesystem are included in the OAI sets advertised to the harvesting clients!

This is in contrast to how the sets used to be managed in DVN v.3, where sets had to be exported manually before any such changes had effect.

Important: Note however that changes made to the actual dataset metadata do not automatically trigger any corresponding OAI sets to be updated immediately! For example: let’s say you have created an OAI set defined by the search query authorName:king, that resulted in 43 dataset records. If a new dataset by the same author is added and published, this does not immediately add the extra record to the set! It would simply be too expensive, to refresh all the sets every time any changes to the metadata are made.

The OAI set will however be updated automatically by a scheduled metadata export job that runs every night (at 2AM, by default). This export timer is created and activated automatically every time the application is deployed or restarted. Once again, this is new in Dataverse 4, and unlike DVN v3, where export jobs had to be scheduled and activated by the admin user. See the “Export” section of the Admin guide, for more information on the automated metadata exports.

It is still possible however to make changes like this be immediately reflected in the OAI server, by going to the Harvesting Server page and clicking the “Run Export” icon next to the desired OAI set.

Introduction¶

Before you embark on customizing metadata in Dataverse you should make sure you are aware of the modest amount of customization that is available with the Dataverse web interface. It’s possible to hide fields and make field required by clicking “Edit” at the dataverse level, clicking “General Information” and making adjustments under “Metadata Fields” as described in the context of dataset templates in the Dataverse Management section of the User Guide.

Much more customization of metadata is possible, but this is an advanced topic so feedback on what is written below is very welcome. The possibilities for customization include:

• Editing and adding metadata fields
• Editing and adding instructional text (field label tooltips and text box watermarks)
• Editing and adding controlled vocabularies
• Changing which fields depositors must use in order to save datasets (see also “dataset templates” in the Dataverse Management section of the User Guide.)
• Changing how saved metadata values are displayed in the UI

Generally speaking it is safer to create your own custom metadata block rather than editing metadata blocks that ship with Dataverse, because changes to these blocks may be made in future releases of Dataverse. If you’d like to make improvements to any of the metadata blocks shipped with Dataverse, please open an issue at https://github.com/IQSS/dataverse/issues so it can be discussed before a pull request is made. Please note that the metadata blocks shipped with Dataverse are based on standards (e.g. DDI for social science) and you can learn more about these standards in the Appendix section of the User Guide. If you have developed your own custom metadata block that you think may be of interest to the Dataverse community, please create an issue and consider making a pull request as described in the Version Control section of the Developer Guide.

In Dataverse 4, custom metadata are no longer defined as individual fields, as they were in Dataverse Network (DVN) 3.x, but in metadata blocks. Dataverse 4 ships with a citation metadata block, which includes mandatory fields needed for assigning persistent IDs to datasets, and domain specific metadata blocks. For a complete list, see the Appendix section of the User Guide.

Definitions of these blocks are loaded into a Dataverse installation in tab-separated value (TSV). [1]^, [2] While it is technically possible to define more than one metadata block in a TSV file, it is good organizational practice to define only one in each file.

The metadata block TSVs shipped with Dataverse are in this folder in the Dataverse github repo and the corresponding ResourceBundle property files are here. Human-readable copies are available in this Google Sheets document but they tend to get out of sync with the TSV files, which should be considered authoritative. The Dataverse installation process operates on the TSVs, not the Google spreadsheet.

About the metadata block TSV¶

Here we list and describe the purposes of each section and property of the metadata block TSV.

1. metadataBlock
   • Purpose: Represents the metadata block being defined.
   • Cardinality:
     • 0 or more per Dataverse
     • 1 per Metadata Block definition
2. datasetField
   • Purpose: Each entry represents a metadata field to be defined within a metadata block.
   • Cardinality: 1 or more per metadataBlock
3. controlledVocabulary
   • Purpose: Each entry enumerates an allowed value for a given datasetField.
   • Cardinality: zero or more per datasetField

Each of the three main sections own sets of properties:

Metadata Block Setup¶

Now that you understand the TSV format used for metadata blocks, the next step is to attempt to make improvements to existing metadata blocks or create entirely new metadata blocks. For either task, you should have a Dataverse environment set up for testing where you can drop the database frequently while you make edits to TSV files. Once you have tested your TSV files, you should consider making a pull request to contribute your improvement back to the community.

Reloading a Metadata Block¶

As mentioned above, changes to metadata blocks that ship with Dataverse will be made over time to improve them and release notes will sometimes instruct you to reload an existing metadata block. The syntax for reloading is the same as reloading. Here’s an example with the “citation” metadata block:

curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file --upload-file citation.tsv

Great care must be taken when reloading a metadata block. Matching is done on field names (or identifiers and then names in the case of controlled vocabulary values) so it’s easy to accidentally create duplicate fields.

The ability to reload metadata blocks means that SQL update scripts don’t need to be written for these changes. See also the SQL Upgrade Scripts section of the Dev Guide.

Tips from the Dataverse Community¶

If there are tips that you feel are omitted from this document, please open an issue at https://github.com/IQSS/dataverse/issues and consider making a pull request to make improvements. You can find this document at https://github.com/IQSS/dataverse/blob/develop/doc/sphinx-guides/source/admin/metadatacustomization.rst

Alternatively, you are welcome to request “edit” access to this “Tips for Dataverse metadata blocks from the community” Google doc: https://docs.google.com/document/d/1XpblRw0v0SvV-Bq6njlN96WyHJ7tqG0WWejqBdl7hE0/edit?usp=sharing

The thinking is that the tips can become issues and the issues can eventually be worked on as features to improve the Dataverse metadata system.

Footnotes¶

Automatic Exports¶

Publishing a dataset automatically starts a metadata export job, that will run in the background, asynchronously. Once completed, it will make the dataset metadata exported and cached in all the supported formats:

• Dublin Core
• Data Documentation Initiative (DDI)
• Schema.org JSON-LD
• native JSON (Dataverse-specific)

A scheduled timer job that runs nightly will attempt to export any published datasets that for whatever reason haven’t been exported yet. This timer is activated automatically on the deployment, or restart, of the application. So, again, no need to start or configure it manually. (See the “Application Timers” section of this guide for more information)

Batch exports through the API¶

In addition to the automated exports, a Dataverse admin can start a batch job through the API. The following 2 API calls are provided:

/api/admin/metadata/exportAll

/api/admin/metadata/reExportAll

The former will attempt to export all the published, local (non-harvested) datasets that haven’t been exported yet. The latter will force a re-export of every published, local dataset, regardless of whether it has already been exported or not.

Note, that creating, modifying, or re-exporting an OAI set will also attempt to export all the unexported datasets found in the set.

Export Failures¶

An export batch job, whether started via the API, or by the application timer, will leave a detailed log in your configured logs directory. This is the same location where your main Glassfish server.log is found. The name of the log file is export_[timestamp].log - for example, export_2016-08-23T03-35-23.log. The log will contain the numbers of datasets processed successfully and those for which metadata export failed, with some information on the failures detected. Please attach this log file if you need to contact Dataverse support about metadata export problems.

Downloading Metadata via GUI¶

The Dataset + File Management section of the User Guide explains how end users can download the metadata formats above from the Dataverse GUI.

Downloading Metadata via API¶

The Native API section of the API Guide explains how end users can download the metadata formats above via API.

Dedicated timer server in a Dataverse server cluster¶

When running a Dataverse cluster - i.e. multiple Dataverse application servers talking to the same database - only one of them must act as the dedicated timer server. This is to avoid starting conflicting batch jobs on multiple nodes at the same time.

This does not affect a single-server installation. So you can safely skip this section unless you are running a multi-server cluster.

The following JVM option instructs the application to act as the dedicated timer server:

-Ddataverse.timerServer=true

IMPORTANT: Note that this option is automatically set by the Dataverse installer script. That means that when configuring a multi-server cluster, it will be the responsibility of the installer to remove the option from the domain.xml of every node except the one intended to be the timer server. We also recommend that the following entry in the domain.xml: <ejb-timer-service timer-datasource="jdbc/VDCNetDS"> is changed back to <ejb-timer-service> on all the non-timer server nodes. Similarly, this option is automatically set by the installer script. Changing it back to the default setting on a server that doesn’t need to run the timer will prevent a potential race condition, where multiple servers try to get a lock on the timer database.

Note that for the timer to work, the version of the PostgreSQL JDBC driver your instance is using must match the version of your PostgreSQL database. See the ‘Timer not working’ section of the Troubleshooting guide.

Harvesting Timers¶

These timers are created when scheduled harvesting is enabled by a local admin user (via the “Manage Harvesting Clients” page).

In a multi-node cluster, all these timers will be created on the dedicated timer node (and not necessarily on the node where the harvesting clients were created and/or saved).

A timer will be automatically removed when a harvesting client with an active schedule is deleted, or if the schedule is turned off for an existing client.

Metadata Export Timer¶

This timer is created automatically whenever the application is deployed or restarted. There is no admin user-accessible configuration for this timer.

This timer runs a daily job that tries to export all the local, published datasets that haven’t been exported yet, in all supported metadata formats, and cache the results on the filesystem. (Note that normally an export will happen automatically whenever a dataset is published. This scheduled job is there to catch any datasets for which that export did not succeed, for one reason or another). Also, since this functionality has been added in version 4.5: if you are upgrading from a previous version, none of your datasets are exported yet. So the first time this job runs, it will attempt to export them all.

This daily job will also update all the harvestable OAI sets configured on your server, adding new and/or newly published datasets or marking deaccessioned datasets as “deleted” in the corresponding sets as needed.

This job is automatically scheduled to run at 2AM local time every night. If really necessary, it is possible (for an advanced user) to change that time by directly editing the EJB timer application table in the database.

Known Issues¶

We’ve received several reports of an intermittent issue where the application fails to deploy with the error message “EJB Timer Service is not available.” Please see the Troubleshooting section of this guide for a workaround.

Getting Data In¶

A variety of integrations are oriented toward making it easier for your researchers to deposit data into your installation of Dataverse.

Analysis and Computation¶

Discoverability¶

Integration with DataCite is built in to Dataverse. When datasets are published, metadata is sent to DataCite. You can futher increase the discoverability of your datasets by setting up additional integrations.

Future Integrations¶

The Dataverse roadmap is a good place to see integrations that the core Dataverse team is working on.

The Dev Efforts by the Dataverse Community spreadsheet is the best way to track integrations that are being worked on by the Dataverse community.

Please help us keep this page up to date making a pull request! To get started, see the Documentation section of the Developer Guide.

Update “mapitlink”¶

SQL commands to point a Dataverse installation at different Geoconnect servers:

Geoconnect Production geoconnect.datascience.iq.harvard.edu

update worldmapauth_tokentype set mapitlink = 'https://geoconnect.datascience.iq.harvard.edu/shapefile/map-it', hostname='geoconnect.datascience.iq.harvard.edu' where name = 'GEOCONNECT';

Heroku Test geoconnect-dev.herokuapp.com

update worldmapauth_tokentype set mapitlink = 'https://geoconnect-dev.herokuapp.com/shapefile/map-it', hostname='geoconnect-dev.herokuapp.com' where name = 'GEOCONNECT';

View Current Settings

SELECT * from worldmapauth_tokentype;

Removing Dead Explore Links¶

After a map has been created in WorldMap (assuming all the setup has been done), an “Explore” button will appear next to the name of the file in Dataverse. The “Explore” button should open the map in WorldMap. In rare occasions, the map has been deleted on the WorldMap side such that the “Explore” button goes nowhere, resulting in a dead link, a 404.

Functionality has been added on the Dataverse side to iterate through all the maps Dataverse knows about (stored in the maplayermetadata database table) and to check for the existence of each map in WorldMap. The status code returned from WorldMap (200, 404, etc.) is recorded in Dataverse along with a timestamp of when the check was performed. To perform this check, you can execute the following curl command:

curl -X POST http://localhost:8080/api/admin/geoconnect/mapLayerMetadatas/check

The output above will contain the layerLink being checked as well as the HTTP response status code (200, 404, etc.) in the lastVerifiedStatus field. 200 means OK and 404 means not found. 500 might indicate that the map is only temporarily unavailable. The lastVerifiedStatus and lastVerifiedTime will be persisted to the maplayermetadata database table.

Armed with this information about WorldMap returning a 404 for a map, you may want to delete any record of the map on the Dataverse side so that the “Explore” button goes away (and so that thumbnail files are cleaned up). To accomplish this, use the following curl command, substituting the id of the file:

curl -H "X-Dataverse-key: $API_TOKEN" -X DELETE http://localhost:8080/api/files/{file_id}/map

End users can also run the DELETE command above (if they have permission to edit the dataset) but it’s more likely that the sysadmin reading this guide will run it for them. It is recommended that you add the “check” command above to cron so that Dataverse periodically checks on the status of maps in WorldMap. In addition to the “check all maps” command above, you can also check an individual map with the following curl command, substituting the id of the row from the maplayermetadata table:

curl -X POST http://localhost:8080/api/admin/geoconnect/mapLayerMetadatas/check/{id}

Manage Users¶

The Manage Users table gives the network administrator a list of all user accounts in table form. You can access it by clicking the “Manage Users” button on the Dashboard, which is linked from the header of all Dataverse pages (if you’re logged in as an administrator). It lists username, full name, email address, affiliation, the authentication method they use, the roles their account has been granted, and whether or not they have Superuser status.

Users are listed alphabetically by username. The search bar above the table allows you to search for a specific user. It performs a right-truncated wildcard search of the Username, Name, and Email columns. This means, if you search “baseba” then it will search those three columns for any string of text that begins with “baseba”, e.g. “baseball” or “baseballfan”.

If you would like to assign or remove a user’s Superuser status, then you can do so by checking or unchecking their checkbox under the “Superuser” column.

If you would like to remove all roles/permissions from a user’s account (in the event of their leaving your organization, for example) then you can do so by clicking the “Remove All” button under the Roles column. This will keep the user’s account active, but will revert it to put the account on the level of a default user with default permissions.

curl -H "X-Dataverse-key: $API_TOKEN" http://localhost:8080/api/admin/authenticatedUsers

curl -H "X-Dataverse-key: $API_TOKEN" http://localhost:8080/api/admin/list-users

Confirm Email¶

Dataverse encourages builtin/local users to verify their email address upon signup or email change so that sysadmins can be assured that users can be contacted.

The app will send a standard welcome email with a URL the user can click, which, when activated, will store a lastconfirmed timestamp in the authenticateduser table of the database. Any time this is “null” for a user (immediately after signup and/or changing of their Dataverse email address), their current email on file is considered to not be verified. The link that is sent expires after a time (the default is 24 hours), but this is configurable by a superuser via the :MinutesUntilConfirmEmailTokenExpires config option.

Should users’ URL token expire, they will see a “Verify Email” button on the account information page to send another URL.

Sysadmins can determine which users have verified their email addresses by looking for the presence of the value emailLastConfirmed in the JSON output from listing users (see the “Admin” section of the Native API). As mentioned in the Account Creation + Management section of the User Guide, the email addresses for Shibboleth users are re-confirmed on every login.

Deleting an API Token¶

If an API token is compromised it should be deleted. Users can generate a new one for themselves as explained in the Account Creation + Management section of the User Guide, but you may want to preemptively delete tokens from the database.

Using the API token 7ae33670-be21-491d-a244-008149856437 as an example:

delete from apitoken where tokenstring = '7ae33670-be21-491d-a244-008149856437';

You should expect the output DELETE 1 after issuing the command above.

Dataverses¶

curl -H "X-Dataverse-key: $API_TOKEN" -X POST http://$SERVER/api/dataverses/$id/move/$destination-id

curl -H "X-Dataverse-key: $API_TOKEN" -X PUT http://$SERVER/api/dataverses/$linked-dataverse-alias/link/$linking-dataverse-alias

curl -H "X-Dataverse-key: $API_TOKEN" -X DELETE http://$SERVER/api/dataverses/$linked-dataverse-alias/deleteLink/$linking-dataverse-alias

curl -H "X-Dataverse-key: $API_TOKEN" http://$SERVER/api/admin/dataverse/$dataverse-alias//addRoleAssignmentsToChildren

Datasets¶

curl -H "X-Dataverse-key: $API_TOKEN" -X POST http://$SERVER/api/datasets/$id/move/$alias

curl -H "X-Dataverse-key: $API_TOKEN" -X PUT http://$SERVER/api/datasets/$linked-dataset-id/link/$linking-dataverse-alias

curl -H "X-Dataverse-key: $API_TOKEN" -X DELETE http://$SERVER/api/datasets/$linked-dataset-id/deleteLink/$linking-dataverse-alias

curl -H "X-Dataverse-key: $API_TOKEN" -X POST http://$SERVER/api/admin/$dataset-id/reregisterHDLToPID

curl -H "X-Dataverse-key: $API_TOKEN" -X POST http://$SERVER/api/datasets/$dataset-id/modifyRegistrationMetadata

Full Reindex¶

There are two ways to perform a full reindex of the Dataverse search index. Starting with a “clear” ensures a completely clean index but involves downtime. Reindexing in place doesn’t involve downtime but does not ensure a completely clean index.

Listing IP Groups¶

IP Groups can be listed with the following curl command:

curl http://localhost:8080/api/admin/groups/ip

Creating an IP Group¶

IP Groups must be expressed as ranges in IPv4 or IPv6 format. For illustrative purposes, here is a example of the entire IPv4 and IPv6 range that you can download and edit to have a narrower range to meet your needs. If you need your IP Group to only encompass a single IP address, you must enter that IP address for the “start” and “end” of the range. If you don’t use IPv6 addresses, you can delete that section of the JSON. Please note that the “alias” must be unique if you define multiple IP Groups. You should give it a meaningful “name” since both “alias” and “name” will appear and be searchable in the GUI when your users are assigning roles.

{
  "alias": "ipGroupAll",
  "name": "IP group to match all IPv4 and IPv6 addresses",
  "ranges": [
    [
      "0.0.0.0",
      "255.255.255.255"
    ],
    [
      "::",
      "ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"
    ]
  ]
}

Let’s say you download the example above and edit it to give it a range used by your library, giving it a filename of ipGroup1.json and putting it in the /tmp directory. Next, load it into Dataverse using the following curl command:

curl -X POST -H 'Content-type: application/json' http://localhost:8080/api/admin/groups/ip --upload-file /tmp/ipGroup1.json

Note that you can update a group the same way, as long as you use the same alias.

Listing an IP Group¶

Let’s say you used “ipGroup1” as the alias of the IP Group you created above. To list just that IP Group, you can include the alias in the curl command like this:

curl http://localhost:8080/api/admin/groups/ip/ipGroup1

Deleting an IP Group¶

It is not recommended to delete an IP Group that has been assigned roles. If you want to delete an IP Group, you should first remove its permissions.

To delete an IP Group with an alias of “ipGroup1”, use the curl command below:

curl -X DELETE http://localhost:8080/api/admin/groups/ip/ipGroup1

Operating System Monitoring¶

In production you’ll want to monitor the usual suspects such as CPU, memory, free disk space, etc. There are a variety of tools in this space but we’ll highlight Munin below because it’s relatively easy to set up.

HTTP Traffic¶

HTTP traffic can be monitored from the client side, the server side, or both.

Database Connection Pool used by Glassfish¶

https://github.com/IQSS/dataverse/issues/2595 contains some information on enabling monitoring of Glassfish, which is disabled by default. It’s a TODO to document what to do here if there is sufficient interest.

actionlogrecord¶

There is a database table called actionlogrecord that captures events that may be of interest. See https://github.com/IQSS/dataverse/issues/2729 for more discussion around this table.

EJB Timers¶

Should you be interested in monitoring the EJB timers, this script may be used as an example:

#!/usr/bin/env bash

# example monitoring script for EBJ timers.
# currently assumes that there are two timers
# real monitoring commands should replace the echo statements for production use

r0=`curl -s http://localhost:8080/ejb-timer-service-app/timer`

if [ $? -ne 0 ]; then
	echo "alert - no timer service" # put real alert command here
fi

r1=`echo $r0 | grep -c "There are 2 active persistent timers on this container"`

if [ "1" -ne "$r1" ]; then
	echo "alert - no active timers" # put real alert command here
fi

Glassfish¶

server.log is the main place to look when you encounter problems. Hopefully an error message has been logged. If there’s a stack trace, it may be of interest to developers, especially they can trace line numbers back to a tagged version.

For debugging purposes, you may find it helpful to increase logging levels as mentioned in the Debugging section of the Developer Guide.

Our guides focus on using the command line to manage Glassfish but you might be interested in an admin GUI at http://localhost:4848

Deployment fails, “EJB Timer Service not available”¶

Sometimes the Dataverse application fails to deploy, or Glassfish fails to restart once the application is deployed, with the following error message: “remote failure: Error occurred during deployment: Exception while loading the app : EJB Timer Service is not available. Please see server.log for more details.”

We don’t know what’s causing this issue, but here’s a known workaround:

• Stop Glassfish;
• Remove the generated and osgi-cache directories;
• Delete all the rows from the EJB__TIMER__TBL table in the database;
• Start Glassfish

The shell script below performs the steps above. Note that it may or may not work on your system, so it is provided as an example only, downloadable here. Aside from the configuration values that need to be changed to reflect your environment (the Glassfish directory, name of the database, etc.) the script relies on the database being configured in a certain way for access. (See the comments in the script for more information)

#!/bin/sh

# EBJ timers sometimes cause problems; utility to clear generated directories and database rows

# assumes this script is being run as root, and that the postgres user had passwordless 
# access to the database (local sockets, or appropriate environmental variables).

# will restart glassfish if it's stopped; comment out the `start-domain` command at the end
# if you'd like to avoid that.

# directory where glassfish is installed
GLASSFISH_DIR=/usr/local/glassfish4

# directory within glassfish (defaults)
DV_DIR=${GLASSFISH_DIR}/glassfish/domains/domain1

# name of dataverse database
DV_DB=dvndb

# OS user for the database
DB_USER=postgres

# stop the glassfish domain (generates a warning if glassfish is stopped)
${GLASSFISH_DIR}/bin/asadmin stop-domain

rm -rf ${GLASSFISH_DIR}/${DV_DIR}/generated/
rm -rf ${GLASSFISH_DIR}/${DV_DIR}/osgi-cache/felix

sudo -u ${DB_USER} psql ${DV_DB} -c 'delete from "EJB__TIMER__TBL"';

# restart the domain (also generates a warning if glassfish is stopped)
${GLASSFISH_DIR}/bin/asadmin start-domain

Timer not working¶

Dataverse relies on EJB timers to perform scheduled tasks: harvesting from remote servers, updating the local OAI sets and running metadata exports. (See Dataverse Application Timers for details.) If these scheduled jobs are not running on your server, this may be the result of the incompatibility between the version of PostgreSQL database you are using, and PostgreSQL JDBC driver in use by your instance of Glassfish. The symptoms:

If you are seeing the following in your server.log…

Handling timeout on …

followed by an Exception stack trace with these lines in it:

Internal Exception: java.io.StreamCorruptedException: invalid stream header …

Exception Description: Could not deserialize object from byte array …

… it most likely means that it is the JDBC driver incompatibility that’s preventing the timer from working correctly. Make sure you install the correct version of the driver. For example, if you are running the version 9.3 of PostgreSQL, make sure you have the driver postgresql-9.3-1104.jdbc4.jar in your <GLASSFISH FOLDER>/glassfish/lib directory. Go here to download the correct version of the driver. If you have an older driver in glassfish/lib, make sure to remove it, replace it with the new version and restart Glassfish. (You may need to remove the entire contents of <GLASSFISH FOLDER>/glassfish/domains/domain1/generated before you start Glassfish).

How This Guide is Organized¶

We document the Dataverse API in five sections:

• SWORD API: For depositing data using a standards-based approach rather than the Native API.
• Search API: For searching dataverses, datasets, and files.
• Data Access API: For downloading and subsetting data.
• Native API: For performing most tasks that are possible in the GUI.
• Metrics API: For query statisics about usage of a Dataverse installation.

We use the term “native” to mean that the API is not based on any standard. For this reason, the Search API and Data Access API could also be considered “native” and in the future we may reorganize the API Guide to split the Native API section into “Datasets API”, “Files” API, etc.

Authentication¶

Most Dataverse APIs require the use of an API token. (In code we sometimes call it a “key” because it’s shorter.) Instructions for getting a token are described in the Account Creation + Management section of the User Guide.

There are two ways to pass your API token to Dataverse APIs. The preferred method is to send the token in the X-Dataverse-key HTTP header, as in the following curl example:

curl -H "X-Dataverse-key: 8b955f87-e49a-4462-945c-67d32e391e7e" https://demo.dataverse.org/api/datasets/:persistentId?persistentId=doi:TEST/12345

Throughout this guide you will often see Bash shell envionmental variables being used, like this:

export API_TOKEN='8b955f87-e49a-4462-945c-67d32e391e7e'
curl -H "X-Dataverse-key: $API_TOKEN" https://demo.dataverse.org/api/datasets/:persistentId?persistentId=doi:TEST/12345

The second way to pass your API token is via an extra query parameter called key in the URL like this:

curl "https://demo.dataverse.org/api/datasets/:persistentId?persistentId=doi:TEST/12345&key=$API_TOKEN"

Use of the X-Dataverse-key HTTP header form is preferred because putting the query parameters in URLs often results in them finding their way into web server access logs. Your API token should be kept as secret as your password because it can be used to perform any action as you in the Dataverse application.

Testing¶

Rather than using a production installation of Dataverse, API users are welcome to use http://demo.dataverse.org for testing.

Support¶

If you are using the APIs for an installation of Dataverse hosted by your institution, you may want to reach out to the team that supports it. At the top of the Dataverse installation’s home page, there should be a form you can fill out by clicking the “Support” link.

If you are having trouble with http://demo.dataverse.org or have questions about the APIs, please feel free to reach out to the Dataverse community via https://groups.google.com/forum/#!forum/dataverse-community .

Metrics¶

APIs described in this guide are shipped with the Dataverse software itself. Additional APIs are available if someone at your institution installs the “miniverse” application from https://github.com/IQSS/miniverse and gives it read only access to a production Dataverse database. http://dataverse.org/metrics is powered by miniverse.

About¶

Introduced in Dataverse Network (DVN) 3.6, the SWORD API was formerly known as the “Data Deposit API” and data-deposit/v1 appeared in the URLs. For backwards compatibility these URLs continue to work (with deprecation warnings). Due to architectural changes and security improvements (especially the introduction of API tokens) in Dataverse 4.0, a few backward incompatible changes were necessarily introduced and for this reason the version has been increased to v1.1. For details, see Backward incompatible changes.

Dataverse implements most of SWORDv2, which is specified at http://swordapp.github.io/SWORDv2-Profile/SWORDProfile.html . Please reference the SWORDv2 specification for expected HTTP status codes (i.e. 201, 204, 404, etc.), headers (i.e. “Location”), etc.

As a profile of AtomPub, XML is used throughout SWORD. As of Dataverse 4.0 datasets can also be created via JSON using the “native” API. SWORD is limited to the dozen or so fields listed below in the crosswalk, but the native API allows you to populate all metadata fields available in Dataverse.

Backward incompatible changes¶

For better security than in DVN 3.x, usernames and passwords are no longer accepted. The use of an API token is required.

Differences in Dataverse 4 from DVN 3.x lead to a few minor backward incompatible changes in the Dataverse implementation of SWORD, which are listed below. Old v1 URLs should continue to work but the Service Document will contain a deprecation warning and responses will contain v1.1 URLs. See also Known issues.

• Newly required fields when creating/editing datasets for compliance with the Joint Declaration for Data Citation principles.
  • dcterms:creator (maps to authorName)
  • dcterms:description
• Deaccessioning is no longer supported. An alternative will be developed at https://github.com/IQSS/dataverse/issues/778
• The Service Document will show a single API Terms of Use rather than root level and dataverse level Deposit Terms of Use.

New features as of v1.1¶

• Dataverse 4 supports API tokens and requires them to be used for APIs instead of a username and password. In the curl examples below, you will see curl -u $API_TOKEN: showing that you should send your API token as the username and nothing as the password. For example, curl -u 54b143b5-d001-4254-afc0-a1c0f6a5b5a7:.
• SWORD operations no longer require “admin” permission. In order to use any SWORD operation in DVN 3.x, you had to be an “admin” on a dataverse (the container for your dataset) and similar rules were applied in Dataverse 4.4 and earlier (the EditDataverse permission was required). The SWORD API has now been fully integrated with the Dataverse 4 permission model such that any action you have permission to perform in the GUI or “native” API you are able to perform via SWORD. This means that even a user with a “Contributor” role can operate on datasets via SWORD. Note that users with the “Contributor” role do not have the PublishDataset permission and will not be able publish their datasets via any mechanism, GUI or API.
• Dataverses can be published via SWORD.
• Datasets versions will only be increased to the next minor version (i.e. 1.1) rather than a major version (2.0) if possible. This depends on the nature of the change. Adding or removing a file, for example, requires a major version bump.
• “Author Affiliation” can now be populated with an XML attribute. For example: <dcterms:creator affiliation=”Coffee Bean State University”>Stumptown, Jane</dcterms:creator>
• “Contributor” can now be populated and the “Type” (Editor, Funder, Researcher, etc.) can be specified with an XML attribute. For example: <dcterms:contributor type=”Funder”>CaffeineForAll</dcterms:contributor>
• “License” can now be set with dcterms:license and the possible values are “CC0” and “NONE”. “License” interacts with “Terms of Use” (dcterms:rights) in that if you include dcterms:rights in the XML, the license will be set to “NONE”. If you don’t include dcterms:rights, the license will default to “CC0”. It is invalid to specify “CC0” as a license and also include dcterms:rights; an error will be returned. For backwards compatibility, dcterms:rights is allowed to be blank (i.e. <dcterms:rights></dcterms:rights>) but blank values will not be persisted to the database and the license will be set to “NONE”.
• “Contact E-mail” is automatically populated from dataset owner’s email.
• “Subject” uses our controlled vocabulary list of subjects. This list is in the Citation Metadata of our User Guide > Metadata References. Otherwise, if a term does not match our controlled vocabulary list, it will put any subject terms in “Keyword”. If Subject is empty it is automatically populated with “N/A”.
• Zero-length files are now allowed (but not necessarily encouraged).
• “Depositor” and “Deposit Date” are auto-populated.

curl examples¶

<?xml version="1.0"?>
<entry xmlns="http://www.w3.org/2005/Atom"
       xmlns:dcterms="http://purl.org/dc/terms/">
   <!-- some embedded metadata -->
   <dcterms:title>Roasting at Home</dcterms:title>
   <dcterms:creator>Peets, John</dcterms:creator>
   <dcterms:creator affiliation="Coffee Bean State University">Stumptown, Jane</dcterms:creator>
   <!-- Dataverse controlled vocabulary subject term -->
   <dcterms:subject>Chemistry</dcterms:subject>
   <!-- keywords -->
   <dcterms:subject>coffee</dcterms:subject>
   <dcterms:subject>beverage</dcterms:subject>
   <dcterms:subject>caffeine</dcterms:subject>
   <dcterms:description>Considerations before you start roasting your own coffee at home.</dcterms:description>
   <!-- Producer with financial or admin responsibility of the data -->
   <dcterms:publisher>Coffee Bean State University</dcterms:publisher>
   <dcterms:contributor type="Funder">CaffeineForAll</dcterms:contributor>
   <!-- production date -->
   <dcterms:date>2013-07-11</dcterms:date>
   <!-- kind of data -->
   <dcterms:type>aggregate data</dcterms:type>
   <!-- List of sources of the data collection-->
   <dcterms:source>Stumptown, Jane. 2011. Home Roasting. Coffeemill Press.</dcterms:source>
   <!-- related materials -->
   <dcterms:relation>Peets, John. 2010. Roasting Coffee at the Coffee Shop. Coffeemill Press</dcterms:relation>
   <!-- geographic coverage -->
   <dcterms:coverage>United States</dcterms:coverage>
   <dcterms:coverage>Canada</dcterms:coverage>
   <!-- license and restrictions -->
   <dcterms:license>NONE</dcterms:license>
   <dcterms:rights>Downloader will not use the Materials in any way prohibited by applicable laws.</dcterms:rights>
   <!-- related publications -->
   <dcterms:isReferencedBy holdingsURI="http://dx.doi.org/10.1038/dvn333" agency="DOI" IDNo="10.1038/dvn333">Peets, J., &amp; Stumptown, J. (2013). Roasting at Home. New England Journal of Coffee, 3(1), 22-34.</dcterms:isReferencedBy>
</entry>

Known issues¶

• Deleting a file from a published version (not a draft) creates a draft but doesn’t delete the file: https://github.com/IQSS/dataverse/issues/2464
• The Service Document does not honor groups within groups: https://github.com/IQSS/dataverse/issues/3056
• Should see all the fields filled in for a dataset regardless of what the parent dataverse specifies: https://github.com/IQSS/dataverse/issues/756
• SWORD 2.0 Profile 6.4 “Retrieving the content” has not been implemented: https://github.com/IQSS/dataverse/issues/183
• Deaccessioning via API is not supported (it was in DVN 3.x): https://github.com/IQSS/dataverse/issues/778
• Let file metadata (i.e. description) be specified during zip upload: https://github.com/IQSS/dataverse/issues/723
• SWORD: Display of actual dcterms xml element for equivalent of required field not found: https://github.com/IQSS/dataverse/issues/1019

Bug fixes in v1.1¶

• Fix Abdera ArrayIndexOutOfBoundsException with non-existent atom-entry-study.xml in SWORD jar (upstream ideally) https://github.com/IQSS/dataverse/issues/893
• Sword API: Can’t create study when hidden characters are introduced in atom.xml https://github.com/IQSS/dataverse/issues/894

Client libraries¶

• Python: https://github.com/swordapp/python-client-sword2
• Java: https://github.com/swordapp/JavaClient2.0
• R: https://github.com/IQSS/dataverse-client-r
• Ruby: https://github.com/swordapp/sword2ruby
• PHP: https://github.com/swordapp/swordappv2-php-library

Parameters¶

Basic Search Example¶

https://demo.dataverse.org/api/search?q=trees

{
    "status":"OK",
    "data":{
        "q":"trees",
        "total_count":4,
        "start":0,
        "spelling_alternatives":{
            "trees":"[tree]"
        },
        "items":[
            {
                "name":"Trees",
                "type":"dataverse",
                "url":"https://demo.dataverse.org/dataverse/trees",
                "image_url":"https://demo.dataverse.org/api/access/dvCardImage/7",
                "identifier":"trees",
                "description":"A tree dataverse with some birds",
                "published_at":"2016-05-10T12:53:38Z"
            },
            {
                "name":"Chestnut Trees",
                "type":"dataverse",
                "url":"https://demo.dataverse.org/dataverse/chestnuttrees",
                "image_url":"https://demo.dataverse.org/api/access/dvCardImage/9",
                "identifier":"chestnuttrees",
                "description":"A dataverse with chestnut trees and an oriole",
                "published_at":"2016-05-10T12:52:38Z"
            },
            {
                "name":"trees.png",
                "type":"file",
                "url":"https://demo.dataverse.org/api/access/datafile/12",
                "image_url":"https://demo.dataverse.org/api/access/fileCardImage/12",
                "file_id":"12",
                "description":"",
                "published_at":"2016-05-10T12:53:39Z",
                "file_type":"PNG Image",
                "file_content_type":"image/png",
                "size_in_bytes":8361,
                "md5":"0386269a5acb2c57b4eade587ff4db64",
                "dataset_citation":"Spruce, Sabrina, 2016, \"Spruce Goose\", http://dx.doi.org/10.5072/FK2/NFSEHG, Root Dataverse, V1"
            },
            {
                "name":"Birds",
                "type":"dataverse",
                "url":"https://demo.dataverse.org/dataverse/birds",
                "image_url":"https://demo.dataverse.org/api/access/dvCardImage/2",
                "identifier":"birds",
                "description":"A bird dataverse with some trees",
                "published_at":"2016-05-10T12:57:27Z"
            }
        ],
        "count_in_response":4
    }
}

Advanced Search Example¶

https://demo.dataverse.org/api/search?q=finch&show_relevance=true&show_facets=true&fq=publicationDate:2016&subtree=birds

In this example, show_relevance=true matches per field are shown. Available facets are shown with show_facets=true and of the facets is being used with fq=publicationDate:2016. The search is being narrowed to the dataverse with the identifier “birds” with the parameter subtree=birds.

{
    "status":"OK",
    "data":{
        "q":"finch",
        "total_count":2,
        "start":0,
        "spelling_alternatives":{
        },
        "items":[
            {
                "name":"Finches",
                "type":"dataverse",
                "url":"https://demo.dataverse.org/dataverse/finches",
                "image_url":"https://demo.dataverse.org/api/access/dvCardImage/3",
                "identifier":"finches",
                "description":"A dataverse with finches",
                "published_at":"2016-05-10T12:57:38Z",
                "matches":[
                    {
                        "description":{
                            "snippets":[
                                "A dataverse with <span class=\"search-term-match\">finches</span>"
                            ]
                        }
                    },
                    {
                        "name":{
                            "snippets":[
                                "<span class=\"search-term-match\">Finches</span>"
                            ]
                        }
                    }
                ],
                "score": 3.8500118255615234
            },
            {
                "name":"Darwin's Finches",
                "type":"dataset",
                "url":"http://dx.doi.org/10.5072/FK2/G2VPE7",
                "image_url":"https://demo.dataverse.org/api/access/dsCardImage/2",
                "global_id":"doi:10.5072/FK2/G2VPE7",
                "description": "Darwin's finches (also known as the Galápagos finches) are a group of about fifteen species of passerine birds.",
                "published_at":"2016-05-10T12:57:45Z",
                "citationHtml":"Finch, Fiona, 2016, \"Darwin's Finches\", <a href=\"http://dx.doi.org/10.5072/FK2/G2VPE7\" target=\"_blank\">http://dx.doi.org/10.5072/FK2/G2VPE7</a>, Root Dataverse, V1",
                "citation":"Finch, Fiona, 2016, \"Darwin's Finches\", http://dx.doi.org/10.5072/FK2/G2VPE7, Root Dataverse, V1",
                "matches":[
                    {
                        "authorName":{
                            "snippets":[
                                "<span class=\"search-term-match\">Finch</span>, Fiona"
                            ]
                        }
                    },
                    {
                        "dsDescriptionValue":{
                            "snippets":[
                                "Darwin's <span class=\"search-term-match\">finches</span> (also known as the Galápagos <span class=\"search-term-match\">finches</span>) are a group of about fifteen species"
                            ]
                        }
                    },
                    {
                        "title":{
                            "snippets":[
                                "Darwin's <span class=\"search-term-match\">Finches</span>"
                            ]
                        }
                    }
                ],
                "score": 1.5033848285675049,
                "authors":[
                    "Finch, Fiona"
                ]
            }
        ],
        "facets":[
            {
                "subject_ss":{
                    "friendly":"Subject",
                    "labels":[
                        {
                            "Medicine, Health and Life Sciences":2
                        }
                    ]
                },
                "authorName_ss": {
                    "friendly":"Author Name",
                    "labels": [
                        {
                            "Finch, Fiona":1
                        }
                    ]
                },
                "publicationDate":{
                    "friendly":"Publication Date",
                    "labels":[
                        {
                            "2016":2
                        }
                    ]
                }
            }
        ],
        "count_in_response":2
    }
}

Iteration¶

Be default, up to 10 results are returned with every request (though this can be increased with the per_page parameter). To iterate through many results, increase the start parameter on each iteration until you reach the total_count in the response. An example in Python is below.

#!/usr/bin/env python
import urllib2
import json
base = 'https://demo.dataverse.org'
rows = 10
start = 0
page = 1
condition = True # emulate do-while
while (condition):
    url = base + '/api/search?q=*' + "&start=" + str(start)
    data = json.load(urllib2.urlopen(url))
    total = data['data']['total_count']
    print "=== Page", page, "==="
    print "start:", start, " total:", total
    for i in data['data']['items']:
        print "- ", i['name'], "(" + i['type'] + ")"
    start = start + rows
    page += 1
    condition = start < total

Output from iteration example

=== Page 1 ===
start: 0  total: 12
-  Spruce Goose (dataset)
-  trees.png (file)
-  Spruce (dataverse)
-  Trees (dataverse)
-  Darwin's Finches (dataset)
-  Finches (dataverse)
-  Birds (dataverse)
-  Rings of Conifers (dataset)
-  Chestnut Trees (dataverse)
-  Sparrows (dataverse)
=== Page 2 ===
start: 10  total: 12
-  Chestnut Sparrows (dataverse)
-  Wrens (dataverse)

Basic File Access¶

Basic access URI:

/api/access/datafile/$id

GET http://$SERVER/api/access/datafile/:persistentId/?persistentId=doi:10.5072/FK2/J8SJZB

Multiple File (“bundle”) download¶

/api/access/datafiles/$id1,$id2,...$idN

Returns the files listed, zipped.

“All Formats” bundled access for Tabular Files.¶

/api/access/datafile/bundle/$id

This is convenience packaging method is available for tabular data files. It returns a zipped bundle that contains the data in the following formats:

• Tab-delimited;
• “Saved Original”, the proprietary (SPSS, Stata, R, etc.) file from which the tabular data was ingested;
• Generated R Data frame (unless the “original” above was in R);
• Data (Variable) metadata record, in DDI XML;
• File citation, in Endnote and RIS formats.

Data Variable Metadata Access¶

These methods are only available for tabular data files. (i.e., data files with associated data table and variable objects).

/api/access/datafile/$id/metadata/ddi

In its basic form the verb above returns a DDI fragment that describes the file and the data variables in it.

The DDI returned will only have two top-level sections:

• a single fileDscr, with the basic file information plus the numbers of variables and observations and the UNF of the file.
• a single dataDscr section, with one var section for each variable.

Example:

http://localhost:8080/api/access/datafile/6/metadata/ddi

<codeBook version="2.0">
   <fileDscr ID="f6">
      <fileTxt>
         <fileName>_73084.tab</fileName>
         <dimensns>
            <caseQnty>3</caseQnty>
            <varQnty>2</varQnty>
         </dimensns>
         <fileType>text/tab-separated-values</fileType>
      </fileTxt>
      <notes level="file" type="VDC:UNF" subject="Universal Numeric Fingerprint">UNF:6:zChnyI3fjwNP+6qW0VryVQ==</notes>
   </fileDscr>
   <dataDscr>
      <var ID="v1" name="id" intrvl="discrete">
         <location fileid="f6"/>
         <labl level="variable">Personen-ID</labl>
         <sumStat type="mean">2.0</sumStat>
         <sumStat type="mode">.</sumStat>
         <sumStat type="medn">2.0</sumStat>
         <sumStat type="stdev">1.0</sumStat>
         <sumStat type="min">1.0</sumStat>
         <sumStat type="vald">3.0</sumStat>
         <sumStat type="invd">0.0</sumStat>
         <sumStat type="max">3.0</sumStat>
         <varFormat type="numeric"/>
         <notes subject="Universal Numeric Fingerprint" level="variable" type="VDC:UNF">UNF:6:AvELPR5QTaBbnq6S22Msow==</notes>
      </var>
      <var ID="v3" name="sex" intrvl="discrete">
         <location fileid="f6"/>
         <labl level="variable">Geschlecht</labl>
         <sumStat type="mean">1.3333333333333333</sumStat>
         <sumStat type="max">2.0</sumStat>
         <sumStat type="vald">3.0</sumStat>
         <sumStat type="mode">.</sumStat>
         <sumStat type="stdev">0.5773502691896257</sumStat>
         <sumStat type="invd">0.0</sumStat>
         <sumStat type="medn">1.0</sumStat>
         <sumStat type="min">1.0</sumStat>
         <catgry>
            <catValu>1</catValu>
            <labl level="category">Mann</labl>
         </catgry>
         <catgry>
            <catValu>2</catValu>
            <labl level="category">Frau</labl>
         </catgry>
         <varFormat type="numeric"/>
         <notes subject="Universal Numeric Fingerprint" level="variable" type="VDC:UNF">UNF:6:XqQaMwOA63taX1YyBzTZYQ==</notes>
      </var>
   </dataDscr>
</codeBook>

More information on DDI is available in the Tabular Data, Representation, Storage and Ingest section of the User Guide.

Advanced options/Parameters:

It is possible to request only specific subsets of, rather than the full file-level DDI record. This can be a useful optimization, in cases such as when an application needs to look up a single variable; especially with data files with large numbers of variables. See variables=123,127 in the example above.

Preprocessed Data¶

/api/access/datafile/$id/metadata/preprocessed

This method provides the “preprocessed data” - a summary record that describes the values of the data vectors in the tabular file, in JSON. These metadata values are used by TwoRavens, the companion data exploration utility of the Dataverse application. Please note that this format might change in the future.

Authentication and Authorization¶

Data Access API supports both session- and API key-based authentication.

If a session is available, and it is already associated with an authenticated user, it will be used for access authorization. If not, or if the user in question is not authorized to access the requested object, an attempt will be made to authorize based on an API key, if supplied. All of the API verbs above support the key parameter key=... as well as the newer X-Dataverse-key header. For more details, see “Authentication” in the Introduction section.

Access Requests and Processing¶

All of the following endpoints take the persistent identifier as a parameter in place of ‘id’.

curl -H "X-Dataverse-key:$API_TOKEN" -X PUT -d true http://$SERVER/api/access/{id}/allowAccessRequest

curl -H "X-Dataverse-key:$API_TOKEN" -X PUT -d true http://$SERVER/api/access/:persistentId/allowAccessRequest?persistentId={pid}

curl -H "X-Dataverse-key:$API_TOKEN" -X PUT http://$SERVER/api/access/datafile/{id}/requestAccess

curl -H "X-Dataverse-key:$API_TOKEN" -X PUT http://$SERVER/api/access/datafile/{id}/grantAccess/{@userIdentifier}

curl -H "X-Dataverse-key:$API_TOKEN" -X PUT http://$SERVER/api/access/datafile/{id}/rejectAccess/{@userIdentifier}

curl -H "X-Dataverse-key:$API_TOKEN" -X DELETE http://$SERVER/api/access/datafile/{id}/revokeAccess/{@userIdentifier}

curl -H "X-Dataverse-key:$API_TOKEN" -X GET http://$SERVER/api/access/datafile/{id}/listRequests

Dataverses¶

POST http://$SERVER/api/dataverses/$id?key=$apiKey

{
  "name": "Scientific Research",
  "alias": "science",
  "dataverseContacts": [
    {
      "contactEmail": "pi@example.edu"
    },
    {
      "contactEmail": "student@example.edu"
    }
  ],
  "affiliation": "Scientific Research University",
  "description": "We do all the science.",
  "dataverseType": "LABORATORY"
}

GET http://$SERVER/api/dataverses/$id

DELETE http://$SERVER/api/dataverses/$id?key=$apiKey

GET http://$SERVER/api/dataverses/$id/contents

GET http://$SERVER/api/dataverses/$id/roles?key=$apiKey

GET http://$SERVER/api/dataverses/$id/facets?key=$apiKey

POST http://$SERVER/api/dataverses/$id/roles?key=$apiKey

{
  "alias": "sys1",
  "name": “Restricted System Role”,
  "description": “A person who may only add datasets.”,
  "permissions": [
    "AddDataset"
  ]
}

GET http://$SERVER/api/dataverses/$id/assignments?key=$apiKey

PUT http://$SERVER/api/dataverses/$id/defaultContributorRole/$roleAlias?key=$apiKey

POST http://$SERVER/api/dataverses/$id/assignments?key=$apiKey

{
  "assignee": "@uma",
  "role": "curator"
}

DELETE http://$SERVER/api/dataverses/$id/assignments/$id?key=$apiKey

GET http://$SERVER/api/dataverses/$id/metadatablocks?key=$apiKey

GET http://$SERVER/api/dataverses/$id/metadatablocks/isRoot?key=$apiKey

PUT http://$SERVER/api/dataverses/$id/metadatablocks/isRoot?key=$apiKey

curl -H "X-Dataverse-key: $API_TOKEN" -X POST $SERVER_URL/api/dataverses/$DV_ALIAS/datasets --upload-file dataset-finch1.json

curl -H "X-Dataverse-key: $API_TOKEN" -X POST $SERVER_URL/api/dataverses/$DV_ALIAS/datasets/:import?pid=$PERSISTENT_IDENTIFIER&release=yes --upload-file dataset.json

{
  "datasetVersion": {
    "termsOfUse": "CC0 Waiver",
    "license": "CC0",
    "protocol":"doi",
    "authority":"10.502",
    "identifier":"ZZ7/MOSEISLEYDB94",
    "metadataBlocks": {
      "citation": {
        "fields": [
          {
            "typeName": "title",
            "multiple": false,
            "value": "Imported dataset with package files No. 3",
            "typeClass": "primitive"
          },
          {
            "typeName": "productionDate",
            "multiple": false,
            "value": "2011-02-23",
            "typeClass": "primitive"
          },
          {
            "typeName": "dsDescription",
            "multiple": true,
            "value": [
              {
                "dsDescriptionValue": {
                  "typeName": "dsDescriptionValue",
                  "multiple": false,
                  "value": "Native Dataset",
                  "typeClass": "primitive"
                }
              }
            ],
            "typeClass": "compound"
          },
          {
            "typeName": "subject",
            "multiple": true,
            "value": [
              "Medicine, Health and Life Sciences"
            ],
            "typeClass": "controlledVocabulary"
          },
          {
            "typeName": "author",
            "multiple": true,
            "value": [
              {
                "authorAffiliation": {
                  "typeName": "authorAffiliation",
                  "multiple": false,
                  "value": "LibraScholar Medical School",
                  "typeClass": "primitive"
                },
                "authorName": {
                  "typeName": "authorName",
                  "multiple": false,
                  "value": "Doc, Bob",
                  "typeClass": "primitive"
                }
              },
              {
                "authorAffiliation": {
                  "typeName": "authorAffiliation",
                  "multiple": false,
                  "value": "LibraScholar Medical School",
                  "typeClass": "primitive"
                },
                "authorName": {
                  "typeName": "authorName",
                  "multiple": false,
                  "value": "Prof, Arthur",
                  "typeClass": "primitive"
                }
              }
            ],
            "typeClass": "compound"
          },
          {
            "typeName": "depositor",
            "multiple": false,
            "value": "Prof, Arthur",
            "typeClass": "primitive"
          },
          {
            "typeName": "datasetContact",
            "multiple": true,
            "value": [
              {
                "datasetContactEmail": {
                  "typeName": "datasetContactEmail",
                  "multiple": false,
                  "value": "aprof@mailinator.com",
                  "typeClass": "primitive"
                }
              }
            ],
            "typeClass": "compound"
          }
        ],
        "displayName": "Citation Metadata"
      }
    },
    "files": [
      {
        "description": "",
        "label": "pub",
        "restricted": false,
        "version": 1,
        "datasetVersionId": 1,
        "dataFile": {
          "id": 4,
          "filename": "pub",
          "contentType": "application/vnd.dataverse.file-package",
          "filesize": 1698795873,
          "description": "",
          "storageIdentifier": "162017e5ad5-ee2a2b17fee9",
          "originalFormatLabel": "UNKNOWN",
          "rootDataFileId": -1,
          "checksum": {
            "type": "SHA-1",
            "value": "54bc7ddb096a490474bd8cc90cbed1c96730f350"
          }
        }
      }
    ]
  }
}

curl -H "X-Dataverse-key: $API_TOKEN" -X POST $SERVER_URL/api/dataverses/$DV_ALIAS/datasets/:importddi?pid=$PERSISTENT_IDENTIFIER&release=yes --upload-file ddi_dataset.xml

POST http://$SERVER/api/dataverses/$identifier/actions/:publish?key=$apiKey

Datasets¶

Note Creation of new datasets is done with a POST onto dataverses. See Dataverses section.

Note In all commands below, dataset versions can be referred to as:

• :draft the draft version, if any
• :latest either a draft (if exists) or the latest published version.
• :latest-published the latest published version
• x.y a specific version, where x is the major version number and y is the minor version number.
• x same as x.0

GET http://$SERVER/api/datasets/:persistentId/?persistentId=doi:10.5072/FK2/J8SJZB

GET http://$SERVER/api/datasets/:persistentId/versions/:draft?persistentId=doi:10.5072/FK2/J8SJZB

GET http://$SERVER/api/datasets/$id?key=$apiKey

DELETE http://$SERVER/api/datasets/$id?key=$apiKey

GET http://$SERVER/api/datasets/$id/versions?key=$apiKey

GET http://$SERVER/api/datasets/$id/versions/$versionNumber?key=$apiKey

GET http://$SERVER/api/datasets/export?exporter=ddi&persistentId=$persistentId

GET http://$SERVER/api/datasets/$id/versions/$versionId/files?key=$apiKey

GET http://$SERVER/api/datasets/$id/versions/$versionId/metadata?key=$apiKey

GET http://$SERVER/api/datasets/$id/versions/$versionId/metadata/$blockname?key=$apiKey

curl -H "X-Dataverse-key: $API_TOKEN" -X PUT $SERVER_URL/api/datasets/:persistentId/versions/:draft?persistentId=$PID --upload-file dataset-update-metadata.json

curl -H "X-Dataverse-key: $API_TOKEN" $SERVER_URL/api/datasets/:persistentId/versions/:latest?persistentId=$PID | jq '.data | {metadataBlocks: .metadataBlocks}' > dataset-update-metadata.json

vi dataset-update-metadata.json

curl -H "X-Dataverse-key: $API_TOKEN" -X PUT $SERVER_URL/api/datasets/:persistentId/editMetadata/?persistentId=$PID --upload-file dataset-add-metadata.json

curl -H "X-Dataverse-key: $API_TOKEN" -X PUT $SERVER_URL/api/datasets/:persistentId/editMetadata?persistentId=$PID&replace=true --upload-file dataset-update-metadata.json

curl -H "X-Dataverse-key: $API_TOKEN" -X PUT $SERVER_URL/api/datasets/:persistentId/deleteMetadata/?persistentId=$PID --upload-file dataset-delete-author-metadata.json

POST http://$SERVER/api/datasets/$id/actions/:publish?type=$type&key=$apiKey

DELETE http://$SERVER/api/datasets/$id/versions/:draft?key=$apiKey

PUT http://$SERVER/api/datasets/$id/citationdate?key=$apiKey

DELETE http://$SERVER/api/datasets/$id/citationdate?key=$apiKey

GET http://$SERVER/api/datasets/$id/assignments?key=$apiKey

POST http://$SERVER/api/datasets/$id/privateUrl?key=$apiKey

GET http://$SERVER/api/datasets/$id/privateUrl?key=$apiKey

DELETE http://$SERVER/api/datasets/$id/privateUrl?key=$apiKey

POST http://$SERVER/api/datasets/$id/add?key=$apiKey

curl -H "X-Dataverse-key:$API_TOKEN" -X POST -F 'file=@data.tsv' -F 'jsonData={"description":"My description.","categories":["Data"], "restrict":"true"}' "https://example.dataverse.edu/api/datasets/:persistentId/add?persistentId=$PERSISTENT_ID"

from datetime import datetime
import json
import requests  # http://docs.python-requests.org/en/master/

# --------------------------------------------------
# Update the 4 params below to run this code
# --------------------------------------------------
dataverse_server = 'https://your dataverse server' # no trailing slash
api_key = 'api key'
dataset_id = 1  # database id of the dataset
persistentId = 'doi:10.5072/FK2/6XACVA' # doi or hdl of the dataset

# --------------------------------------------------
# Prepare "file"
# --------------------------------------------------
file_content = 'content: %s' % datetime.now()
files = {'file': ('sample_file.txt', file_content)}

# --------------------------------------------------
# Using a "jsonData" parameter, add optional description + file tags
# --------------------------------------------------
params = dict(description='Blue skies!',
            categories=['Lily', 'Rosemary', 'Jack of Hearts'])

params_as_json_string = json.dumps(params)

payload = dict(jsonData=params_as_json_string)

# --------------------------------------------------
# Add file using the Dataset's id
# --------------------------------------------------
url_dataset_id = '%s/api/datasets/%s/add?key=%s' % (dataverse_server, dataset_id, api_key)

# -------------------
# Make the request
# -------------------
print '-' * 40
print 'making request: %s' % url_dataset_id
r = requests.post(url_dataset_id, data=payload, files=files)

# -------------------
# Print the response
# -------------------
print '-' * 40
print r.json()
print r.status_code

# --------------------------------------------------
# Add file using the Dataset's persistentId (e.g. doi, hdl, etc)
# --------------------------------------------------
url_persistent_id = '%s/api/datasets/:persistentId/add?persistentId=%s&key=%s' % (dataverse_server, persistentId, api_key)

# -------------------
# Update the file content to avoid a duplicate file error
# -------------------
file_content = 'content2: %s' % datetime.now()
files = {'file': ('sample_file2.txt', file_content)}


# -------------------
# Make the request
# -------------------
print '-' * 40
print 'making request: %s' % url_persistent_id
r = requests.post(url_persistent_id, data=payload, files=files)

# -------------------
# Print the response
# -------------------
print '-' * 40
print r.json()
print r.status_code

curl -H "X-Dataverse-key: $API_TOKEN" -X POST "$SERVER_URL/api/datasets/:persistentId/submitForReview?persistentId=$DOI_OR_HANDLE_OF_DATASET"

{
  "reasonForReturn": "You forgot to upload any files."
}

curl -H "Content-type:application/json" -d @reason-for-return.json -H "X-Dataverse-key: $API_TOKEN" -X POST "$SERVER_URL/api/datasets/:persistentId/returnToAuthor?persistentId=$DOI_OR_HANDLE_OF_DATASET"

curl -H "X-Dataverse-key: $API_TOKEN" -X PUT http://$SERVER/api/datasets/$linked-dataset-id/link/$linking-dataverse-alias

curl "$SERVER_URL/api/datasets/{database_id}/locks

curl "$SERVER_URL/api/datasets/{database_id}/locks?type={lock_type}

{"status":"OK","data":
    [
            {
             "lockType":"Ingest",
             "date":"Fri Aug 17 15:05:51 EDT 2018",
             "user":"dataverseAdmin"
            },
            {
             "lockType":"Workflow",
             "date":"Fri Aug 17 15:02:00 EDT 2018",
             "user":"dataverseAdmin"
            }
    ]
}

POST /api/datasets/{database_id}/lock/{lock_type}

curl -X POST "$SERVER_URL/api/datasets/1234/lock/Ingest?key=$ADMIN_API_TOKEN"
or
curl -X POST -H "X-Dataverse-key: $ADMIN_API_TOKEN" "$SERVER_URL/api/datasets/:persistentId/lock/Ingest?persistentId=$DOI_OR_HANDLE_OF_DATASET"

DELETE /api/datasets/{database_id}/locks

DELETE /api/datasets/{database_id}/locks?type={lock_type}

curl -X DELETE -H "X-Dataverse-key: $ADMIN_API_TOKEN" "$SERVER_URL/api/datasets/1234/locks?type=pidRegister"

Files¶

curl -H "X-Dataverse-key:$API_TOKEN" -X PUT -d true http://$SERVER/api/files/{id}/restrict

curl -H "X-Dataverse-key:$API_TOKEN" -X PUT -d true http://$SERVER/api/files/:persistentId/restrict?persistentId={pid}

curl -H "X-Dataverse-key:$API_TOKEN" -X POST -F 'file=@data.tsv' \
-F 'jsonData={"description":"My description.","categories":["Data"],"forceReplace":false}'\
"https://demo.dataverse.org/api/files/$FILE_ID/replace"

from datetime import datetime
import json
import requests  # http://docs.python-requests.org/en/master/

# --------------------------------------------------
# Update params below to run code
# --------------------------------------------------
dataverse_server = 'http://127.0.0.1:8080' # no trailing slash
api_key = 'some key'
file_id = 1401  # id of the file to replace

# --------------------------------------------------
# Prepare replacement "file"
# --------------------------------------------------
file_content = 'content: %s' % datetime.now()
files = {'file': ('replacement_file.txt', file_content)}

# --------------------------------------------------
# Using a "jsonData" parameter, add optional description + file tags
# --------------------------------------------------
params = dict(description='Sunset',
            categories=['One', 'More', 'Cup of Coffee'])

# -------------------
# IMPORTANT: If the mimetype of the replacement file differs
#   from the origina file, the replace will fail
#
#  e.g. if you try to replace a ".csv" with a ".png" or something similar
#
#  You can override this with a "forceReplace" parameter
# -------------------
params['forceReplace'] = True


params_as_json_string = json.dumps(params)

payload = dict(jsonData=params_as_json_string)

print 'payload', payload
# --------------------------------------------------
# Replace file using the id of the file to replace
# --------------------------------------------------
url_replace = '%s/api/v1/files/%s/replace?key=%s' % (dataverse_server, file_id, api_key)

# -------------------
# Make the request
# -------------------
print '-' * 40
print 'making request: %s' % url_replace
r = requests.post(url_replace, data=payload, files=files)

# -------------------
# Print the response
# -------------------
print '-' * 40
print r.json()
print r.status_code

POST http://$SERVER/api/files/{id}/uningest?key={apiKey}

POST http://$SERVER/api/files/{id}/reingest?key={apiKey}

GET http://$SERVER/api/files/{id}/prov-json?key=$apiKey

GET http://$SERVER/api/files/{id}/prov-freeform?key=$apiKey

POST http://$SERVER/api/files/{id}/prov-json?key=$apiKey&entityName=$entity -H "Content-type:application/json" --upload-file $filePath

POST http://$SERVER/api/files/{id}/prov-freeform?key=$apiKey -H "Content-type:application/json" --upload-file $filePath

DELETE http://$SERVER/api/files/{id}/prov-json?key=$apiKey

GET http://$SERVER/api/admin/datafiles/integrity/fixmissingoriginalsizes{?limit=N}

Builtin Users¶

Builtin users are known as “Username/Email and Password” users in the Account Creation + Management of the User Guide. Dataverse stores a password (encrypted, of course) for these users, which differs from “remote” users such as Shibboleth or OAuth users where the password is stored elsewhere. See also “Auth Modes: Local vs. Remote vs. Both” in the Configuration section of the Installation Guide. It’s a valid configuration of Dataverse to not use builtin users at all.

{
  "firstName": "Lisa",
  "lastName": "Simpson",
  "userName": "lsimpson",
  "affiliation": "Springfield",
  "position": "Student",
  "email": "lsimpson@mailinator.com"
}

curl -d @user-add.json -H "Content-type:application/json" "$SERVER_URL/api/builtin-users?password=$NEWUSER_PASSWORD&key=$BUILTIN_USERS_KEY"

Roles¶

POST http://$SERVER/api/roles?dvo=$dataverseIdtf&key=$apiKey

GET http://$SERVER/api/roles/$id

DELETE http://$SERVER/api/roles/$id

Explicit Groups¶