Link Search Menu Expand Document

Setup of NFDI-MatWerk Metadata Repository

Instructions on how the NFDI-MatWerk Metadata Repository can be installed locally with basic settings are provided here. A prerequisite for this is an accessible database which needs to be created beforehand.

The NFDI-MatWerk Metadata Repository is based on the MetaStore. All neccessary dependencies like java and git needs to be installed. Follow the instructions here to check if they are installed.

Create Directories for Saving the Data

Create a sub-directory called metastore under a directory called data. You are free to choose other names, but make sure to use the right names and paths always.

For Unix based systems (eg. Mac, Linux) run the following commands from the command line:

mkdir -p /data/metastore

For windows, the same can be done by removing the “-p” from the command above.

If you are using a server and have created an unprivileded user called “matwerk” in the beginning, you can use the following commands to give owner rights of the repository to this user and to switch to this user. For this, run the following commands from the command prompt:

chown matwerk /data/base-repo
su - matwerk

Clone MetaStore

Next create a folder dem-services/metastore to store the configuration files of the metadata repository. Thereafter, create another folder called software. If you have already installed a lab instance of the NFDI-MatWerk Data Repository (based on the base-repo), then the folders dem-services and software would already have been created. In this case it would be enough to just make the new folder metastore in the folder dem-services. Clone MetaStore in the folder software. For executing these steps, run the following commands from the command prompt for unix based systems and for windows run the same commands without using the “-p”:

mkdir -p dem-services/metastore
mkdir software
cd software
git clone https://github.com/kit-data-manager/metastore2

The output would look like this:

Cloning into 'metastore2'...
remote: Enumerating objects: 10813, done.
remote: Counting objects: 100% (2020/2020), done.
remote: Compressing objects: 100% (944/944), done.
remote: Total 10813 (delta 1087), reused 1765 (delta 901), pack-reused 8793
Receiving objects: 100% (10813/10813), 6.84 MiB | 3.49 MiB/s, done.
Resolving deltas: 100% (5417/5417), done.

Build MetaStore

Now, change to the cloned metastore2 directory to build the MetaStore service using the shell script build.sh which is already provided in the metastore2 folder. To execute the shell script build.sh, first it needs to be set as an executable in Mac and Unix. This is done using the chmod 755 command. In Windows systems, no additional steps are needed as it would be treated already as an executable based on its extension .sh.

cd metastore2
chmod 755 build.sh

Next build the Metadata Repository (MetaStore) in the folder dem-services/metastore which was created before, by using the following command, but modify the path /home/matwerk/dem-services/metastore/ in the command based on the actual location where the folder dem-services is located.

./build.sh /home/matwerk/dem-services/metastore/

The output would look like this after succesful installation: 

---------------------------------------------------------------------------
Build microservice of metastore2 at '/home/matwerk/dem-services/metastore'
[...]
BUILD SUCCESSFUL in 46s
15 actionable tasks: 14 executed, 1 up-to-date
Copy configuration to 'home/matwerk/dem-services/metastore'...
Copy jar file to 'home/matwerk/dem-services/metastore'...
Create config directory
Create lib directory
---------------------------------------------------------------------------
Create run script ...
---------------------------------------------------------------------------
.
---------------------------------------------------------------------------
Now you can start the service by calling 'home/matwerk/dem-services/metastore/run.sh'
---------------------------------------------------------------------------

Configure MetaStore

The NFDI-MatWerk Metadata Repository based on MetaStore needs to be connected to the Postgres database. For this, configure the Metadata Repository to work with the database and set the paths for schema and metadata. The configurations are to be copied into the application.properties file, located in the dem-services/metastore folder.

  1. Edit the application.properties file to link MetaStore to the database and to change other settings. In the example below, the nano editor is used. Any other commandline file editor can also be used. 
     cd ~/dem-services/metastore
 cp application.properties config/
 nano config/application.properties
  2. In the application.properties file, first go to the section “Setup path for schema and metadata” and change the paths for schema and metadata to file:///data/metastore/schema and file:///data/metastore/metadata, respectively. 
     ###############################################################################
 # Setup paths for schema and metadata
 ###############################################################################
 metastore.schema.schemaFolder:file:///data/metastore/schema
 metastore.metadata.metadataFolder:file:///data/metastore/metadata
 ###############################################################################
  3. Next go to the section “Database” and modify the contents by copying and pasting the lines below: 
     ###############################################################################
 # Database
 ###############################################################################
 #spring.datasource.driver-class-name: org.h2.Driver
 #spring.datasource.url:  jdbc:h2:file:/tmp/metastore2/database;MODE=LEGACY;NON_KEYWORDS=VALUE
 #spring.datasource.username: sa
 #spring.datasource.password: sa
 #spring.jpa.hibernate.ddl-auto: update
 spring.datasource.platform: postgres
 spring.jpa.database: POSTGRESQL
 spring.jpa.database-platform: org.hibernate.dialect.PostgreSQLDialect
 spring.datasource.driverClassName: org.postgresql.Driver
 spring.datasource.url: jdbc:postgresql://localhost:5432/metastore
 spring.datasource.username: matwerk_db_user
 spring.datasource.password: DB_USER_ADMIN_PASSWORD
 spring.jpa.hibernate.ddl-auto: update
  4. Now, press Ctrl+O and press enter, followed by Ctrl+X, to save the file and exit the nano editor.

Test locally installed MetaStore

To test the local installation, start MetaStore by typing home/matwerk/dem-services/metastore/run.sh in the command prompt and press enter. Replace this path with the path shown after installation of MetaStore to start the service, if this is different.

If MetaStore started successfully, the output would look as shown below:

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v3.1.0)

Spring is running!

The built-in web front end of MetaStore by clicking on http://localhost:8040 or by copying and pasting the address in a browser. Alternativly, a REST-Call can be made in another terminal window other than the one in which the message “Spring is running” is shown.

curl http://localhost:8040/api/v1/schemas
[]

For the first time, the response is naturally empty as nothing is ingested yet in the Metadata Repository.

It is preferable to have SSL and to not directly specify the port or use stand ports due to firewall restrictions. The section Advanced Setup addresses these issues and provides tips for creating “clean” URLs and to use the frontend-collection as a web GUI for the services.

Stopping the MetaStore

To stop MetaStore, press Ctrl+C. Please note, this is the control(^) key and not the command key in Mac. It is recommended to stop the MetaStore before you shut down your computer.

Configurating MetaStore to start on boot (for linux servers)

First stop the MetaStore service by pressing Ctrl+C. To automatically start MetaStore on boot, create a file metastore.service in /etc/systemd/system folder:

  1. In the Terminal window, navigate to the /etc/systemd/system folder using the command cd /etc/systemd/system and press enter

  2. Using a commandline editor like nano, create the file metastore.service. For this, type nano metastore.service and press enter.

  3. Write the following content inside the editor and press Ctrl+O and press enter, again followed by Ctrl+X to close the editor and save the file. 
     [Unit]
 Description=metastore
 After=syslog.target

 [Service]
 User=matwerk
 ExecStart=/home/matwerk/dem-services/metastore/run.sh

 [Install]
 WantedBy=multi-user.target
  4. Finally, enable the script and start the service by typing the following commands in the command prompt: 
     systemctl enable metastore.service
 systemctl start metastore.service
 systemctl status metastore.service