2101737 – Recovery of a Multitenant Database Container fails in SAP HANA

Symptom

The recovery of a Multitenant Database Container fails with a message which contains the text fragments “recovery could not be completed” and “No such file or directory” or “service does not match”.

Reason and Prerequisites

A recovery into a Multitenant Database Container needs a target database which has the same configuration especially the same volume numbers like the source database which wrote the backup before. In case of a system copy or a disaster recovery which needs a recreation of the Multitenant Database Container this precondition can not always be fulfilled by the HANA itself.
You are using a SAP HANA Revision 90 – 92.

Solution

The volume numbers of the target database to which the backup will be applied has to be configured identically to the source database. This SAP Note contains a workaround to reconfigure these volume numbers manually. SAP will provide a solution which applies the appropriate volume numbers to the target database automatically. This solution is available with HANA Revision 93.
To reconfigure the volume number you will need the privilege “DATABASE ADMIN” in the target instance.
Please be carefully with using copy and paste to copy statements and commands from this SAP Note. Depending from the tool which you are using to view this SAP Note the text copied may contain control characters which prevents the successful execution of the statement or the command.
Caution: The solution requires changes to the topology and it can be a risk to make such changes. Please contact the support of SAP if you have any doubt about the correct procedure.

1. Determine the volumes and services of the source database which wrote the backup using the tool hdbbackupdiag

Locate the backup files in the file system and apply the hdbbackupdiag command to this backup.
hdbbackupdiag -v -d <directory> -b <backupprefix> | grep “ServiceName|VolumeId”
e.g.
hdbbackupdiag -v -d /usr/sap/M13/HDB00/backup/data/DB_M13 -bCOMPLETE_DATA_BACKUP | grep “ServiceName|VolumeId”
    ServiceName: indexserver
    VolumeId: 4
    ServiceName: indexserver
    VolumeId: 4
Please save the values given in the output. The master indexserver will occur twice.

2. Configure the target database with the same number and types of services

Please use the appropriate commands to create a Multitenant Database Container and to add the same number of services to the database. Please check the chapter “Setting up Multitenant Database Containers” in the SAP HANA Administration Guide for details.

3. Determine the volumes and services of the target database

Connect to the system database and execute the SQL statement:
SELECT S.DATABASE_NAME, S.HOST, S.SERVICE_NAME, S.PORT, V.VOLUME_ID
FROM SYS_DATABASES.M_SERVICES S, SYS_DATABASES.M_VOLUMES V
WHERE S.HOST = V.HOST AND S.PORT = V.PORT AND S.DATABASE_NAME = V.DATABASE_NAME AND S.DATABASE_NAME = ‘<DATABASE_NAME>
e.g.
SELECT S.DATABASE_NAME, S.HOST, S.SERVICE_NAME, S.PORT, V.VOLUME_ID
FROM SYS_DATABASES.M_SERVICES S, SYS_DATABASES.M_VOLUMES V
WHERE S.HOST = V.HOST AND S.PORT = V.PORT AND S.DATABASE_NAME = V.DATABASE_NAME AND S.DATABASE_NAME = ‘M13
Please save the values given in the output.

4. Define a mapping for the volume ids

In the next steps the volume ids of the target database will be changed to the volume ids of the source database. To prepare these steps define a mapping between the volumes of the source database (step 1.) and the volumes of the target database (step 3.). Please keep in mind that the assigned service type can’t be changed. Please map the volume id of the master indexserver in the backup to the master indexserver of the target database.

5. Determine the configuration values for every volume of the target database

Connect to the system database and execute the SQL statement:
SELECT CONCAT(SUBSTRING(PATH, 1, LOCATE(PATH, ‘/’, 0, 2)), SUBSTR_BEFORE(NAME, ‘@’)) PATH, SUBSTR_BEFORE(NAME, ‘@’) DB_VOLUME_ID, SUBSTR_AFTER(NAME, ‘@’) NAME, VALUE
FROM M_TOPOLOGY_TREE
WHERE PATH = ‘/volumes/*+|@’ AND NAME LIKE CONCAT((SELECT SUBSTR_BEFORE(name, ‘@’) DB_ID FROM M_TOPOLOGY_TREE WHERE PATH=’/databases/*+|@’ and NAME like ‘%name’ and value = ‘<DATABASE_NAME>‘), ‘:%’)
e.g.
SELECT CONCAT(SUBSTRING(PATH, 1, LOCATE(PATH, ‘/’, 0, 2)), SUBSTR_BEFORE(NAME, ‘@’)) PATH, SUBSTR_BEFORE(NAME, ‘@’) DB_VOLUME_ID, SUBSTR_AFTER(NAME, ‘@’) NAME, VALUE
FROM M_TOPOLOGY_TREE
WHERE PATH = ‘/volumes/*+|@’ AND NAME LIKE CONCAT((SELECT SUBSTR_BEFORE(name, ‘@’) DB_ID FROM M_TOPOLOGY_TREE WHERE PATH=’/databases/*+|@’ and NAME like ‘%name’ and value = ‘M13‘), ‘:%’)
You will receive a table with the columns PATH, NAME and VALUE like
PATH;DB_VOLUME_ID;NAME;VALUE
/volumes/3:2;3:2;path;mnt00001/hdb00002.00003
/volumes/3:2;3:2;active;yes
/volumes/3:2;3:2;catalog;yes
/volumes/3:2;3:2;location;berl30052174:30240
/volumes/3:2;3:2;database;3
/volumes/3:2;3:2;tenant;-
/volumes/3:2;3:2;servicetype;indexserver
Please save the values for the steps following.

6. Stop the target database

Connect to the system database and execute the SQL statement below:
ALTER SYSTEM STOP DATABASE <DATABASE_NAME>
e.g.
ALTER SYSTEM STOP DATABASE M13

7. Change the volume ids of the services of the target database

Connect to the system database and change for each service found in step 3 the number of the assigned volume according to the mapping defined in step 4. To change the volume number use this statement:
ALTER SYSTEM ALTER CONFIGURATION (‘topology.ini’, ‘system’) SET (‘/host/<host>/<service_name>/<port>‘, ‘volume’)= ‘<volume_id>
e.g.
ALTER SYSTEM ALTER CONFIGURATION (‘topology.ini’, ‘system’) SET (‘/host/berl30052174/indexserver/30240‘, ‘volume’)= ‘4
Caution: It can be a risk to make changes to the topology. Therefore, carefully check the values in the statements.

8. Delete configuration values for every volume from target database

Connect to the system database and execute an UNSET statement for each distinct DB_VOLUME_ID value found in step 5.
ALTER SYSTEM ALTER CONFIGURATION (‘topology.ini’, ‘system’) UNSET (‘volumes’, ‘<DB_VOLUME_ID>‘);
e.g.
ALTER SYSTEM ALTER CONFIGURATION (‘topology.ini’, ‘system’) UNSET (‘/volumes’, ‘3:2‘)
Keep in mind that after this step you cannot anymore retrieve these values again like in step 4.
Caution: It can be a risk to make changes to the topology. Therefore, carefully check the values in the statements.

9. Insert new configuration values for every volume into the target database

Connect to the system database and set the configuration values retrieved in step 5 for each volume according to the mapping defined in step 4.
Change the last number of the path column (after the colon) to the new volume id (e.g. ‘/volumes/3:2‘ to ‘/volumes/3:4‘). The number before the colon is the database number and must not be changed.
Change the second number with the prefix ‘hdb’ of the value with the name ‘path’ to the new volume id (e.g. ‘mnt00001/hdb00002.00003′ to ‘mnt00001/hdb00004.00003′)
Use this statement repeatedly to insert the values
ALTER SYSTEM ALTER CONFIGURATION (‘topology.ini’, ‘system’) SET (‘<path>‘, ‘<name>‘)= ‘<value>
e.g.
ALTER SYSTEM ALTER CONFIGURATION (‘topology.ini’, ‘system’) SET (‘/volumes/3:4‘, ‘active‘)= ‘yes
ALTER SYSTEM ALTER CONFIGURATION (‘topology.ini’, ‘system’) SET (‘/volumes/3:4‘, ‘catalog‘)= ‘yes
ALTER SYSTEM ALTER CONFIGURATION (‘topology.ini’, ‘system’) SET (‘/volumes/3:4‘, ‘database‘)= ‘3
ALTER SYSTEM ALTER CONFIGURATION (‘topology.ini’, ‘system’) SET (‘/volumes/3:4‘, ‘location‘)= ‘berl30052174:30240
ALTER SYSTEM ALTER CONFIGURATION (‘topology.ini’, ‘system’) SET (‘/volumes/3:4‘, ‘path‘)= ‘mnt00001/hdb00004.00003
ALTER SYSTEM ALTER CONFIGURATION (‘topology.ini’, ‘system’) SET (‘/volumes/3:4‘, ‘servicetype‘)= ‘indexserver
ALTER SYSTEM ALTER CONFIGURATION (‘topology.ini’, ‘system’) SET (‘/volumes/3:4‘, ‘tenant‘)= ‘
Caution: It can be a risk to make changes to the topology. Therefore, carefully check the values in the statements.

The target database is now prepared to recover the backup. You may always interrupt this procedure and drop the target database to start from the beginning.

Leave a Reply