SAP HANA Database startup fails during revision upgrade

Note : 1918943

Symptom

When upgrading the HANA revision, the first startup with the new revision which takes place during the upgrade fails. The HLM logs display the following error:

INFO:       Waiting for hdbdaemon
ERR :       FAIL: process hdbdaemon HDB Daemon not running
ERR :     Start instance 00 on master host <hostname> failed.
INFO:     Messages from trace files
ERR :     Check the log file for SAP HANA Database error messages
ERR :   Cannot start system
ERR :   error upgrading
ERR :   Upgrade failed
INFO:   Summary of critical errors
ERR :     Upgrade failed
ERR :       error upgrading
ERR :         Cannot start system
ERR :           Start instance 00 on master host <hostname> failed.
ERR :             FAIL: process hdbdaemon HDB Daemon not running
ERR :           Check the log file for SAP HANA Database error messages

Subsequent startup attempts (for example directly using ‘HDB start’ on the master node) either fail immediately or hang forever. It is important to note that the only database process which at least is attempted to be started is the hdbdaemon process. Other processes like indexserver or nameserverare not even started and hence, their logfiles are not updated by the startup.

The trace file of the HANA daemon on the master node (daemon_<hostname>_<port>_<>number) contains the following entries:

program “nameserver” has runlevel 1, which is larger than the target runlevel 0, it won’t be started
program “preprocessor” has runlevel 2, which is larger than the target runlevel 0, it won’t be started
program “indexserver” has runlevel 3, which is larger than the target runlevel 0, it won’t be started
program “indexserver.c” has runlevel 3, which is larger than the target runlevel 0, it won’t be started
program “scriptserver” has runlevel 3, which is larger than the target runlevel 0, it won’t be started
program “statisticsserver” has runlevel 3, which is larger than the target runlevel 0, it won’t be starte
program “xsengine” has runlevel 3, which is larger than the target runlevel 0, it won’t be started
program “sapwebdisp” has runlevel 4, which is larger than the target runlevel 0, it won’t be started
program “compileserver” has runlevel 2, which is larger than the target runlevel 0, it won’t be started

OR

Basis LttImportProvider.cpp(00280) : >>>> DUMP: exception occurred:
file ltt/impl/memory.cpp, line 000063, errno 1000002: Out of memory ;
$size$=549755813897; $name$=<malloc allocator>; $type$=MallocAllocatorImpl; $inuse_count$=0; $allocated_size$=0
exception throw location:
1: … in ltt::string_base … at memory.hpp:737 (hdbdaemon)
2: … in ltt::basic_stringbuf … string.hpp:718 (hdbdaemon)
3: … in ltt::basic_streambuf … streambuf.hpp:693 (hdbdaemon)
4: … in ltt::basic_ostream … streambuf.hpp:340 (hdbdaemon)
5: … in ltt::basic_ostream … ostream.hpp:600 (hdbdaemon)
6: … in SAPStartsrvInfoWriter::add … TrexTrace.h:81 (hdbdaemon)
7: … in TeeInfoWriter::add …

Environment

SAP HANA

Reproducing the Issue

Upgrade of the HANA database using HLM.

Cause

The database parameters are stored on the filesystem in different paths. Directory/usr/sap/<SID>/HDB<xx>/exe/config contains those ini files that hold the default configuration of the database. These files are not supposed to be changed. Directory/hana/shared/<SID>/global/hdb/custom contains changes to the default configuration which are valid for all nodes of the HANA system.

When the mentioned symptoms occur, a daemon.ini file, holding customer specific changes, exists in directory /hana/shared/<SID>/global/hdb/custom. It may contain entries like this:

[scriptserver]
instances

OR

[daemon]
runlevel

In both cases, the configuration file is not consistent (it just contains the name of the parameter, but no value).

Resolution

Rename the customer specific configuration file /usr/sap/<SID>/HDB<xx>/exe/config/daemon.ini so that it is not taken into account during system startup.

One Response

  1. Dana Bodenstein September 28, 2016

Leave a Reply