Monthly Archives: March 2013

The Value of Managing Your Data Effectively

We recently came across a very good presentation that described concisely, in business terms, why a strategic approach to data and masterdata management is a must for any serious organisation.

I highly recommend that anyone who is struggling to understand the benefits of these critical areas, or needs a reminder, to take a few minutes out of their day to view the slides below:

Do you agree or are there other areas we all should also be considering….?

Tagged , , ,

Oracle Hyperion DRM technical issue | Application Restart Failure

We’re back blogging again!

Firstly, please accept our apologies for the lack of submissions over the last few months. It has been a busy time for Seismi. We have had major implementations underway with a number of our clients that has distracted us from this blog! It is only now that we have time to draw breath.

I thought that it would be useful to report back an issue that we recently experienced with Oracle Hyperion Database Relationship Manager (DRM) in a client’s development environment. Hopefully this blog will achieve two objectives.

• help you understand this particular issue; and
• explain the process we went through to diagnose and fix the issue.

The issue

During our development cycle we needed to restart our DRM server. The server stopped fine, but refused to start again. A review of the Windows event viewer showed us that two errors had been produced. The first related to a validation incorrectly referencing a global property and the second, showed the DRM application was shutting down. Our immediate thought was that the two must be related and so we raised an service request with Oracle. Oracle gave us a fix to allow us to change the property used in the validation to a global property. With much anticipation we ran the fix and then tried to bring up the application. Unfortunately, it still would not start. A further review of the Windows event log now showed a different error message.

System.ArgumentException: An item with the same key has already been added.
at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
at Oracle.Drm.Engine.Core.VersionMgr.b__9a()
at Oracle.Drm.Engine.Core.VersionMgr.Write(Action func)
at Oracle.Drm.Engine.Core.VersionMgr.b__b6()
at Oracle.Drm.Engine.Core.VersionMgr.Write(Action func)
at Oracle.Drm.Engine.Core.Engine.Start()
at Oracle.Drm.Engine.Host.Program.Main(String[] args)

We notified Oracle that their fix had not resolved the issue. We also felt it was necessary to restore the DRM database from the last back up prior to the issue occurring (i.e. the day before). Disappointingly this restore did not resolve the issue, implying that the issue had been lying dormant for an unspecified period of time.

With very little information to go on we revisited the Windows event viewer and, starting from the first time we experienced the issue, we worked backwards through all the DRM messages to identify any unexpected entries. After a few hours of (mind-numbing!) analysis, we discovered the following entry dated three days prior to the eventual problem:

Oracle.DataAccess.Client.OracleException ORA-03114: not connected to ORACLE at Oracle.DataAccess.Client.OracleException.HandleErrorHelper(Int32 errCode, OracleConnection conn, IntPtr opsErrCtx, OpoSqlValCtx* pOpoSqlValCtx, Object src, String procedure) at Oracle.DataAccess.Client.OracleTransaction.Rollback()
at Oracle.Drm.DalFramework.DatabaseContext.Rollback()
at Oracle.Drm.DalFramework.DataAccessLayer.Rollback()
at Oracle.Drm.DalFramework.DataAccessLayer.ExecuteTransaction(Action codeBlock, Action afterCommit, Action afterRollback)
at Oracle.Drm.Engine.Core.VersionMgr.DeleteVersionFromDB(Int32 versionID)
at Oracle.Drm.Engine.Core.ReaderWriterLockSlimExtensions.WriteLock(ReaderWriterLockSlim rwLock, Action action)
at Oracle.Drm.Engine.Core.VersionMgr.DeleteVersion(String versionAbbrev, Action`1 updateProgress, Action fireEvent)
at Oracle.Drm.Engine.Core.ReaderWriterLockSlimExtensions.ReadLock(ReaderWriterLockSlim rwLock, Action action)

Please note that this entry was not logged as an “Error”, but as a “Warning”. Looking at these two messages side-by-side we saw that both errors mentioned the “Oracle.Drm.Engine.Core.VersionMgr”. The “Warning” suggested a loss of connectivity during the delete of a DRM version (“Oracle.Drm.Engine.Core.VersionMgr.DeleteVersionFromDB”) and the error at start-up suggested some form of duplication with the versions (“System.ArgumentException: An item with the same key has already been added.”). At this point we decided to connect directly to the DRM database to have a look at the version table. When we queried the table we saw two versions with identical names, aside from case. One was called “LIVE” and the other “Live”. This seemed unusual and so we used a different DRM server to try and replicate the issue. The DRM interface prohibited it.

We suspected that if we changed the name of one of the versions, the duplication issue would be resolved and the server would restart. With the upmost caution, and only as a last resort, we manually updated the RM_VERSION table to remove the duplicate names and restarted the server successfully. We are still in the process of trying to identify why the database allowed this issue to occur. The RM_VERSION table has a unique that should have prevented this and so we are starting to look at the schema settings. If we track anything down, we will publish an update to this blog.

Hopefully this has helped those of you trying to diagnose this particular issue. We also hope that it has given you some guidance on how to diagnose an issue with DRM. Remember that some issues only become apparent after an application restart and can be hidden by other errors. Always concentrate your analysis on the period between the last successful restart and the failure. Finally, remember that you need to review all log entries, not only those that are tagged as “Error”.

Until next time – hopefully next week….