#dominoforever | Product Ideas Portal

 

Welcome to the #dominoforever Product Ideas Forum! The place where you can submit product ideas and enhancement request. We encourage you to participate by voting on, commenting on, and creating new ideas. All new ideas will be evaluated by HCL Product Management & Engineering teams, and the next steps will be communicated. While not all submitted ideas will be executed upon, community feedback will play a key role in influencing which ideas are and when they will be implemented.

For more information and upcoming events around #dominoforever, please visit our Destination Domino Page

Improve View Update performance in NSF dramatically

As an example, the problem with slow View Update performance becomes most visible with large domlog.nsf or large log.nsf NSF databases.

 

If there are several million documents, and if several documents are added per second, the View Update may take too long, or the View will "never" open.

 

Apparently, updating a View seems to be a single-threaded process. If too many documents were added/changed, the Update task will initiate a full View Rebuild, which makes it even worse.

 

This model limits NSF-based applications to scale per se.

 

Typical workarounds to this weakness are to use faster storage, or to distribute data accross more databases, which have a smaller amount of documents.

 

I believe that it should be possible to raise this limitation substantially by introducing a multi-threaded View Update model (multiple threads working on updating a single View in parallel)

 

Toni Feric, Belsoft Collaboration

  • Avatar32.5fb70cce7410889e661286fd7f1897de Guest
  • Jul 31 2018
  • Likely to implement
  • Attach files
  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    13 Jun 10:03

    @Thomas Hampel:

    Yes - but "horizontal scaling" (if we want to call it that way) does not help to update individual Views faster.

    So, this Thread is about assigning available system resources more efficiently, in order to update individual Views much faster.

    Toni Feric

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    10 Jun 16:14

    Yes, we are using multiple update tasks at once but 1 update task is updating 1 view and is using only 1 processor not all 28. Is the same of using only 1 update task.

    Yes, multiple views are updating in parallel if using multiple update tasks but we need to update very fast very large views.

  • Admin
    Thomas Hampel commented
    10 Jun 07:21

    Have you considered running multiple update tasks at once ?

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    08 Jun 12:04

    We need notes update view index multithreading. We have applications with many views (business request) and many documents (over 10 millions) and the bottleneck is the view and database index, Database is very active, 100 new doc per minute. Users are waiting over 30 seconds to 1 minute to open the views. We have SSD and 28 processors on the server but the view index is using only 1 thread/1 processor and is very slow slow slow.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    31 Mar 21:16
    1. Are you creating views? That is, building indexes from scratch? If so, is the operation performance-critical? It's VERY important that we know the use case, because incremental view updating and rebuilding are to very different code paths.

    2. inline affects only incremental updating. (Re)build is unaffected

    3. All due respect, multi-threaded view updating is a solution for some problem, though until we're convinced it addresses a real bottleneck, it's just throwing ideas around.

    4. I will look into the rebuild operation as that seems to be the target. I will report what I find.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    22 Jan 20:55

    @JohnCurtis:

    Inline Views work indeed much faster. But in some cases not fast enough.

    Other questions may be:

    • Even if Inline Views are much faster, are they suitable for millions of documents and/or very complex Views?
    • As far as I understand, Inline Views are meant to be a measure of tuning, i.e. make very specific Views much faster. It does not seem to be a solution for all Views yet, nor does it seem to address the problem of single-threaded View updates.

     

    I understand the concept of lazy View Updates. As far as I know, an administrator can tune settings on how aggressively Update tasks should work on the queue. More aggressive settings imply more extensive use of system ressources in favour of lower probability of wait time by users, which are randomly accessing Views. But there is nothing we can do, if a View Update is triggered interactively by a user.

     

    I would like to show an example:

    Our database db1.nsf has 445521 documents.

    We use "INLINE_VIEW_INDEX=db1.nsf" in the notes.ini, and we use NIFNSF pointing to a separate folder on a separate, dedicated volume.

    I clicked on a View triggering an interactive View Update:

    Finished update of database: db1.nsf View: All\Editors and Days|Docs.By.Editors.Day by user: CN=Toni Feric/O=Belsoft Finished at: 22.01.2020 21:23:05 Total time (milliseconds) 331560


    So, I was waiting 5mins 31s.

    What if this database had, say,  50 million documents?

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    November 24, 2019 03:48

    (continued) please try that feature and provide feedback.

     

    Thanks,

    John

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    November 24, 2019 03:47

    [John Curtis here]

     

    1. Domino's view indexes were built to be updated lazily because there were so many of them created as part of a database design.  The comments that call for SQL-like indexing get that wrong - that is, tell your Oracle DBA you want 1000 indexes on your favorite table. 

    2. Furthermore there are many indexes per view, so a view != an index.

    3. The locking model around views creates contention which can drastically slow down view updating AND reading.


    Please try the 9.01FP8+ feature described here

    https://www.ibm.com/support/knowledgecenter/SSKTMJ_10.0.1/admin/admn_inline_index_enabling_with_updall.html

    It will allow you to choose views or entire databases and update views with the document.  It's available now.  We aren't stopping with that feature, but if this is a problem

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    August 13, 2019 15:06

    "If too many documents were added/changed, the Update task will initiate a full View Rebuild, which makes it even worse." - no, it does not do that.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    January 13, 2019 15:05

    Another method to speed up View indexing would be to use "learned indexing" structures.

    https://www.arxiv-vanity.com/papers/1712.01208/

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    November 05, 2018 10:42

    +1.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    September 13, 2018 09:46

    Many thanks for pointing to the feature 'Dedicated View Thread'. Obviously, it is a very specific workaround. It does not solve the fundamental problem of Domino databases not being able to scale.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    August 27, 2018 14:23

    Since 9.0.1 FP3 the feature 'Dedicated View Thread' https://www-01.ibm.com/support/docview.wss?uid=swg21960257 exists. This let you dedicate a separate thred to a specific view, up to a maximum (as of 9.0.1 FP3) of 25 dedicated threads.  The feature also allows access to the view while it is being updated. This might alleviate some specific pain points in a server.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    August 24, 2018 15:14

    It should be possible to mark view indices in applications as non-discardable (like DBMT does to specific mailbox views).

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    August 01, 2018 15:44

    helo

    +1

    it would be excellent to gain an updated view with one million documents in 7sec as in SQL.
    to build into Domino relational structure of view