Skip to Main Content
HCL Domino Ideas Portal

Welcome to the #dominoforever Product Ideas Forum! The place where you can submit product ideas and enhancement request. We encourage you to participate by voting on, commenting on, and creating new ideas. All new ideas will be evaluated by HCL Product Management & Engineering teams, and the next steps will be communicated. While not all submitted ideas will be executed upon, community feedback will play a key role in influencing which ideas are and when they will be implemented.

For more information and upcoming events around #dominoforever, please visit our Destination Domino Page

231 VOTE
Status Shipped
Workspace Domino
Categories Administration
Created by Guest
Created on Jul 31, 2018

Improve View Update performance in NSF dramatically

As an example, the problem with slow View Update performance becomes most visible with large domlog.nsf or large log.nsf NSF databases.

 

If there are several million documents, and if several documents are added per second, the View Update may take too long, or the View will "never" open.

 

Apparently, updating a View seems to be a single-threaded process. If too many documents were added/changed, the Update task will initiate a full View Rebuild, which makes it even worse.

 

This model limits NSF-based applications to scale per se.

 

Typical workarounds to this weakness are to use faster storage, or to distribute data accross more databases, which have a smaller amount of documents.

 

I believe that it should be possible to raise this limitation substantially by introducing a multi-threaded View Update model (multiple threads working on updating a single View in parallel)

 

Toni Feric, Belsoft Collaboration

  • Attach files
  • Guest
    Reply
    |
    May 19, 2022

    The claim that -inline, being specific to a database and view is completely in keeping with the breadth of views present in a production environment. There are THOUSANDS of view indexes. And not all views are necessary to push updates to.

    Another parameter in the mix is the presence of permuted views (those with "Show multiple values as separate entities"). Those views are MUCH more expensive to update; truly requiring a separate effort.

    But I do hear you regarding parallelism; we need a locking model to support that but we do know what threads are :-D .

  • Guest
    Reply
    |
    Jun 13, 2020

    @Thomas Hampel:

    Yes - but "horizontal scaling" (if we want to call it that way) does not help to update individual Views faster.

    So, this Thread is about assigning available system resources more efficiently, in order to update individual Views much faster.

    Toni Feric

  • Guest
    Reply
    |
    Jun 10, 2020

    Yes, we are using multiple update tasks at once but 1 update task is updating 1 view and is using only 1 processor not all 28. Is the same of using only 1 update task.

    Yes, multiple views are updating in parallel if using multiple update tasks but we need to update very fast very large views.

  • Admin
    Thomas Hampel
    Reply
    |
    Jun 10, 2020

    Have you considered running multiple update tasks at once ?

  • Guest
    Reply
    |
    Jun 8, 2020

    We need notes update view index multithreading. We have applications with many views (business request) and many documents (over 10 millions) and the bottleneck is the view and database index, Database is very active, 100 new doc per minute. Users are waiting over 30 seconds to 1 minute to open the views. We have SSD and 28 processors on the server but the view index is using only 1 thread/1 processor and is very slow slow slow.

  • Guest
    Reply
    |
    Mar 31, 2020
    1. Are you creating views? That is, building indexes from scratch? If so, is the operation performance-critical? It's VERY important that we know the use case, because incremental view updating and rebuilding are to very different code paths.

    2. inline affects only incremental updating. (Re)build is unaffected

    3. All due respect, multi-threaded view updating is a solution for some problem, though until we're convinced it addresses a real bottleneck, it's just throwing ideas around.

    4. I will look into the rebuild operation as that seems to be the target. I will report what I find.

  • Guest
    Reply
    |
    Jan 22, 2020

    @JohnCurtis:

    Inline Views work indeed much faster. But in some cases not fast enough.

    Other questions may be:

    • Even if Inline Views are much faster, are they suitable for millions of documents and/or very complex Views?
    • As far as I understand, Inline Views are meant to be a measure of tuning, i.e. make very specific Views much faster. It does not seem to be a solution for all Views yet, nor does it seem to address the problem of single-threaded View updates.

     

    I understand the concept of lazy View Updates. As far as I know, an administrator can tune settings on how aggressively Update tasks should work on the queue. More aggressive settings imply more extensive use of system ressources in favour of lower probability of wait time by users, which are randomly accessing Views. But there is nothing we can do, if a View Update is triggered interactively by a user.

     

    I would like to show an example:

    Our database db1.nsf has 445521 documents.

    We use "INLINE_VIEW_INDEX=db1.nsf" in the notes.ini, and we use NIFNSF pointing to a separate folder on a separate, dedicated volume.

    I clicked on a View triggering an interactive View Update:

    Finished update of database: db1.nsf View: All\Editors and Days|Docs.By.Editors.Day by user: CN=Toni Feric/O=Belsoft Finished at: 22.01.2020 21:23:05 Total time (milliseconds) 331560


    So, I was waiting 5mins 31s.

    What if this database had, say,  50 million documents?

  • Guest
    Reply
    |
    Nov 24, 2019

    (continued) please try that feature and provide feedback.

     

    Thanks,

    John

  • Guest
    Reply
    |
    Nov 24, 2019

    [John Curtis here]

     

    1. Domino's view indexes were built to be updated lazily because there were so many of them created as part of a database design.  The comments that call for SQL-like indexing get that wrong - that is, tell your Oracle DBA you want 1000 indexes on your favorite table. 

    2. Furthermore there are many indexes per view, so a view != an index.

    3. The locking model around views creates contention which can drastically slow down view updating AND reading.


    Please try the 9.01FP8+ feature described here

    https://www.ibm.com/support/knowledgecenter/SSKTMJ_10.0.1/admin/admn_inline_index_enabling_with_updall.html

    It will allow you to choose views or entire databases and update views with the document.  It's available now.  We aren't stopping with that feature, but if this is a problem

  • Guest
    Reply
    |
    Aug 13, 2019

    "If too many documents were added/changed, the Update task will initiate a full View Rebuild, which makes it even worse." - no, it does not do that.

  • Guest
    Reply
    |
    Jan 13, 2019

    Another method to speed up View indexing would be to use "learned indexing" structures.

    https://www.arxiv-vanity.com/papers/1712.01208/

  • Guest
    Reply
    |
    Nov 5, 2018

    +1.

  • Guest
    Reply
    |
    Sep 13, 2018

    Many thanks for pointing to the feature 'Dedicated View Thread'. Obviously, it is a very specific workaround. It does not solve the fundamental problem of Domino databases not being able to scale.

  • Guest
    Reply
    |
    Aug 27, 2018

    Since 9.0.1 FP3 the feature 'Dedicated View Thread' https://www-01.ibm.com/support/docview.wss?uid=swg21960257 exists. This let you dedicate a separate thred to a specific view, up to a maximum (as of 9.0.1 FP3) of 25 dedicated threads.  The feature also allows access to the view while it is being updated. This might alleviate some specific pain points in a server.

  • Guest
    Reply
    |
    Aug 24, 2018

    It should be possible to mark view indices in applications as non-discardable (like DBMT does to specific mailbox views).

  • Guest
    Reply
    |
    Aug 1, 2018

    helo

    +1

    it would be excellent to gain an updated view with one million documents in 7sec as in SQL.
    to build into Domino relational structure of view

49 MERGED

Improvement of database performance including large number of documents

Merged
When 500,000 documents, 1 million documents and the number of documents become large, creation, updating and other operations of the view index become very heavy. I want you to improve.
over 5 years ago in Domino 3 Shipped