Improve View Update performance in NSF dramatically

The claim that -inline, being specific to a database and view is completely in keeping with the breadth of views present in a production environment. There are THOUSANDS of view indexes. And not all views are necessary to push updates to.

Another parameter in the mix is the presence of permuted views (those with "Show multiple values as separate entities"). Those views are MUCH more expensive to update; truly requiring a separate effort.

But I do hear you regarding parallelism; we need a locking model to support that but we do know what threads are :-D .

@Thomas Hampel:

Yes - but "horizontal scaling" (if we want to call it that way) does not help to update individual Views faster.

So, this Thread is about assigning available system resources more efficiently, in order to update individual Views much faster.

Toni Feric

Yes, we are using multiple update tasks at once but 1 update task is updating 1 view and is using only 1 processor not all 28. Is the same of using only 1 update task.

Yes, multiple views are updating in parallel if using multiple update tasks but we need to update very fast very large views.

Have you considered running multiple update tasks at once ?

We need notes update view index multithreading. We have applications with many views (business request) and many documents (over 10 millions) and the bottleneck is the view and database index, Database is very active, 100 new doc per minute. Users are waiting over 30 seconds to 1 minute to open the views. We have SSD and 28 processors on the server but the view index is using only 1 thread/1 processor and is very slow slow slow.

1. Are you creating views? That is, building indexes from scratch? If so, is the operation performance-critical? It's VERY important that we know the use case, because incremental view updating and rebuilding are to very different code paths.

2. inline affects only incremental updating. (Re)build is unaffected

3. All due respect, multi-threaded view updating is a solution for some problem, though until we're convinced it addresses a real bottleneck, it's just throwing ideas around.

4. I will look into the rebuild operation as that seems to be the target. I will report what I find.

@JohnCurtis:

Inline Views work indeed much faster. But in some cases not fast enough.

Other questions may be:

• Even if Inline Views are much faster, are they suitable for millions of documents and/or very complex Views?
• As far as I understand, Inline Views are meant to be a measure of tuning, i.e. make very specific Views much faster. It does not seem to be a solution for all Views yet, nor does it seem to address the problem of single-threaded View updates.

I understand the concept of lazy View Updates. As far as I know, an administrator can tune settings on how aggressively Update tasks should work on the queue. More aggressive settings imply more extensive use of system ressources in favour of lower probability of wait time by users, which are randomly accessing Views. But there is nothing we can do, if a View Update is triggered interactively by a user.

I would like to show an example:

Our database db1.nsf has 445521 documents.

We use "INLINE_VIEW_INDEX=db1.nsf" in the notes.ini, and we use NIFNSF pointing to a separate folder on a separate, dedicated volume.

I clicked on a View triggering an interactive View Update:

Finished update of database: db1.nsf View: All\Editors and Days|Docs.By.Editors.Day by user: CN=Toni Feric/O=Belsoft Finished at: 22.01.2020 21:23:05 Total time (milliseconds) 331560

So, I was waiting 5mins 31s.

What if this database had, say,  50 million documents?

(continued) please try that feature and provide feedback.

Thanks,

John

[John Curtis here]

1. Domino's view indexes were built to be updated lazily because there were so many of them created as part of a database design.  The comments that call for SQL-like indexing get that wrong - that is, tell your Oracle DBA you want 1000 indexes on your favorite table. 

2. Furthermore there are many indexes per view, so a view != an index.

3. The locking model around views creates contention which can drastically slow down view updating AND reading.

Please try the 9.01FP8+ feature described here

https://www.ibm.com/support/knowledgecenter/SSKTMJ_10.0.1/admin/admn_inline_index_enabling_with_updall.html

It will allow you to choose views or entire databases and update views with the document.  It's available now.  We aren't stopping with that feature, but if this is a problem

"If too many documents were added/changed, the Update task will initiate a full View Rebuild, which makes it even worse." - no, it does not do that.

Another method to speed up View indexing would be to use "learned indexing" structures.

https://www.arxiv-vanity.com/papers/1712.01208/

+1.

Many thanks for pointing to the feature 'Dedicated View Thread'. Obviously, it is a very specific workaround. It does not solve the fundamental problem of Domino databases not being able to scale.

Since 9.0.1 FP3 the feature 'Dedicated View Thread' https://www-01.ibm.com/support/docview.wss?uid=swg21960257 exists. This let you dedicate a separate thred to a specific view, up to a maximum (as of 9.0.1 FP3) of 25 dedicated threads.  The feature also allows access to the view while it is being updated. This might alleviate some specific pain points in a server.

It should be possible to mark view indices in applications as non-discardable (like DBMT does to specific mailbox views).

helo

+1

it would be excellent to gain an updated view with one million documents in 7sec as in SQL.
to build into Domino relational structure of view