Long delays on "Open": display of TOC

Everything related to our flagship word processor.
David Sharp
Posts: 67
Joined: 2008-07-06 23:21:27
Location: Paris, France
Contact:

Long delays on "Open": display of TOC

Post by David Sharp »

Hi,
I work almost every day on a long document which is a kind of diary. For reasons of searchability I've so far resisted the temptation to split it into several parts and it currently runs to just over 500,000 words. I use styles a lot, to define different types of entry along with key words. Each day's entry generally has a title, which means that the TOC contains thousands of entries.
Recently, Nisus has been showing me the spinning beachball when I reopen the document. The process is taking between three and five minutes each time.

While waiting to be able to start work, I notice that the "Table of Contents" is displaying an early part of the document (where I was working a few days ago), even though the main window is showing the part where was working when I last shut down the computer.
I get the impression that NWP is "misremembering" the part of the TOC that corresponds to the place I stopped working at during the previous session, and that it's taking a long time to get things back into synch.

Could this be the reason for the delays, and if so what can I do to fix it? Many thanks for any advice.
David Sharp
Posts: 67
Joined: 2008-07-06 23:21:27
Location: Paris, France
Contact:

Re: Long delays on "Open": display of TOC

Post by David Sharp »

I'm sorry that there has not been anyone available to address my problem, which I think may be relatively simple to solve. Perhaps it has something to do with the preferences for Nisus?

Pending a solution I go on opening the file every day, and waiting around five minutes for the spinning beachball to go away before I can start writing.
adryan
Posts: 561
Joined: 2014-02-08 12:57:03
Location: Australia

Re: Long delays on "Open": display of TOC

Post by adryan »

Re: Long delays on "Open": display of TOC

G’day, David et al

I’m not sure what’s going on here. I assume you have sufficient memory to ensure a memory bottleneck is not the root of the problem. It would also be worthwhile ensuring that no other applications are running (including Internet-related software), just in case something besides Nisus Writer is spinning that beachball.

I would suggest you duplicate the document, delete the Table of Contents, and see whether you still get the spinning beachball. If you do, the problem is presumably unrelated to the TOC and the solution will have to be sought elsewhere.

If performance is snappier with the TOC deleted, use the Insert TOC command to compile a TOC afresh in the stripped-down document. If the spinning beachball returns after this, I think I would next try reinstalling Nisus Writer. If the problem persists then, I would seek help from Martin.

It occurs to me that a TOC with thousands of entries is going to occupy many pages in itself. As more entries are added, the TOC itself will expand, and the referenced text of all existing entries (already acknowledged to be numerous) will eventually be moved to new pages, so that the TOC will need to be rebuilt to refer correctly even to entries towards the beginning of the document. I wouldn’t have thought that this would necessarily be a problem, even with a document of your size, but I guess you’re making everything (eg, Nisus Writer, Time Machine, read/write from disk) work harder, so there’s more risk of something tripping over itself or its associates.

So another suggestion is to allocate a set of blank pages in the Section containing the TOC, so that the TOC can expand into them without having to keep shifting all the text it is referencing. I wouldn’t regard this as a true solution of the underlying problem — more a workaround that may give some relief. I wouldn’t have thought it would be necessary, though.

Sorry I couldn’t have been of more help.

Cheers,
Adrian
MacBook Pro (M1 Pro, 2021)
macOS Ventura
Nisus Writer user since 1996
User avatar
phspaelti
Posts: 1313
Joined: 2007-02-07 00:58:12
Location: Japan

Re: Long delays on "Open": display of TOC

Post by phspaelti »

Hello David,
just to clear up some obvious things: I assume you are opening your document in Draft view. Obviously switching to Page View for a document of this size is just going to take forever, and should be avoided at all costs. This is particularly true if the file contains things that are "costly" as far as page layout is concerned. (I'm not sure here, just guessing, things like: images (especially floating), tables, footnotes, styles with attributes like "Keep together", "Keep with next", etc.) If you need page layout, you may be able to speed it up using hard page breaks regularly.

Similarly to Adryan I was wondering about the TOC, but I assume you were maybe only talking about the TOC in the Navigator, not an inserted TOC. Also I am guessing you are applying the TOC style using the heading styles. Have you tried removing the TOC attribute from the styles?

More generally you might consider trying to make a copy of the file with all the styles removed, just as a test. Does that kind of file open any faster?
philip
David Sharp
Posts: 67
Joined: 2008-07-06 23:21:27
Location: Paris, France
Contact:

Re: Long delays on "Open": display of TOC

Post by David Sharp »

Hi Adrian and Philip, many thanks for those suggestions.

In reply to Philip, yes I learned very early on *never* to try and open the file in page view, and I take great care never to click on the icon by mistake! Accordingly, the TOC I've been using does not include page numbers. If one day I decide I need to print the whole thing out, I'll set aside an hour or two to let it paginate.

I've now tried getting rid of the TOC and then reinstating it, and unfortunately it's had no effect. The problem disappeared when I deleted the TOC, and then reappeared as soon as I reinstalled it, which tends to indicate that it has some connection to the TOC.

To delete the TOC I removed all references to it in my styles. I don't have a huge array of styles, and am only using two of them to create the TOC.
When I first did this, the document still took a long time to open, with the navigator panel displaying as blank. It was only when I deactivated "Display Navigator" that the document became manageable, opening immediately without the dreaded spinning beachball.

However once I'd reconnected the two styles to the "Default TOC" and reactivated "Display Navigator", the document reverted to the previous behaviour. When I open it, the text appears at the point where I was last working, but the navigator pane displays the very first entries in the document. It takes several minutes to synchronise the TOC display with the displayed text, and then several more before the beachball finally disappears.

As regards Adrian's suggestion concerning memory usage, I've always understood that OS X had removed any concerns on that front, and that one could work with as many applications open as desired. As a general rule I have Mail and Spamsieve, Firefox, Bookends, Ember and Preview and of course Nisus open at the same time, and often Photos as well. If that poses a problem, it'd be good to know: I'll need to change my user habits.

One of my habits that might pose a problem is that I make intensive use of what Apple used to call "Spaces", attributing different applications to different screen views. I have the impression that recent versions of OS X have deprecated that feature; again if that's the case it would be good to know but it may be outside the scope of this forum. In any case I intend to get an actual human being, a Mac specialist, to come round to my place to tell me if there are things are doing that I shouldn't, or thing I'm not doing that I should be!
User avatar
phspaelti
Posts: 1313
Joined: 2007-02-07 00:58:12
Location: Japan

Re: Long delays on "Open": display of TOC

Post by phspaelti »

And working without the Navigator/TOC is not an option? Since this point changes performance by that much it might be worth it, no?
Of course you want navigation in such a large document. Do Bookmarks also slow things down? How about Find?
philip
David Sharp
Posts: 67
Joined: 2008-07-06 23:21:27
Location: Paris, France
Contact:

Re: Long delays on "Open": display of TOC

Post by David Sharp »

Philip writes:
And working without the Navigator/TOC is not an option? Since this point changes performance by that much it might be worth it, no?
Of course you want navigation in such a large document. Do Bookmarks also slow things down? How about Find?
Working without the navigator and a TOC would pretty well eliminate the main attraction of Nisus for this kind of document. And implementing the use of bookmarks for such a complex and long document would take days of work, for a result that might not be any better.
As regards relying on Find, that doesn't correspond to the way I use the TOC. The latter is a neat summary of the document's contents, which I can scan to go to items that interest me. Which, it seems to me, is the one of the basic purposes of a table of contents.
NB: In reply to an earlier comment, I don't have any graphics or other complex objects in my document. It's nothing but text.

I really think something must have gone wrong with this file, as the problem I'm reporting only started appearing a few weeks ago, after I'd done a lot of editing in an early part of the document. Before that, I don't recollect a problem with the display of the TOC. When I opened the file, the latter would appear automatically synchronised with the point in the text at which I'd last been working.
NisusUser
Posts: 317
Joined: 2011-01-12 05:32:38

Re: Long delays on "Open": display of TOC

Post by NisusUser »

David,

You might consider redacting the entire file (make a copy first!) by using the attached macro (redact-creates-new-file.nwm). It will scramble the text so it is confidential. I think it handles all the "hidden" stuff, too, like author, etc. I'm not sure, but I think Martin provided this macro originally.

Then you could use Help > Send Feedback to send the file to Martin or others at Nisus.

They might be able to analyze the file and see what if anything is amiss.

Also, after you've scrambled the data into a new file, you can check it and see if it acts just like the offending original file.

redact-creates-new-file.nwm
(20.68 KiB) Downloaded 495 times
User avatar
phspaelti
Posts: 1313
Joined: 2007-02-07 00:58:12
Location: Japan

Re: Long delays on "Open": display of TOC

Post by phspaelti »

David Sharp wrote:Working without the navigator and a TOC would pretty well eliminate the main attraction of Nisus for this kind of document. And implementing the use of bookmarks for such a complex and long document would take days of work, for a result that might not be any better.
I totally understand you on this point.
I really don't have any experience with 1/2 million word documents, so I can't say whether Nisus crossed some threshold or why you're getting the problem.

I was going to suggest exploration of other possibilities, generally using macros:
  • Using a macro to generate the TOC from the styles. Since the generated TOC would be a freestanding document, it wouldn't cause any delay in the opening of the file. Depending on how this is set up, you could still use this file for navigation. Of course it would need to be periodically updated, so it depends on how much editing goes on in your file (vs. simple appending)
  • Strip the file of all formatting and then recreate the formatting with a macro on a "new" file. This would only be worth the trouble if the issue with the file really is some kind of corruption. But if the problem is the sheer weight of the file, this will leave you with the same problem.
  • Similarly you could use a macro to create a TOC via bookmarks, so creating this might not take that much work. But again I have no idea if a heavily bookmarked file might not have the same problems as one with a heavy TOC.
philip
ptram
Posts: 280
Joined: 2007-10-21 14:59:09

Re: Long delays on "Open": display of TOC

Post by ptram »

Generating a ToC of this huge document is slow by nature, due to the live process going on.

My suggestion is to break the document into several shorter documents, each one with its own shorter ToC, save them into the same folder, and do your search by using the Finder's search function to locate the topic you are looking for. Searching with the Seach field in the folder's window immediately limits your seach to that folder and the contained documents. You will immediately locate the document(s) containing that topic, and be able to open it/them in NWP.

Paolo
David Sharp
Posts: 67
Joined: 2008-07-06 23:21:27
Location: Paris, France
Contact:

Re: Long delays on "Open": display of TOC

Post by David Sharp »

Thanks Philip and Paolo for those suggestions.
Splitting up the document is a possibility, but unless mistaken I would lose the ability to carry out "attribute sensitive" searches using Nisus's style tools, which as far as I know is not possible via Spotlight. Of course what would be really nice would be if Nisus's search tool could itself be used across several documents, but as far as I know that's not on the cards.

Philip suggests
Using a macro to generate the TOC from the styles. Since the generated TOC would be a freestanding document, it wouldn't cause any delay in the opening of the file. Depending on how this is set up, you could still use this file for navigation. Of course it would need to be periodically updated, so it depends on how much editing goes on in your file (vs. simple appending)
I think that's a potential solution. I could have the extracted TOC and the main document open next to one another, and jump from one to the other.

It is also possible that my document is becoming so complex that I should really be using a database, such as Devonthink. I'll look at that once I've got another writing project out of the way.
Þorvarður
Posts: 410
Joined: 2012-12-19 05:02:52

Re: Long delays on "Open": display of TOC

Post by Þorvarður »

Hello David,
Splitting up the document is a possibility, but unless mistaken I would lose the ability to carry out "attribute sensitive" searches using Nisus's style tools, which as far as I know is not possible via Spotlight. Of course what would be really nice would be if Nisus's search tool could itself be used across several documents, but as far as I know that's not on the cards.
You can carry out "attribute sensitive" searches across all open Nisus documents. There is a macro for that called “Find in All Documents” or “Find in All Open Documents”.
I've started using different paragraph styles to format different types of subject matter, such as nature notes, household events, dreams and so on.
[See: <viewtopic.php?f=18&t=5561&hilit=david+Sharp>]

It seems you are trying to use NWP as a database, which may not be a very good idea. However, it’s hard for us to say for sure, because we don’t know exactly how your document looks like and why you are using paragraph styles to tag different types of subject matter instead of using, for example, inline tags in brackets, which won’t slow down performance. You could, for example, tag your nature notes with [bnn] and [nne] respectively, where “b” stands for “the beginning of a nature note” and “e” for “the end of a nature note”.
Inline tagging would enable you to quickly find all your nature notes; and with Find All plus Copy you could easily extract all these notes and paste them into a separate document for further editing.

The screenshot shows how to search for all nature notes:
1.png
1.png (58.09 KiB) Viewed 16822 times
[bhe] = beginning household events
[ehe] = household events end
2.png
2.png (67.84 KiB) Viewed 16822 times
3.png
3.png (75.09 KiB) Viewed 16822 times
Plus of course, all my date headers (year, month, day) have a different style.
Does that mean that years, months and days all have different styles? How does this work exactly?

________
Þorvarður
Last edited by Þorvarður on 2016-06-27 05:40:15, edited 2 times in total.
Þorvarður
Posts: 410
Joined: 2012-12-19 05:02:52

Re: Long delays on "Open": display of TOC

Post by Þorvarður »

David Sharp wrote:It is also possible that my document is becoming so complex that I should really be using a database, such as Devonthink.
I use FileMaker Pro for projects like this. It is extremely flexible, and it works well with Nisus Writer Pro.

I use DEVONthink Pro Office too, and I find it is good as a *container* for files created by other programs. I’m afraid you wouldn’t gain much if you imported your Nisus file into DEVONthink. You would lose Nisus’ unique writing features, and DEVONthink can’t make searches based on your Nisus styles.

However, when in DEVONthink, you can prompt Nisus to open a Nisus file, but then you’re back in Nisus, and that probably wasn’t your intention when you decided to use DEVONthink.

DEVONthink relies on Apple’s text engine and this engine has problems with certain RTF features, such as footnotes. If you open a Nisus file in DEVONthink and you do some changes on the file in DEVONthink, all footnotes and styles will disappear! You won’t notice this until it’s too late, because DEVONthink can’t display footnotes.

In order to avoid this, you must tell DEVONthink that you want to open the file in the *original* program. To make sure you don’t forget this, you should *lock* the document from within DEVONthink. This will ensure that the file will *always* be opened by the original program and no text elements will be (accidentally) destroyed.
4.png
4.png (8.79 KiB) Viewed 16817 times
Most DEVONthink tutorials will just tell you that there is no one right way to use DEVONthink, and then leave it at that. This Information isn’t awfully helpful for a beginner, I’m afraid.
The following tutorial, on the other hand, shows you practical tips how to create and organize a very large project. It’s an advanced tutorial for Centre for Criminal Appeals volunteers.

https://www.youtube.com/watch?v=eP93fbBr33k
adryan
Posts: 561
Joined: 2014-02-08 12:57:03
Location: Australia

Re: Long delays on "Open": display of TOC

Post by adryan »

G’day, all

The way I see it, we probably need an opinion from Nisus regarding David’s original problem; viz, the aberrant TOC behavior and long delay on opening his long document. They may have a solution that saves him from a major redesign exercise. It would also be of interest to others thinking of embarking on such a major project.

Having said that, there is no reason why Nisus Writer Pro cannot be used as a database. Indeed, I have done so myself when faced with the deplorable state of database software for the Mac. (I used to like working with FileMaker Pro, but I could not justify the exorbitant cost of the upgrades.) But I don’t think David is using his document as a sort of database. To me, his seems a not unreasonable use of a word processor. Which is why we need an answer from Nisus about whether he has bumped up against some wall inherent in the software.

If it turns out that a single Nisus Writer document is no longer capable of delivering what David wants, I would advise a serious re-think of the whole project before attempting to migrate it to some other solution. Our collective wisdom may well be able to help with this. I think we would need to know, among other things, the intended purpose of the document, the types of access required and the reasons for the current formatting or styling choices. A major consideration is how “pretty” the document needs to be.

The reminder by Þorvarður that a Nisus Writer macro can search multiple open documents may mean that the document can indeed be split into multiple files and life continue pretty much as before. You would probably want to amend the macro so that it searched all relevant files without your having to open them all manually yourself first. And you would use a macro to augment the TOC which would have entries hyperlinked to the appropriate files.

The suggestion by Þorvarður that inline tags might be a useful alternative is also a good one. They may well speed things up, as well as offer a range of Find enhancements. However, Nisus Writer offers no way of toggling the visibility of such tags for “ordinary” reading — at least not in a way that invisible tags will survive relaunch of the application. Which is why the “prettiness” question is important. If tags are used and the document still needs to read normally, I would seriously consider migrating the document to BBEdit and recasting it as an HTML document. I would also create a CSS document in BBEdit, then view the HTML document in a Web browser. Everything could be made as pretty as one wanted. Incorporation of images would be easier, too. The Table of Contents would be a set of hyperlinks. The Web browser’s Find command would give rudimentary search capability, but for serious work (ie, using regular expressions and searching through multiple folders, whether open or not) one could use BBEdit’s sophisticated Find/Replace mechanism. Scripts of various sorts could be used for certain purposes. Nisus Writer can be used to do all this — in fact, I have used it extensively for such work — but I find the workflow for Web development is more seamless with BBEdit. A word of caution: I would first check with Bare Bones Software about any filesize limitations the program might have.

So there are other solutions, depending on David’s precise needs. But I reiterate: it is better to try to solve the original problem before taking any drastic measures!

Cheers,
Adrian
MacBook Pro (M1 Pro, 2021)
macOS Ventura
Nisus Writer user since 1996
User avatar
xiamenese
Posts: 543
Joined: 2006-12-08 00:46:44
Location: London or Exeter, UK

Re: Long delays on "Open": display of TOC

Post by xiamenese »

I concur with Adryan—I too enjoyed using FileMaker Pro but got priced out of the market—but I would suggest looking at importing it into a Scrivener project. Without knowing how his text is structured—I presume he uses something like headings—he should be able to use "Import and Split", or whatever its proper name is, without too much difficulty.

I am a long-term user of both Scrivener and NWP but not related in any way to either; I generally refrain from pushing Scrivener on these forums, but it seems to me that what David is doing is precisely the sort of thing for which Scrivener would be my first choice. I'll leave it at that, but if David doesn't know Scrivener and wishes to contact me via PM to find out more, I will be pleased to help in any way I can.

:)

Mark
Post Reply