problem with Sente scanning

Everything related to our flagship word processor.
Post Reply
chabotd
Posts: 11
Joined: 2014-03-03 10:20:49

problem with Sente scanning

Post by chabotd »

Hi,
I recently switched from MS Word to Nisus Writer Pro, but although Sente is supposed to scan RTF files, I have a problem which I've submitted to Sente and am also submitting here in case the solution should come from Nisus.

When the first author's name has diacritical marks, Sente preserves them in the automatic citation ID it creates. Thus, for

Pörtner, H.O., Berdal, B., Blust, R., Brix, O., Colosimo, A., De Wachter, B., Giuliani, A., Johansen, T., Fischer, T. and Knust, R. 2001. Climate induced temperature effects on growth performance, fecundity and recruitment in marine fish: developing a hypothesis for cause and effect relationships in Atlantic cod (Gadus morhua) and common eelpout (Zoarces viviparus). Cont. Shelf Res., 21(18-19): 1975-1997.

the automatic ID is Pörtner 2001a. In Word, Sente is able to associate this ID with the reference. In my RTF Nisus document, I get
Po?rtner 2001 no match found.

I suppose I can manually edit the citation ID, but then my already written Word documents will no longer find some of the references they contain. And this solution would mean that automatic key creation would no longer be truly automatic, which is bad news.

Is there a reason why ö and other such things appear differently in .docx and a .rtf documents? Can this be fixed?

Otherwise, I will have to hope that Sente can recognize both during it scans (in a future version) or that they can give me the option of controlling the automatic generation of citation IS.

Thanks,

Denis
chabotd
Posts: 11
Joined: 2014-03-03 10:20:49

Re: problem with Sente scanning

Post by chabotd »

I did further testing and the problem is not the presence of diacritical marks. When the same citation appears several times in one document, sometimes it is recognized, formatted inline, and present in the bibliography. Other times it is not recognized. In my document, all references that were not recognized had diacritical marks.
I made a short test document containing randomly selected paragraphs from my longer document. Several references are not recognized by Sente. Yet if I copy and paste this short document into a Word file, all references are recognized, formatted, and found in the bibliography.
Back to the Nisus file, sometimes adding a few spaces before an recognized citation makes it recognizable, but this does not work reliably.
Another annoying difference in Sente's scanning behaviour for Word and Nisus files: in Word, {citation ID1; citation ID2} gives the same result as {citation ID1}{citation ID2}: you get a set of bracket and 2 citations. When the file is RTF (Nisus), the second method gives 2 citations.
The 2 test files, before and after scanning, and the small Sente library, are included for whoever wants to test this. Woops, file too big at 600K, here is a link to get it: https://dl.dropboxusercontent.com/u/612 ... 20test.zip

I thought the problem was with Sente, but I just tried something new and I am not so sure anymore: I took the Word document and exported (from Word) into the RTF format. I asked Sente to scan that file. All references were recognized, although the problem with {citation ID1}{citation ID2} giving 2 citations remained. This result suggests that Nisus does something different than Word even if the file format is RTF in both cases, and this difference makes it difficult for Sente to recognize citations ids containing diacritical marks. Sometimes it succeeds, but not always. Any thoughts?

DC
User avatar
xiamenese
Posts: 543
Joined: 2006-12-08 00:46:44
Location: London or Exeter, UK

Re: problem with Sente scanning

Post by xiamenese »

I suspect this is yet another point at which the problem arises of the way Word references UTF-8 characters in the non-ASCII — or is it non-ANSI — set, and the way the Apple text-engine on which Nisus is based does it. There was a thread some time ago concerning French guillemets not appearing properly, and as I can personally testify, makes collaborating on a Scrivener project involving Chinese with a PC-using friend impossible.

Your saying that it works sometimes and not others suggests to me that some of your citations have been entered in Word others in Nisus.

No, expert here, but … I suspect the problem will potentially affect anyone using an Apple text engine-based app — Nisus, Scrivener, Yojimbo, Bean and many others — in conjunction with Word.

Mark
chabotd
Posts: 11
Joined: 2014-03-03 10:20:49

Re: problem with Sente scanning

Post by chabotd »

Hi Mark,

I do not think this is the problem in this particular case. I have tried copying my text and pasting it in Textmate, with the option "show invisibles", and there was nothing weird. Also, one of the first things I tried when I discovered that the same citation was recognized some times, but not always, was to copy and paste a successful citation to replace unsuccessful ones. This had no effect.

DC
User avatar
xiamenese
Posts: 543
Joined: 2006-12-08 00:46:44
Location: London or Exeter, UK

Re: problem with Sente scanning

Post by xiamenese »

I was only guessing, but the problem is not the introduction of invisibles, but rather it seems that there are two different legal ways of addressing glyphs in UTF-8 and Apple uses one, Microsoft the other, so problems arise, particularly when other apps — Sente, Scrivener — are involved. I don’t use Sente as I found when I tried it that the interface got in the way, whereas Bookends suits me fine, so I can’t really comment on Sente; and I don’t have Word at all, though I spend a lot of time working with Word users on both platforms. I always use RTF for exchanges, it saves so many problems.

Mark
chabotd
Posts: 11
Joined: 2014-03-03 10:20:49

Re: problem with Sente scanning

Post by chabotd »

I have not received any help from Sente's team yet, but found something on an older message on Sente's site that suggests that Mark was on something very relevant: <https://sente.tenderapp.com/discussions ... n-scanning>

I am not sure I follow the explanations given there because I copied a citation that Sente had recognized and pasted it on top that Sente did not recognize, and this pasted citation was not recognized by Sente.

Can someone from Nisus help me understand what is going on with these question marks followed by carriage returns?

Thanks,

Denis
chabotd
Posts: 11
Joined: 2014-03-03 10:20:49

Re: problem with Sente scanning

Post by chabotd »

chabotd wrote:I have not received any help from Sente's team yet, but found something on an older message on Sente's site that suggests that Mark was on something very relevant: <https://sente.tenderapp.com/discussions ... n-scanning>

I am not sure I follow the explanations given there because I copied a citation that Sente had recognized and pasted it on top that Sente did not recognize, and this pasted citation was not recognized by Sente.

Can someone from Nisus help me understand what is going on with these question marks followed by carriage returns?

Thanks,

Denis
I tried the workaround mentioned in the thread cited above: I opened my RTF file with TextMate and replaced all "carriage return followed by ?" by "space followed by ?". Saved this, asked Sente to format the bibliography, it worked. All is well, you might say? No, every time I open and save my file with NWP, some of these "carriage return followed by ?" are recreated. Unless you guys (at Nisus) discover why this happens, I am condemned to do this every time I need to reformat my bibliography: close the file in NWP, open it in TextMate, do the global search and replace, save the file, finally ask Sente to format the bibliography". Not quite the "trouble-free power word processing" I was hoping for. More complicated than in Word, to my dismay.

I am also disappointed that nobody at Nisus is getting involved and letting me know that they are aware of the problem.

Denis
NisusUser
Posts: 317
Joined: 2011-01-12 05:32:38

Re: problem with Sente scanning

Post by NisusUser »

chabotd wrote:
chabotd wrote:I am also disappointed that nobody at Nisus is getting involved and letting me know that they are aware of the problem.Denis
It's a bit strange that no one from Nisus has chimed in yet. Usually they do so quickly. Maybe you could try Help > Send Feedback and see if they get back to you. You might want to paste in the URL to this thread.
chabotd
Posts: 11
Joined: 2014-03-03 10:20:49

Re: problem with Sente scanning

Post by chabotd »

Thanks, I did this 5 days ago. Nothing concrete yet except I confirmation that my message was received. Denis
User avatar
martin
Official Nisus Person
Posts: 5227
Joined: 2002-07-11 17:14:10
Location: San Diego, CA
Contact:

Re: problem with Sente scanning

Post by martin »

We're sorry about the delay in getting back to you on this issue, and apologize for this problem you're having with Nisus Writer Pro (NWP) and Sente.

We've investigated the problem and here's the situation: RTF is a flexible file format, and there are a variety of ways to save/encode the same information. When it comes to the "ö" character in your document, Microsoft Word and NWP are saving the information in a slightly different way. Both are proper RTF (you may observe that opening NWP's file in Word shows the correct text), but Sente is stumbling on this during its scan.

However, we strive to make NWP's RTF as compatible as possible. We've found a way to adjust our RTF so that Sente no longer has trouble scanning these umlauted characters (and others that require special handling in RTF). We plan to include this fix in the next update to NWP.

That said, Sente's scanning process could still be improved, and it's good that you submitted the problem to them. In fact, if I take your file and open it in Microsoft Word, save it as RTF, then Sente still has some problems with one of the references: no match is found for "Pöörtner 2001a". The error is different than with NWP, but Sente also sometimes has trouble scanning Word's RTF.

In the meantime I'm not sure there are any workarounds (aside from editing the RTF every time, as you discovered). The errant Sente scan is triggered by the internal sequencing of the RTF codes, and isn't something you'll be able to affect using NWP or Word directly. Sorry to not have better news.

Please let me know if you have any questions.
chabotd
Posts: 11
Joined: 2014-03-03 10:20:49

Re: problem with Sente scanning

Post by chabotd »

Hi Martin,

I am really looking forward to the update that will help with this problem. Other examples of problems that are unfortunately common in my documents: the capital M with a dot above (most of them are followed by a question mark after scanning. Same for the degree symbol.

If you need beta testers...

Denis
Post Reply