Tinderbox User-to-User Forum (for formal tech support please email: info@eastgate.com)
http://www.eastgate.com/Tinderbox/forum//YaBB.cgi
Tinderbox Users >> Exporting from Tinderbox >> OPML for Scrivener import (ex:Notes for a Novel)
http://www.eastgate.com/Tinderbox/forum//YaBB.cgi?num=1262280542

Message started by AsafKeller on Dec 31st, 2009, 12:29pm

Title: OPML for Scrivener import (ex:Notes for a Novel)
Post by AsafKeller on Dec 31st, 2009, 12:29pm

HTML might be considered "easy", but, unless I am missing something, it does not retain the outline order; the exported folders and files are sorted alphabetically. The copy/paste approach is nice, but it does not create the individual files I would like to import into Scrivener, nor does it transfer the contents of the notes. A simple, user-friendly OPML export would be great.

Title: Re: Notes for a Novel
Post by Mark Bernstein on Dec 31st, 2009, 12:37pm

Your HTML export may be in any order you like, though the most natural order is, in fact, the outline structure.

OPML export is just a special case of HTML export -- several OPML templates have been posted in the forum and (if I recall) in the public file exchange.  I'll be away from the office for the next few days, visiting my aged mother; if you can't find the templates in the meantime, drop my an email next week and I'll dig them up.

Title: Re: Notes for a Novel
Post by Mark Anderson on Jan 1st, 2010, 1:36pm

Asaf, Exported HTML files never have an order in the way your describe it - they are just files. It is the links within the HTML themselves that give effective structure. OPML is just another form of mark-up (like HTML, XML, etc.) only less standardised than HTML. The latter fact means you need to plan which parts of your TBX need exporting - TB can't always guess your intent.

Please download and try this demo TBX file which shows how to do OPML export. Do the demo and see if the output imports correctly into Scrivener (an app I don't have/know). If so, just copy the 2 templates used by the demo across to your own documents (you can drag-drop between TBXs). If stuck on how to use the templates in your own TBX, do ask back here. Tip: to help us help you in the latter case, it would be useful if you post a small specimen TBX showing your style of TBX layout so we've a common reference.

Everyone does their export a little differently, so don't be downhearted if your first guess at wiring up your own TBX for OPML export doesn't work. I'm sure we can help.

Title: Re: Notes for a Novel
Post by AsafKeller on Jan 1st, 2010, 7:23pm

Mark: The template creates an opml file that exports flawlessly into Scrivener. Thanks much! The only thing I am having difficulty with is modifying the template so that the contents of the notes, and not only their title, are also exported. Thanks so much for your helpful, speedy and courteous response!

Title: Re: Notes for a Novel
Post by Mark Anderson on Jan 2nd, 2010, 5:28am

OK, I've prepared a new demo. NOTE: I don't believe OPML was really intended for multi-paragraph data like this so if it doesn't work you might need to look at using a different format. Let us know if this works.

Also, if this doesn't work, please post an example of an OPML file that does work as you expect.

Title: Re: Notes for a Novel
Post by AsafKeller on Jan 2nd, 2010, 6:13am

The new template does not export the notes' contents. Here is a small opml file I created with a demo of OmniOutliner; it correctly exports 'note' contents into Scrivener.
Code:
<?xml version="1.0" encoding="utf-8"?>
<opml version="1.0">
 <head>
   <title>test file</title>
   <expansionState>0,2,3</expansionState>
 </head>
 <body>
   <outline text="1 Level A" _note="this is the text that will go here">
     <outline text="1 Level B"/>
   </outline>
   <outline text="2 Level A">
     <outline text="2 Level B">
       <outline text="2 Level C"/>
     </outline>
   </outline>
 </body>
</opml>

Title: Re: Notes for a Novel
Post by Mark Anderson on Jan 2nd, 2010, 9:54am

Well, the OPML exports from Tinderbox as asked for so the error is on the Scrivener end. The code you post proves the point. OmniOutliner is exporting 'text' in a non-standard OPML <oultine> attribute called '_note'. Scrivener has clearly been hardwired to use Omni's private extension to the OPML. I did look at the Scrivener site and its manual but the latter neglects to mention how its OPML import works and what OPML extensions it requires; you might help yourself by posted a request on the Scrivener forums that they add something to the manual about OPML; if the app expects non-standard input it helps to know what the variations are.

I've posted another demo that - based on the OO example to give above - outputs code that is compatible Scrivener custom OPML extensions.

(Tinderbox can't create the <extensionState> tag as the export code doesn't give access to that info but again, the tag is a non-standard Omni addition to OPML.)

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by AsafKeller on Jan 2nd, 2010, 12:42pm


Quote:
you might help yourself by posted a request on the Scrivener forums
I shall do that. Again: thanks much for your help with this!
[edit]done:http://bit.ly/4SUIMZ[/edit]

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by Mark Anderson on Jan 2nd, 2010, 1:07pm

I think the last demo ought to solve the problem as I've fixed the TB OPML export to mirror that of OmniOutliner (OO).  The problem may be if you are putting more than one paragraph of text in your TB notes and XML (the code used for OPML) created doesn't sit well with Scrivener's XML parser. Anyway please tell me how you got on with the last demo I made for you (OPML-for-Scrivener.tbx). Specifically, you need to try the export/import process with TB notes that have zero or one paragraphs of text, before trying export with notes that may have greater than one paragraph of text. Let me know the results of that.

I appreciate you don't want to get involved in the tech side of this, but unfortunately you're asking for something for which there isn't a simple answer. A 'universal translator' is still in the realms of science fiction! The problem here is you want a push-and-go solution for something which doesn't use a consistent syntax.

My hunch is Scrivener has a kludge to accept whatever OO's OPML output is.  Hover, the latter';s not documented. Can you make an OO note with more than one paragraph of text and successfully get it into scrivener.  If so I'd need to see the file (not just the code pasted into a forum post) so I can see how OO deals with paragraph delimiters.

If OO OPML only imports to Scrivener only works when there's one paragraph of text then that's a limitation on the Scrivener end that we can't work around here.

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by AsafKeller on Jan 2nd, 2010, 1:41pm

The OPML-for-Scrivener.tbx failed to transfer text even when it originated from notes having only a single paragraph. Perhaps these posts from the Scrivener forum would be helpful? http://bit.ly/6zTu2F
Thanks again.

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by Mark Anderson on Jan 2nd, 2010, 3:20pm

It looks like there's already a solution as quoted here, in the Scrivener support thread. AmberV - the forum moderator over at Scrivener support forums - is also a Tinderbox user and pretty code savvy.  My hunch is this is a problem better followed up on the scrivener support forum. TB can emit HTML, XML, OPML, etc., but what we don't know is what Scrivener will import. From the app's author's comments it doesn't parse any OPML but rather only a very particular layout.

The forums at Scrivener sound pretty helpful so I think that's your best bet at this point.  If you know exactly what's needed but not how to produce it, call back here and we can have a go at making something suitable.

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by Mark Anderson on Jan 2nd, 2010, 6:11pm

Asaf, I got the file you sent.  It looks like we need to find a way to coerce TB note text paragraph breaks (carriage return character) into XML entity "&#10;". I'm not sure of any of the existing action or export codes do this. I think TB does it for OPML import - you might want to check directly with Eastgate support..

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by AsafKeller on Jan 2nd, 2010, 7:58pm

Thanks. I sincerely appreciate your efforts on this.
Keith, author of Scrivener, is keen to resolve the TBx-->Scrivener export issue. How do I get you folks to talk directly?

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by Mark Anderson on Jan 3rd, 2010, 5:38am

I'm just a community helper here - a fellow TB user (I don't work for Eastgate). The best thing might be for Scrivener's author to contact Eastgate directly via email (info@eastgate.com). Though I don't use Scrivener myself I'm aware a number of Tinderboxers do use both apps. I suspect 'all' we need is an extra action code to do the line break -> vertical tab transform.

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by Amber Vaesca on Jan 10th, 2010, 10:05pm

I still maintain that the best way to get a book-like structure from Tinderbox to Scrivener is via the HTML exporter engine. I have posted new, detailed instructions in the Scrivener forum in response to the sister thread, here:

http://www.literatureandlatte.com/forum/viewtopic.php?f=2&t=7244&p=59063#p59063

Using the method I describe, you'll have a fully retained outline order, rich text with long document support, and styled titles. The method is simple enough that it could be tweaked fairly easily to produce different results. There are sample files provided, and using those you can see how you can from from Tb to Scriv in ten seconds, once you have the system built.

The only downside to the method is that by working around file system definitions, titles become somewhat generic and numbered. This may or may not be a deal breaker, but if you keep your titles short (very short), you may still be able to work something in by adjusting the scene and chapter rules to `$HTMLExportFileName = $SiblingOrder + " - " + $Name`

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by Mark Anderson on Jan 11th, 2010, 5:37am

Can I suggest one small tweak to your TBX? In the 'Chapter' prototype, set it OnAdd to: $Prototype="Scene";. That way, the user only has to set the prototype for chapters as their content prototype is auto-set.

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by Amber Vaesca on Jan 11th, 2010, 8:49am

Good point, I'll make that update to the description. That's the kind of thing that would have definitely evolved out of regular usage. Likewise the Draft container could have a similar rule set up to set added children to Chapters.

For anyone else reading this: I'm also working on a variant that uses simple MultiMarkdown and exports a single file. This way, outlines of arbitrary depth can be exported and full note titles relayed into Scrivener Binder order, rather than the sequential numbering system currently used. I should have that posted later today.

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by Mark Bernstein on Jan 11th, 2010, 9:20am

The next Tinderbox beta, incidentally, has an experimental feature that encodes paragraph breaks as &#10; for the benefit of Scrivener and related OPML importers.

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by Mark Anderson on Jan 11th, 2010, 11:44am

I don't have Scrivener, but (further to Mark's comment above) I believe this specimen OPML NewOPMLTest.opml should now import correctly into Scrivener if someone could give it a try.

The outline items are from TB note names and the '_note' attributes have TB note text (where there is any) including multi-paragraph text. If it works this should give a nice fast/light interchange method, with Amber's MultiMarkdown solution providing a more comprehensive alternate approach.

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by AsafKeller on Jan 11th, 2010, 2:31pm

Mark: I cannot open the file. Is it a TBX? It downloads with XML extension. Scrivener cannot import it; IT returns "Some files could not be imported because they were of an unsupported file type or contained data that could not be read." I opened it in TextWrangler and cannot see anything obviously wrong.

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by Mark Anderson on Jan 11th, 2010, 3:08pm

No, it's an OPML file (which is an XML of sorts, I guess). If Safari tries to open it, use the menu File -> Save As and save as "NewOPMLTest.opml". Then try importin the OPML file to Scrivener.  Sometimes, web browsers tend to try a bit too hard at opening things ... and get it wrong.

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by Amber Vaesca on Jan 11th, 2010, 5:10pm

I'm getting a no-go on the OPML file, too. I've saved it properly and dragged it into the Binder with the error message stating it is unrecognised. Examining the file, it appears there are syntax errors in it. Should it really be:


Code:
<outline text="<outline text="More stuff &gt; less stuff"/>


Rather than:


Code:
<outline text="More stuff &gt; less stuff" />


If this is attempting nesting, OmniOutliner does it like:


Code:
<outline text="Blah" _note="More blah"><outline ... /></outline>


But perhaps I don't understand what is going on. And indeed see other cases where nesting is done properly.

Once I S&R `"<outline text="` with `"` and add a final </outline> to the file, it works.

There are also a lot of entities in the last note that do not relay well.

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by Mark Anderson on Jan 11th, 2010, 5:37pm

Ah, mea culpa. Let me take another look...

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by Mark Anderson on Jan 11th, 2010, 5:46pm

Try it again now (same URL - here). I'd left out a closure for a non-nested element and a closing ^endIf^. I've walked the tag nesting in BBEdit and it looks OK now.  I've put in deliberate characters that will cause problem if I've not encoded them correctly. Sorry for the earlier typo.

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by Amber Vaesca on Jan 11th, 2010, 6:28pm

Latest revision works with no modification.

The mentioned encoded characters pass straight through to Scrivener, by the way. The "standard" ones are fine in the other nodes, but the final import looks like &#xa1;Hola! in the text editor, and the title is full of these as well. Perhaps that is what you are going for, but if so there are going to be limitations in character sets, potentially.

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by Mark Anderson on Jan 11th, 2010, 6:52pm

Re your last, I'm just the monkey-with-typewriter in this, testing features! I've been mainly working on getting decent recursing OPML export templates with some level-indenting in code (to help when/if triaging the source code) and the new vertical tab feature to which Mark's alluded.

The &#xa1 is a valid hex-based XML/HTML entity for an upside-down exclamation mark. I guess Scrivener doesn't parse all HTML entities. I wonder what Scrivener outputs for text containing "ˇHola!" when exporting to OPML (does it export OPML?).

If the TB user knows what Scrivener does/doesn't parse out of XML it's probably possible to make up a rubric for TB users to follow. As a bystander, I don't see much mileage in pushing for more engineering at the TB end as it now seems to export valid OPML code (including the vertical tab for line/paragraph breaks).

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by Amber Vaesca on Jan 11th, 2010, 11:04pm

I've posted instructions and an example Tinderbox file, using the MMD method, here:

http://www.literatureandlatte.com/forum/viewtopic.php?f=2&t=7244&p=59129#p59129

As mentioned before and there as well, this will not preserve rich text styling in the note text. If you need that, it is better to suffer with generic title names for now, using the originally posted method.

P.S. Is there any way to turn off entities entirely? That is one thing that is bugging me and it looks like the only way to get around it is post-processing the exported file.

P.P.S. It looks like whatever XML engine Scrivener is using (probably a Cocoa solution) to decode the OPML file can handle named entities, but not Unicode entities.

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by Mark Anderson on Jan 12th, 2010, 3:51am

Turning off enitities - look into HTMLMarkupText, HTMLQuoteHTML and QuickLists.

Re named vs Unicode entities, I see what you mean. Though if Scrivener aims to import OPML and to support characters beyond ASCII, it really needs to extend its XML parser and I believe from comments up thread that it's author is open to fixing/changing the  OPML import.

Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by Amber Vaesca on Jan 12th, 2010, 8:42am

Yes, those were the first two attributes I played with, but to no avail. HTMLMarkupText only impacts the generation of lists, links, and so forth; HTMLEntities only impacts characters that would otherwise disrupt HTML, such as ampersand and great-than. Typographic quotes, umlauts, and other such things still get converted to entities.

With above settings tweaked to logical states:


Code:
Third “level & depth” test.


Becomes:


Code:
Third &#x201c;level & depth&#x201d; test.


Title: Re: OPML for Scrivener import (ex:Notes for a Novel)
Post by Mark Anderson on Jan 12th, 2010, 9:16am

With ^value($Text)^, I get your example to export with no enitity creation - regardless of HTMLQuoteHTML ands HTMLMarkupText state.  Even with both of those attributes as 'false' if instead I use ^text(plain)^, the smart quotes in your example are entified. The latter code variant is older and doubtless works off different assumptions about the consuming application for the output.

The ^value()^ export approach assumes you'll encode the attributes as required, e.g. ^value(escapeHTML($Text))^ is in effect doing the $HTMLQuoteHTML task.

One 'gotcha' with the ^value()^ and $Text is inline ^export^ codes aren't evaluated. That means that it may be hard to use one set of notes for HTML and XML export if the note $Text includes ^code^; you may need to re-structure your codes/templates to fit the sort of output you're making. The old and new export method don't tear down in the same way (and I'm deliberately not taking a view as to whether they should or shouldn't).

Tinderbox User-to-User Forum (for formal tech support please email: info@eastgate.com) » Powered by YaBB 2.2.1!
YaBB © 2000-2008. All Rights Reserved.