Welcome, Guest. Please Login
Tinderbox
  News:
IMPORTANT MESSAGE! This forum has now been replaced by a new forum at http://forum.eastgate.com and no further posting or member registration is allowed. The forum is still accessible via read-only access for reference purposes. If you wish to discuss content here, please use the new forum. N.B. - posting in the new forum requires a fresh registration in the new forum (sorry - member data can't be ported).
  HomeHelpSearchLogin  
 
Pages: 1
Send Topic Print
Tinderbox to DOT (Graphviz) (Read 8947 times)
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Tinderbox to DOT (Graphviz)
Apr 12th, 2010, 5:04pm
 
Looking at ways to export TB maps, I had a go at exporting to the 'DOT' format used by Graphviz but which can also be used by Omnigraffle. My initial efforts for Graphviz (here and here) and Omnigraffle (here).

Attempting to get closer to a TB map, I added 'subgraphs' to the DOT - approximating to TB adornments - and the Graphviz results were far less useful: here. compare this with the lowest zoom of the source TB map, here. In fairness the Graphviz maps use an automatic layout whilst the TB map was constructed by using smart adornments, then turning of the 'smart' queries and doing a little manual rearrangement.

Last note for anyone contemplating this. You'll need to do a little post-processing to get from what TB can make to the final DOT file but I was happily impressed by what TB would do with appropriate templates.
Back to top
 
 

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Paul Walters
Full Member
*
Offline



Posts: 267

Re: Tinderbox to DOT (Graphviz)
Reply #1 - Apr 12th, 2010, 6:41pm
 
Very interesting, Mark - would you be able to share the templates?
Back to top
 
 
  IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Tinderbox to DOT (Graphviz)
Reply #2 - Apr 13th, 2010, 2:25am
 
Make a template called 'text' whose code is just:
Code:
^text^ 

Now make a note - mine was called 'DOT Export' and set the 'text' template as its HTML Export template.  Make $Text of the note this:
Code:
digraph GRAPH_0 {

  edge [ arrowhead=closed ];
  graph [ style=filled, rankdir=LR ];
  node [
    fontsize=11,
    fillcolor=white,
    style=filled,
    shape=box ];

^include(/DATA,"/Templates/Dot-Wrapper")^

} 

This assumes a note 'DATA' is the container of all the export data. The 'Dot-Wrapper' template code:
Code:
^children("/Templates/Dot-Item")^ 

The 'Dot-Item'template code:
Code:
^if(!$Separator)^^value(MyString)^^endIf^ 

N.B. the COIN map has separators we don't want to export. No recursion downwards needed as all data is in the one map.

To make the $MyString data, I added string attribute $MyStringand set attribute $MySet. In the prototype for all the data notes I set this rule:
$MySet=links().outbound..$Name;$MyString=format($MySet,"","*"+$Name+"* -> *","*\n,"");

Unfortunately format() doesn't allow use of single quotes vice double as what DOT requires are double-quote enclosed node names (unless single words/no odd chars). So we use an asterisk * instead (so no asterisks in names please!) that we'll clean elsewhere. I used BBEdit but you could use HTMLExportCommand with tr or sed.

Export 'DOT Export' to HTML. Replace the * with " and you're good to go.

DOT has UNIX-style documentation - i.e. a list of settings but no explanation of how to use them or how they work. For instance, the order things appears in the DOT file likely affects the layout. Possibly obvious after the fact but just the sort of thing it should say at the top of the docs. So, you'll find yourself experimenting a bit.

There's certainly no way to make a generic TB that will deal with any note data, e.g. the issue with quotes above, but the rubric I've given is easy to apply.

If you're going to Omnigraffle, IIRC you need to pick the 'directed' layout option when asked by OG.

I think that's it...

[Later] I should add, this site was useful for figuring DOT syntax - use the option to show raw output code.
Back to top
 
« Last Edit: Apr 13th, 2010, 12:04pm by Mark Anderson »  

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Paul Walters
Full Member
*
Offline



Posts: 267

Re: Tinderbox to DOT (Graphviz)
Reply #3 - Apr 13th, 2010, 11:57am
 
Mark

Excellent work.  Thank you for posting this.  Exporting maps will be very helpful; so I best learn Graphviz.

One small change to the rule - quotes in the third token in the format() string:
Code:
$MyString=format($MySet,"","*"+$Name+"* -> *","*\n",""); 

Back to top
 
 
  IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Tinderbox to DOT (Graphviz)
Reply #4 - Apr 13th, 2010, 12:41pm
 
Good spot (my original now corrected). Guilty secret - I originally used ' to get around the format() problem before realising some note names have apostrophe. Only after doing the export work did I change the delimiter to an asterisk (actually with a hint, off forum, from Mark B). So a poor update edit by me of my original code!
Back to top
 
 

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Paul Walters
Full Member
*
Offline



Posts: 267

Re: Tinderbox to DOT (Graphviz)
Reply #5 - Apr 14th, 2010, 9:02pm
 
Hello Mark,

I've made a few revisions to the model you proposed.  At this point I wanted to streamline the export to DOT format.  My suggestions are in the file at this drop.io location: http://drop.io/TBX2DOT.

Here are the changes:

1) The use of *, with later substitution in a text editor can be avoided because Graphviz accepts <> as a string delimiter in label statements.  So, [label=<this is a label>] is legal.

2) A node in Graphviz can be defined, given a label, and later used in a reference - and the label is displayed rather than the node name.  To make simpler files for human readers to examine, I make use of this feature to give each note a unique node name ($NodeID) that is a string composed of OutlineDepth & OutlineOrder.  I also create an attribute, $NodeString, to serve as the node definition.  So, if the second note in a container has $NodeID of 21, and the note name is "This Test Note", then the exporter creates a node definition statement, thus:

Code:
21 [label=<This Test Note>]; 


is a node named "21" that will display with the label, rather than "21", including in nodes with edges (the Graphviz term for links).

3) I modified the export template scheme.  The container for the exported notes has a shell template (named "digraph" in the linked file) that merely includes ^children()^.  The prototype for exported notes is assigned an HTMLExportTemplate (named "node" in the file) that exports $NodeString and nodes for each outbound link.  (I'm using essentially the same technique you do for creating a link set and a formatted export string for that set).

4) I'm using text exports rather than HTML exports.  For some reason, HTML export files, even with proper form and gremlins zapped in BBEDIT, cause crashers in the Graphviz engine.

(I'm not using OmniGraffle - it honors a subset of the Graphviz standard and doesn't render complex digraphs well.  The standard Graphviz engine for Mac (download from graphviz.org)is powerful and can save rendered digraphs in an enormous range of formats.]

There's one vexing problem.  Graphviz edges (link lines) can have labels as in Tinderbox.  But I haven't worked out how to access the Tinderbox link label.  If there is something obvious I'm missing, I'd be pleased to hear it.  Otherwise, it would be nice if Tinderbox had a feature to access the link label text.  With that, then adding the Tinderbox label to Graphviz graphs as edge labels would be trivial.

On the plus side, Graphviz supports URLs and has a "record" type node that has interesting possibilities for combining any number of notes into one structure in the graph.  I plan to play with those things next.

Thanks again for posting the possibilities of using Graphviz exports.
Back to top
 
 
  IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Tinderbox to DOT (Graphviz)
Reply #6 - Apr 15th, 2010, 5:25am
 
Very interesting. I wondered if anyone would take this further - glad to see it happen.  I'd agree on Graphviz vs. Omnigraffle. Indeed, I see Graphviz can export DOT. I've not tried but I suspect it might 'tidy' our learner DOT into something cleaner as it is reversing the graph as rendered to DOT regardless of the original source being DOT (worth a try!). Of course, OG offers scope for prettifying the final output.

Re your points...

1. Doing inline substitution via a command line in $HTMLExportCommand is not a big deal.  My early demo didn't include that. The < > idea is neat but reviewing the docs you're using the HTML form of annotation (of which there are no clear examples) and that form should be XML compliant so you should either hard code &lt; vice <, &gt; vice > and &amp; vice & or choose an action/export function that does that for you. Otherwise, as you scale your test to production level data unexpected things may happen. Pending TB offering support for format allowing both ' and " as format() delimiters (don't know if it will) then a simple tr or sed call in $HTMLExportCommand should yield clean DOT data.

2. Yes, much better! I was intending something like that but my test got bogged down in the mere basics. This label approach is much better. Don't bother with manually set note identity strings (too much for real would use!). Instead, make a sequential number user attribute.  This will give all notes/agents/adornments including pre-existing ones a discrete number value. As at v5.0.2, aliases are an exception to this as sequential numbers aren't intrinsic; the issue has been raised though**.

3. Actually my digraph 'wapper' isn't so different from yours.  the extra node/edge definition are effectively prototype info. It's the look nodes/edges will use unless altered at sub-graph or object level by further code.

4. I'm 99% certain this issue is the 'BOM' added to HTML export files as from v5.0.2 (and likely to go away again in subsequent releases***). My understanding is the Text Export shouldn't really use includes (due to the way Nakakoji view works) so I'd recommend sticking with HTML Ex[port for now. When your file is output, simply open in BBEdit or the like and check the file is UTF=8 with no BOM. If not, change to that mode and save; that should solve the issue until BOM-less export returns.

5. Exporting list of links by type. you need to (manually, I'm afraid) tease out this:
$LinksOut=links().outbound..$NoteID;
to
$LinksOut1=links().outbound.agree.$NoteID;
$LinksOut2=links().outbound.disagree.$NoteID;
$LinksOut3=links().outbound."some label".$NoteID;
etc.

TB has no arrays or loops so you can't iterate all link type or an array of some link type names. Basically, you need one 'links out' construction per desired link type (a pain if you've lots!) then either
  • simply include each them in your note template, perhaps in a if() clause so the data's only written if the attribute holds any value
  • concatenate all the links data to one attribute and then call that in the template.
The former method may be better as it offer finer control. Also, in your digraph 'root' template - or in an include to it, predefine the look of each link type (bold, dotted, label, etc.) and remember to alter the individual link -> statements to indicate the correct link type for DOT to draw.

Quote:
if Tinderbox had a feature to access the link label text

Indeed, this would offer a different approach. I sense the need is niche - in terms of engineering effort vs. the number of active users of the feature - so the above is the way to go for now. Of course, by all means make a feature request.

~~~~~~~~~~~~

** The answer is not necessarily as simple as giving an alias a discrete value. Aliases get created and deleted a lot, so using up sequential numbers are a massively faster rate than ordinary note creation. Anyway, if this issue affects your use, I'd suggest dropping a line directly to support explaining your problem/needs. It's probably a bit niche to warrant forum discussion.

*** BOM = Byte Order Marker.  In theory a good thing and designed to help BOM-aware apps to get the correct encoding of a file. However, it seems real-world development took a different path and BOMs do more harm than good - for instance they make most HTML pages fail any common validation test. On balance, BOMs haven't helped and I believe will likely get dropped in the next release. Again, if people have particular desires in this context, email TB support direct outlining your issues (it's very techie stuff most users don't need to see/understand).
Back to top
 
 

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Paul Walters
Full Member
*
Offline



Posts: 267

Re: Tinderbox to DOT (Graphviz)
Reply #7 - Apr 15th, 2010, 6:07am
 
Mark

Thank you for your thoughts on this.  Very helpful.  I had not considered the BOM problem and was thinking there was something wrong with my file system - good to know it's a fixable thing.  I realize that accessing link label might be a niche request - and complex.  OTOH, the data is all in the .tbx file so I might look into writing a transform on a copy of that to produce the digraph markup.  The <links> section of the file has just about everything that's needed to drive the transform.
Back to top
 
 
  IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Tinderbox to DOT (Graphviz)
Reply #8 - Apr 16th, 2010, 5:27am
 
Googling for some code examples I found a few more useful wrinkles.
Code:
graphG{
node[shape=box,style=filled];
{node[width=.3,height=.3,shape=octagon,style=filled,color=skyblue] A1 A2 A3} 


In this only the nodes A1, A2 and A3 will have the special shape/colouring, all others will be a filled box. Note this isn't a sub-graph being defined, simply a sub-clause - if you will - delimited by { } that defines a particular node style.

Pulling off the same for edges (think - different TB link type styles) is not so easy, given how we're generating them on a source note by note basis. However, we can leverage the top-down parsing of the DOT code. An objects style remains the same until a new global declaration is made. So as we pass out each discrete link (edge) type we start it with a style declaration:

[post-edit - forgot the " around the colour values in this and the other code snippet in the original post]
Code:
edge [label="likes", fontcolor="#009900", style="solid,bold"]
Curly -> Moe
Larry -> Moe

edge [label="dislikes", fontcolor="#990000", style="dashed"]
Curly -> Larry 



It may be easier to turn the link listing export on its head and export all links of a given type in one go as each style only needs to be declared once. To do this I'd run template code in the contect of the (actual) parent of the original notes  - 'DATA' in my COIN ,map example. Then we can make calls like:

  links(child).outbound."Significant Delay".$Name

The above will get all outbound Significant Delay type links for the whole map enabling you to declare the styling of that link type only once.  If there are are more than a few I think this will make for a less verbose DOT file saving you doing this:

Code:
edge [label="likes", fontcolor="#009900", style="solid,bold"]
Curly -> Moe
Larry -> Moe

edge [label="dislikes", fontcolor="#990000", style="dashed"]
Curly -> Larry 



OK, you still have to set up the link type names/style info by hand but for a stable doc you do that once - perhaps amending if link types change.

I hope that helps
Back to top
 
« Last Edit: Apr 16th, 2010, 8:50am by Mark Anderson »  

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Paul Walters
Full Member
*
Offline



Posts: 267

Re: Tinderbox to DOT (Graphviz)
Reply #9 - Apr 16th, 2010, 7:13am
 
Mark, your recent post is very interesting.  The syntax hints and approach is going to be helpful.  I find that I need to put the hex codes for colors into double quotes else the Graphviz interpreter ignores them.

Here's another approach (sorry, this code wraps because it is very long - the spaces before [label are needed in the DOT).

Code:
^outboundBasicLinks("",^getFor(this,$NoteID)^->++,++^getFor(destination,$NoteID)^  [label="Type 1 Link"]"\n","","type 1")^ 


in the node template, with one such statement for each link type, will yield all the properly labeled nodes.  If a note has no links or no links of a type, then the statement is skipped.

Two problems with this approach:  

(1) The html link is output - hence the "++" offsetting the link so that the text inside the crosses can be zapped in a BBEDIT textfactory (or other means).  I don't like this postprocessing stage, but haven't worked around it yet.

(2) To get proper line breaks the code "\n" is needed - this outputs a proper line break - but with the preceding and succeeding double quotes, which then need to be zapped in BBEDIT.  This is my own limitation with Tinderbox syntax, and maybe you have a suggested revision?
Back to top
 
« Last Edit: Apr 16th, 2010, 7:16am by Paul Walters »  
  IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Tinderbox to DOT (Graphviz)
Reply #10 - Apr 16th, 2010, 8:48am
 
Colours  - ah yes. In my last post I was quoting code that used a named colour (which we can't map from TB - at least not without more code) and in adding a hex colour I forgot to add the quotes. IIRC they're needed as otherwise the # hash symbol has DOT code syntax significance. (I'll amend the earlier code for subsequent forum readers).

Nice outbound links trick.  I'd change the "->++" bit to " -> ++" as I think you may need a space either side of the '->'.

Although - like in JavaScript - terminating lines with a semi-colon is optional, I think it best to always give a deliberate line end to your DOT code.  As with not using closing ^ with TB export code, by using line endings the app reading the DOT has to make fewer guesses [sic] as to what you intended.

Why not replace the \n with a @@ and we'll replace that too. Here's a simple TB p-o-c. Make $MyString, $MyStringA. Make a note and give $MyStringA this value:

ant -> ++<a href="bee.html">bee</a>++bee;@@ant -> ++<a href="cow.html">cow</a>++cow;

Note set this Rule in the note:

$MyString = runCommand("sed 's/++\([^+]*\)++//g' | sed 's/@@/\n/g'",$MyStringA);

Now select the contents of $MyString and paste it into the text pane - as it (should be) more than one line:

ant -> bee;
ant -> cow;

You can run the command line anywhere - this is just so you can test. $HTMLExportCommand would make more sense. You make its value:

sed 's/++\([^+]*\)++//g' | sed 's/@@/\n/g'

The only problem with the above outbound links process is the it only works if the note names are single words. Otherwise they must be double-quote enclosed; yes only ", as DOT doesn't allow single quotes. OK so put ~~ where you'd put the ", e.g.

~~ant~~ -> ++<a href="bee.html">bee</a>++~~bee~~;@@ ...etc.

Now add to our CL:

sed 's/++\([^+]*\)++//g' | sed 's/@@/\n/g' | sed 's/~~/"/g'

Except we can't. That won't compile. TB won't let us escape a single ". Bah!  Let's put the CL in the $Text of another note called 'CL', i.e.:

sed 's/++\([^+]*\)++//g' | sed 's/@@/\n/g' | sed 's/~~/"/g'

Now change the original $Rule to:

$MyString = runCommand($Text(CL),$MyStringA);

Hmm. Now the line break's gone AWOL. But, if we now change the code in CL and replace the \n with an actual line break, as in:

sed 's/++\([^+]*\)++//g' | sed 's/@@/
/g' | sed 's/~~/"/g'


Tada! At least, it works here. Took some prodding but Tinderbox delivers again!

[Later edit for typos]
Back to top
 
« Last Edit: May 5th, 2011, 10:10am by Mark Anderson »  

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Pages: 1
Send Topic Print