Welcome, Guest. Please Login
Tinderbox
  News:
IMPORTANT MESSAGE! This forum has now been replaced by a new forum at http://forum.eastgate.com and no further posting or member registration is allowed. The forum is still accessible via read-only access for reference purposes. If you wish to discuss content here, please use the new forum. N.B. - posting in the new forum requires a fresh registration in the new forum (sorry - member data can't be ported).
  HomeHelpSearchLogin  
 
Pages: 1
Send Topic Print
Importing XML into tinderbox (Read 28047 times)
minty
Full Member
*
Offline



Posts: 3

Importing XML into tinderbox
Jan 27th, 2012, 9:31am
 
Hi all -I've been looking to really 'break into' tinderbox for years, and have faithfully upgraded while waiting in the wings.

I have an interest in medical informatics, (but have been in a primarily clinical role - which doesn't afford the time at a computer to 'break in', and doesn't afford a whole bunch of extra time to do so).  Currently though, I am working in the area of medical ontology.

While an eventual destination for the work is an ontology tool like protege / OWL ; I think Tinderbox would be a good environment to 'conceptualize' the relationships between entities.

I have been able to generate a list of medical concepts in both text and xml.

Acute rheumatic fever                                                
     Rheumatic fever without mention of heart involvement                                          
           Pneumonia in rheumatic fever without mention of heart involvement                                    
           Erythema marginatum rheumaticum                                    
     Rheumatic fever with heart involvement                                          
           Acute rheumatic pericarditis                                    
           Acute rheumatic endocarditis                                    
           Acute rheumatic myocarditis                                    
           Other acute rheumatic heart disease                                    
           Acute rheumatic heart disease, unspecified                                    
     Rheumatic chorea      ...

Or in XML..                                    

<Heading0>Chronic rheumatic heart diseases</Heading0>
       <Heading1></Heading1>
       <Heading2></Heading2>
       <Heading3></Heading3>
       <Heading4></Heading4>
       <Heading5></Heading5>
       <Heading6></Heading6>
       <Heading7></Heading7>
       <Heading8></Heading8>
   
   
       <Heading0></Heading0>
       <Heading1>Rheumatic mitral valve diseases</Heading1>
       <Heading2></Heading2>
       <Heading3></Heading3>
       <Heading4></Heading4>
       <Heading5></Heading5>
       <Heading6></Heading6>
       <Heading7></Heading7>
       <Heading8></Heading8>
   
   
       <Heading0></Heading0>
       <Heading1></Heading1>
       <Heading2>Mitral stenosis</Heading2>
       <Heading3></Heading3>
       <Heading4></Heading4>
       <Heading5></Heading5>
       <Heading6></Heading6>
       <Heading7></Heading7>
       <Heading8></Heading8>
   
   
       <Heading0></Heading0>
       <Heading1></Heading1>
       <Heading2>Rheumatic mitral insufficiency</Heading2>
       <Heading3></Heading3>
       <Heading4></Heading4>
       <Heading5></Heading5>
       <Heading6></Heading6>
       <Heading7></Heading7>
       <Heading8></Heading8>

But can't figure out how one can get an app like Tinderbox to recognize the hierarchy (which makes sense, as it was exported from a table).  I recognize that this is not a tinderbox weakness at all- and am looking into the best way for a newbie to break into the world of XML and transformation scenarios, but unsure if anyone here has done something like this within tinderbox itself, (or has suggestions about where to turn).

Once I get this stuff INTO tinderbox, I'm sure you'll be hearing from me about the world of adornments and agents etc.  Looking forward to this!

Evan
Back to top
 
 
  IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Importing XML into tinderbox
Reply #1 - Jan 27th, 2012, 11:50am
 
Generate the list as OPML. The latter is XML based. For drag/drop imported OPML dataInbound TB maps the OPML attributes as follows:
  • 'text' -> $Name
  • '_note' -> $Text
  • '_status' -> $Checked
TB ignores the OPML expanded state data (i.e. which branches are expanded. (more here).
Back to top
 
 

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Mark Bernstein
YaBB Administrator
*
Offline

designer of
Tinderbox

Posts: 2871
Eastgate Systems, Inc.
Re: Importing XML into tinderbox
Reply #2 - Jan 27th, 2012, 11:54am
 
You should be able to transform your XML to OPML easily through XSLT, and then you're home free.
Back to top
 
 
WWW   IP Logged
minty
Full Member
*
Offline



Posts: 3

Re: Importing XML into tinderbox
Reply #3 - Jan 30th, 2012, 7:40am
 
Dammit Jim!!

Alright, so I had to learn XSLT.  Building on that 1 semester of C++ 12 years ago.  But the data is in there now.  Thanks for the tips!

Back to top
 
 
  IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Importing XML into tinderbox
Reply #4 - Jan 30th, 2012, 10:11am
 
Worth noting if you subsequently want to push TB data back out to XML that's no problem; no XLST needed your end. TB's 'HTML' export process is for any form of markup so just set up your own XML templates as required. Note too, the action code function attributeEncode() which will safely encode data for insertion in XML attributes, etc.
Back to top
 
 

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Jacek Gwizdka
Full Member
*
Offline



Posts: 6

Re: Importing XML into tinderbox
Reply #5 - Jul 16th, 2012, 4:15pm
 
(Another TB newbie question) Following on this older thread - so, if I want to write out data from any other application (and I want to do it from Zotero) for import into TB, I'd need to write it in OPML format. Where can I find a detailed specification of TB file formats (for various objects supported by TB). (Is that it: http://www.acrobatfaq.com/atbref5/index/ImportExportFormatting/OPMLImport.html  ?)

thanks
- Jacek
Back to top
 
 
  IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Importing XML into tinderbox
Reply #6 - Jul 16th, 2012, 5:30pm
 
Step back a bit, there is no formal import API in Tinderbox v5, nor is there a general (data) import menu. However, there are number of ways tbring in data via drag/drop or blind pasting into major views (pasting when no notes are selected).  The methods are described in the Import/Export section of aTbRef. In terms of data formats for large sections of data, there are really two:
  • Tab-delimited data. Note that TB doesn't strip enclosing straight double quotes from cell/field values (such as some processes create by default. However, field values containing line break will import correctly if quote enclosed. If you need to support import of quoted values, use of .replace() should enable the unneeded quotes to be stripped after import. The format is essentially 'flat' - i.e. all notes created are siblings within the same container. Rich text import is note supported (though rich-text data can be copy/pasted to TB note $Text on a note-by-note basis.
  • OPML. This is as already discussed. Allowes hierarchical import. Plain text only.  Attributes supported: $Name, $Text, $Checked. Anything else is best packed into $Text and then exploded or action code derived from it post import.
Separately, I believe that post v5.11.2 TB understands Cmd+drag imported RIS format data such as produced by Bookends (see more).

Footnote: if you're unfamiliar with achieving successful inter-app data transfe between apps with no formal interchange method, do understand that 'standards' like those above are quite loosely defined. So, App A and App B may both meet the standard and yet data from each may still not be compatible. I say this as the user's first instinct is to blame this on 'bugs' when in reality its simply different readings of the same loose description (the tab-delimited/quotes issue above is a typical such mismatch issue). Not what the user wants, but not the fault of the vendor. Richer inter-app import, like TB's recent ability to read a Scrivener file involves quite a bit of unseen collaboration and testing by both vendors. Zotero not having an owner/vendor adds to the problem, but doesn't make it insurmountable.
Back to top
 
« Last Edit: Jul 16th, 2012, 5:38pm by Mark Anderson »  

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Jacek Gwizdka
Full Member
*
Offline



Posts: 6

Re: Importing XML into tinderbox
Reply #7 - Jul 16th, 2012, 6:13pm
 
Drag/drop and pasting are slow - for manual use. I want to export from another app and import to TB many items at a time.

So, I guess more appropriately my question is about a detailed specification of .tbx XML format. Does it exist?

I would want to export data from another application and write it directly to a .tbx XML file.

thanks
Back to top
 
 
  IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Importing XML into tinderbox
Reply #8 - Jul 16th, 2012, 6:27pm
 
There is no public documentation of the Tinderbox file format.  For such matters, contact support directly - it's outside the scope of the forum.

OPML can handle lots of data. For a current bit of consultancy (which I can't show here) I'm taking OPML from another app. A single [sic] file drag/drop is creating over 16,000 new notes in Tinderbox and pretty much instantaneously. Slow, it isn't!
Back to top
 
 

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Jacek Gwizdka
Full Member
*
Offline



Posts: 6

Re: Importing XML into tinderbox
Reply #9 - Jul 16th, 2012, 6:54pm
 
Indeed that's quite fast. I was thinking of one-file-per-item and hence the projected slowness.  

Does OMPL allow for specifying links between items?
(not clear if element of type link supports links inside the same document)

Back to top
 
 
  IP Logged
Mark Bernstein
YaBB Administrator
*
Offline

designer of
Tinderbox

Posts: 2871
Eastgate Systems, Inc.
Re: Importing XML into tinderbox
Reply #10 - Jul 16th, 2012, 8:12pm
 
Google OPML for lots of information, or see the specs at OPML.org (server is down right now, but should be back presently.)
Back to top
 
 
WWW   IP Logged
Jacek Gwizdka
Full Member
*
Offline



Posts: 6

Re: Importing XML into tinderbox
Reply #11 - Jul 16th, 2012, 8:24pm
 
My question came after I examined OPML 2.0 docs at http://dev.opml.org/spec2.html . The description of links seems to indicate that links are to outside entities only.    (opml.org is down indeed).
Back to top
 
 
  IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Importing XML into tinderbox
Reply #12 - Jul 17th, 2012, 8:05am
 
[opml.org is back up!]

The basic OPML spec allowed for an outline element whose type attribute was link that would also have a url attribute whose value would be a valid URI. An OPML v1.0 example is here. The whole 'type' attribute method is fairly arbitrary and open.

OPML 2.0 added a new type value of include where the url value is always an OMPL file (example here & sample referenced include). This feature is not currently supported by TB. FWIW, in the several years that OPML has been discussed here by TB users as an interchange format I've not seen anyone ask for include support. So that's likely the reason it isn't supported.

It's worth noting point #3 in the 'Shortcomings' section of the Wikipedia article on OPML, by way of managing expectations for what OPML-based intechange can offer:

Quote:
Due to the arbitrary nature of the "type" attribute, and the acceptance of arbitrary attributes on "outline" elements, interoperability of OPML documents relies almost entirely on the undocumented conventions of content producers.

TB's OPML is closer to following the unofficial 'flavour' of OPML popularised by Omnigroup which centres around just these per-note characteristics:
  • title: OPML "text" attribute, TB's $Name.
  • text: Omni addition OPML "_note" attribute, TB's $Text
  • checked status: Omni addition OPML "_status" attribute, TB's $Checked.
  • creation date: OPML "created" attribute, TB's $Created.
As it happens, TB maps any other OPML attribute to a new TB String-type attribute using the OPML attributes name (and case)So, if an OPML file includes categories (tags) data of "xx/yy/zz" - a slash being OPML's tags delimiter, TB with create $categories and insert the value "xx/yy/zz". To put that into a TB set-type attribute $Tags you'd simply find notes with a value for $categories and then use this code to move/clean the tags values and clear out the source attribute:
$Tags = $Tags+ ($categories.replace("/",";")); $categories=;

It's worth noting that the user has no access to OPML->TB attribute mapping, which is why the extra steps are needed in the last example above.

Quote:
Does OMPL allow for specifying links between items?

No and Yes. No, it doesn't expect/define linkage within the OPML <outline> items. Yes, in that use of the 'url' type attribute can define a line to the represented iyem via some other protocol usch as HTTP or RSS.  DevonThink, Bookends and some other apps extend accessibility further by (on the computers on which they're installed) by special protocols. For instance, on my Mac (only) this reference 'bookends://sonnysoftware.com/40489' opens a reference in my Bookends library; on a different Mac with Bookends it will open Library item 40489; on a Mac without Bookends nothing will happen. FWIW, I've searched but Zotero seems to lacks such an external reference protocol.

Tab-delimited import seems a much easier and simpler method to use for your Zotero -> TB needs. You configure you Zoerto export template, select the Zotero items of interest and export. The column headers you define in line #1 of the output file from Zotero become attributes in TB unless they simply map to existing TB attributes; the first column always maps to $Name. Thus is would be trivial (assuming the Zotero techies will build the export mechanism for you) to export Zotero's item ID for a reference to $ZoteroID whilst the item would export the IDs of all other Zotero items to which is links as a semi-colon delimited list pointed at $ZoteroLinks. Then in TB is simple to set up an agent (i.e. do this job once) whose query finds any note with a ZoteroID value and whose action looks at each note's ZoteroLinks and then links the note's original to the note(s) with each of the IDs in the list using linkTo(). Pretty simple.

In summary, you'd ask your Zotero techs to write something that allowed you to map a Zotero field (attribute?) to a particular column in the export data. Perhaps a TXT file Zotero will read where each line is in the form Zotero field{tab}TB attribute name. Zotero's export would also need to enclose in straight double quotes any Zotero value containing a Tab or line break character.

It strikes me that latter method is much more achievable in the short term than OPML as the TB end is all in place, you just need a Zotero exporter which you'd still need if going the XML route. Effort in TB is focussed on a new Cocoa based version so I suspect building a new OPML import interface might not sit at the top of the new feature list (especially in v5.x). I don't mean the latter unkindly, but if you want to do something now, I'd look at the tab-delim route.
Back to top
 
« Last Edit: Jul 17th, 2012, 12:21pm by Mark Anderson »  

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Jacek Gwizdka
Full Member
*
Offline



Posts: 6

Re: Importing XML into tinderbox
Reply #13 - Jul 17th, 2012, 4:56pm
 
Mark, many thanks for such a detailed answer with many great ideas! Indeed going through TSV/CSV text file format seems achievable. I will need to work on links on the Zotero end (as it does not seem to be a very frequently used feature). I intend to do the Zotero exporter myself.

I actually did a proof of concept bare bones script that exports some data from Zotero. The data is then saved into TBX XML format (with one manual step). And - voila! Worked for my very simple test case.

Back to top
 
« Last Edit: Jul 17th, 2012, 4:56pm by Jacek Gwizdka »  
  IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Importing XML into tinderbox
Reply #14 - Jul 17th, 2012, 5:09pm
 
That sounds neat - well done!
Back to top
 
 

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Pages: 1
Send Topic Print