Tinderbox User-to-User Forum (for formal tech support please email: info@eastgate.com)
http://www.eastgate.com/Tinderbox/forum//YaBB.cgi
Tinderbox Users >> Questions and Answers >> Linking into DEVONThink items
http://www.eastgate.com/Tinderbox/forum//YaBB.cgi?num=1346268662

Message started by Mark Anderson on Aug 29th, 2012, 3:31pm

Title: Linking into DEVONThink items
Post by Mark Anderson on Aug 29th, 2012, 3:31pm

In DT 'item' links use a form like:

x-devonthink-item://F02133F3-6520-40A6-A5B0-9CC590B8BDD6

…which if used on the Mac with both the DT  app & the relevant DT database installed will open the target DT-stored item in DT.

There is an extended syntax with allows things like

x-devonthink-item://F02133F3-6520-40A6-A5B0-9CC590B8BDD6?page=14

..which opens the target document at page #15**. This extra syntax is, I believe for PDFs only. Sadly the syntax is doesn't seem to be documented (at least anywhere one would expect)…and there's no aDTRef to fall back on!

I believe there are a couple of extra such link arguments for things like addressing individual Mail emails. If I find a reference, I'll add it here.

** somewhat unhelpfully the 'page' parameter is zero-based (ever seen a book with a page #0?). IOW, a ?page=2 parameter targets page #3 (count 0,1,2). Remember too, in figuring out the target page to use the document page not the 'printed' number in the page text as they will be different; the latter will not account for the preface, TOC, etc.

Title: Re: Linking into DEVONThink items
Post by Mark Anderson on Aug 29th, 2012, 3:33pm

This article might also be of interest.

Title: Re: Linking into DEVONThink items
Post by Mark Anderson on Aug 29th, 2012, 5:48pm

OK, it seems there is one more PDF (only?) text file related DT link parameter you can add:

x-devonthink-item://F02133F3-6520-40A6-A5B0-9CC590B8BDD6?search=styling%20a%20link

The 'search' string will be used as a search term in the targeted PDF. In the above case the linked PDF will be opened and scrolled to the first (case-insensitive) match to the string "styling a link", with the matched text string selected. note that the string must be URL encoded, so spaces must be substituted with %20, etc. The PDF opens at the first match; find next cannot be invoked to move to a subsequent match. If the search term is not found the PDF opens at the last viewed (in DT) page or page #0 if never viewed.

Only one (or no modifier can be used). At least, I've not tried using page= and search= in one DT URL.

The DT links were new to v2 of DT and aren't as yet fully documented, and may well be added to in functionality in due course. Best place to ask for more info is the DT forums.

Title: Re: Linking into DEVONThink items
Post by Mark Anderson on Aug 30th, 2012, 8:12am

This is from the DTPro Applescript Dictionary (where few might look), the description of a record's reference URL property, i.e. an item link:


Quote:
The URL (x-devonthink-item://...) to reference/link back to a record. Append ?page= to specify the zero-based index of a page of a PDF document, ?time= to specify the time of a movie or ?search= to specify a string to search.

So the extra this shows is that one can link into a DT movie record and specify the display start point using a time-based value. Unfortunately the time format isn't described. I'd assume hh:mm:ss would be a good first try. It's seconds - see post #6 below.

Title: Re: Linking into DEVONThink items
Post by Sumner Gerard on Aug 30th, 2012, 12:27pm

Thanks for these pointers!  Can see they will be very useful for linking to long documents in DT.


Quote:
there's no aDTRef to fall back on!

aDtRef, now that's a capital idea!

Title: Re: Linking into DEVONThink items
Post by Mark Anderson on Aug 30th, 2012, 1:05pm

Yes indeed, but I'm not offering!

Not least it's because I'm still at the baby steps with DT, albeit the very first thing I tried was scripting DTPro having added just two PDFs to the entire database! I did pick up a copy of the TakeControl series eBook for DT which has been useful (as has the Scrivener one), but such apps repay continued use which - as yet - my daily work doesn't need.

Title: Re: Linking into DEVONThink items
Post by John Sidi on Sep 3rd, 2012, 3:14pm

I played a little with the parameter "?time=" and I have found that you should write the time point in seconds. For example, the URL x-devonthink-item://DB470689-620E-4831-A428-651B08010E3D?time=80 creates a link to the 1st minute and 20th second of the movie which has this URL.

Title: Re: Linking into DEVONThink items
Post by Mark Anderson on Sep 3rd, 2012, 3:54pm

Thanks very much for that - I've edited my earlier post to point to your correction.

Title: Re: Linking into DEVONThink items
Post by Sumner Gerard on Sep 5th, 2012, 11:55am


Quote:
OK, it seems there is one more PDF (only?) related DT link parameter you can add:

x-devonthink-item://F02133F3-6520-40A6-A5B0-9CC590B8BDD6?search=styling%20a%20link


The 'search=' parameter works not just with PDFs but with RTF and plain text documents as well!  Very useful for going immediately to the relevant location(s) of a long document in DEVONthink. If there is more than one hit, all are highlighted. Thanks for that pointer.

Title: Re: Linking into DEVONThink items
Post by Sumner Gerard on Sep 5th, 2012, 11:20pm

Here's a script that takes selected DEVONthink RTF and plain text documents, explodes them into paragraphs, and exports an OPML file that can be opened by or dragged into Tinderbox. Each exported paragraph in the OPML file includes, among other Tinderbox attributes, a link back to the specific passage in the source DEVONthink document. This facilitates leveraging DEVONthink's strengths as a general repository for documents long and short, and Tinderbox's strengths as a tool for analyzing and discovering connections between ideas. I've worked with AppleScript only a few days. Comments/suggestions welcome.

Title: Re: Linking into DEVONThink items
Post by Mark Anderson on Sep 6th, 2012, 4:59am

That works nicely and creates TB attributes $DTComment, $DTTags, $OrigParaNo, $OrigURL with system attribute $URL being set to the DT link. If using this method regularly, I'd add a prototype (or a stamp) that sets some/all of these as  key attributes.

One error that does occur but which in most cases has no seeming side effect. The paragraph backlink URLs include an unintended space after the '=' character.  The error is in the paratrunc() routine. I suggest this replacement (tested):


Code:
on paratrunc(str, lgth)
     -- This function grabs a specified number of WORDS from the beginning of a paragraph (or the whole paragraph if that's shorter).
     -- The conditional in 'Explode the text…' lines in the "main" script below ensures that the string passed to this function is not empty.      
     set theWords to words of str --> this tells DEVONTHINK to split str into words
     set numWords to count of theWords
     if numWords > lgth then set numWords to lgth
     set nameStr to ""
     repeat with i from 1 to numWords
           set wrd to word i of str
           if i is equal to 1 then
                 set nameStr to wrd
           else
                 set nameStr to nameStr & " " & wrd
           end if
     end repeat
     return nameStr
end paratrunc


An unavoidable issue with the 'search' string link method is if the article has multiple paragraphs with the same start. I'd also consider URLEncoding the search string, e.g. a space becomes %20, etc.). AS can't do this directly but can use tolls installed in the Mac's Unix shell; here a Perl-based example.

Separately, I suspect there could be some useful AS code in this DT forum thread on highlighting and annotation.

Title: Re: Linking into DEVONThink items
Post by Sumner Gerard on Sep 6th, 2012, 12:11pm

Thanks for the pointers, Mark. I have (I think) incorporated your suggestions in this revised script.

If multiple paragraphs have the same start, DEVONthink jumps to the first one. As a workaround, if unique matches are important and numerous paragraphs start with the same 5 words (as could happen, say, in a speech), the maxSearch variable can be increased to match on more than just the first 5 words.

That DT forum thread does indeed look useful.  Thanks.

Title: Re: Linking into DEVONThink items
Post by Mark Anderson on Sep 6th, 2012, 12:55pm

Thanks. BTW, your script let me to start tinkering with (RTF) highlight detection in DT, where I tripped over some more AppleScript weirdness which I've documented in another DT forum thread. I hope is saves a few others the time I wasted figuring out the answer.

Note to self, as there seems to be a goodly cross-over of DT & TB users, I'll try and roll some of this up into an expanded version of this aTbRef page (but probably not today!).

Title: Re: Linking into DEVONThink items
Post by Mark Anderson on Sep 7th, 2012, 7:22am

OK, now see the updated aTbRef page re DEVONThink/TB use.

I'm no DT expert as I'm only a few days in as a DT user, so do let me know if there's other pertinent DT/TB info that might benefit from being added. I'd point out, that's 'missing manual' type info, not how-tos (someone else can write that!).

Title: Re: Linking into DEVONThink items
Post by Sumner Gerard on Sep 7th, 2012, 12:11pm

The first seeds of aTbDtRef!

I had trouble following the part about
Quote:
copy a 'bare' link icon


Separate but related topic prompted by the mention of highlight detection up thread: should it be possible to set up an automated method to extract just the highlighted passages from selected DT document(s) and bring them and their DT internal link into Tinderbox via opml (or tab-delimited table in the clipboard)? That would be useful, playing to the strengths of both applications.

Title: Re: Linking into DEVONThink items
Post by Mark Anderson on Sep 7th, 2012, 12:44pm

By a 'bare' link I just meant the normal one, bare of any query suffix (time/page/search). I'm just heading off on leave, but in due course I can review that article and perhaps improve the sense of it.

Re OPML. You'd need to write a custom OPML export script for DT - there seem plenty of examples in the DT scripting sub-forum that would help you get going and people there seem pretty responsive and helpful.

By default, in DT's built-in export the OPML URL attribute contains the DT item's URL info, i.e. to what it links or from where it came (whatever - I'm not 100% sure, but certainly not the DT link info you're after). Therefore, to use the DT 'item link' URL value you need to get the AppleScript reference URL property for the item and insert it into the OPML code.

You can write it into the URL OPML attribute to map to $URL in TB, or use a custom OPML attribute name. To ensure the latter data goes into a TB URL-type attribute, ensure that you make an attribute of the correct data type in your TBX before you import the OPML data. Then, if the custom OPML attribute name matches an existing TB attribute name (case-sensitive!), TB will use the existing attribute. Otherwise TB will make a String-type attribute for your custom OPML attribute. That's fair enough as TB has no way to know, from the OPML, what data type a custom OPML attribute's value should be. Only a few OPML-TB attribute mappings are hard-coded, such as OPML's _note to TB's $Text and text to $Name.

Title: Re: Linking into DEVONThink items
Post by Sumner Gerard on Sep 7th, 2012, 2:11pm

It was the reference to 'icon' that made me think I was missing something. Probably a typo to be fixed in due course. Enjoy the leave!

I now know a bit about custom OPML export script from DEVONthink to Tinderbox, of course, learned in the course of modifying and posting the script up-thread, which works well for chopping a long document in DEVONthink into paragraphs for exploration in Tinderbox, maintaining the links back to specific passages in the DEVONthink document.  

What I am interested in now is how to detect and grab highlighted text only, not paragraphs. The mention of 'highlight detection' made me think maybe you'd already come across something useful on that. Looking for it, I got lost in the thicket of threads over on the DEVONthink scripting forum. It's somehow related to 'annotation' I think, but I haven't been able to narrow the search down enough.

Title: Re: Linking into DEVONThink items
Post by Mark Anderson on Sep 7th, 2012, 2:24pm

'icon'? Ouch, how did that creep in? Sorry!.  Now corrected.

Highlight detection? I think it's possible.  Spun my wheels yesterday trying this (inadequacies of AppleScript language definition rather than DT, in fairness). I think it is possible, but it's not going to be blink-of-the-eye fast.

I'd float the concept under a topic of that name in the DT scripting forum., IOW, export only highlighted text to OPML. From what i've seen, I'm pretty sure it's possible though for more than the odd doc or two the process will likely give you time to relax while you wait (again, speed of AppleScript not a knock at DT). I suspect this is faster simpler to do for a single DT item (doc) than for your whole database. some of the knowledgable hands in the DT scripting forum also turn up here so they might have some idea what you're about.

Title: Re: Linking into DEVONThink items
Post by Sumner Gerard on Sep 7th, 2012, 4:49pm

From what I can tell from this thread I just found, grabbing just the text from highlights may not be possible in DEVONthink. Or if possible, not (as intimated by your comments about having time to relax) practical. For pdf's the way to ago apparently is via Skim. For rtf documents not sure.

Title: Re: Linking into DEVONThink items
Post by Mark Anderson on Sep 7th, 2012, 5:33pm

Mining a bit deeper, it seems DT can't script read PDF annotations (in fairness this stuff isn't simple under the hood - lots of gnarly Postscript). I do think the task is do-able with RTF and TXT (and DOC, etc.) files but I suspect PDF is where most people's DT data lives.

Skim (free!) does seem to be AppleScriptable. In reality, we're very often doing something similar  but not the same as others for which a one-tool solution simply doesn't exist and thus a degree of flexibility is required. Tools that do one (or several tasks) very well and have configurable import export are the things to looks for.

TB may not be AppleScrip-table, it does consume formats like UTF-8 encoded OPML and tab-delim text fairly well so is pretty good at importing formated plain text data. Ergo, if the upstream app has built-in configurable export or is scriptable (to make such a data file) it then becomes a matter of trial and error testing to get it into TB. User forums exist to take up the slack - not least as some of the building blocks of a workflow solution can be quite arcane. Also, if it's a really big task maybe it's maybe worth someone with XML skills to enable simple read-write to the source TB data. Sometimes having a UI blinds us to the possibilities (even if it means other people/cost in the loop).

[edited for typos]

Title: Re: Linking into DEVONThink items
Post by Sumner Gerard on Sep 7th, 2012, 11:38pm


Quote:
I do think the task id do-able with RTF and TXT (and DOC, etc.) files but I suspect PDF is where most people's DT data lives.

Right you are that it is in fact doable with TXT and RTF. Via the DTPO Scripts/More Scripts menu one can download 'extras'. These 'extras' happen to include two 'Merge Highlights' scripts. These scripts sweep up the highlighted text in selected plain text or RTF documents and dump them into a newly created DT plain text or RTF note. Presumably those scripts can be modified to output an OPML file for Tinderbox with links back to the relevant passages in the original DT document using the same search-string-appended-to-the-DT-link approach discussed upthread.

For PDFs, though, the best route seems to be through Skim, which out of the box apparently can export highlights and annotations... Or first convert the PDF to RTF.

Title: Re: Linking into DEVONThink items
Post by Sumner Gerard on Oct 11th, 2012, 11:35pm

Here are some enhancements to Mark A's script here for exporting multiple selected DEVONthink records to the clipboard for pasting into Tinderbox, retaining the links back to DEVONthink.


Code:
-- exports selected DEVONthink records to clipboard for pasting into Tinderbox
-- http://www.eastgate.com/Tinderbox/forum//YaBB.pl?num=1346083940/7#7
-- http://forum.devontechnologies.com/viewtopic.php?f=20&t=15873
-- expanded to include "quotified" multi-paragraph text

set pTab to tab -- "tab" has a specific meaning to DEVONthink so use "pTab" in tell block

--Set up tab-separated-values table "header" with Tinderbox attribute names:
set dataString to "Name" & pTab & "URL" & pTab & "Text" & return

tell application id "com.devon-technologies.thinkpro2"
     set itemList to selection
     if itemList is {} then
           tell application "System Events" to display dialog "Oops! ... First select some DEVONthink records and try again." buttons "OK" default button "OK"
     else
           repeat with anItem in itemList
                 set aText to rich text of anItem
                 --replace any straight quote with curly/smart, and surround with straight double quotes:
                 set quotifiedText to quote & my quotify(aText) & quote
                 --add a tab-separated-values "data row" to the pile:
                 set dataString to dataString & (name of anItem) & pTab & (reference URL of anItem) & pTab & quotifiedText & return
           end repeat
           set the clipboard to dataString as Unicode text
           tell application "System Events" to display dialog "DEVONthink data placed successfully on clipboard! … Now click Tinderbox outline view (not an open note) and hit <command-v> to paste." buttons "OK" default button "OK"
     end if
end tell

on quotify(txt)
     -- replace straight quotes with curly/smart versions
     --"quoted form" of mwra quotify() <http://www.eastgate.com/Tinderbox/forum//YaBB.pl?num=1265006946/0#4>;
     set scrptCom to "sed 's:\"\\([^\"]*\\)\":“\\1”:g' | sed \"s:'\\([d|ll|re|s|t|ve]\\):’\\1:g\" | sed \"s:'\\([^']*\\)':‘\\1’:g\""
     return do shell script ("echo " & quoted form of txt & " | " & scrptCom)
end quotify

return dataString -- (optional) display in AppleScript Editor Result pane


This brings over text as well as the name and the link.

Initially, it would choke on text with multiple paragraphs and/or quotes. Surrounding the text with straight double quotes and "quotifying" it using an AppleScript-friendly version of this gem solved that problem.

But, sadly, Tinderbox doesn't discard the surrounding quotes after the import as does, say, Excel...and probably other programs that accept tab-delimited data. And it *seems* to insert an extra line return before the closing quote. Harmless (I think) but not sure it should be there. Any workaround?


Title: Re: Linking into DEVONThink items
Post by Mark Anderson on Oct 12th, 2012, 3:27am

Re the stripping of enclosing quotes, the problem is contextual. It wouldn't be useful if any quote enclosed data was stripped of those quotes when pasting. It's only in certain defined contexts (as definable from an app's code perspective, not a human one) where we might want TB to strip quotes. Not sure how easy the latter is - though I agree it would be nice if everything 'just' worked as intuited!

Meanwhile, whilst the app may be unable to guess, the user's easily able to see if quotes have strayed in. Why not make a stamp that uses String.substr() to trim the quotes? Thus:

$Text=$Text.substr(1,($Text.size-2))

Code above tested in v5.11.2.

Title: Re: Linking into DEVONThink items
Post by Mark Anderson on Oct 12th, 2012, 5:47am

Also see this thread which links to more examples building on my last answer above.

Title: Re: Linking into DEVONThink items
Post by Sumner Gerard on Oct 12th, 2012, 12:15pm


Quote:
It's only in certain defined contexts (as definable from an app's code perspective, not a human one) where we might want TB to strip quotes.

Pasting of tab-separated-value data in table form definitely, I think, qualifies as one of the defined contexts in which we would expect TB to strip surrounding double straight quotes. Stripping them seems to be the de facto standard in Excel, Google Spreadsheets, Bento, and probably many (most?) other programs that accept (and export) tab-separated-value format.

That would avoid, I think, the problems with import from Aeon Timeline discussed here as well as with import of multi-paragraph text from DEVONthink.

Meanwhile, thanks for the useful stamps to add to the collection!  (Someday, maybe, regular expressions will seem less inscrutable.)

Title: Re: Linking into DEVONThink items
Post by Sumner Gerard on Nov 27th, 2012, 12:38pm

Here's an AppleScript that grabs chunks of highlighted text from DevonTHINK rich text records for pasting onto a Tinderbox view, using TB's "spreadsheet import" capability. This imports just the chunks of text wanted for analysis in Tinderbox, with each chunk retaining a link back to its specific location in the DT source document, making it easy to review the original context whenever needed.

I've found it useful for making sense of long, meandering conference call and interview transcripts and the like. I read through, highlighting what seems most relevant, then with the DT document(s) selected, run the script, and paste the results onto a Tinderbox view, where the notes can be manipulated in the usual ways.

This grabs all highlight colors, though it could easily be modified to grab only highlights of specific colors if one is using a highlight color-coding system. The subroutines with one-liner shell scripts may be of interest to those working with applications other than DT too.


Code:
-- Gathers highlights in selected DEVONthink Pro rich text documents and copies them to the clipboard for pasting into Tinderbox
-- Appends search string to URL$ that links back to the specific passage in the source DT document
-- Handles Chinese characters, and other multi-byte languages
-- A highlight must be followed by a space or punctuation mark for the DT search string to work
-- Loop logic simplified from "Merge Highlights" scripts by Christian Grunenberg
-- To use: select one or more DT records containing highlights, run this script, and paste onto a Tinderbox view

-----------Initialization-----------
set data_str to "Name" & tab & "Text" & tab & "URL" & return --header row column names map to TB attribute names
set start_length to length of data_str as string --for error handling
set p_tab to tab --tab has specific meaning in DT so use p_tab instead in the DNtp tell block
set n to 10 --maximum number of words desired for Tinderbox attribute $Name and $URL search string
-----------Main routine-----------
tell application id "DNtp"
     try --the try block allows error processing
           set the_selection to the selection --list of all DT records selected by user
           if (not (exists think window 1)) or (the_selection is {}) then error "Please select some rich text documents."
           set the_window to missing value
           set the_main_window to think window 1
           show progress indicator "Gathering highlights..."
           repeat with a_record in the_selection -- loop through the selected DT records
                 if type of a_record is rtf or type of a_record is rtfd then
                       if the_window is not missing value then
                             set record of the_window to a_record
                       else
                             set the_window to open window for record a_record
                       end if
                       tell text of the_window
                             repeat with an_attribute in attribute runs --loop through an "attribute runs" list of strings grabbing those with a non-white "background" property      
                                   try
                                         set the_properties to properties of an_attribute
                                         set the_background to background of the_properties
                                         if the_background is not missing value and the_background is not {65535, 65535, 65535} and length of (an_attribute as string) > 1 then
                                               set raw_highlight to an_attribute as text
                                               set cleaned_highlight to my tsvify(raw_highlight) --replace any tabs or line feeds with spaces, for $Name and $Text
                                               set dt_uri to "x-devonthink-item://" & uuid of a_record & ¬
                                                     "?search=" & my pct_encode(my first_n_wrds(raw_highlight, n)) --DT search string--already encoded so do not clean      
                                               set data_str to data_str & my first_n_wrds(cleaned_highlight, n) & p_tab & cleaned_highlight & p_tab & dt_uri & return --add data row
                                         end if --> highlighted                                          
                                   end try
                             end repeat -->until no more strings from "attribute runs"                  
                       end tell
                 end if -->DT record type is rtf or rtfd
           end repeat -->until no more DT records in selection      
           set the clipboard to data_str --put the tab-separated-values table on the clipboard
           hide progress indicator
           if the_window is not missing value and the_window is not the_main_window then close the_window
           if length of data_str is not greater than start_length then error "No highlights found."
           display dialog "Highlights placed on clipboard… Now click a Tinderbox view (not an open note) and hit <command-v> to paste." buttons "OK" default button "OK"
     on error error_message number error_number
           if the error_number is not -128 then display alert "DEVONthink Pro" message error_message as warning
     end try
end tell

----------- subroutines ------------------
on tsvify(str) --replace any tabs and line feeds with spaces - syntax:  tr 's' 'r'
     do shell script "echo " & quoted form of str & " | tr  '" & tab & "' ' ' | tr '" & linefeed & "' ' '"
end tsvify

on pct_encode(str) -- "percent encode" search string for uri
     do shell script "echo " & quoted form of str & " | php -r \"print urlencode(trim(file_get_contents('php://stdin')));\""
end pct_encode

on first_n_wrds(str, n) --return first n words - syntax: cut -d' ' -f1-n
     do shell script "echo " & quoted form of str & " | " & "cut -d' ' -f1-" & n
end first_n_wrds

Title: Re: Linking into DEVONThink items
Post by Simon on Jan 21st, 2014, 9:51am

I'm no coder and don't really understand all this. I have a simple question:

How do I link multiple DT items to a single TB Note? I noticed that keynote files can be dragged onto the "Drop Files and Folder" section out of DT. However, this does not work with pages files. I can create on one URL attribute to paste an item link.

Title: Re: Linking into DEVONThink items
Post by Mark Anderson on Jan 21st, 2014, 10:58am

Pages 'files' aren't actually files - they're folders that the OS shows as files. That might be one problem.

Separately, if you want to have only note links to, for example ,3 discrete URLs then you need to have at least 3 such attributes. TB has $URL configured by default, so you'd need to add 2 URL-type user attributes. Also, although $URL gets a button on the sidebar, to interact with the other two URL attributes you'd ned to show them as key attributes.

You can also make web links from selections of text. So I guess (not tested) you could make a web link from a selection in the note's $Text add paste the local DT pseudo-protocol URL into that.

Bear in mind that whatever you do, these DT URLs only work if TB and the DT database are on the same Mac.

Title: Re: Linking into DEVONThink items
Post by Ellen Madono on Jun 11th, 2015, 10:31pm

Use of codes is out of my experience.  I have Keyboard Maestro, DevonThinkOfficePro and TBx.  I want to go between DTOP and TBx.  

I am looking at Sumner Gerard's code to bring text into TBx.  I am wondering where to use it.  It is apple script, so I could put in a Keyboard Maestro macro.

I imagine there are multiple approaches, but I am looking for the most handy.  I know how to make KM macros.

Best,
EllenM

Tinderbox User-to-User Forum (for formal tech support please email: info@eastgate.com) » Powered by YaBB 2.2.1!
YaBB © 2000-2008. All Rights Reserved.