Welcome, Guest. Please Login
Tinderbox
  News:
IMPORTANT MESSAGE! This forum has now been replaced by a new forum at http://forum.eastgate.com and no further posting or member registration is allowed. The forum is still accessible via read-only access for reference purposes. If you wish to discuss content here, please use the new forum. N.B. - posting in the new forum requires a fresh registration in the new forum (sorry - member data can't be ported).
  HomeHelpSearchLogin  
 
Pages: 1 2 
Send Topic Print
Academic Workflow (Revisited) (Read 25877 times)
Mark Cubberley
Full Member
*
Offline



Posts: 10

Academic Workflow (Revisited)
May 05th, 2009, 12:09pm
 
I'm interested in creating/streamlining an academic workflow with the following components: Bookends, Skim, DevonThink Pro, Tinderbox, and Mellel.  When I find an article of interest, I read, highlight, and annotate (via anchored notes) the article using Skim.  I then create a citation for the article in Bookends and attach the corresponding pdf.  (Regardless of when or if I use the citation, I'd rather do this bibliographic work now than later.)  I then index the article in the appropriate database in DevonThink Pro.

Now this is part that I've been struggling with...I would like to export my highlights and anchored notes from Skim and import them into a container in Tinderbox.  The name of the container would be the article title as well as have a link to the citation in Bookends.  In this container, I would like to explode the rtf file created in Skim in such a way as to make the anchored notes children of their respective highlights.  In other words, I'd like to make my primary notes (highlights) containers for my secondary notes (anchored notes).
(The format that Skim exports notes has been explored in another post.)

I will have more time to explore Tinderbox this summer once classes and labs are done for the year.  But I would like to start creating Tinderbox documents that are formatted as described above.  I'm I can follow some of the agent syntax from earlier posts on importing text, but I'm not convinced I can implement it with my limited experience with Tinderbox.

Of course, maybe there's a better way.  I'd really like to use Tinderbox to convert from a product to a process writer.  Any help, direction, or comments would be greatly appreciated.

Mark
Back to top
 
 
  IP Logged
Jean Goodwin
Full Member
*
Offline



Posts: 136
North Carolina
Re: Academic Workflow (Revisited)
Reply #1 - May 6th, 2009, 10:21am
 
Hello, Mark:  Everyone has a different work process, so the following remark may be useless to you.  Luckily, online communication is easily ignorable!

It strikes me that your workflow process emphasizes "the article" (other people's ideas) over your own thinking.  In your description, your notes are attached to articles, articles are what get indexed in DevonThink, and you're planning on articles being top level containers in Tinderbox.  

I also maintain an article-oriented database (I'm pre-committed to Endnote, for better or worse):  that's inevitable in academic work.  But I use Tinderbox primarily for my own thinking--mostly notes of my own ideas, sometimes sparked by reading, sometimes just random.  And my top-level organization of these notes is according to my own projects.  As more and more content gets poured in, the document increasingly resembles a map not of my field, but of my perspective on it, skewed by my evolving interests.

So:  If you're going to invest your time in automating your workflow, you might want to consider what you want the process to highlight for you, and what you want it to hide.  For example:  what about making your annotations the container, with the quote and article cite as the children?  [Note:  I have no idea of whether this is technically possible!]

Good luck with your project!  I join you in looking forward to some Tinderbox "professional development" time this summer--Jean
Back to top
 
 
  IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Academic Workflow (Revisited)
Reply #2 - May 6th, 2009, 11:16am
 
@Mark. I've not got Skim so can't replicate the whole workflow but if you've a sample Skim export that can be used as a real-world 'test' file, I might be able to explore the TB end of things a bit. Made-up test files are rarely as good as real-world data as the latter tend to include details (and resulting issues) often overlooked in a general description of a task.
Back to top
 
 

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Charles Turner
Full Member
*
Offline



Posts: 180
New York, USA
Re: Academic Workflow (Revisited)
Reply #3 - May 6th, 2009, 1:08pm
 
Hi Mark-

Skim can export a text file, as well as RTF, so they're a breeze to drag-n-drop into TBX. I've recently imported about 1000 Skim text files into TBX from DEVONthink. I used an Applescript in DT to add an (x-devonthink-item://) link back, so whenever I need to look at the original PDF in DT, I can just click on the URL tab.

It's very slick- I really love TBX!

:-) Charles

Back to top
 
 
WWW   IP Logged
Gregory
Full Member
*
Offline



Posts: 21

Re: Academic Workflow (Revisited)
Reply #4 - May 6th, 2009, 10:59pm
 
I have to admit, I have three parts of my academic work that are related, but not formally linked:

1. Tinderbox takes whatever crazy ideas come to mind.  I try to add as many keywords as possible and let agents organize away.  
2. I do my searching, downloading, organizing, and annotating in Sente, using keywords to organize the documents into smart folders.
3. Then I write, flipping back and forth between Tinderbox, Sente, and Word.  

There are probably much better ways - but this one seems to be the one that works naturally for me.

Gregory
Back to top
 
 
  IP Logged
briankpenney
Full Member
*
Offline



Posts: 7

Re: Academic Workflow (Revisited)
Reply #5 - May 18th, 2009, 10:54pm
 
Hi Mark-

As Charles notes, TBX is great for importing text as drag and drop and other ways. For the other part of your question: as you take notes in Skim if you make a habit of ending each thought with a hard return (or other keystroke), you can have that be the delimiter for the explode action to turn those into separate notes. I am not sure about a way to automatically make the highlighted notes a container and the anchored notes children of that container; look at the text exported by Skim to see if there are consistent characters inserted for which you can have an agent search. (aside: I am not sure what you mean by Primary vs Secondary notes here)

I know the new version of TBX allows agents to move notes (and therefore make them children of others), but you should try this on a test file that is not crucial to ensure the action goes the way you want, before committing serious research to be rearranged!

However, as Charles also notes, you can use TBX notes to link to other files, so maybe you don't have to import all of this? You could take your raw notes in Skim and just save them in that format or as txt files. Then as you read your notes in Skim you can have  TBX window open to record your own thoughts, and simply link these to the relevant sources instead of trying to import them. This wouldn't create as much extra work, and you would have the advantage of ensuring you are not altering the original documents nor confusing notes representing your thoughts with those from your reading.

Incidentally, a link that might be of interest. Cal Newport discusses on his blog how he structures his research and writing using a wiki (http://calnewport.com/blog/2009/05/11/how-to-build-a-paper-research-wiki/). It seems to me that the principle is easily transferable to TBX, while gaining all of the linking/agent/etc. power that TBX provides.

I hope your TBX exploration and writing time is productive (and comes soon!)


Cheers- Brian
Back to top
 
 
  IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Academic Workflow (Revisited)
Reply #6 - May 19th, 2009, 5:50am
 
I'd note that I've been working - off-forum - with Mark Cubberley and by tweaking the Skim export template we're beginning to make it easier to tease apart quote/highlight text from actual annotation text.  It's working on progress and the aim is to share the process when done but we've hit a few issues we yet need to resolve.

So points/observations/constraints/ to share based on use of actual Skim export data (as kindly supplied by Mark C:
  • Skim export 2 items where a highlight or annotation runs across a source (PDF) page boundary. That's something that needs to be manually re-stitched once the data's in TB (or possibly by manual edit of the the Skim RTF file.
  • Although the text, as RTF does import, hidden control characters in the text make it difficult to do some of the tasks we want - e.g. further splitting content via regexp - within TB. I'd suggest re-saving the RTF as plain text before importing it into TB.
  • Skim exports can include the source doc's page number but there appears to per annotation numbering.  If you've got 3 quotes/annotations on page #20 how to know which is which.
  • TB's notes aren't - I suspect - designed to hold large amounts of text whereas if a highlighted passage in Skim is several sentences long, the normal explode process means you're asking TB to put all that in the note name. Using Explode, if the resulting $Name is over c.64 characters in length an ellipsis is added after that point, truncating the source string.  Now, that limit would have been set before T stated allowing greater display of long note title is Map, Outline, etc., so there's scope (in the future) for TB to allow more than 64 characters, but it begs the question how many - or based on what delimiter(s)? Of course, if your references/annotations are short you won't see these issues at all!
  • By tweaking Skims exports templates you can set things up to make it easier to use an agent on the imported & exploded data to do things like set $Name to a different string to the Explode default or to add $DisplayExpression. However, this is where I got bitten by seeming under-the-hood differences between incoming RTF versus plain text data and also with trying to use regex to match across paragrpah boundaries.  The latter, admittedly, may be my lack of regex smarts.

So for all I haven't figured yet, my gut feeling is that with a little planning the desired import from Skim (and other such tools) to Tinderbox should be possible though it may require use of non-default procedures.
Back to top
 
« Last Edit: May 19th, 2009, 5:50am by Mark Anderson »  

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Charles Turner
Full Member
*
Offline



Posts: 180
New York, USA
Re: Academic Workflow (Revisited)
Reply #7 - May 19th, 2009, 6:49am
 
Quote:
Skim export 2 items where a highlight or annotation runs across a source (PDF) page boundary.
This is because Skim doesn't take a note across a page boundary. I don't think it's part of the export process.
Quote:
I'd suggest re-saving the RTF as plain text before importing it into TB.
As I mentioned above, Skim exports as Text, so you shouldn't be dealing with RTF conversion here.
Quote:
By tweaking Skims exports templates
Aren't these in the application's bundle? Is there a roadmap for the export tags?
Quote:
If you've got 3 quotes/annotations on page #20 how to know which is which.

Quote:
Now, that limit would have been set before T stated allowing greater display of long note title
As for the above 2 quotes, Skim is very Applescript-able, so if you're monkeying with Skim's export templates, I think you'd get more mileage from your effort if you studied Skim's Applescript dictionary instead. Much greater reward there!

Best, Charles
Back to top
 
 
WWW   IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Academic Workflow (Revisited)
Reply #8 - May 19th, 2009, 7:40am
 
Sorry, I missed the export-as-text thing.  I don't have Skim and got an export file that had an 'rtf' extension. Thus the false assumption.

I'm not surprised about the page boundary thing.  PDF is a page-centric format - PostScript pages at its core (I believe) so I guess other more recent additions to the format like annotation have to be linked to a page; if an annotation is drawn across a page boundary (possible in an non-1-up view) then the annotation contents belong to the page(s) over which it lies. I a mechanical sense this figures, it's just a bit dumb from an end user context. Moral: don't create annotations across page boundaries pages!

Skim allows custom templates via ~/Library/Application Support/Skim/Templates.

Having found the Skim documentation for templates & export my hunch is that the best route here would be to get help in the skim community to make an OPML export template as that might obviate the need for (much) post-processing when importing into TB. TB does support OPML import. I should add I'm not the person to ask about writing OPML, sadly!
Back to top
 
 

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Charles Turner
Full Member
*
Offline



Posts: 180
New York, USA
Re: Academic Workflow (Revisited)
Reply #9 - May 19th, 2009, 10:10am
 
Another thing I should mention, perhaps more about PDFs than Skim, although you may be affected by it: when you work with multi-column highlights/notes via Option-select, the note order in the resulting text/RTF file is based on its "vertical" precedence, not both vertical and "horizontal."

So if you have a highlight that goes from the bottom of column one and continues to the top of column two, you'll (of course!) have two highlights for a single note, but the end will come before the beginning.

This can get pretty messy with more columns (ie newspapers) and several highlights to a page.

I should point out that Skim enables you to edit highlights, so you can cut 'n' paste, or add tags to help the import to TBX. I hacked my copy of Skim to not display the annoying tooltips in the note sidebar so I could type more quickly. (There's a small discussion of this on the Skim mailing list.)

Skim appears to have (somewhat understandably) based their interface design on previous paradigms for note-taking, which seem decidely "executive/business" in nature. If they'd really thought about providing a usable note-taking interface, they would have put the sidebar across the bottom of the screen to maximize horizontal line length.

Best, Charles

Back to top
 
 
WWW   IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Academic Workflow (Revisited)
Reply #10 - May 19th, 2009, 11:02am
 
Raising the curtain of the import process a bit further, the way we go is to get a Skim template emitting one or more 'elements' like this, in text form:

[bullet] Page 20
"Quotation - i.e. highlighted source passage - in double quotes... rhubarb rhubarb rhubarb."

"Quatation migh have > 1 paragraph. more rhubarb..."

Annotation sans quotes. Spiffy comment on above

Possible extra spiffy comment


Import to TB and the Explode on the bullet symbol, removing the delimiter. TB takes first para of text and trims leading/trailing whitespace so we get a $Name like "Page 20", or whatever page number for the quotation. at this point the notes $Text is a whole single element minus it's opening bullet but with a space preceding 'page' on line 1.

Now we create an agent to look for anything inside the note "Exploded Note" plus a Text() pattern match. The neat thing is this will see any such note, so we can write one agent to (re-use) every time we import and explode. By putting back references into the regex pattern in the query we can pull substrings out from the default text.  I aimed for:
$1 = the number from the page number
$2 = the second para onwards (i.e. everything except the opening page number line)
$3 = the second para only (i.e. first para of quotation)

So the agents OnAdd sets a user attribute 'PageNumber' = $1.  We make DisplayExpression = $3. We make Text = $2.  Or that's the plan.... my regex have been having strange results, probably due to inexpertise on my part. Not least, I had issues matching across paragraph breaks  - .+ doesn't match a line return and the like.

The rationale for using DisplayExpression is that if $3 is too big to display, TB doesn't show the DE but simply the existing name (e.g. page 20).

One other problem is how we tell apart 3 quotations from page 20 and know the order in which they occur.  It does look as if Skims export template syntax might help there.

As you may see, an OPML approach might be simpler as one Skim template might resolve all/most of the above at source.





Back to top
 
 

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Mark Cubberley
Full Member
*
Offline



Posts: 10

Re: Academic Workflow (Revisited)
Reply #11 - May 19th, 2009, 4:55pm
 
I appreciate everyone's comments.  Here are some of my thoughts at this point in the discussion:

I want to read more in my work as an academic.  I also want to keep track of what I read more.  And I want to write more.   I want to leverage the power of TBX to work with my highlights, notes, and ideas (sometimes disparate, sometimes not) from the former to promote the latter.

I think this idea of highlights as note titles and annotations as note bodies could be a subconscious product of my current unproductive and unsatisfying writing process and my naiveness of the utility of TBX.  For better or worse, I'm essentially creating virtual index cards in TBX where the quotation is the front of the card and my annotation is the back.  Although TBX will allow me to un-/group these virtual index cards, yet, as Jean G. pointed out in an earlier post, this workflow process emphasizes other people's ideas over my own.

So...I have no doubt that we can get text and annotations out of Skim and into TBX, but I'm not convinced the resulting structure in TBX will help me to develop a new and improved way of thinking and writing.  (Did I mention how dissatisfied I am with my current writing process?)

I think Skim is useful for actively reading electronic documents.  As others have pointed out, maybe there's no need to import quotations and annotations into TBX at all...

Mark


Back to top
 
 
  IP Logged
Charles Turner
Full Member
*
Offline



Posts: 180
New York, USA
Re: Academic Workflow (Revisited)
Reply #12 - May 20th, 2009, 8:24am
 
Quote:
Although TBX will allow me to un-/group these virtual index cards, yet, as Jean G. pointed out in an earlier post, this workflow process emphasizes other people's ideas over my own.

I think Skim is useful for actively reading electronic documents.  As others have pointed out, maybe there's no need to import quotations and annotations into TBX at all...


Hi Mark-

To crudely reduce the point of Jean's message, she's making a distinction between "top-down" and "bottom-up" approaches to creating structure. (There are others also: "sideways," "easy-hard," etc.)

Most of the Tbox philosophy emphasizes manipulation of the "emergent" properties of your work, which has more affinity to a "bottom-up" approach. But Tbox can support many kinds of organizational paradigms.

As you mull over the interrelations of applications, just keep in mind that getting back to the original document, citation, etc., for me at least, is pretty important. No need to move stuff into an application if you don't intend to manipulate it there, but if you want to refer to it, how can you make that happen?

Best, Charles
Back to top
 
 
WWW   IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Academic Workflow (Revisited)
Reply #13 - May 20th, 2009, 10:53am
 
Meanwhile, on a more mechanical level I've had a tinker round with Skim, TB and the OPML spec this morning and I'm less sure of OPML as a best way forward as I don't think (happy to be corrected) that OPML's really best for long (multi-paragraph) sections of text. Using that - or other formats - there's also a text encoding issue at the Skim/TB divide which means that, for example, curly quotes and other non-basic-ASCII characters turn up garbled in TB (even using Skim's XML export template call).

Looking around the Skim templates it ought to be possible to do some more focused export but as the built-in template uses undocumented syntax (that perhaps unsurprisingly doesn't work!) so it's definitely worth figuring out what info in what order/syntax people might want to use for Skim/TB transfer before looking much further as my hunch it will certainly require some time on the Skim user list to figure some of the apps template internals.

I hope that helps!
Back to top
 
 

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Charles Turner
Full Member
*
Offline



Posts: 180
New York, USA
Re: Academic Workflow (Revisited)
Reply #14 - May 20th, 2009, 11:37am
 
Hi Mark A.-

Not that Applescript is any Garden of Eden, but Skim is very Applescript-able. (As is DEVONthink)

Best, C.

Back to top
 
 
WWW   IP Logged
Pages: 1 2 
Send Topic Print