Tinderbox User-to-User Forum (for formal tech support please email: info@eastgate.com)
Tinderbox Users >> Tinderbox applications >> Sweeping up after a long text 'Explode...'

Message started by Sumner Gerard on Feb 28th, 2010, 7:05pm

Title: Sweeping up after a long text 'Explode...'
Post by Sumner Gerard on Feb 28th, 2010, 7:05pm

Those who use Tinderbox's unique power to 'Explode' long text files and manipulate the many resulting "fragments" may find this tip saves much fiddling and fossicking about.  It leverages TB's automatic placement of fragments in convenient containers always named 'Exploded Text', and uses one simple agent and a prototype with a self-canceling rule.  Altogether three short lines of code.

The problem: After using 'Explode' on a long note, often one needs (like investigators of a physical explosion) to identify where each fragment came from, and keep track of that information after removal from the scene. In Tinderbox one way to do this is to populate some variety of 'MySourceID' attribute and/or other metadata such as the source's url. That can take care and feeding.

One solution: Instead of scrolling to select potentially hundreds of fragments and applying a stamp corresponding to each source note, or fiddling with various and sundry agents to automate the stamps, perhaps leaving rules running constantly in hundreds or thousands of fragment notes with unintended effects when the notes are moved, try this:

1) In each original unexploded source note enter in user attribute '$MyString' an alphanumeric value identifying the source (or whatever other metadata will be wanted in the fragments).

2) Create an 'AllExplodedText' agent that gathers all exploded fragments.

    AgentQuery:  $Name(parent)=="Exploded Text"

    AgentAction: $Prototype=protoFragment

Note: inside("Exploded Text") in the query should also work if only one 'Exploded Text' container is present. But if there is more than one the agent only finds the "first" one.

3) Create the prototype 'protoFragment'

    KeyAttributes:  As desired to define how a fragment will look when opened

    Rule: $MyString=$MyString(../..); if (!IsPrototype) {$Rule=""}

Now, whenever a source text note is exploded the agent gathers up each fragment and assigns the prototype, whose rule is inherited by each fragment, causing each fragment to grab the desired value from its original source text note (its grandparent) and then clear its own rule in a way that breaks the rule inheritance from the prototype. (Hence $Rule="" , not  $Rule=;  ). The 'if' statement prevents the rule from being cleared from the prototype, so the prototype stands ready to pass the rule on to any "new" fragment, which applies it once and cancels it.

This approach has an added benefit. After applying 'Explode' and looking at fragments, the metadata from the source note may seem inadequate or wrong. Just delete the relevant 'Exploded Text' container and contents, change the offending value in the source note, and 'Explode' again.

Or if the inadequacy is discovered after many links or values already have been added to the fragments, then change the offending value in the source note, and consider applying a 'Reset rule' stamp to all the notes in the relevant 'Exploded Text' container with this action:  $Rule=; .  This restores the inheritance from the prototype. The rule runs once, then cancels itself, once again breaking the inheritance.

Note: Using '../.' or 'parent' as the designator instead of '../..' or 'grandparent' also works in the prototype rule. Though I couldn't find documentation, it seems each 'Exploded Text' container inherits many attributes from its parent, the original unexploded note.

Comments/critiques appreciated.  Any misinformation here? Even easier way to accomplish the same thing? What code would automatically create a link from each exploded fragment to its source, so the links could be visible after fragments and sources are moved to a map and moved around?

Resources and references:

Forum and aTbRef tips and references: here, here, and here.   (Thanks to Mark, Mark, and Amber.)

User's Manual references: 'Explode' is described in the (long…so scroll) section on 'Importing Notes'.

Example TBX ( 'Exploded Text example' ) posted here.   (Novice coder alert; use at your own risk)

Title: Re: Sweeping up after a long text 'Explode...'
Post by Mark Bernstein on Feb 28th, 2010, 8:58pm


a   AgentQuery:  $Name(parent)=="Exploded Text"

If you're worried that there might be several containers named "Exploded Text", you could also write

      inside("/path/to/Exploded Text")

and specify the complete path to the container "Exploded Text" you want to work on.

Title: Re: Sweeping up after a long text 'Explode...'
Post by Mark Anderson on Mar 1st, 2010, 6:50am

The 'Exploded Text' container inherits all the source note's attributes except $Rule and $OnAdd. Why not the latter two? Up to v4.7.1 these were inherited but were found not to scale well, when using either/both large source text and/or many imports.

So exploded notes can pick up source attributes from their parent or grandparent designators, and access Rule or OnAdd of the original via the grandparent designator.

If using agents, remember that you should use parent(original) and grandparent(original). The children of an agent are aliases and their parent is the agent, not the parent of the original - thus the different notation.

What code would automatically create a link from each exploded fragment to its source, so the links could be visible after fragments and sources are moved to a map and moved around

Use an agent:
Q:  $Name(parent)=="Exploded Text";
A:  linkTo("grandparent(original)","Source Text");

Note how the linkTo() operator doesn't need a left-side argument (i.e. $SomeThing=linkTo()...), and the use of the 'original' designator. The above will create a link of link type name "Source Text" - by all means change that to something else.

Because the agent uses reference to the generic 'Exploded Text' and uses designators it can be left in situ (perhaps turned off between explodes) and used with any explode results.

Title: Re: Sweeping up after a long text 'Explode...'
Post by Jean Goodwin on Mar 1st, 2010, 11:03am

It's interesting that Tinderbox allows different easy ways of getting this task done--maybe because users have different learning styles?  or at least different habits?

Anyhow, I tend to think in terms of prototypes--they're nice and concrete.  So for this important task, I'd rely on a cascade of one-off OnAdd actions and not agents/self-canceling rules;  something like:

1.  Give the source note to be exploded an OnAdd action:


(If I'm exploding a lot, I'd create a stamp to do this.)

2.  Create ExplodePrototype; give it OnAdd actions just like in Sumner's/MarkA's examples:


Hey--makes me feel secure; no mysterious agents running in the background!

Title: Re: Sweeping up after a long text 'Explode...'
Post by Sumner Gerard on Mar 1st, 2010, 8:23pm

Many thanks for the pointers.

@Mark B:  Though usually I want the agent to gather fragments from all the Exploded Text containers, it is good to know how to include the path to refer to one particular Exploded Text when needed.  (I'm never sure when to use quotes.)

@Mark A: I see there is reference to Exploded Text in release notes to 4.7.1.  On reflection it seems that it cannot just be $OnAdd and $Rule that are not inherited: also $Name, $Text, $Created, etc., etc. (I know, I know, that is obvious…  but…we insecure learners tend to be literal!)  At some point perhaps the inherited properties and the uses of the Exploded Text container deserve an additional line or two in the documentation somewhere. When I first encountered it, it seemed an unneeded "layer" rather than the great convenience that it is.

Thanks much for the pointer on how to use linkTo(). Wouldn't have gotten 'original' right. I got it to work in an agent, then worried about your mention of perhaps turning off the agent between explodes, and decided to try linkTo() in the self-canceling rule in the prototype, leaving out 'original'. It works. The rule is now:

$MyString=$MyString(../..);$URL=$URL(../..); linkTo("grandparent","Source");if(!IsPrototype) {$Rule=""}

[Admin note: as discussed here and further below in this thread, if you have problems using paths containing punctuation and non-alphanumeric characters is can be a useful to enclose the path string in double quotes.]

@ Jean Goodwin: Fascinating to see the "OnAdd cascading with prototype" approach. Since I tend to add multiple sources, I added a step at the top. I created a 'Source texts' container with this (entire, from $ to final ') expression entered in the OnAdd Action box:

$OnAdd=' $Prototype="ExplodePrototype" '

That way whenever I add a new source text, its OnAdd action sets automatically.  No need to remember to stamp.

I see the cascade technique leaves OnAdd actions in each Exploded Text container. So if one were to happen to manually add a note there (perhaps a brilliant thought of one's own) it would automatically be attributed to that source... A good thing or a bad thing, I suppose, depending on one's habits.

Not sure how to implement linkTo() with the cascade technique.  Must be easy too.

I know what you mean about mysterious agents!  My one agent is actually a double, gathering all fragments and setting the prototype.  Its gathering function is convenient for dragging aliases onto a sorting map whose adornments color code and assign topics, etc., leaving the original notes in place.

Either approach takes just three lines of code.  A marvel how terse one can be with Tinderbox.

Title: Re: Sweeping up after a long text 'Explode...'
Post by Mark Anderson on Mar 2nd, 2010, 5:24am

Quoting & Paths answered separately in a new thread.

Exploding & attributes. What's not inherited (excluding calculated & read-only attributes)? From brief perusal: $Created, $Modified, $Name - these are self-evident as to why. $OnAdd, $Rule - for reasons stated re performance. $Height, $Width, $Xpos, $Ypos - the new note is a container on a new map so these 'changes' are a by-product though they use app defaults for the 'new' values'. However, $DisplayExpression is inherited as, I presume, is $TableExpression (latter not tested). In other words, the only attributes you might expect to be inherited that aren't are $Rule and $OnAdd. Anyway, you can easily and quickly test this for yourself - TB does change over time so it's always a good idea to re-check your assumptions.  Tb offers many ways to do things as some most users' TBXs differ. As such don't expect to always be informed if something you really rely on changes - you might be doing something very niche. The release notes are there for all to read - they may be terse but they are informative.

Agents & $AgentPriority - turning agents on ant off.  Again, don't take too hide bound an approach.  I menyioned this aspect as Sumner refers to doing a lot of imports.  Agent(s) scanning everything have trivial impact in a small TBX with tens of notes. Scale out to thousands of notes and - depending on how you set things up, there may be some impact.  I assume you don't like to turn an agent 'off' in case you forget to turn it back 'on'.  We turn lights off when we leave the room to save power (costs), but we know to turn them back on when returning as it's dark. So, by analogy, how to you see an agents status. for outline view, it's on screen (though you need to look). But in a map? Well, add code to alter the colour, or badge/shape/pattern/whatever, of agent processed exploded notes.  No change on new explodes' appearance?  You probably left the agent turned off!

An alternate apporach to agent impact is to consider restring the scope.  For instance all data to be exploded might be created as notes inside a (container) note called 'Sources'. Now we can make our agent query:

 descendedFrom("Sources") & $Name == "Exploded Text"

If Sources and it's contents are, say, only 10% of the whole TBX, then the first part of the query ensure we don't waste time scanning the other 90% of the TBX.

Even so, as the content within Sources grows, so does the number of notes scanned  - including processed ones, unless/until you move the exploded notes out of the 'Exploded Text' container and move/delete the latter. How might that be done.

First, you might add an agent (or add to existing agent code) that after doing whatever (e.g. set $SourceText, make links) adds code to move the notes up a level as siblings to 'Exploded Text'. Thus:


...or move them wherever.  Note the references to 'original' as were using the an agent.  If we don't, we act on the alias in the agent causing a never ending loop as each time the alias is moved out the next query cycle the aliases is re-created, moved again, etc.

Now you have orphaned 'Exploded Text' notes. TB won't delete notes, but it can move them. Make an agent:

Q: $Name=="Exploded Text" & $ChildCount==0 & $Name(parent)!="Bin"
A: $Container(original)="/Bin"

This finds empty explode containers and moves them to a root -level note called 'Bin' (set whatever name/path you like). Every so often you can delete the contents of 'Bin' knowing those notes aren't required. On the first use of this do check you get what you expect if you move a container with children, the children go too - so if you set your query incorrectly you could move an 'Exploded Text' container that's not empty.

Note the above query checks for notes already in 'Biin - we don't need to work on those. Going back to the first example, as long as 'Bin is outside 'Sources', we could have used:

Q: descendedFrom("Sources") & $Name=="Exploded Text" & $ChildCount==0

Remember - there's no single 'right' way to do all this so don't be too literal in your use of the above ideas - adapt them as need to suit your TBX.

Title: Re: Sweeping up after a long text 'Explode...'
Post by Sumner Gerard on Mar 2nd, 2010, 1:35pm

Thanks much for these fantastic pointers!  As I learn TB, I often can understand reasonably well a particular feature or capability but still struggle to draw a concrete connection to "how" I might use it. So specific suggestions like these really help. Sort of like training wheels or mental scaffolding. With luck cast off in due course, but indispensable to learning.

Very helpful to know about the visual cue in outline view for sleeping agents.  For me it was hidden in plain sight. Of course many Americans like me grew up with cheap electricity and don't always turn lights off when they should! Even with TB's power maybe I'll have to adapt. But assuming bad habits die hard and I forget to turn out the lights, I'm wondering in terms of TB power conservation:

Would having $Prototype|= protoFragment in the action of my example agent collecting thousands of fragments be noticably more efficient than  $Prototype=protoFragment?

And now a  'Humpty Dumpty' problem:

Suppose I explode long text into fragments.  Then to slim down the TBX I want to delete the text of the source note, but keep the note for its values in $MyString and other relevant attributes describing the soource.  If the fragments have been moved around in the outline order and otherwise tortured for insights, can I be confident it is easy to put them back together again in their original order. Or will it take all the King's men?  I know I can easily gather them with an agent query on the relevant value for $MyString. But what attribute is available for the sort?  I see that $Created is identical for all children of  a given Exploded Text container, presumably because they were all created in the same instant in the big bang.  Does TB have a unique note identifier?  (If so, though, like a UniqueID in a relational database table I suppose it might not be in sequence anyway…).

Title: Re: Sweeping up after a long text 'Explode...'
Post by Jean Goodwin on Mar 2nd, 2010, 6:42pm

You can get a unique "serial number" for notes by creating a number Attribute and ticking the "sequential" checkbox (which I didn't notice until a forum post pointed it out about a year ago).  Each new note takes the "next" (+1) value of this attribute, and in my quick test, the exploded notes are created in the expected order (1st paragraph-->2nd paragraph--> etc.).  

Warning:  the count is going to start at the number of notes you already have on the map--e.g., in my case, 3652.  But you could get a serial number starting at 1 for the explosion results with another OnAdd action in the ExplodePrototype (or self-cancelling rule), something like:


Title: Re: Sweeping up after a long text 'Explode...'
Post by Sumner Gerard on Mar 2nd, 2010, 8:03pm

Yes indeed! Thank you for putting me on to that tick box. And now that I know to search the forum on "sequential" I see lots of relevant discussion. And as you say Explode does indeed create the notes with values in $SequentialAttribute in sequence as expected, starting with the number for Exploded Text, which I see enables your subtraction of the parent value if one wants to start with 1 or whatever (with your number of notes I can see how that might be convenient!). TB doesn't seem to 'reuse' numbers, say if you Explode, fix something in the source note, delete the fragments, and try again. But that is as I would want and expect. Many thanks for the pointer.

Title: Re: Sweeping up after a long text 'Explode...'
Post by Mark Anderson on Mar 3rd, 2010, 4:42am

TB doesn't seem to 'reuse' numbers
Yes, but only for number attributes with the sequential option on (default is off). This makes the attribute a 'counter' whereby the count is only ever incremented upwards. If a sequential number attribute is added to
a TBX with existing notes, the latter are all numbered as well; if I recall, they're numbered in current outline order but the main point is everything gets a number.

Following up Jean's idea, if you set a $SourceText string (that is unique to each explode) and a sequential $ExplodeOrder, then an agent could reassemble the note.  Let's assume our source is 'Review Q4-09' then we make an agent:

Query:  $SourceText=="Review Q4-09"
Sort:  $ExplodeOrder  (set via create/rename dialog, or use a rule: $Sort=$ExplodeOrder)

Assuming you've deleted no exploded notes (or tinkered with the referenced attributes) you should have all the text of the original note.

To get the source text back:
  • Set the $Text of the agent ot some other note to all the agent's child aliases' $Text via the AgentAction - don't forget to add a line break "\n" after each item's input or paragraphs will run together at explode margins. Indeed, you need to put back in whatever delimiter you stripped when exploding - so if those vary by source keep a note of setting used.
  • Use ^include^ in an HTML export - you'll need to write your own template.
  • Do the same in Nakakoji via for plain text export.
Doing this reassembly process a lot? Then consider using a prototype agent that sets it's query value from the agent name - there's some info about the technique on aTbRef. If you're going either of the export routes above, use internal (note-based) template for your export template and the prototype agent can be pre-set to use that.

Title: Re: Sweeping up after a long text 'Explode...'
Post by Mark Anderson on Mar 3rd, 2010, 5:11am

In reply #6 above, Sumner asks about using |= versus = when setting prototype values and the effect on performance. I'd say the answer is 'it' depends.  First, the possible downside of |= is that it runs once.  If the right-side value (here, the prototype $Name) changes, then notes using a |= rule would need to be manually updated.  If you don't foresee the latter occurrence then I'd definitely use the |= construct.

[For those unfamiliar with this syntax, in basic terms using '|=' to set a value results in the left side attribute only being set if it has an empty value (or what equates to that for the relevant data type - e.g. 'never' for dates, etc.]

Performance impact is really related to size and to the complexity of the code on the right side of the expression.  For instance, if instead of simply holding a prototype name string there was a complex conditional query, then the latter is more code to run each cycle. Thus for simple rules there may be far less impact leaving an '=' .

This links neatly to one other useful issue. I note earlier you include this in some code:

if(!IsPrototype) {$Rule=""}

There's nothing wrong with that but the code merely empties the rule, it does not also re-enable inheritance for $Rule - the difference is explained here and here.

Title: Re: Sweeping up after a long text 'Explode...'
Post by Sumner Gerard on Mar 3rd, 2010, 8:02pm

So I've taken away the general idea that it very much depends, but that TB finds things in a flash (e.g. an agent gathering aliases or me typing in the find box) and the more "power-consuming" aspect of agents left on can be in the running of the code that acts on the originals of the gathered aliases.   Anyway, 5.0.2 is fast.

And putting Humpty Dumpty back together again, just as he was, is simple.  Getting Humpty Dumpty off the wall and out of TB, if needed, requires "a few more steps."  Of course the whole idea is to turn him into something else, not put him back together just the way he was.  But it's good to know it's possible, if needed. Sort of like taking comfort in knowing how to get back to the shallow end of the pool should there be momentary confusion and panic in the deep.

I'm wondering per guidelines here about quoting paths whether it might not be wise insert a note on notation or a correction at in the example at the top of this thread.  I didn't know "../.." without the quotes is just the sort of "spelling mistake" that can cause problems.  

Title: Re: Sweeping up after a long text 'Explode...'
Post by Mark Anderson on Mar 4th, 2010, 4:38am

Sumner, feel free to edit your original post (noting that might make parts of the latter thread not make sense. It might better to note that something might be an issue and to warn readers to read the whole thread.

The odd thing re ../.. is that I assume the code you posted was code that worked in your TBX but not mine - which isn't what I'd expect.  It took me a while to think of quoting as you don't expect a working example to not work when copy pasted verbatim.  I'm at TB weekend next week so my be able to look into this more deeply. Whilst I think guidelines help less tech-based users there is a danger of creating rules where they don't actually exist and thus effectively removing access to features.

Title: Re: Sweeping up after a long text 'Explode...'
Post by Sumner Gerard on Mar 4th, 2010, 3:46pm

At least a note to try "../.." if ../.. doesn't work sounds like a good idea.

Problem is, I'm allowed to 'modify' my yesterday's post, but none prior. There the only option I'm given is 'delete'.  Hence the suggestion, on the assumption that a YaBB administrator perhaps has permissions to 'modify' old posts.

The path without the quotes did work for me. I just doublechecked, and it still does in the example TBX to which I posted a link. Strange.  In any case, I'm eager to follow "best practice" wherever it exists, and appreciate your pointers to reduce those "I did everything right and it just doesn't work" moments that I suspect even the tech savvy encounter from time to time.

Title: Re: Sweeping up after a long text 'Explode...'
Post by Mark Anderson on Mar 4th, 2010, 5:07pm

I've annotated your earlier post where use of '../..' first got a mention. I didn't realise the forum software didn't allow ordinary users continued editing of their posts.

Tinderbox User-to-User Forum (for formal tech support please email: info@eastgate.com) » Powered by YaBB 2.2.1!
YaBB © 2000-2008. All Rights Reserved.