Tinderbox User-to-User Forum (for formal tech support please email: info@eastgate.com)
http://www.eastgate.com/Tinderbox/forum//YaBB.cgi
Tinderbox Users >> Tinderbox applications >> Mass-creating agents for unique values of a set
http://www.eastgate.com/Tinderbox/forum//YaBB.cgi?num=1276575622

Message started by Les Orchard on Jun 15th, 2010, 12:20am

Title: Mass-creating agents for unique values of a set
Post by Les Orchard on Jun 15th, 2010, 12:20am

Consider an attribute $Tags and lots of notes with various permutations of values as lightweight categorization. I wanted to easily and repeatedly build a set of searches for tags collecting notes by unique tag values.

Well, it seems like I've got something working, though it's very hacky:

http://decafbad.com/2010/06/set-agent-generator.tbx

I've wanted this off and on for a few years, but have never really seen a solution until now. In fact, this demo combines a slew of Tinderbox features that I've only just started understanding in the past month or so.

If anyone can point me to a better way to do this, I'd be happy. But, I'd also be just as happy if anyone finds this puzzle solution handy.


Title: Re: Mass-creating agents for unique values of a set
Post by Charles Turner on Jun 15th, 2010, 7:10am

Nicely done, Les!

So when's the Wrox "Programming Tinderbox" book going to come out?

Best wishes, Charles

Title: Re: Mass-creating agents for unique values of a set
Post by Les Orchard on Jun 15th, 2010, 8:43am


Quote:
So when's the Wrox "Programming Tinderbox" book going to come out?


Heh, not any time soon. But, if I come up with another dozen or so hacks like this, I might just self-publish something. It's been awhile since I had a book in progress, so I might get the itch again. The few I've done so far all started in Tinderbox, so it's only fitting. :)

Title: Re: Mass-creating agents for unique values of a set
Post by Mark Anderson on Jun 15th, 2010, 8:43am

Les, I enjoyed this. As there's no explode() action it's hard to automate this particular route further. I'd looked at self-configuring agents a while back and it's good to see this moved forward.

To my surprise, I still haven't figured where the 'delay' referred to is set - all agents appears to be set to normal priorities. In due course a logic walk-through of the explode process & agent generation would probably help more users get what this is about.  That's not critique - I know documentation for the general reader is more difficult to write than most suppose. Thanks again for this neat demo.

Title: Re: Mass-creating agents for unique values of a set
Post by Les Orchard on Jun 15th, 2010, 9:05am


Quote:
To my surprise, I still haven't figured where the 'delay' referred to is set


The overall process is:

* The container note's Rule continually collects unique $Tags set values in $Text, one per line.

* Performing Explode on the container note produces an Exploded Text child, and the container note's OnAdd applies the *tagSearchExplosion prototype to it.

* The children of Exploded Text are all given the prototype *tagSearchTmpFolder, thanks to the OnAdd of *tagSearchExplosion.

* Each *tagSearchTmpFolder comes along with an agent of prototype *tagSearchAgent, thanks to $PrototypeBequeathsChildren

* Once it sees that its parent is not a prototype, the *tagSearchAgent Rule assigns an AgentQuery based on its parent's name (the set value); renames the agent after its parent; moves the agent up to its great-grandparent (the parent of "Exploded Text"); and finally self-cancels.

* The Rule for *tagSearchExplosion replaces itself with a subsequent self-cancelling Rule to flag itself for the trash.

* An agent elsewhere moves flagged notes to the trash.

* By this point, all the *tagSearchAgent agents should have found their great-grandparent and escaped the exploded structure before it gets tossed away.

Lots of Rube-Goldbergian hacks here, but I think the worst part of this thing is the Rule-that-sets-a-Rule delay in "Exploded Text" sending itself to the trash. It seems to work reliably, and I think I understand why (ie. multiple Rules across many update cycles), but it's a bit of smoke and mirrors.

The other thing buried in there is the trash mechanism, which I've found handy elsewhere:

* An agent moves notes with $IsTrashed=true to a trash folder, preserving the original location in $ContainerBeforeTrash.

* Setting $IsTrashed=false causes another agent to move the note back to that container.

* Stamps to set $IsTrashed work as handy menu commands to trash/restore sets of notes.

Title: Re: Mass-creating agents for unique values of a set
Post by Mark Anderson on Jun 15th, 2010, 11:03am

Excellent, I saw all this in the code but having written down sure helps.  The set syntax shortcomings, stumbled over here are another factor of which to beware. In summary, syntax you'd intuit from use with other attribute types fails (silently) when used with sets. In the examples below blue = valid and red is invalid syntax:

To first set, or completely replace, a set's value:
  $MySet = "dog"
  $MySet = $MyString
  $MySet = $MySet(other note)


A different syntax is used to add/subtract/toggle a set value reuiring the set to be stated both sides of the assignment:

  $MySet = $MySet + "dog"  (add)
  $MySet = $MySet - "dog"  (remove)
  $MySet = $MySet ~ "dog"  (add or remove if already present)
...but, hazards await:
  $MySet = $MySet + $MyString
  $MySet = $MySet + eval($MyString)

For a a $MyString value of "dog"

  if($MySet==$MyString){...

...is true if $MySet is the single value "dog", but false if any other values are present, e.g. "dog;cat". But even where this is true:

  if($MySet==$MyString){...
...this is always false...
  if($MySet==$MyString(some note){...

IOW, equality tests on sets only return false where the entire set's string equals the other side - as opposed to completely matching a single value. So, testing sets in if() conditions using attribute-stored values is very difficult. It is slightly easier using an agent and the AttributeName(pattern) operator but even then we have to precompile the query. So:

AgentQuery:  MySet($MyString)
AgentQuery:  MySet(eval($MyString))
...but...
Agent's Rule:   $AgentQuery="MySet("+$MyString+")"

Still the need for any given syntax isn't necessarily obvious until someone actually needs it  ;)

Title: Re: Mass-creating agents for unique values of a set
Post by Jody Foo on Mar 15th, 2011, 11:42am

Thanks for posting this!

When I tried it however, the result was that one agent was created for each instance of the tag used.

I suspect that this is due to some change in Tinderbox between your post and my trial (5.8.0).

I had to change the $Rule of *tagSearchFolder from

if (!$IsPrototype) {$Text=format(collect(all,$Tags), "\n") }

to the following:

if (!$IsPrototype) { $tmpSet = collect(all,$Tags); $Text=$tmpSet.format("\n") }

The difference is that I added a $tmpSet variable to the Tinderbox of the type Set. I am guessing that collect(<Set>) no longer outputs a <Set>, but a <List>. By directing the output to a Set, duplicates are eliminated.

I also changed the formatting syntax from format() to <Set>.format(). What syntax is preferred?

Title: Re: Mass-creating agents for unique values of a set
Post by Mark Anderson on Mar 15th, 2011, 12:15pm

collect() was switched to a List from a Set in v5.6.0 (see more). As at v5.8.0, the aTbRef note is correct but TB Help lists collect() as returning set data; I've submitted a change for the latter.

Your solution is quite the right one. To get a de-duped list pass the List-type output to a Set-type attribute. With aSset you can use either the old style format() or the new Set.format() dot operator. At present both have the same effect.

Title: Re: Mass-creating agents for unique values of a set
Post by J Fallows on Mar 22nd, 2011, 3:54am

I am still learning the innards of TB, but I have a basic question. I collected this (useful) file for the "Set Agent Generator," and then I made the change mentioned above, to avoid having duplicate agents appear for each occurrence of a tag. (Ie, to go from "list"-like to "set"-like behavior.) But even after that change, exactly copied from the screen here, I still get a long set of duplicate results. Behavior is the same before and after the change. What might I be missing here? Am using the latest release of Tbox.   Thanks  J Fallows / Beijing 

For reference, this is the change previously mentioned, which I made:
>>I had to change the $Rule of *tagSearchFolder from

if (!$IsPrototype) {$Text=format(collect(all,$Tags), "\n") }

to the following:

if (!$IsPrototype) { $tmpSet = collect(all,$Tags); $Text=$tmpSet.format("\n") }      <<

Title: Re: Mass-creating agents for unique values of a set
Post by Mark Anderson on Mar 22nd, 2011, 4:45am

OK, I downloaded Les' files, added a Set attribute $tmpSet and a note with this rule:

$tmpSet = collect(all,$Tags); $Text=$tmpSet.format("\n")

...and the result is a de-duped list, one item per line.

A Set will de-dupe a list. So if the output of $tmpSet has dupes, your attribute may not be the correct data type.  When creating a new attribute, it is very easy to type the name and forget to set the data type pop-up; I regularly trip up on this especially when my attribute name includes the intended data type (I must intuit TB will guess the needed type <g>).  So, I'd open User Attributes (Cmd+2) and confirm your $tmpSet is a Set and not the default String type.

Title: Re: Mass-creating agents for unique values of a set
Post by J Fallows on Mar 23rd, 2011, 2:17am

Thank you. I figured out how to apply these tips and feel as if I have taken another step forward along the learning curve. Appreciate it jf.

Title: Re: Mass-creating agents for unique values of a set
Post by Marcos Ramos on Oct 16th, 2014, 8:15am

The overall process is:

* The container note's Rule continually collects unique $Tags set values in $Text, one per line.

* Performing Explode on the container note produces an Exploded Text child, and the container note's OnAdd applies the *tagSearchExplosion prototype to it.

* The children of Exploded Text are all given the prototype *tagSearchTmpFolder, thanks to the OnAdd of *tagSearchExplosion.

* Each *tagSearchTmpFolder comes along with an agent of prototype *tagSearchAgent, thanks to $PrototypeBequeathsChildren

* Once it sees that its parent is not a prototype, the *tagSearchAgent Rule assigns an AgentQuery based on its parent's name (the set value); renames the agent after its parent; moves the agent up to its great-grandparent (the parent of "Exploded Text"); and finally self-cancels.

* The Rule for *tagSearchExplosion replaces itself with a subsequent self-cancelling Rule to flag itself for the trash.

* An agent elsewhere moves flagged notes to the trash.

* By this point, all the *tagSearchAgent agents should have found their great-grandparent and escaped the exploded structure before it gets tossed away.

Lots of Rube-Goldbergian hacks here, but I think the worst part of this thing is the Rule-that-sets-a-Rule delay in "Exploded Text" sending itself to the trash. It seems to work reliably, and I think I understand why (ie. multiple Rules across many update cycles), but it's a bit of smoke and mirrors.

The other thing buried in there is the trash mechanism, which I've found handy elsewhere:

* An agent moves notes with $IsTrashed=true to a trash folder, preserving the original location in $ContainerBeforeTrash.

* Setting $IsTrashed=false causes another agent to move the note back to that container.

* Stamps to set $IsTrashed work as handy menu commands to trash/restore sets of notes.

Awesome you shared out the entire process.. I was bit confused but you sort out my confusion and I am able to get the menu now..

Title: Re: Mass-creating agents for unique values of a set
Post by Mark Anderson on Oct 16th, 2014, 9:06am

Some follow-up points.

Collect/collect_it changed from returning sets to returning lists in v5.6.0, released 15 Sep 2010. If you want a set back from collect, just chain .unique to the source list or pass output to a set:

$MyList = collect(children,$SomeAttr).unique;  (Result is list, but no dupe values)
$MySet = collect(children,$SomeAttr);  (Passing a list to a set creates a set, inherently de-duping input)

If you'll explode regularly you may find it useful to use 2 cascading prototypes to set the exploded items' prototype. However, if you just want the latter to receive a protoype in a one-off explode, set the $OnAdd of the to-be-exploded note to:

$OnAdd='$Prototype="pSomeProto";';

Note the single quotes wrapping the whole OnAdd string to be placed in Exploded notes.  As the inner string uses double quotes, we use single quotes for the outer pair. You can nest quotes either way around - as long as the nesting is correct.

If the idea is for each exploded note to find values of itself and hold a reference to it, we don't need agents, but can use find(). So the prototype for exploded notes might have a rule:

$ListOfPathsToNotesUsingThisTag = find($Tags.contains($Name(that)));

Of course, it's likely the find() will have extra query terms to limit search scope (always good for performance). Also, rather than than just get the paths to matching notes you could perhaps store their titles:

$SetOfNames = collect(find($Tags.contains($Name(that))),$Name)

Or link back to each one:

linkTo(find($Tags.contains($Name(that))),"uses tag")

If you do use a custom linkType name (e.g. above) as at v6.0.4 you must define the link type in the TBX before using the link type name in code like that above.

Consider performance (important!). Don't simply set all this code up and leave it running continuously. All this back-referencing via .contains()** pushes a large document quite hard.  Design your code such that you run the code once, or as required. That way you get all the upside of action code automation but don't wonder why the docs running a bit slow.

** Operators like .contains() use a log of regex searching and this is an inherent overhead. Use when needed except in trivial size documents.

Title: Re: Mass-creating agents for unique values of a set
Post by J Fallows on Oct 16th, 2014, 9:27am

Maybe I am missing something here, but I believe that in most cases, the need to mass-create agents has been overtaken/obviated by the new Attribute Browser feature in TB6.

That is, my purpose in joining this discussion several years ago was to be sure that if I created a new value for an attribute, I would know that a new agent would be created to collect all notes with that value. That is what the Attribute Browser now does.

Not sure if that covers the exact use case being discussed here. But for the basic task of making sure that notes will be grouped according to all values of a certain attribute, that now happens automatically.

Title: Re: Mass-creating agents for unique values of a set
Post by Mark Anderson on Oct 16th, 2014, 12:28pm

@JF, yes probably so for most strictly internal UI based stuff, like just looking at the value split in an attribute. But for some more action code/network work and in bigger docs, I'm less sure. But this is edge case stuff - I'm doing heavy lift about 30 attributes and some have c.500 discrete values which make the app chuff a bit (it's having to work hard!). From a quick test, I think the above is probably easier/more stable for *big* data sets than Attribute Browser which is a really cool feature and which you're quite right to mention.

Also having an actual note per value in an attribute (AB view just reorganises existing notes based on an attributes values  - nicely, too!) then allows you to add notes about that value.  So, in my discourse analysis, each subject data post records $UserName (the poster's ID), the topics record a $UserNameSet of all posters in that thread and the root container's $UserNameSet aggregates a set off all discrete user names. I then set another note to read that set, sort it, and split it into value per line .format("\n") and pop it in the note's $Text. Explode, and now I have a note for each user and store info about them. I can't o the latter in Attribute Browser. So, it's a case of what you're trying to do. For just viewing the allocation of the values of $SomeAttribute across all/some existing notes, the Attribute Browser is the way to go. I hope that helps clarify the choices use cases. If anyone's confused, do ask, as finding out the right approach from several possibilities can be a boon to making progress with your work.

Title: Re: Mass-creating agents for unique values of a set
Post by jwiegley on Jan 14th, 2015, 5:49pm

I wonder if my recent feature suggestion about pivot agents would make this request easier to accomplish.  A pivot agent on tag values would be almost exactly what the original poster asked for.

Title: Re: Mass-creating agents for unique values of a set
Post by Mark Anderson on Jan 14th, 2015, 5:57pm

This thread goes back to 2010. Notwithstanding your feature request, I do think Attribute Browser view (v6.0+) actually takes care of this. As discussed in the linked thread, the suggested feature whilst making absolute sense doesn't scale well whilst Attribute Browser provides a perfectly usable alternative.

Title: Re: Mass-creating agents for unique values of a set
Post by jwiegley on Jan 14th, 2015, 5:59pm

I understand that, except the attribute browser solution doesn't allow me to save queries within my outline, the way that the original poster demonstrated with his tags example; or have I missed some aspect of how the browser should be used?

Title: Re: Mass-creating agents for unique values of a set
Post by Mark Anderson on Jan 14th, 2015, 6:10pm

Indeed not, but as explained in a separate thread, the proposed solution simply doesn't scale. I know, from experience, having tried - which is why I genuinely think AB is a better solution.

Yes, you could expand collapse (2 levels of) agent but in truth it is easier in practice to simple select a different attribute in AB. If you need an expression value then use an agent or rule to store the per-note value in a user attribute then view that in AB. If the expression needs value(s) that alter simply store those too in an attribute.

Otherwsise there's a lot of engineering cost to something already possible and which few users will use. Please understand, I'm not being argumentative or dismissive. I've had to deal with the issues you raise in actual research already done in TB; the experience is that once you get into the detail it won't scale in the way one might imagine. As it is, TB already offers the simple building blocks to do this albeit via a slightly different method and with no extra engineering needed. If you'd like more detail as to how, please ask - I'd be happy to help.

Title: Re: Mass-creating agents for unique values of a set
Post by jwiegley on Jan 14th, 2015, 6:25pm

I wonder, can a container rule be used to create agents as children?  If so, I might be able to script what I'm thinking of for the case of smaller databases.

Otherwise, yes, I'd be quite interested in whatever you might write up to take better advantage of the attributes view.

And no, I haven't read you to be argumentative.  I realize this is a thin medium for us to communicate ideas, and I'm brand new to all of this, so please bear with me. :)

Title: Re: Mass-creating agents for unique values of a set
Post by Mark Anderson on Jan 15th, 2015, 11:16am

Actions can't create notes or agents. My understanding of the design intent in this regard that the risk of user error in action code creating runaway note creation (and support issues arising) outweighed he benefit for the few who'd use such a feature with the care needed. Regardless, that's the status quo as at v6.1.1.

However, you can make an agent into a prototype - the process is the same as with a note. A good tip if doing this is to turn the prototype agent 'off'. Thus you then add a new agent to your doc and before changing anything on it you set the prototype. The new agent now inherits the 'off' state whilst you go configure the query, etc. On a small doc this might be a seemingly slow process but on a big doc it's a time saver as you're not trying to search everything whilst still fully configuring the agent.

Agents can use the 'agent' designator in agent queries and actions to access the agent's attribute values. Thus we might add a string attribute $MyString as a key attribute (KA) to the agent and set the query to $Text.contains($MyString(agent)). If $MyString is 'cat', the agent will find all notes whose $Text contains the string 'cat'. To target a different attribute or type of match, just change the query:  $SomeUserAttribute==$MyString(agent). Now the agent matches an note whose $SomeUserAttribute has the exact value of 'cat'. To change the searched-for value, just edit $MyString as shown in the agents KA table and the query is immediately updated. If other notes hold different values of $MyString then you don't even need to type a new value; simply click the value list pop-up at the right side of the item's value box in the KA table (value lists).

Title: Re: Mass-creating agents for unique values of a set
Post by jwiegley on Jan 15th, 2015, 11:27am

That's a great idea, Mark!  I could have the query key off of $Name, and then make a stamp for re-enabling the agent, and it would be quite fast indeed to build "canned queries" as sub-items within a grouping container.

Title: Re: Mass-creating agents for unique values of a set
Post by Mark Anderson on Jan 15th, 2015, 11:50am

Here's a useful stamp to toggle agents on/off (i.e off if on or vice versa):

if($AgentPriority>-1){$AgentPriority=-1}else{$AgentPriority=;}

Tinderbox User-to-User Forum (for formal tech support please email: info@eastgate.com) » Powered by YaBB 2.2.1!
YaBB © 2000-2008. All Rights Reserved.