Tinderbox User-to-User Forum (for formal tech support please email: info@eastgate.com)
http://www.eastgate.com/Tinderbox/forum//YaBB.cgi
Tinderbox Users >> Agent, Actions, Rules & Automation >> Query help (all agents missing certain items)
http://www.eastgate.com/Tinderbox/forum//YaBB.cgi?num=1452091918

Message started by Derek Van Ittersum on Jan 6th, 2016, 9:51am

Title: Query help (all agents missing certain items)
Post by Derek Van Ittersum on Jan 6th, 2016, 9:51am

I'm struggling to construct a query that would find sets of notes that do not have certain properties.

My data looks like this:

Each note has these relevant attributes:

$CiteKey, $Name, $Text

$CiteKey will be the same for many notes (e.g., 10 notes with Smith86, 20 notes with Jones95, etc.)

$Name will be different for each note, but will follow a pattern of LetterNumber (e.g., H1, H2, H3 ... ; S1, S2, S3 ... ; F1, F2, F3 ...)

$Text will be unique for each note.

What I would like to find, in regular language, is this:

Any group of notes with the same $CiteKey that are missing a certain Letter in the $Name.  

For example, there are notes with the $CiteKey Jones95 with the Names H1, S1, and F1, which means all categories are covered. But there are only notes H1 and F1 for the $CiteKey Smith86. I want to create an agent that looks for $CiteKeys that don't have "S" notes and would return "Smith86" cite key.

Here's what I've tried:

I created Agents (in a container called Agents) that look for specific CiteKeys. So, I have multiple agents that have this query: $CiteKey==$CiteKey(agent), then I set each one to a different $CiteKey.

Then, I created an agent to search those with this query:

descendedFrom("Agents") & any(children, !$Name.contains("S"))

This query returned each Agent, whether it had S-named notes or not. Then I tried:

descendedFrom("Agents") & any(grandchildren, !$Name.contains("S"))

That query returned nothing.

I then had the idea that each Agent with this query: $CiteKey==$CiteKey(agent)  Could create a set with names of its children, then I could create an agent to find sets that are missing specific names. I used this action:

$MySet=$Name(children);

But that didn't work: the Set remained empty.

Any suggestions? I'm at the very beginning of this project, so it's easy enough to restructure the data or Tinderbox organization to make it easier to achieve my desired result.

Title: Re: Query help (all agents missing certain items)
Post by Mark Anderson on Jan 6th, 2016, 10:48am

[edit]Corrected suggested solution code due to conversation further down thread[/edit]
TB doesn't mind extra attributes.  Why not get a rule or edict to set:

$TheLetter= $Name.substr(0,1)

$TheLetter now holds 'P' or 'S' etc.  Now for any $CiteKey value's agent, the agent rule can set:

$LetterSet = collect(children,$TheLetter)

Or skip the first step and just do (that will save continuously running lots of rules/edicts):

$LetterSet = collect(children,$Name.substr(0,1))

In the latter I'm assuming - not tested - that String.substr() is valid as a collect parameter.

collect() returns a List but passing that to a set de-dupes it. Now, the agent's $LetterSet, has a list of the letters used in all its cited notes' titles. You can't use .contains(pattern) on a list but you can use != or == again a complete single list value. And, it's less resource intensive than a .contains(). Yay! Oops that's wrong - it's the reverse. You can't use == or != with a list. You can use .contains but only for matches to complete individual list values (not partial ones).

Now you can poll the per-CiteKey agents and return only those not having the letter of interest - which we'll store as $MyString in the calling agent. We'll assume the cite agents are of prototype "pCiteAgent". Now we'll set $MyString to "S". To find its missing an 'S' value the agent query is

$Prototype=="pCiteAgent" & $LetterSet!=$MyString(agent)

$Prototype=="pCiteAgent" & $LetterSet.contains($MyString(agent))

To test for 'P' instead of 'S', simply set the $MyString value of the agent accordingly.

I hope I've understood the scenario correctly - if not please say.

N.B. Not tested as I'm buried in some deadline stuff.

Title: Re: Query help (all agents missing certain items)
Post by Derek Van Ittersum on Jan 6th, 2016, 11:12am

thanks Mark--Very useful.  A few questions:


Quote:
$TheLetter now holds 'P' or 'S' etc.  Now for any $CiteKey value's agent, the agent rule can set:

$LetterSet = collect(children,$TheLetter)


Why does this need to be a rule? I'm not sure I get why this can't be in the action code: $LetterSet(agent)=collect(children,$TheLetter)     (I tried it and it doesn't work)


Quote:
collect() returns a List but passing that to a set de-dupes it. Now, the agent's $LetterSet, has a list of the letters used in all its cited notes' titles. You can't use .contains(pattern) on a list but you can use != or == again a complete single list value. And, it's less resource intensive than a .contains(). Yay!

$Prototype=="pCiteAgent" & $LetterSet!=$MyString(agent)


I'm having trouble with this. The above query didn't work for me (it returned all agents, even those missing the letter), but this did:

$Prototype=="pCiteAgent" & !$LetterSet.contains($MyString(agent))

when I have two agents with LetterSet of:

Agent A: h,s,f
Agent B: h, f

and MyString == s

Title: Re: Query help (all agents missing certain items)
Post by Mark Anderson on Jan 6th, 2016, 11:44am

Q1. It's the agent collecting the first character of the name of each child.  If you want to use the agent action, it would be (not tested):

$LetterSet(agent) = $LetterSet(agent) + $Name.substr(0,1)

You're adding to a set so dupe new values are ignored. Whoever, you're doing the same task in a different context.  Actually, the agent is perhaps better as if the agent is turned off the agent action not longer runs whilst the agent's action still does.

Q2. Hmm, a list value should have commas. It should look like "h;s;f" (without the quotes) when viewed as a key attribute. Anyway, my error for posting without running code. I have it back wards - you can't test ;lists with ==/!= but can use .contains() by only against complete discrete values. So your query

$Prototype=="pCiteAgent" & !$LetterSet.contains($MyString(agent))

Is correct (I ran a quick and dirty test here in the current beta and it works as you describe! Sorry for the misstep above. i'll correct when I get a moment. Yrs in haste...

Title: Re: Query help (all agents missing certain items)
Post by Derek Van Ittersum on Jan 7th, 2016, 6:28am

Thanks for the help Mark. Don't miss your deadline on my behalf!

Your code from above works as you say:


Quote:
$LetterSet(agent) = $LetterSet(agent) + $Name.substr(0,1)


However, I realized there is a drawback to this approach. If the data is changed in some way, that is not reflected in the agent unless the agent's $LetterSet is cleared first. In other words, if a particular agent is collecting notes with a;f;h as Letters, but the one note with "a" in it is deleted, the agent's LetterSet still contains "a".

Because of this drawback, I'll likely return to the previous method,


Code:
$LetterSet = collect(children,$TheLetter)


However, I'm a bit wary about the number of Rules this might create in the document. I'm thinking the data set will be approx 400 agents examining 5000 notes. Do you think 400 Rules to perform the "collect" function will degrade performance? Is there some way to perform the "collect" outside of a Rule?

At any rate, thanks much for your assistance here. I have what I want working as I hoped--just trying to look ahead and head off problems.

Title: Re: Query help (all agents missing certain items)
Post by Mark Anderson on Jan 7th, 2016, 9:22am

This is an area where we need to consider personal work style - and tendency to charge data (or simply mis-key it). Rules are 'always on' so the more (and more complex) rules you have, the greater the need to consider the impact on the file as a whole. Rules can be backed off to edicts, which run on first creation, then every hour for the rest of there session. Also, if not now then soon, edicts can be run on demand by using File->Update agents.

Unless you make lots of typos or the names change a lot, I'd experiment using edicts. Then when you know you've made an edit that affects this 'rule' you'd update agents to run all edits. You might need to do that command several times( 1 run all edicts, 2 refresh agents to reflect edict-breasted data, 3 refresh agents polling other data). That said 2 & 3 will sort themselves out within several agent cycles.

Broadly, find something that works and consider changing it if you think your action code is making things bog down. For later readers - this issue of updating is something very few users will see so don't worry about it. If you're giving Tinderbox too much to think about it's usually soon self-evident, at which point (and probably not before) you should look at ways to make your action code more efficient. I write that as someone with an unfortunate knack for writing unintentionally complex action code, and the experience of resolving performance.  :)

Title: Re: Query help (all agents missing certain items)
Post by Derek Van Ittersum on Jan 7th, 2016, 9:42am

Yes, edicts are a good idea here. However, I kept banging my head on a way to construct a query "action" that would do the same as the rule you suggested. I came up with the following, which seems to work:

$LetterSet(agent)=collect(find(inside(agent)),$TheLetter)

Do you see any issues with this? Is it performing essentially the same as this rule applied to the same agent?

$LetterSet = collect(children,$TheLetter)

Title: Re: Query help (all agents missing certain items)
Post by Mark Anderson on Jan 7th, 2016, 10:13am

'agent' only works in the scope of an agent action. You can't - or shouldn't, as I understand it - use it elsewhere, even in the rule/edict of an agent. The idea of the 'agent' designator is to bridge from the context of an agent child alias to the agent itself (as 'parent' can be ambiguous in such a setting).

Testing for the absence of something is generally harder than the opposite a a fail (false) is the default return from lots of errors - i.e. you get false positive results. A further complication is you want to test for multiple omissions, even if only for one letter at a time.

I think you might need to take a different approach. How many discrete 'letters' are there? Is this a closed list (i.e. doesn't change over time)? worst case is this naming set one forced on you by external data or is the design self-inflicted - in which case perhaps you want to find a completion marker system that's less hard to track.

Bear in mind:

$LetterSet = $LetterSet + "some value";

only ever adds to the set (which you don't want):

$LetterSet = collect(...

builds a completely new set each time the code fires (better).

Don't overlook breaking out only the logic in immediate use into a smaller treat rig TBX making it easier to test edge cases. Testing in the full file may be adding in effects not part of the problems as discussed thus far.

Title: Re: Query help (all agents missing certain items)
Post by Derek Van Ittersum on Jan 7th, 2016, 10:17am

Grr. My mistake! I meant to say that I had constructed an "action" instead of a "rule" (I wrote "query" above, but meant "action").

At any rate, the action appears to do what I want. I have reworked the data set somewhat to get better analysis. Instead of naming the notes using letters, I have moved these to a set of codes for each note ($Code).

Title: Re: Query help (all agents missing certain items)
Post by Mark Anderson on Jan 7th, 2016, 10:49am

Wonderful. It'll all be plain sailing from here...  ;)

Tinderbox User-to-User Forum (for formal tech support please email: info@eastgate.com) » Powered by YaBB 2.2.1!
YaBB © 2000-2008. All Rights Reserved.