Welcome, Guest. Please Login
Tinderbox
  News:
IMPORTANT MESSAGE! This forum has now been replaced by a new forum at http://forum.eastgate.com and no further posting or member registration is allowed. The forum is still accessible via read-only access for reference purposes. If you wish to discuss content here, please use the new forum. N.B. - posting in the new forum requires a fresh registration in the new forum (sorry - member data can't be ported).
  HomeHelpSearchLogin  
 
Pages: 1
Send Topic Print
Query help (all agents missing certain items) (Read 2083 times)
Derek Van Ittersum
Full Member
*
Offline



Posts: 168

Query help (all agents missing certain items)
Jan 06th, 2016, 9:51am
 
I'm struggling to construct a query that would find sets of notes that do not have certain properties.

My data looks like this:

Each note has these relevant attributes:

$CiteKey, $Name, $Text

$CiteKey will be the same for many notes (e.g., 10 notes with Smith86, 20 notes with Jones95, etc.)

$Name will be different for each note, but will follow a pattern of LetterNumber (e.g., H1, H2, H3 ... ; S1, S2, S3 ... ; F1, F2, F3 ...)

$Text will be unique for each note.

What I would like to find, in regular language, is this:

Any group of notes with the same $CiteKey that are missing a certain Letter in the $Name.  

For example, there are notes with the $CiteKey Jones95 with the Names H1, S1, and F1, which means all categories are covered. But there are only notes H1 and F1 for the $CiteKey Smith86. I want to create an agent that looks for $CiteKeys that don't have "S" notes and would return "Smith86" cite key.

Here's what I've tried:

I created Agents (in a container called Agents) that look for specific CiteKeys. So, I have multiple agents that have this query: $CiteKey==$CiteKey(agent), then I set each one to a different $CiteKey.

Then, I created an agent to search those with this query:

descendedFrom("Agents") & any(children, !$Name.contains("S"))

This query returned each Agent, whether it had S-named notes or not. Then I tried:

descendedFrom("Agents") & any(grandchildren, !$Name.contains("S"))

That query returned nothing.

I then had the idea that each Agent with this query: $CiteKey==$CiteKey(agent)  Could create a set with names of its children, then I could create an agent to find sets that are missing specific names. I used this action:

$MySet=$Name(children);

But that didn't work: the Set remained empty.

Any suggestions? I'm at the very beginning of this project, so it's easy enough to restructure the data or Tinderbox organization to make it easier to achieve my desired result.
Back to top
 
 
  IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Query help (all agents missing certain items)
Reply #1 - Jan 6th, 2016, 10:48am
 
Edited:
Corrected suggested solution code due to conversation further down thread

TB doesn't mind extra attributes.  Why not get a rule or edict to set:

$TheLetter= $Name.substr(0,1)

$TheLetter now holds 'P' or 'S' etc.  Now for any $CiteKey value's agent, the agent rule can set:

$LetterSet = collect(children,$TheLetter)

Or skip the first step and just do (that will save continuously running lots of rules/edicts):

$LetterSet = collect(children,$Name.substr(0,1))

In the latter I'm assuming - not tested - that String.substr() is valid as a collect parameter.

collect() returns a List but passing that to a set de-dupes it. Now, the agent's $LetterSet, has a list of the letters used in all its cited notes' titles. You can't use .contains(pattern) on a list but you can use != or == again a complete single list value. And, it's less resource intensive than a .contains(). Yay! Oops that's wrong - it's the reverse. You can't use == or != with a list. You can use .contains but only for matches to complete individual list values (not partial ones).

Now you can poll the per-CiteKey agents and return only those not having the letter of interest - which we'll store as $MyString in the calling agent. We'll assume the cite agents are of prototype "pCiteAgent". Now we'll set $MyString to "S". To find its missing an 'S' value the agent query is

$Prototype=="pCiteAgent" & $LetterSet!=$MyString(agent)

$Prototype=="pCiteAgent" & $LetterSet.contains($MyString(agent))

To test for 'P' instead of 'S', simply set the $MyString value of the agent accordingly.

I hope I've understood the scenario correctly - if not please say.

N.B. Not tested as I'm buried in some deadline stuff.
Back to top
 
« Last Edit: Jan 6th, 2016, 3:36pm by Mark Anderson »  

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Derek Van Ittersum
Full Member
*
Offline



Posts: 168

Re: Query help (all agents missing certain items)
Reply #2 - Jan 6th, 2016, 11:12am
 
thanks Mark--Very useful.  A few questions:

Quote:
$TheLetter now holds 'P' or 'S' etc.  Now for any $CiteKey value's agent, the agent rule can set:

$LetterSet = collect(children,$TheLetter)


Why does this need to be a rule? I'm not sure I get why this can't be in the action code: $LetterSet(agent)=collect(children,$TheLetter)     (I tried it and it doesn't work)

Quote:
collect() returns a List but passing that to a set de-dupes it. Now, the agent's $LetterSet, has a list of the letters used in all its cited notes' titles. You can't use .contains(pattern) on a list but you can use != or == again a complete single list value. And, it's less resource intensive than a .contains(). Yay!

$Prototype=="pCiteAgent" & $LetterSet!=$MyString(agent)


I'm having trouble with this. The above query didn't work for me (it returned all agents, even those missing the letter), but this did:

$Prototype=="pCiteAgent" & !$LetterSet.contains($MyString(agent))

when I have two agents with LetterSet of:

Agent A: h,s,f
Agent B: h, f

and MyString == s
Back to top
 
 
  IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Query help (all agents missing certain items)
Reply #3 - Jan 6th, 2016, 11:44am
 
Q1. It's the agent collecting the first character of the name of each child.  If you want to use the agent action, it would be (not tested):

$LetterSet(agent) = $LetterSet(agent) + $Name.substr(0,1)

You're adding to a set so dupe new values are ignored. Whoever, you're doing the same task in a different context.  Actually, the agent is perhaps better as if the agent is turned off the agent action not longer runs whilst the agent's action still does.

Q2. Hmm, a list value should have commas. It should look like "h;s;f" (without the quotes) when viewed as a key attribute. Anyway, my error for posting without running code. I have it back wards - you can't test ;lists with ==/!= but can use .contains() by only against complete discrete values. So your query

$Prototype=="pCiteAgent" & !$LetterSet.contains($MyString(agent))

Is correct (I ran a quick and dirty test here in the current beta and it works as you describe! Sorry for the misstep above. i'll correct when I get a moment. Yrs in haste...
Back to top
 
 

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Derek Van Ittersum
Full Member
*
Offline



Posts: 168

Re: Query help (all agents missing certain items)
Reply #4 - Jan 7th, 2016, 6:28am
 
Thanks for the help Mark. Don't miss your deadline on my behalf!

Your code from above works as you say:

Quote:
$LetterSet(agent) = $LetterSet(agent) + $Name.substr(0,1)


However, I realized there is a drawback to this approach. If the data is changed in some way, that is not reflected in the agent unless the agent's $LetterSet is cleared first. In other words, if a particular agent is collecting notes with a;f;h as Letters, but the one note with "a" in it is deleted, the agent's LetterSet still contains "a".

Because of this drawback, I'll likely return to the previous method,

Code:
$LetterSet = collect(children,$TheLetter) 



However, I'm a bit wary about the number of Rules this might create in the document. I'm thinking the data set will be approx 400 agents examining 5000 notes. Do you think 400 Rules to perform the "collect" function will degrade performance? Is there some way to perform the "collect" outside of a Rule?

At any rate, thanks much for your assistance here. I have what I want working as I hoped--just trying to look ahead and head off problems.
Back to top
 
 
  IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Query help (all agents missing certain items)
Reply #5 - Jan 7th, 2016, 9:22am
 
This is an area where we need to consider personal work style - and tendency to charge data (or simply mis-key it). Rules are 'always on' so the more (and more complex) rules you have, the greater the need to consider the impact on the file as a whole. Rules can be backed off to edicts, which run on first creation, then every hour for the rest of there session. Also, if not now then soon, edicts can be run on demand by using File->Update agents.

Unless you make lots of typos or the names change a lot, I'd experiment using edicts. Then when you know you've made an edit that affects this 'rule' you'd update agents to run all edits. You might need to do that command several times( 1 run all edicts, 2 refresh agents to reflect edict-breasted data, 3 refresh agents polling other data). That said 2 & 3 will sort themselves out within several agent cycles.

Broadly, find something that works and consider changing it if you think your action code is making things bog down. For later readers - this issue of updating is something very few users will see so don't worry about it. If you're giving Tinderbox too much to think about it's usually soon self-evident, at which point (and probably not before) you should look at ways to make your action code more efficient. I write that as someone with an unfortunate knack for writing unintentionally complex action code, and the experience of resolving performance.  Smiley
Back to top
 
 

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Derek Van Ittersum
Full Member
*
Offline



Posts: 168

Re: Query help (all agents missing certain items)
Reply #6 - Jan 7th, 2016, 9:42am
 
Yes, edicts are a good idea here. However, I kept banging my head on a way to construct a query "action" that would do the same as the rule you suggested. I came up with the following, which seems to work:

$LetterSet(agent)=collect(find(inside(agent)),$TheLetter)

Do you see any issues with this? Is it performing essentially the same as this rule applied to the same agent?

$LetterSet = collect(children,$TheLetter)
Back to top
 
« Last Edit: Jan 7th, 2016, 10:17am by Derek Van Ittersum »  
  IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Query help (all agents missing certain items)
Reply #7 - Jan 7th, 2016, 10:13am
 
'agent' only works in the scope of an agent action. You can't - or shouldn't, as I understand it - use it elsewhere, even in the rule/edict of an agent. The idea of the 'agent' designator is to bridge from the context of an agent child alias to the agent itself (as 'parent' can be ambiguous in such a setting).

Testing for the absence of something is generally harder than the opposite a a fail (false) is the default return from lots of errors - i.e. you get false positive results. A further complication is you want to test for multiple omissions, even if only for one letter at a time.

I think you might need to take a different approach. How many discrete 'letters' are there? Is this a closed list (i.e. doesn't change over time)? worst case is this naming set one forced on you by external data or is the design self-inflicted - in which case perhaps you want to find a completion marker system that's less hard to track.

Bear in mind:

$LetterSet = $LetterSet + "some value";

only ever adds to the set (which you don't want):

$LetterSet = collect(...

builds a completely new set each time the code fires (better).

Don't overlook breaking out only the logic in immediate use into a smaller treat rig TBX making it easier to test edge cases. Testing in the full file may be adding in effects not part of the problems as discussed thus far.
Back to top
 
 

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Derek Van Ittersum
Full Member
*
Offline



Posts: 168

Re: Query help (all agents missing certain items)
Reply #8 - Jan 7th, 2016, 10:17am
 
Grr. My mistake! I meant to say that I had constructed an "action" instead of a "rule" (I wrote "query" above, but meant "action").

At any rate, the action appears to do what I want. I have reworked the data set somewhat to get better analysis. Instead of naming the notes using letters, I have moved these to a set of codes for each note ($Code).
Back to top
 
 
  IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Query help (all agents missing certain items)
Reply #9 - Jan 7th, 2016, 10:49am
 
Wonderful. It'll all be plain sailing from here...  Wink
Back to top
 
 

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Pages: 1
Send Topic Print