Welcome, Guest. Please Login
Tinderbox
  News:
IMPORTANT MESSAGE! This forum has now been replaced by a new forum at http://forum.eastgate.com and no further posting or member registration is allowed. The forum is still accessible via read-only access for reference purposes. If you wish to discuss content here, please use the new forum. N.B. - posting in the new forum requires a fresh registration in the new forum (sorry - member data can't be ported).
  HomeHelpSearchLogin  
 
Pages: 1
Send Topic Print
Export to JSON (Read 11689 times)
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Export to JSON
Feb 15th, 2010, 4:32am
 
To get string/set data to JSON, I need to backslash-escape all literal ( straight) single and double quotes, backslashes and line breaks. I thought I could use an approach like:

^(value(eval($MyText=runcommand([command]);$MyText)^

In command I was using echo and a chain of sed commands to nibble away at the replacements. I'm putting the command in another notes text to remove one 'layer' of quoting in the above code, i.e.:

$MyText=runCommand('echo ' +$Text+$Text(MyCommand));

However, I'm tripping over the fact that I've several processes all using quotes and backslashes in the same way.

Any bright ideas?

[Why JSON? Well, I'm scouting a method that might allow easy export of TB treemaps to interactive web pages and the technique uses JSON as the content data format]
Back to top
 
 

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Export to JSON
Reply #1 - Feb 15th, 2010, 7:22am
 
Some progress, but still need a nudge. OK, note 'JSON_source' has this rule:

$Text(JSON)=runCommand($Text(sed), $Text)

'sed' has this text:

sed 's/\\/@/g' | sed 's/"/\\"/g' | sed "s/'/\\\'/g" | sed 's/$/\\n/g' | tr -d '\012'

The @ is just a text to spot I'm finding backslashes - I'm not. The 'tr' command is stripping the actual newline characters. In fact, eventually, the paragraph breaks will be encoded as HTML but I need to get JSON-safe output first. Note how the single quote needs an extra escape backslash in order to work - presumably that's to escape TB-side parsing.

Source text:
"This is John's "place" pad."
"More \stuff."


Ends up as:
\"This is John\'s \"place\" pad.\"\n\"More stuff.\"\n

Still to fix: find and escape backslashes.

Anyone?
Back to top
 
« Last Edit: Feb 16th, 2010, 11:38am by Mark Anderson »  

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Charles Turner
Full Member
*
Offline



Posts: 180
New York, USA
Re: Export to JSON
Reply #2 - Apr 12th, 2010, 9:59am
 
Hi Mark A:

I think I would approach it with Python (or Ruby, Perl, etc.) where there are some nice facilities to keep you from going crazy. Here's a Python solution that uses System Python (2.5.1) on Leopard:


#! /usr/bin/python

import re, sys


def escape(m):
 if (m.group(0) == '"'):
   return '\\"'
 elif (m.group(0) == '\t'):
   return '\\t'
 elif (m.group(0) == '\n'):
   return '\\n'
 elif (m.group(0) == '\f'):
   return '\\f'
 elif (m.group(0) == '\r'):
   return '\\r'
 elif (m.group(0) == '\b'):
   return '\\b'
 elif (m.group(0) == '/'):
   return '\/'
 elif (m.group(0) == '\\'):
   return '\\\\'
 else:
   return 'FOO'

print re.sub('(")|(\t)|(\n)|(\f)|(\r)|(\b)|(/)|(\\\\)', escape, sys.stdin.read())



It's meant to be used like a "filter," thusly:

cat test.txt | ./jsonify.py -u

where "test.txt" had your sample string and the "-u" argument specifies binary treatment of STDIN's data.

Not completely sure how you are using this as the sample was a pretty illegal string in lots of languages. Also, I'm sure your know that JSON doesn't need the asterisk escaped.

Anyway, using sub() to set up a little state machine makes life a lot simpler: only one pass through your data!

HTH, Charles


Back to top
 
 
WWW   IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Export to JSON
Reply #3 - Apr 12th, 2010, 10:45am
 
Interesting, but doesn't as much as it might as it gives an external dependency prone to breakage. Copy the TBX to a different machine and everything needs reconnecting, script permissions (may) need resetting, etc.
Back to top
 
 

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Charles Turner
Full Member
*
Offline



Posts: 180
New York, USA
Re: Export to JSON
Reply #4 - Apr 12th, 2010, 1:38pm
 
Hahaha!

Given how complicated Tinderbox is, I find that pretty comic, but if it really worries you, just use it as a source file:

cat jsonify-test.txt | python -u jsonify.py

You can fiddle with the path name so it could reside in Tinderbox application support or whatever.

Charles

Back to top
 
 
WWW   IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Export to JSON
Reply #5 - Apr 12th, 2010, 4:48pm
 
Complexity is in the eye of the beholder! For me, TB is far less complex than trying to learn Unix stuff. What was behind my earlier point was that often I'm trying to figure stuff for the least tech of TB users. Being able to just give someone a TBX that does everything without having to connect up external files is much easier.  For personal use it doesn't matter where stuff is.

Actually, (feature request!) I'd love it if there was a switch for runCommand(), $HTMLCommand, etc., so TB assumes (or sets) the current directory is that of the active TBX.
Back to top
 
 

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Charles Turner
Full Member
*
Offline



Posts: 180
New York, USA
Re: Export to JSON
Reply #6 - Apr 15th, 2010, 3:34pm
 
Hi Mark-

You mentioned this earlier:
Quote:
Note how the single quote needs an extra escape backslash in order to work - presumably that's to escape TB-side parsing.


I'm still working on this, and I see that your sample text has its backslashes removed when I pass it as "input" to runCommand(). Backslashing the backslash allows it through.

You mention in my quote that Tbox is doing some parsing? Is this explained somewhere?

Thanks, Charles

Back to top
 
 
WWW   IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Export to JSON
Reply #7 - Apr 15th, 2010, 7:49pm
 
In short, there's no precise documentation of runCommand(), using both single and double quote or backslashes takes a degree of care.  IIRC you can 'double-escape' an escape backslash but not a literal backslash. I can't say why but I assume it's pushing the regex parsing of the string being done inside Tinderbox.
Back to top
 
 

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Charles Turner
Full Member
*
Offline



Posts: 180
New York, USA
Re: Export to JSON
Reply #8 - Apr 16th, 2010, 7:20am
 
Well, yes, I guess. (It'd be great to get a small statement from Mr. Bernstein on this issue.)

I did rewrite my code as a Python one-liner:

import re, sys; print re.sub('(")|(\t)|(\n)|(\f)|(\r)|(\b)|(/)|(\\\\)', lambda x : '\\"' if x.group(0) == '"' else '\\t' if x.group(0) == '\t' else '\\n' if x.group(0) == '\n' else '\\f' if x.group(0) == '\f' else '\\r' if x.group(0) == '\r' else '\\b' if x.group(0) == '\b' else '\/' if x.group(0) == '/' else '\\\\' if x.group(0) == '\\' else 'foo', sys.stdin.read())

But as you can see, Python has it's own issues with backslashes. I was able to get it to run as part of a Tbox agent:

$Text = runCommand("python -c "+$Text(thescript) , $Text(thetext))

although it didn't produce the output it should have, because of the need to further escape the backslashes. Pretty hellish trying to figure it all out, if it's indeed solvable.

Perl is a language much more friendly to one-liners, but for me, regardless of language, complex Regex (because of the \s) might be better handled in script files outside of Tbox.

Because that takes us back to your original objection to my contribution, I'll rest my effort for now.

Best wishes, Charles
Back to top
 
 
WWW   IP Logged
Charles Turner
Full Member
*
Offline



Posts: 180
New York, USA
Re: Export to JSON
Reply #9 - Apr 16th, 2010, 8:32am
 
Hi Mark-

One final thing:

Quote:
Actually, (feature request!) I'd love it if there was a switch for runCommand(), $HTMLCommand, etc., so TB assumes (or sets) the current directory is that of the active TBX.


You can do something like:

$Text = `"cd; pwd"

To get the user's home directory. (/Users/cturner, in my case)

So you could ask that users create a Tbox scripts folder there. Its' visible, unlike /usr/local, and the scripts don't need any execute permissions set.

Best, C
Back to top
 
 
WWW   IP Logged
Mark Anderson
YaBB Administrator
*
Offline

User - not staff!

Posts: 5689
Southsea, UK
Re: Export to JSON
Reply #10 - Apr 16th, 2010, 10:06am
 
Charles, all useful stuff. Don't let my personal needs/preferences take a disproportionate stake in proceedings - it's a broad church here! Escape key/control/mark-up characters are always problematic and worse if we try an output language 1 via language 2. Agree that for complex regex, that likely use up all the needed 'control' characters as part of the regex, it may be safest to use external scripts.

The 'lack' of documentation of the command line is - I suspect - partly because it's an 'experimental' feature - like inbound email - so there's an assumption the user knows what they're doing (so I'm clearly unqualified <g>) and will figure it out. The 'experimental' features tends to arise because a user manages to persuade Mark B to support some new task. So, a feature gets added to do that but isn't tested in the widest context.  The feature is mentioned in the release notes and canny users pile in with tasks exploiting the feature but weren't which explicitly designed tested for. I suspect the runCommand got added as the original

Meanwhile, I'd note a few 'rules' that touch on CL use:
  • TB supports both single and double quotes, with double quotes being the expected /default.
  • An exception to the last is format() with lists. I suspect this is an oversight that may change at some point.
  • You can't escape a ' character. Period. At least, I've never never found a workaround.
  • You can sometimes escape backslashes, (though not within TB name/path strings). No hard-and-fast rules, just try it and see.
  • For complex CL code inside TB, it is often best to put the code in $Text of a note (q.v. an example of such just today).
At some point the above list will probably filter into aTbRef - certainly more likely than into the main manual while the feature is still 'experimental'. Some other CL-related TB refs here, here and here.

Back to top
 
 

--
Mark Anderson
TB user and Wiki Gardener
aTbRef v6
(TB consulting - email me)
WWW shoantel   IP Logged
Pages: 1
Send Topic Print