Coppieters, New Zealand

The Tightener project is still moving forward. I’ve just released alpha version 0.0.8.

Tightener is ‘automation glue’ – it ties together all kinds of computer softwares and allows them to interact in a structured and efficient manner.

Tightener is a developer’s tool – it lives in the background and allows developers to integrate disparate systems.

Watch this little Youtube demo (skip to time stamp 08:00 if you don’t want to see the intro):
https://www.youtube.com/watch?v=_r55k54AuBA

https://github.com/zwettemaan/TightenerDocs/wiki/Tightener-Architecture

This example shows how Tightener is used to make a bridge between Jupyter Notebooks (https://jupyter.org/ – originally from the Python world), and ExtendScript or UXPScript in InDesign.

This combo of Jupyter Notebooks and ExtendScript allows would-be scripters a much smoother way to ‘get into’ scripting.

For someone new to scripting, using Jupyter Notebooks is much more ‘natural’, interactive and instructive than using more specialized tools like Visual Studio Code.

Personally, I use Jupyter Notebooks during the ‘first figure it out’ phases of a new project. Once the basic ingredients have been determined, I’ll switch to Visual Studio Code for the actual system building.

Yes, I know, installing Python3 and Jupyter Notebooks is not really straightforward in its own right, and that presents a different kind of threshold, but once they’re installed, trying out scripts and script statements is much easier.

You can download an alpha version of Tightener from the Github repo:

https://github.com/zwettemaan/TightenerDocs

Version 0.0.8 has a link here:

https://github.com/zwettemaan/TightenerDocs/tree/main/Releases/Alpha

Other Stuff

Other things that are new to the 0.0.8 release: I’ve open-sourced and moved all of the supporting code for Tightener (scripts, plug-ins, and so on) into the TightenerDocs repo on Github, so you can now inspect or access a lot of the source code that makes Tightener ‘tick’.

For example, have a look here:

https://github.com/zwettemaan/TightenerDocs/tree/main/CurrentRelease/CommandLine/Scripts

https://github.com/zwettemaan/TightenerDocs/tree/main/CurrentRelease/SampleScripts

Next Steps

I’ll now turn my attention to the licensing module of Tightener. I want to make it easy for scripters to protect and monetize their work, without having to wrestle with complicated packaging and publishing rules, and Tightener will be instrumental in achieving that.

The basic idea is that Tightener will protect scripts by giving the developer multiple choices – for example: a developer could opt to never physically copt their script onto end-users computers. Tightener would only retrieve the script from the developer’s own server when the end-user computer is licensed to do so and the script would never be saved to disk, which would help in protecting the script against downloading and reverse engineering.

The Tightener licensing module will take care of ‘uniquenizing’ the end-user computer and checking licensing allowances before allowing scripts to run…

More info to come…

UPDATE: Feb 13, 2024. This page is outdated, and JSXGetURL 1.x.x has been released. Visit

https://www.rorohiko.com/jsxgeturl

What is it?

JSXGetURL enhances ExtendScript to make it easy to access servers using http or https – e.g. to download assets from a remote server.

JSXGetURL is meant to work with any Adobe Creative Cloud application that has ExtendScript support – InDesign, InDesign Server, InCopy, Illustrator, Photoshop…

JSXGetURL is implemented as an ExtendScript DLL/framework, written in C/C++.

This DLL allows you to access URLs (including https: or ftp:) straight from ExtendScript.

There is nothing to install – all you need to do is //@include a .jsx file which then loads the DLL.

Sample code:

//@include "JSXGetURL/JSXGetURLLoader.jsx"
var getURL = JSXGetURL();

getURL.addRequestHeader("Accept: */*");
var s = getURL.get("https://www.rorohiko.com") + "";

alert(s.substr(0,1000));

var headers = getURL.getResponseHeaders();
alert(headers);

// Some random FTP file for testing

var f = new File("~/Desktop/sha512.sum");
var s = getURL.get("ftp://cygwin.com/pub/gcc/sha512.sum", f.fsName);

// Some zip file to test binary file download

var f = new File("~/Desktop/FrameReporter.1.1.8.zip");
getURL.get("https://www.rorohiko.com/downloads/FrameReporter.1.1.8.zip", f.fsName);

JSXGetURL is a wrapper around libcurl which in turn is using open source code from OpenSSL, Boost, zlib.

The last few weeks, I put a fair bit of work in updating JSXGetURL. Version 0.0.9 is now available:

https://www.rorohiko.com/downloads/rr612412/JSXGetURL.0.0.9.zip

Changes

• Added some simple functions to access request and response headers.

• M1 support on Mac

• Upgraded a bunch of dependencies. JSXGetURL now uses

• zlib 1.2.13
• Boost 1.81.0
• OpenSSL 3.0.7
• Curl 7.81.0
• Visual Studio 2019 on Windows
• Xcode 14.2

Licensing

The .zip file contains a fully functional, time-bombed version of JSXGetURL – it will expire on 30-June-2023.

I am currently still working hard on the Tightener project, and by then I hope to have the licensing module in Tightener functional enough to handle a licensing scheme for JSXGetURL. I’ve tried to run JSXGetURL as a ‘sponsored/donationware’ project for a few years, but that has not worked. From June onwards, there will be some subscription fee to be paid for continuous use.

I was mucking around with InDesign GREP styles for auto-formatting dollar values, and had a bit of trouble getting it to work.

If you’re not interested in wading through the technical information below – you’re just poking around to find some example of how to format prices in InDesign: scroll down. The working solution is at the end of this post.

After a lot of hemming and hawing, I figured out what it was that made it behave in a way I did not expect.

Lookbehind does not allow for variable-length patterns

I was formatting prices, and had the following patterns set up:

While experimenting, I had given all the character styles involved a different colored stroke, so the characters would ‘glow’ in different colors depending on the character style applied to them.

This makes things a lot easier when used it together with the Preview option on the Paragraph Style dialog.

The raw text looks like this:

After applying the paragraph style, the result looked like this:

which means it did not format the cent values as expected.

As it turns out, positive lookahead (?=... allows variable length patterns, but positive lookbehind (?<=... does not.

So these patterns, which have lookbehind, did not work:

(?<=\d+)\.(?=\d{2})
(?<=\d+\.)\d{2}

Both patterns look behind for one or more decimal digits.

\d means a decimal digit;
\d+ means one or more digits
(?<= … ) means: look behind the character we’re currently working on
\. means: a period
(?= … ) means: look ahead from the character we’re currently working on
\d{2} means exactly two decimal digits

The first pattern means: look for a period, and then look behind (i.e. to the left of) that period and verify you can see one or more digits. Then look ahead of the period and verify you can see exactly two decimal digits.

But this pattern, a lookahead, does work:

\$(?=\d+\.\d{2})

Removing the + from the positive lookbehind patterns makes it all work:

My working solution:

I’ve put the styles into style groups ‘GREPStyles’ (to keep things organized):

The character styles are:

DollarSign: [None] + superscript
DollarValue: [None]
DecimalPoint: [None] + size 0.1pt + color: [None]
CentValue: [None] + superscript

The paragraph GREP style is set to:

Apply DollarSign to:
\$(?=\d+\.\d{2})

Apply DollarValue to:
(?<=\$)\d+(?=\.\d{2})

Apply DecimalPoint to:
(?<=\d)\.(?=\d{2})

Apply CentValue to:
(?<=\d\.)\d{2}

This is not perfect, but it’ll do for me.

You could easily extend this to also handle thousands separators – I leave that as an exercise.

Postscriptum:

David Blatner gave me a great tip which can be used to achieve a more precise matching and less ‘iffy’ results: the \K pattern.

This pattern allows us to do ‘lookbehind without lookbehind’ and does not have the same issues as lookbehind.

More info here:
https://www.regular-expressions.info/keep.html

Note 13-Feb-2024: Creative Developer Tools is the first tool which uses Tightener to offer new features for UXP and ExtendScript developers. More info about the alpha version here:
https://www.rorohiko.com/crdt

The second public alpha of Tightener is available.

It consists of a single .zip file which works on Mac, Linux and Windows.

This version adds new integrations of Tightener into any Adobe® Creative Cloud applications that support ExtendScript.

It adds the Tightener Daemon CEP panel which helps Tightener run its event loop.

The Tightener Daemon CEP panel is available as a ZXP file which can be installed by way of Anastasiy’s Extension Manager.

To download, visit:

https://github.com/zwettemaan/TightenerDocs/tree/main/Releases/Alpha

For documentation, visit:

https://github.com/zwettemaan/TightenerDocs/wiki/Tightener-Docs

To see a few of the planned features, visit:

https://github.com/zwettemaan/TightenerDocs/issues

The first public alpha of Tightener is available.

It consists of a single .zip file which works on Mac, Linux and Windows.

To download, visit:

https://github.com/zwettemaan/TightenerDocs/tree/main/Releases/Alpha

For documentation, visit:

https://github.com/zwettemaan/TightenerDocs/wiki/Tightener-Docs

To see a few of the planned features, visit:

https://github.com/zwettemaan/TightenerDocs/issues

For a YouTube video demo, visit:

Quick status update on Tightener: the project is still moving; I am slowly gearing up for a first alpha release.

A big chunk of work has been making the TQL glue language support ‘cooperative multitasking’.

Unlike JavaScript, where scripts are not running concurrently, but instead ‘pass the baton’ using constructs like Promises or async/await, TQL provides cooperative multitasking baked into the language, so multiple TQL scripts can run concurrently.

Next step will be to add a ‘yield()’ function to ExtendScript. The idea is that your ExtendScript would call ‘yield()’ frequently, and TQL scripts would then run a bit on each yield.

The use case I am working towards is around InDesign Server. One would use Tightener/TQL to coordinate things.

A web server could interact with the InDesign Server, and launch a TQL script containing an embedded ExtendScript on the InDesign Server.

The ExtendScript would be the ‘meat and potatoes’ script which does pagination, rendering, exporting,… whatever we typically do on InDesign Server.

The TQL script would continue to run concurrently with the ExtendScript and it would be able to easily send live feedback info to the web application (e.g. to show progress bars, update job progress, report failure or errors, interact with the user via the web browser,…)

This would make ExtendScript less of a ‘blocker’ and allow more lively behavior of the InDesign server.

Stay tuned…

UPDATE: Feb 12, 2024. This page is outdated, and JSXGetURL 1.x.x has now been released. Visit

https://www.rorohiko.com/jsxgeturl

Note: 23-Jan-2023: the sponsorship approach did not work, and JSXGetURL now uses an activation system.

Also see

https://coppieters.nz/?p=720

Note: If you are using version 0.0.7 or below, it will have stopped working by now because it was time bombed on December 31 of last year. You will have to update to version 0.0.9 (see link below).

JSXGetURL has been updated – I now have an M1 version available, but only for Github sponsors.

JSXGetURL adds basic http/https/ftp functionality to all Adobe apps that support ExtendScript – this includes InDesign, InDesign Server, Photoshop, Illustrator…

More info: https://coppieters.nz/?p=720

There is nothing to install, no admin privileges needed. You just need to add a folder with a few .jsx files to your ExtendScript project, and you can simply write ExtendScript like:

#include "JSXGetURL/JSXGetURLLoader.jsx"

var getURL = JSXGetURL();

var s = getURL.get("https://www.rorohiko.com");

alert(s.substr(0,1000));

and this works in InDesign Server, ExtendScript Toolkit, Bridge, InDesign, InCopy, Illustrator or whatever else supports ExtendScript, and it works both on Mac and Windows. And it’s very fast because it’s all compiled C/C++ code.

Note 23-Jan-2023: The sponsorship approach I tried in 2022 did not work, so I decided to change the strategy. The plan is that from June 2023 onwards, JSXGetURL will be subject to a subscription fee.

The link below gives you access to a fully functional version 0.0.9. It is time-bombed to stop working at 30-Jun-2023.

If all goes well, the subsequent updated version will be available for a subscription fee.

https://www.rorohiko.com/downloads/rr612412/JSXGetURL.0.0.9.zip

I’ve been working hard on Tightener these last few months, and finally managed to build enough infrastructure to show a meaningful proof-of-concept demo.

The idea for Tightener occurred to me when setting up a web server that interacts with InDesign Server in the backend.

That integration is cumbersome and not straightforward, and developing and testing code is hard.

Doing this requires:
– setting up some method to synchronize data transfer between the various servers. Things like Samba or WebDAV or shared cloud volume. Synchronization problems abound.
– setting up some method to pass data from the web server to the scripts that need to run on the InDesign Server
– setting up some way to test and debug and deploy scripts on the live server
– maintaining multiple pools of code – web server code on one end, InDesign ExtendScript code on the other.
– setting up some method to monitor and track the jobs that are processed by the InDesign Server. Is the job complete? Has it crashed? Where to send the log files?

With Tightener that will all becomes a whole lot easier.

– Tightener is to offer bi-directional data transfer, synchronization and message passing. The web server can receive events from the InDesign server as the jobs progress
– Unified code base and simplified deployment: if so desired, InDesign scripts can be embedded into the web server code repository. Tightener can auto-deploy the scripts to the InDesign server as needed.
– Simplified testing and debug workflow. Interactively run the script against a local InDesign Desktop and a remote InDesign Server ‘as you go’.

There is still a lot of work to be done, but I’ve managed to get past a few sizeable hurdles.

I’ve got Tightener embedded in an initial InDesign plug-in. This extends InDesign with a new language called TQL (in addition to ExtendScript, VBScript and AppleScript).

TQL can access the InDesign DOM (I still have a lot more to cover here, but I’ve figured out how to do it, and finalizing the DOM access is just a matter of ‘more of the same’).

Tightener also handles IPC (InterProcess Communications) so multiple Tightener nodes on a workstation can interact with one another – e.g. a command line script runner can run TQL scripts in InDesign.

Here’s a YouTube playlist with more info:

https://www.youtube.com/playlist?list=PLuW3sE8aHqZa_9BBRIDKAUnLH65JZ-PRC

Also a link to the Tightener documentation project (rudimentary at the moment:

https://github.com/zwettemaan/TightenerDocs/wiki/What-is-Tightener%3F

A few months ago, I started work on new infrastructure to automate Adobe Creative apps (and other apps).

A quick update: work is still steadily progressing.

At the the moment, I have built a framework in pure C++ 11, called ‘Tightener’.

There are no external dependencies – all you need to compile it is a C++ 11 compiler and a standard C++ library.

Tightener has an embedded language, called TQL (Tightener Query Language), pronounced as ‘tickle’.

TQL is similar to JavaScript, but simplified. It has the basics: expressions, functions, scopes, cooperative multitasking….

Tightener provides support for ‘Tightener Nodes’ which are instances of Tightener that are embedded into some environment (a command-line console, a language interpreter, an application, a network stack…).

Tightener nodes running on the same computer can communicate with one another using pipes/shared memory.

I envision many nodes per computer. There is one ‘main’ node and number of satellite nodes embedded into various environments, providing services (e.g. Python, InDesign, Illustrator, node.js…)

Using the messaging mechanism, one Tightener node can ask another Tightener node to run a TQL script.

TQL provides support for some XPath-like queries into entity-attribute Object Models – think of it as ‘XPath for JSON’.

Most embedded Tightener nodes will be able to handle queries into some object model (e.g. a Tightener node embedded in InDesign will handle queries into the InDesign DOM), and hence allow other Tightener nodes to make queries on their behalf.

I am currently integrating Tightener into an InDesign plug-in, so the InDesign DOM will become an integral part of the InDesign Tightener node.

Once that is done, any other Tightener node on the same computer can send TQL scripts to the InDesign node to query or execute.

Next I’ll integrate Tightener into Python, so I can start showing proof-of-concept demos: open a Jupyter Notebook, and interactively construct an InDesign automation script.

TQL is not meant for full-fledged programming (though it is powerful enough to do that).

TQL’s main goal is to reduce the number of round trips between the controlling Tightener node and controlled Tightener node: there is latency between the controller and the controlled. If we can execute some coded logic on the controlled node, that helps reduce the number of round trips we need.

(In short: this is the same thing as why ExtendScript is much faster than AppleScript. AppleScript has some latency each time we ‘cross the fence’ between the controlling AppleScript and the controlled program. ExtendScript lives ‘inside the fence’ and grafts directly only the controlled program).

A remote Tightener node is more ‘like AppleScript’.

An embedded Tightener node is more ‘like ExtendScript’.

So we want any remote Tightener nodes to delegate some of their work to an embedded Tightener node whenever possible. Hence TQL.

TQL is currently fleshed out enough to run all kinds of scripts, but there is still a lot of unimplemented things on my ‘to do’ list – e.g. the ‘switch’ statement.

Tightener nodes are, by design, not network-aware.

Networking functionality has to be achieved by building separate ‘gateway nodes’ which are essentially one or the other network comms stack with a copy of Tightener embedded.

This approach is to avoid introducing unwanted dependencies into Tightener: I don’t want to have to upgrade Tightener each time a new version of OpenSSL is released.

Next steps:
– Finalize Tightener integration into InDesign
– Embed Tightener into Python
– Create a Tightener gateway node by combining Tightener with OpenSSL

I’ve started a public Github repo where I’ll put the Tightener documentation as I write things up.

https://github.com/zwettemaan/TightenerDocs/wiki

There is too much to remember, and I find myself searching for the same information over and over.

I’ve got a bunch of loose tidbits in Evernote, but in practice, Evernote is a bit of a sink: file a lot, rarely retrieve.

I am going to start keep track up useful snippets of code.

Every time I do a search and find some samples, I’ll make a re-usable snippet out of it – sample code for copy-pasting or testing

I’ll try to keep them organized, mainly for my own use, and for anyone else who can use it.

I’ll create a separate page for these: https://coppieters.nz/?page_id=451