It has been about a month since the initial release of JavaScript/TypeScript support on the Moderne platform and OpenRewrite. Development continues as we build out recipes, dogfood the API, and make continual improvements.
I think it would be useful to speak to some further improvements we have planned, framed as a Q&A between a well-meaning developer who took the time to write out their initial thoughts recently. The questions are summarizations of the real questions posed by the developer.
We have a particular vision for where we want OpenRewrite to go, and admittedly not all of that vision is immediately clear with the initial release. So I love the opportunity to share that vision in this blog first, and I look forward even more to be able to show that vision come to life with subsequent releases.
YAML recipe support for Javascript
Developer: The declarative YAML recipe format works wonderfully for Java. Will this same capability be extended to JavaScript/TypeScript recipes? Being able to compose and configure JS transformations declaratively would be valuable.
Jonathan: Yes, we always intended to add the same declarative YAML support on the JavaScript/TypeScript side.
First we needed to carefully consider how recipe packaging works in NPM and how we discover recipes that exist inside that package. We have been working on the next iteration of RecipeMarketplace which combines concepts from CategoryTree and Environment. Both CategoryTree and Environment have been around from really early versions of OpenRewrite (I believe they are 6 major versions old at this point). They’ve served us well, but couldn’t possibly have anticipated what a combined recipe marketplace consisting of bundles of recipes produced in language specific package managers would look like (or even if we would ever have such a thing). I’ve always tried to follow the principle of not designing too far ahead of what I know now.
RecipeMarketplace is taking shape, consisting of contributions of recipes from different RecipeBundles with associated RecipeBundleLoaders for each “ecosystem” that supplies collections of recipes. That includes “Maven” and “NPM” ecosystems of course, but could really be any source that is identifiable by a package name and version. It is in that RecipeBundleLoader for a particular ecosystem that we decide how we discover recipes inside a bundle, and as this API solidifies and we introduce the NpmRecipeBundleLoader, we have an opportunity to consider where we expect YAML recipes to be found inside an NPM bundle.
In the meantime, recipes have recipe lists even in their code form, so it is still possible to compose higher-level recipes from lower-level building block recipes. We’ve been using this for now as we’ve been working out Node migration recipes and other compositional recipes. But this is only meant to be a starting point.
There is something more to say about RecipeMarketplace, but one of your later questions gets into the subject so I’ll put a pin in it for now 😀.
Preserving whitespace WIP
Developer: These abstractions for preserving whitespace, like J.RightPadded, J.Container and implied keywords, feel intrusive when navigating AST elements. When working with JavaScript transformations, accessing simple node properties requires unwrapping through these containers.
Jonathan: This is an insightful question, and indeed something we are not satisfied with. Early forms of OpenRewrite on the Java side also required this unwrapping. As you can see, we long ago inverted LST navigation to avoid having to navigate through padding.
On the Java side to get to the padding looks like this:
fieldAccess.getPadding().getName().getBefore() // for a left padded element
forEach.getPadding().getBody().getAfter() // for a right padded elementI really wanted to solve this in the initial release of JavaScript/TypeScript too, but there was SO much work that went into the recipe dev experience, LST modeling, RPC, etc. that we decided to go ahead and get something out and come back to this problem.
The solution eluded us for a while. After all, TypeScript LSTs are defined as interfaces and not classes (for a number of reasons, but allocation pressure is a big one). The consequence is we can’t add behavior for navigation being inside-out like it is in Java.
I think the design we will pursue is clear now though. On the JS side we will replace:
LeftPadded<T>withPadding & TRightPadded<T>withT & PaddingContainer<T>with(T & Padding)[](parentheses added for emphasis)
This puts padding on the same “level” as the data of the syntax element rather than a wrapper around it. There is no element on Padding consequently. There is also a larger design realization hinted out by this specific example. As long as there is a hole for every piece of data on both sides of the mapping from JS (as defined in Java value objects) and JS as defined in typescript, they don’t have to have exactly the same shape. And that’s a great segue to your next question…
JS model extending J model for cross-language work
Developer: The JS model extending the J model is my primary concern. The inheritance relationship creates significant confusion. The J model is clearly Java-specific (J.ClassDeclaration, J.MethodDeclaration, Java keywords).
I understand the potential rationale: enabling recipes that work across both languages. However, such recipes would be extremely limited (formatting, imports?).
Jonathan: Your understanding of the rationale “enabling recipes that work across both languages” is correct. In the first attempt at OpenRewrite for JS a couple years ago, I thought very similarly to you about them being independent LSTs with no overlap and evolved my thinking over time. I ask that you kindly suspend disbelief for a moment and follow my description of the result of that evolution.
The intuition that “such recipes would be extremely limited” turns out to not be true. We have discovered reuse on several levels of the recipe ecosystem. Naturally, framework migration recipes are language-specific, but this is just one part of the overall recipe ecosystem. Just a few examples in a not-exhaustive list:
- High-value core building blocks like “Find types”, “Change method name”, “Find methods”, etc. Some of these core recipes are used both independently and as preconditions to other recipes. Indeed, some of these preconditions are so common that they led to a whole class of avoidance optimization and pre-computation of precondition evaluation that allow for efficient execution of recipes on billions of lines of code in minutes inside of the Moderne platform.
- For known SAST-like recipes, we see moderate double-digit overlap (30-40%).
- Classes of recipes that do things that look at literal values like “Find SQL”, “Find secrets”, recipes that look at feature flag use, and many others are language-neutral in the C family.
Just how much overlap there really is will become more apparent with the new RecipeMarketplace, because we will be placing recipes in multiple categories. All the recipes that we’ve discovered work well across Java and JavaScript will be placed in both Java and JavaScript categories so one doesn’t have to guess at the implementation of a recipe to determine whether it is cross-language. That same capability will allow folks building up their own marketplace to essentially reorganize the available recipe marketplace in a way that is natural to them as well – kind of like moving furniture around the living room.
In addition to there being categories of recipes that are cross-language, the OpenRewrite advanced program analysis module consisting of data flow, taint flow, etc. has been discovered to be language neutral in the C family because all of the exit points in such analyses happen to belong to the overlapping LST elements. Given that advanced program analysis is the foundation of symbolic execution, differential testing, and other technologies we’re currently developing, I have great hope that many of these too are indeed more mutli-lingual than I would have originally thought.
All of this said, there are ergonomic improvements we still need to make, and you’ve pointed out one of those examples: Java keywords. I am convinced, however, that even if it’s not a perfect topological isomorphism, there is so much overlap between C family languages that the reuse justifies the work. I think what offends developer sensibilities more than anything is to see J.MethodDeclaration rather than JS.MethodDeclaration, even though both have the same structure. It makes a loud and controversial statement before you have the chance to experience reuse, and I think makes it less likely folks are willing to even consider it. We can smooth that over to alienate developers less at the beginning.
There are already several places in the LST model where we have chosen a different “name” for an LST element that has a piece of data of the same shape as the Java LST and that fits into that differently named hole. We can do more of this to further reduce what I know is a natural developer resistance to seeing the topological similarity between these languages, at least at the LST level.
There’s the old joke about topological isomorphism about the mathematician who can’t tell the difference between a coffee cup and a donut because you could bend a donut into a coffee cup with enough effort. I’ve often thought that Shakespeare was a closet mathematician and was thinking of topological isomorphism when he wrote “That which we call a rose by any other name would smell as sweet.”
Visitor overrides: What’s actually important
Developer: As an extension to the above question, how do we know which visitor methods are needed to be overridden to handle all the variations?
Jonathan: For quite a lot of recipes, you don’t care how you get to a piece of data (e.g., which LST elements are traversed through to get to a particular type of LST element), you only care that you arrived.
When I write a simple recipe that makes a change to a library’s method, for example, I don’t need to consider where that method exists in the structure of the code. That’s because visitMethodInvocation and the whole visitor pattern is kind of like an event-driven system for interacting with LST elements of a particular kind. The same visitMethodInvocation gets triggered for both of these sysout calls regardless of how much structure is around them.
So when we design LST elements in a compositional way, we are essentially just adding another element that this event will pass through on the way to the thing you care about.
Only when you care to interact with the truly unique structure of a type of syntax do you implement its visit method.
Why build on OpenRewrite: Encapsulation, types, and lessons learned
Developer: At present, I'm considering working directly with the TypeScript Compiler API and building my own transformation utilities. But I'd prefer to contribute to and build on OpenRewrite if these architectural concerns could be addressed.
Jonathan: Ultimately, this decision is up to you. I hope you’ll choose to consider the direction as a positive one not in need of being “addressed” beyond the incremental improvements I’ve suggested. We’ve shipped something that still needs more polish. And we ship just about every week. In the journey to build my own transformation technology over the years, I’ve messed up more times than I can probably remember. I like to say there’s a reason why we are on major version 8 and that’s because I’ve been seriously wrong 7 times before.
I have two more thoughts that don’t fit neatly into your questions.
Thought #1 – Encapsulation. The first is that OpenRewrite has in many ways been a love letter to the old OOP principle of encapsulation (maybe not so fashionable today as it once was). Encapsulation exists at multiple levels in this ecosystem.
One of them is the LST itself and the visitors that operate on them. But that is only the lowest level. And that’s similar to where you start with the TypeScript Compiler API (or any other language’s AST).
Visitors like those that manipulate imports are the next level up. There’s a lot of subtlety in making visitors run on subtrees, run multiple times, whether they are required to be thread safe or not, etc. Even add/remove import visitors are very difficult building blocks to get right because of the crazy number of permutations for the way imports can function. Any transformation technology can’t just make a syntactically “right” change. It needs to manipulate them in a way that fits the stylistic opinions of the particular repository in question, opinions that will invariably be conflicting with the opinions in the repository next door.
Moving up a level is the Traits mechanism. Traits themselves can inherit this cross-language nature by the way. The SqlQuery trait functions on JavaScript precisely because of the LST isomorphism. And it will function on Python and C# too… and Ruby… and Go…
Recipes themselves are encapsulations, as you well know, as are compositions of recipes.
Encapsulation to me means sharing. I think there is a beauty in the fact that the SqlQuery trait could very well have been written in JavaScript by a JavaScript developer and functioned and added value to Java developers. Rarely do we see the opportunity to extend a helping hand across language boundaries so directly.
Thought #2 – Type attribution. The second is about type attribution (JavaType/Type). One thing I’ve learned in all the years working on OpenRewrite is how essential type attribution information is to a whole lot of recipes, not just transformations but searches too.
One of the examples I often use is this recipe looking for “sensitive data” being returned by an API endpoint. Suppose I wanted to find “last name”. In the type attribution information I can see that Vets returns a personList of Person that in turn contains a lastName. That isn’t apparent from the text of this code or the AST. And lest you think you can cross-reference Vets from somewhere else in the repository, you often need to consider the case where there is some type intermediate or not that is coming from a binary dependency outside your control. All of this is visible in type attribution.
The thing about type attribution is it is a brutal slog to get it. The TypeScript compiler will have ready access to types in a repository who uses all modern libraries that contain type definitions, or have studiously added DefinitelyTyped devDependencies for all of its dependencies. But what about the types of the transitive dependencies of your dependencies? And what if the repository wasn’t built for typescript at all and doesn’t have any of those?
A lot of our work goes into finding ways to interact with the compiler to maximize type coverage. In this example, that could be injecting DefinitelyTyped packages as needed, but there are many similar situations.
YMMV of course depending on how internally consistent your codebase is, but I’ve found most companies of sufficient complexity have an archaeology of code spanning years, people, and miles, and the happy path is rarely the only one.
I am grateful you took the time to write, and I hope this has at least been a fun glimpse into our thinking!
Learn more about Moderne’s JavaScript/TypeScript support
To learn more about how to transform JavaScript and TypeScript apps with Moderne and OpenRewrite:
- Read our launch blog.
- Contact us for a demo.
