Janus Macros – Problem of Permutations

Long live Janus. ;-)

 

I’ve been doing some custom development for an architectural visualisation company in Australia called 3DVIZ recently. During Janus’s commercial phase,  3DVIZ bought a licence. It is usually the case that I don’t know what Janus user actually do with Janus technically-speaking. Few users ask for feature requests, and even fewer ones explain their workflow and how to better improve it through Janus. That’s mainly why Janus developed the way it was: my own personal production needs, and my curiosity in proceduralism.

It is the proceduralism that seemed to draw 3DVIZ into Janus. When I wrote the FOR loop constructs in Janus, it was mostly out of admiration of Houdini and the curiosity of what proceduralism could look like in Janus. No had asked for it, I only had an inkling that I may, just possibly, need it if ever the chance I get into a large-scale project in LightWave. But, if I’m honest, I’ve never actually used FOR loops for any of my commercial projects; none of them were ever big enough to warrant to advantages of using the feature.

When 3DVIZ contacted me for support, I realised that they were using it in a far more advanced way than I personally used it as a TD in commercial work. It was gratifying to see that someone actually had some need to proceduralise their scene management and rendering to that level, and that Janus’s FOR loops actually made all the difference.

3DVIZ approached me again with a permutation problem. From the little I know about it so far, their asset hierarchy is well-organised (ie rigid). And this is the case because they need to be able to render more variants upon variants; from a house, they render parts of the house in varying configurations, each with their own types of materials, and so on so forth.

Part of 3DVIZ’s own request, since they know their problem more than I do, is to enable them to automate scene loading and Janus operations from a more ‘global’ script; as FOR loops handle the breakouts for any given scene, now they want to expand that capability across scenes. The concept is similar to LightWave’s own Render-Q script, where a script resides ‘persistently’ and orders LightWave to do tasks.

The most obvious idea to automate Janus is to allow it to accept macro commands. A ‘controller’ script handles the scene loading, then writes a macro to a file, which contains commands to breakout so-and-so render pass; then signals Janus to receive the macro; when Janus completes the macro, the ‘controller’ loads up another scene in its queue, then writes another macro, and repeats the procedure.

Thanks to the years put into Janus, the implementation of macros was clean, and with some limited testing, the concept works as well as imagined.

However, my main obstacle would be their expansive asset hierarchy. The real challenge is to make sense of it in my head, and design a ‘controller’ script that creates sensible macros that solve 3DVIZ’s particular problem of permutations.

 

Retirement and Retrospect: Janus EOL

After some thought last year, weighing in what I want to do for the future, I  decided that I should ‘clean my closet’ first. And one of the things that stood out in that closet was Janus development. There have been no new sales for Janus for quite a long time now, unsurprisingly, because I never really made any respectable effort in its marketing. Besides a few clients, its userbase has equally been quiet. And so I’ve decided to retire Janus from commercial development, and the main reason is that I can’t see myself guaranteeing the same kind of support that Janus users have enjoyed through the years for free.

It feels like a nominal thing to say that Janus is no longer being developed because Janus dev hasn’t been as active, and I’m pretty sure only a very few are concerned with its development anyway. If anything, announcing the fact will simply get me off that spectral hook of ‘developer obligation’ for EOL products. At least that’s what I hope.

I’ve always said in the past that Janus was never meant to be a mainstream tool. But over the course of the years, I learned one major reason why: many LWers didn’t try it. Part of me comprehends the rationale that the Janus video tutorials described a workflow that they didn’t like, or was confusing. But this is what I couldn’t understand: despite the complaints of the lack of a render layer system in LW, why people wouldn’t even attempt a free try in the hopes that they make something out of it.

Then there was the price point of 200 bucks (later 100 bucks) which might have made it totally incompatible with their idea of a layering system, no matter how well (or badly) designed. I kept hearing their demands to NT to put a layering system in there as part of their LW upgrade path, avoiding the need to invest in a 100-200 dollar plugin. Apparently, they’ve been waiting for a long time: at least 7 years.

What strikes me ironically, in retrospect, is that if they had invested in Janus from the beginning, it would have been a free Janus upgrade path from then on. Of course, I can’t say that I would have guaranteed it, though it seems likely since it was the case despite the minimal community support behind it. It would have been a very small price to pay to have such a tool that early on in LW9.5 — LW 2015 still does not have a functional equivalent of Janus or a render layer system. I say that in retrospect; a few Janus users have been saying it for years.

Most LWers have lived without a proper layering system, because the need is not truly pressing for most of them; I think despite their complaints, they can wait 7 more years if they have to.

Whether or not I would have stopped development regardless of its popularity is something I will never know myself. To me, Janus, as a commercial product, has run its course. I think it does a lot more than what was advertised, which is a good thing for me as a developer; I’m proud of what I’ve accomplished, but more grateful for the things I’ve learned developing this tool. Janus is still available to be bought, but no support will be given (unless I can actually afford to), and I will try my darnedest to ignore bug fix requests: it’s easy to get obsessed with them, and they eat up lots of my time.

This is not to say that I’m through with Janus: I use it daily, and I will continue to code it to solve problems that I encounter myself. I may yet support Janus in the context of a company as a technical consultant, which is a better use of my time, and I can be actually recompensed for my work. I may fork it, or create a derivative for other progs like Maya — who knows, really? The future unknown, and I’d rather not try to plan or predict. I’ve focused on tools for the most part of my vfx career, and left creative pursuits largely untravelled. And that’s where I’m headed next.

User functions FTW

Here I am, few days later talking about user functions in Janus again. Why not? It’s been giving me results and it’s fun to play with — as a TD, I mean.

Back at work I wanted to output an RGB matte for multiple instanced object. LW doesn’t have a way to do a scene-based surface override on an object (perhaps shaderMeister?– though I don’t know if it the Spot Instance node will work with it). Basically, I wanted an easy way of assigning RGB colours to items in a particular group.

Before, Janus’s item processing was strictly group-oriented, where group settings were simply propagated to the items. There was no way to individually and programmatically affect certain items within the group. So in the latest rev, I changed that. But the important bit was that I tied it (again) to user functions, and I embedded a new context — the group item context — through populating two constant variables relating to the group item’s index and name. In this way, the user function can be created that references the group item’s index, and based on that value, the command parameters can be changed on that particular item as it is being processed. It’s basically a ‘last minute’ change in the render pass settings just before the group item is applied its parent group settings.

Since user functions are basically string replacements from within the cmd line, there’s a lot of flexibility (but can also end up a heaping mess if the Janus user doesn’t watch it), so that I can dynamically various bits of settings; I’m not limited to adding one parameter, for example, because I can just concatenate the new subcommands in the return string; so as along as the final resulting cmd line is syntactically valid, Janus will grok it.

Janus for Mac

It’s a bit too early to shout out, so I’m whispering here in my little blog about Janus for Mac. I’ve restarted development for the Mac. It basically just took a weekend to hunker down and sort what was going with it.

The biggest issue that I came up with was the use of LScript’s store() and recall() functions, where in Windows it uses the registry to store the information. In the Mac, I’m not sure where it ends up, but apparently, calling it those functions multiple times seemed to put LW in a state where it could no longer stream more files into memory. It was as if the i/o was full.

I avoided this by redirecting the functions; this was already facilitated by the fact that they were already housed in custom functions, and all I had to do was make it everything cohesive. I also cleaned up the code so that the array that Janus uses to store settings were more consistent, and that the PC version should remain unaffected by the change.

At this point the Mac seems to be very workable, and I’m probably going to do a few iterations of tests in the coming weeks. And who knows? — it may come out officially soon after that.

User functions revisited

User functions have come back, and what seemed to me at the time an excuse to implement a structure that’s both ambitious (because it wishes to liken itself to Houdini) and hacky (because it ain’t Houdini), seems now to have been vindicated.

Talking about user functions is like talking about… well, nothing analogous I can pick out: no one relates to it, not because it isn’t uncommon — cg ops use it all the time in other progs — but because it’s in Janus, and thus LW, and among the LW crowd, it’s weird.

But then I contact this person asking for some scripting help; he’s made his assets and scenes, and needs a way to break out passes; I learn about what he particularly wants, and think Janus fits the bill, albeit requires a bit of modification to get the exact workflow. I ponder how much modification is needed. Then I realise that I already have a system of user customisation in Janus: user functions. But user functions, prior to this new development, was only in the realm of the more broad render pass settings (eg subcommands, and the cmd line). What was needed was to expand the same functionality to partial surface overrides, which were contained in text files.

This modification — user functions implemented in partial surface overrides — was done in one night, because there wasn’t much fudging of code; the functions were all there, and all I needed to do was apply the right timings so that the parsing functions didn’t fall on top of one another, and that the proper variables were being populated first before passing it on to the user function. So now, using the very system I made for Janus users, I’m using that system to get unique results instead of hard-coding things in.

I’m very grateful, and again, feel vindicated, that by not taking the easy route, by sticking to the vision I had for Janus, I am seeing the result of the flexibility, which I had hoped would be more evident to LW users. While I still inwardly bemoan the fact that Janus is not a very popular tool, it’s hard to feel sorry for it when its proving itself up to task in more and more situations.

User functions in Janus

Back somewhere around v1.6 I started developing the pre-/post-breakout script feature, where a script is run contextually to coincide with the breakout of a render pass. This matured to the script system in Janus in 1.7. But it was back then when I was in the mental thoroughfare thinking of ways to get user-created scripts to work with Janus. Specifically, I wanted to call functions from within the render pass cmd line; these functions should be completely user-defined, and obviously, referred to at run-time. This meant that I needed Janus to dynamically call a script. The wall I encountered was that because Janus is sourced and compiled, it can’t source (e.g. include/insert) any other script once it is distributed. I thought the gate was closed at the time.

Recently, I’ve broken through the wall by going through a surprisingly obvious route. In a word, the solution was to execute a single pre-defined script (at least for the initial implementation of the feature) which the user modifies or adds user functions.

The first step was to re-use the structure of how subcommand variables are processed, since they share pretty much the same procedure.

The second bit was to implement the Queue Object Agent in LScript into the procedure. The Queue creates slots which serve to communicate to the user-defined functions the function call name and its argument(s). The function name and arguments are derived from parsing the cmd line. Once the function name is located, and arguments identified, this information is inserted into the slots of the Queue.

Then comes the third step, which was to create the base user function script that Janus will call once the Queue is populated. This base script calls upon the Queue that Janus has created, and starts querying the slots for information. It gets the function name and arguments, then checks if the function name exists; runs the function within and expects a return; the return value is passed on back to a reserved Queue slot, whereby Janus picks it up again and replaces the whole function call with the return value.

Neat.

It took about thirty minutes to prototype and two and half hours to implement. More tests required, but I was surprised that I got it working that night.

I suppose it also bears noting that the motive of this effort didn’t just come out from the blue. I received an email from a client very recently asking how he can fine-tune his output file paths. He wanted to be able to rearrange the output path so that his version number stuck at the end. Because filenames are just single strings, there is no inbuilt way of identifying which is the base name against the version number; the whole filename is effectively the base name. A parsing solution is needed. But because this is Janus, hard-coding is frowned upon. The solution must come in a form of being able to programmatically process the render pass parameters so that the user can do anything with it, and free to modify it in case their circumstances change.

B52 R4

Janus 1.7 B52 R4: released it, and with it a price adjustment to $99 (from the original $199). But the maintenance update marked only bugfixes, and nothing spectacular in of itself. I think the price adjustment is more of the headline.

It’s a bit difficult to keep on reasoning that Janus should maintain the price point it has had for five years. If inflation had anything to say it should have gone up. Of functionality, Janus has increased in bang: in 1.6 where the interface was really streamlined and new workflows were introduced to form a more sophisticated base for later development, and in 1.7 where the offline method was implemented. But Janus dwarfs the price point of its fellow LW plugins – though it has no functional off-the-shelf peer in LW-land – and I’ve received hints from more than a couple of people of the price being the main turn-off.

But some also complain that’s its too technical, and I think that no price is going to compensate for a system that doesn’t feel right for them. Fair enough.

The many faces of Janus

When I typed in that title I wasn’t intentionally alluding to Janus, the Roman god. But it’s cool nonetheless.

I’ve been developing Janus for seven or so years, and I’ve faced, time and time again, the difficulty in presenting the Janus in a way that satisfies me – because I think it’s a good concept, it makes it even more difficult.

In my previous post about Dynamic Mesh Combining, and my recent re-vamping PSAT rendering I’ve been reminded of how cool coding ventures (from my point of view) have been dwarfed up by the pressure to promote Janus as a mainstream product, which presents the difficulty of consolidating the features that make up Janus into something that most LW users can understand and appreciate. And thus, I laboured two weeks on writing and creating an overview in the hopes that I have explained in a little bit better this time.

But this didn’t change the fact that there are a lot of stuff in Janus that’s difficult to present. On one hand you have For Loops, and on another you have Dynamic Mesh Combining; there is PSAT rendering, plugin support, buffers support, the proxy system, search-and-replace functionality (in object replacement, and throughout the system), the script system, the offline method, etc. Then you have these strange little terms like ‘composite command types’, which, if you were a Janus user, would make more sense – though it is most probable that even Janus users don’t touch it.

It’s though I’ve put in so much functionality that I wanted to have because I didn’t want to run another script to do it for me. One may venture to consolidate these features into a superlative blurb that it has ‘everything’. It doesn’t have everything, but it has a whole lot that no one knows about.

For Loops in Janus

In the recent 1.7 release of Janus I’ve implemented For Loops, which is a way to get a looping construct in the Janus context. It’s rather interesting, and it all came because a prospective (Janus) client emailed me one day asking me if it was possible to procedurally tell Janus to replace objects in a specific directory. There was none at the time, and the solution to that was to either to create render passes dynamically via definition files, or use composite command types. He was happy enough for such a solution. But for my part I felt things could be done better: it was still not procedural insofar as the generation of the data strings was external to Janus: what I wanted to do was get something procedural so that extra scripting was not necessary to do these sort of things.

Enter For Loops, which are represented as subcommands. For Loops tell Janus to ‘recycle’ the render pass (similar to what composite command types already do). In For Loop ‘recycling’ the render pass is internally duplicated; this duplicate passes through a string filter and replaces variable names which are a specific subset for the given For Loop. This contextualises the data that is being accessed; if the loop is around a directory, for example, then you can’t use the variable to get file line information, etc. However, being a construct, they do share some data, such as the iterator; perhaps more can be added in the future.

The prospective client didn’t end up getting Janus, though, as it often happens. But the great thing about it is his question sparked a good idea in my brain. It may also have been through my exposure with Houdini and its Wedge ROP that I’ve implemented it the way I have. These emails from prospecting clients was the root of another less-known feature of Janus called Dynamic Mesh Combining, which combines two or more mesh items into one and it does so during breakout so that the master scene is not modified in any way. Dynamic Mesh Combining came because a person asked me how to prevent shadows from being cast by separate shadow catcher. The only solution to his specific problem was to combine all of his shadow catching objects into one and then turn the Self Shadow render flag off. Because combining meshes is a tedious task, why not implement it?

Janus 1.7

I’ve just released Janus 1.7 which is a major update since a long time.

This cycle started when Mikael Burman, a Technical Supervisor working at Ubisoft Massive looked for a feature where Janus could break out passes without loading the scene in. Well, as it turns out, more than seven years ago, I was wishing that very same feature. Back then, however, the engine wasn’t mature enough; there were too many things to address, and the feature itself was a daunting task which, I had thought then, required an extended development. This feature was called the ‘offline method’; offline denotes that the scene file is not loaded into memory, but that the file will be parsed directly by Janus without pre-processing. I don’t know if this is Janus’s claim to fame (because Janus is not really that popular), but I  don’t know of any other render layer offering that can do that.

Had I attempted to do this back when I wanted it to, I very much think it would have taken me at least three times longer to do, and the coding wouldn’t probably be as clean that I could build a lot of future development on top of it. I credit the relatively slow and deliberate updates that led up to 1.6 with the robust base that I built the offline method on.

Janus’s breakout engine has two processing phases. The first phase is scene pre-processing, which prepares all objects for scene parsing; in this phase the items are ‘tagged’ with their own particular settings based on final evaluation of that pass. Key Layout functions are employed here which help in exporting shader-based overrides, and other peculiar tools such as mesh combine (combining separate meshes into one object during breakout).

The second phase is scene parsing, which opens the scene file and starts parsing the scene out, line-by-line, looking for keywords and tags, and dumping it to another scene – the breakout scene – with the data modified.

In the offline method, the second phase was generally left untouched, except perhaps for some additions to new tags that were related to the offline method that needed to be considered. It was actually the first phase that was particular different. In fact, the engine’s first phase is, for the most part, bypassed when the offline method is being used. Some of the code was still used, of course, because I couldn’t possibly think of changing the main ‘trunk’ procedures. I changed some of the called functions to imitate what was already going on in the ‘online’ method.

But this didn’t change the very base limitation of not being able to deal with shader-based overrides and dynamic mesh combination. Of course, these are actually possible, but it entails a lot more work. While I have some functions designed to write shader-based overrides directly into .lwos, my suite is not comprehensive enough for what is required. And dynamic mesh combination is a bit further away from my reach because I don’t have any code pertaining to handling .lwo mesh data.

I’m not sure if it’s worth the time, as well.

Lastly, there are still some things that can be addressed in the offline method. One disclaimed limitation is the fact that once you create your groupings in Janus, you cannot change them when the scene is unloaded, and the user is working in the offline method. This is because the groupings have been tagged into the items, and this used the parenting information from Layout to get those items tagged correctly. Perhaps there is a way to re-assemble the hierarchy internally when the the file is loaded as offline so that groupings can be changed around. It is a bag of bugs, I’m sure, but maybe it can be done on the side.

At any rate, I’m still busy with other things non-technical in nature. Truth be told, at this point I’m just more eager to use Janus for the stuff I’m doing than coding it.