Perception-oriented graphics

I'm going to try and express an understanding that is forming in my brain as a convergence of several interests I've expressed through the years.

Recently I've mentioned some first impressions from Vision Science: From Photons to Phenomenology. I noted that it seems like its models of visual perception mirror the way a modern graphics pipeline is structured. Reading further in, that impression remains. It's not entirely surprising - I can't imagine the designers and engineers who built up the modern pipeline were ignorant of theories and practices of computer vision, which is one of the three disciplines covered by this book (the other two are psychology and neurophysiology), and, for example, the explanation for the importance of effects like diffuse shading in a computer graphics guide I have read recently, OpenGL SuperBible, mirror those in Vision Science.

Much earlier in the year, I mentioned that I'd gotten into folklore the previous year, and specifically Edward T. Hall's The Hidden Dimension, which discussed how people from different backgrounds perceive and make use of space differently when they interact with each other, as well as how different cultures and genres express spatial information visually.

Even before that, I reflected on a talk I gave years ago about virtual reality and various types of immersion. I expressed an idiosyncratic and limited version of rejecting the immersive fallacy, the notion that the more video games are able to reproduce the sensory impressions that a real experience would provide, the more players will enjoy and be taken in by the game world.

To this let me add something I've not discussed, mostly because I don't have as much personal experience with it yet: physically-based rendering (PBR for short), which is a set of techniques on the graphics side of video games, film, and other visual media, meant to put sensory reproduction to practice by trying to approach a close physical simulation of the interaction of light with various real materials. This encourages moving towards computationally expensive techniques such as ray tracing to generate the visuals, practically expensive techniques such as photogrammetry to extract light interaction information from real materials, and then storing this information and making use of that when rendering, which tends to increase the memory and processing requirements further.

If you happen to be thinking about the next game or digital movie you're going to create, I think it's worth it to take a step back and ask: what is it that rendering is supposed to do for a video game or a movie? Its main task to provide information to a person, a player or viewer, about a virtual world; or possibly to pass along a mood. Perspective 3D rendering of a view of an environment and objects within it is one way of doing this, but it is not necessarily the best. Even in modern games and realistic visualizations, other forms are used. For example, map applications usually provide orthographic projections rather than perspective projections, because that is a better way of providing regional information in graphical form. Even in realistic modern 3d video games, a lot of information is still provided through UI and menus. What are those? Stylistic, non-perspective descriptions of information, laid over the 3D view, instead of being placed within it. And yet, after a bit of a learning curve, they are a very effective way to present information, better than if a perspective rendered substitute were required.

Making use of these non-perspective, non-simulative representations was a recurring problem for me when working on virtual reality applications. Since control of the visual field is given over to the computer, it needs to provide something close to what your own visual field provides, in terms of reactivity and the organization of information, so entirely stylized overlays, like subtitles placed at a constant place in the visual image, which work pretty well when watching a movie on a stationary display, can be jarring and even lead to simulation sickness. This results in solutions such as placing them inside of artificial objects in the environment, which leads to challenges such as making sure that they are not obscured by other elements in the environment.

But it is possible to think about this differently. Instead of starting with the idea of having a three dimensional world built towards being presented through a perspective projection (with at most a parallel orthographic projection for an automap), it should be possible to create a more sophisticated logical framework connecting them, and decide how to turn this into visual data later.

Lateral thinking about how to render still exists in contemporary game design. Take shadows, for example. Shadows require a bit of work to render, over just ignoring them. But as Vision Science explains, shadows are an important tool for the visual system to extract at least two pieces of information from the environment: their detailed spatial structure, and how far away they are from the viewer. However, these two uses of shadows don't have to be rendered the same way, and sometimes it's best not to. If you've played any modern third-person 3D platformer, you would have noticed that yes, there's naturalistic shadowcasting that is consistent with the lighting conditions, but there's also usually a small shadow right under your character, regardless of the direction of light. That's because when light is coming from above, the vertical distance between the object and the shadow gives you very good information as to where the object is in 3D space, and it also tells you where the object will fall if it were dropped. That's essential when you want your in-game avatar to perform complicated jumps, a basic problem of the platformer genre.

Instead of thinking of shadows as a physical effect that must be simulated as realistically as possible to create an immersive scene, you can think of shadows as a way of conveying information to the player. That is a creative choice. An ideal version of what I would call "perception-oriented graphics" would let you make decisions like this for the overall display of the system - whether you want to display a perspective projection of the space or an orthographic overview - as well as for individual objects or phenomena - like the question of whether to use light-reactive or drop-down shadows. Maybe some day I will have the opportunity to work on something like this.

Deep Space 9 vignettes

I've decided to collect a few thoughts I've had about Star Trek: Deep Space 9. There are some spoilers for particular episodes and the general thrust of the show.

Not just a Babylon 5 knock-off

When I was a teenager in Israel in the 1990s, there was a sense of rivalry between fans of Star Trek: Deep Space 9 and Babylon 5. Sure, I knew many of people who were fans of both, but since they were both in the same niche of a space station-centric longform science fiction drama, and since there were a lot of specific parallels, and even a story about how one stole the premise from the other, there was definitely tension.

For broadcast availability reasons, it was B5 which I ended up watching through, while I only watched DS9 sporadically. I was left with a sense that the former was superior to the latter. I was taken in by the attempt to make a mature piece of work: a TV novel, and avoiding kids and cute robots. But recently I have achieved a more nuanced perspective.

After an almost complete rewatch of B5 last year, I did a complete watch-through of DS9 earlier this year. And I feel I should have given the latter a fair shake earlier. No, it isn't a TV novel. There is a TV novel hiding there, maybe about 5 seasons of it, but it is also attempting to be a particle of the week Star Trek series, and manages to pull that off, as well. The characters are very well-written, engaging, and grow with the plot and the passage of time.

One thing that stands out is how the kids affect the story. The parental relationships add a depth and sense of stakes to the world that, with all of the dramatic development in B5, just wasn't there. Seeing the world through the eyes of Jake and Nog as they grow up, seeing the emotional toll that long parting has had on the O'Briens and the Siskos when those occurred, the fear for Jake and Molly's welfare when they were out of sight, were irreplaceable. Similar situations in B5 involved adults being concerned for their adult relatives, relationships which we were only introduced to remotely, so the impact was lessened. No children grew up on B5, either, instead just coming in for a plot point and leaving.

So I don't think JMS got more maturity out of the bargain. Cute robots were absent in both, though, and neither felt neglected as a result.

Vic Fontaine as an artificial gamemaster

As part of the transition from a paramilitary ship to an essentially civilian space station, the public holodeck became a set of private holosuites, run by everyone's favorite Ferengi bar-owner, Quark.

I didn't cover the holodeck or the holosuite in my talk about virtual reality in fiction and reality, because at that point it had been a while since I've watched The Next Generation or DS9, and frankly, it's a very weird entity, that benefits from thinking more in terms of role-playing games, and specifically, live action role-play. In the holo experience, a program creates artificial beings and an environment, which react to the real visitors naturally, and/or according to the program's dictates. The visitors are expected to dress for the occasion, and also play their parts - so this sits somewhere between augmented reality and larping. There is some interlock that prevents real people from being injured, which is occasionally deactivated to add more tension, but otherwise, the experience feels real. There is a finite room which can be seen when the program is shut down, but the visitors are able to feel like they are moving freely in an open-ended environment, when necesary.

In story terms, the holo experience seems to be a place for unrealistic crossovers to happen with in-fiction logic: TNG's Data can match wits with Moriarty, the crew can play Robin Hood's Merry Men, and, in DS9, a somewhat sanitized experience of 1960s Vegas is recreated, hosted by a lounge singer modeled after Frank Sinatra - Vic Fontaine. In time, the various crew members become his friends - he is very helpful and insightful, especially when it comes to their love lives.

One of these crew members is Quark's nephew, Nog. We have watched him succeed in achieving his dream of joining Starfleet, taking part in battle, and losing his leg. A few episodes later, in "It's Only a Paper Moon", he is back to the station, with leave to acclimate to a prosthetic and become rehabilitated. He visits the Fontaine program to cheer up, and then decides he wants to recover there.

This pushes the boundaries of the program as we have seen it so far: instead of watching a show and getting some advice from Vic, Nog is taken backstage. And an entirely novel part of the premise is revealed: the management of the casino and lounge, and particularly, its finances, which according to Vic, confound him. After an altercation with Jake, Vic tells Nog that he has to improve his attitude or he will not be allowed back to the lounge - until this episode - the only attraction of this program. Nog is finally convinced to have a look at the finances, and, in typical Ferengi fashion (the less said about a race of big-nosed aliens with a natural talent for money, the better), has an easy time sorting them out, and starts investing time in improving the profitability of the place, and even expanding it.

The fascinating thing about all this is how much this holo experience moves towards being an ideal role-playing game. Nog is able to be swept up by a part of the imaginary or simulated world that was only created to engage him, that wasn't planned in advance. This is something a good gamemaster would do; indeed, the framework, which does not require Nog to roll any dice or understand new behind-the-scene rules, much less reprogram anything, makes this closer to an ideal Blackmoor phenomenon, engaged with through pure larping. In our world, it would be very difficult to reproduce the freedom of this experience as a larp, because it would beg the question of how the new systems are designed and made to seem and sound and feel real, and where the new players required, or their clothes, would come from, and how they would be trained.

Odo as a literal referee

Earlier in the season, in "Take Me Out to the Holo-Suite", Captain Sisko manages to convince members of his crew to join him in playing Baseball against a team of Vulcans lead by an old nemesis of his. Since an actual Baseball court would be difficult to recreate on the station, it is reproduced in one of Quark's holosuites.

As we've discussed, holosuite programs are more than capable of creating people with complicated personalities, and a programmed umpire (Baseball referee) would have a perfect view and understanding of the situation in the field, but Sisko insists on having a person do it, specifically Odo. He wants a real person. He doesn't explain exactly why - it seems he mostly rejects a holosuite umpire on principle as what Solok, his nemesis, would do - but later something happens that is apparently not exactly according to the rules as, possibly, a holosuite umpire would do it: one of the Vulcans runs into the dugout without having touched base. Since he can't single out the Vulcan who had run in, he goes to them, one by one, touching them with the ball. It turns out that the one who had run in is the last in the seating order, and as soon as Nog gets to the player right before him, he runs and attempts to make home plate, but Jake is able to catch Nog's ball and tag him, anyway.

Had this been run by a holosuite umpire, maybe something like the MLB rules would have been used, and the runner would have been called out as soon as he entered the dugout. But instead, it was Odo who indicated that something was wrong, in his typical gruff manner, adding to the suspense and thrill of this small victory.

Sisko cheers and comments that it is this unpredictability of Baseball which he loves about the game. And having a living referee, one who can make mistakes, or idiosyncratic judgments, is part of what allows for this exciting unpredictability. Even in this, a competitive game with overall much stricter rules than any role-playing game.

Why was O'Brien self-conscious about his models?

As the series progresses, Bashir and O'Brien bond over holosuite experiences reproducing various desperate historical combat situations. The last one they repeatedly attempt is the Battle of the Alamo. In addition to playing at this scenario on the holosuite, O'Brien starts investing in recreating the Alamo Mission and the order of battle in miniature. And it is fascinating that he is repeatedly teased about it, with others constantly calling them toys, which he rebukes by stating that "it's not a toy - it's a model!" It's interesting because by the time DS9 was produced, actual model armies and miniature warfare were pretty well established as a grown-up hobby, particularly historicals.

This does remind me of a story suggesting why the author Robert Louis Stevenson gave up on miniature wargaming, which he enjoyed playing with his step son in Switzerland: apparently a friend mocked him so badly for playing a children's game that he never did much more with it, much less publish the rules he used, a loss to the history of wargaming.

Virtual reality and immersion - a reflection

In 2017 I was planning my annual trip to Israel, to see my family and friends there. I aimed for the Jewish fall holidays, because they corresponded to one local holiday off that I could take, and meant more of the people I would like to meet would be free. I realized that I could schedule things so that I can attend ICon, the big Israeli science fiction, fantasy, and role-playing game festival. I announced this on Facebook, and a friend asked "well, what will you be presenting?"

I hadn't considered presenting anything, but I realized that I definitely could. I had left academia about a year before, but still had some presentation chops, and I was working in virtual reality, which was really hot at the time, and often featured in genre fiction. So I decided to propose a talk comparing virtual reality in fiction and in, well, reality.

For recent advances in the field, while I needed to do some reading, I mostly could depend on the work I was doing for a living. For a more historical view, and references to fiction beyond my own favorites like William Gibson's Neuromancer or Neal Stephenson's Snow Crash, I read through Howard Rheingold's Virtual Reality, from 1991. This, in turn, lead me to The Machine Stops by E. M. Forster, Pygmalion's Spectacles, by Stanley G. Weinbaum, and other early works. To the written fiction I added a discussion of The Matrix and the Holodeck in Star Trek: The Next Generation. So there, I had a series of comparisons, as promised. But would that make for a good enough talk?

People with a better liberal arts education than mine probably already know that any type of non-fiction work is carried best by some kind of thesis statement. So I needed one. And as I read the fiction and delved into the history of the field, I came to realize something important: immersion didn't really require all of that technology, what was presaged in Cyberpunk literature, or what I was using at work. I could think back to spending the '90s on IRC, arguing with people across the world, feeling that I was together with them, even though it was just text going back and forth. I was reminded of BBSs, and MUDs, and text-parser games that were able to immerse people in worlds, together with others or alone, without high quality graphics or audio, not to mention VR headsets. I compared it to immersion when reading, or in the theater, both as an actor and as spectator.

So here was my thesis statement. I broke down immersion into three types, which I will credit properly if they are not my own invention, once I find my notes:

1. Authenticity - do you feel like you are in the virtual world?

1. Connection - do you feel involved in the goings on in the computer?

1. Belonging - do you feel part of a community?

and stated that the first is not necessary for the other two. In fact, people have felt a connection to virtual worlds and each other through computers for almost as long as there were interactive computers. Furthermore, the kind of control the user has over their immersion when not encased in a headset can be very important for a variety of use cases.

I gave this talk in Israel, in Hebrew. A few months later, the local IGDA chapter needed someone to substitute for a last-minute cancellation, and since I had already redone some of it in English for a small VR interest group, I expanded it and refocused on my thesis, concluding with a long example concerning Chile's Project Cybersyn, as documented in Eden Medina's Cybernetic Revolutionaries. That talk was recorded, and you may watch it for a full treatment.

After that I didn't pick up the material for a while. Instead, a few months further into 2018, I went back to tabletop roleplaying, first as a player at conventions and with a local group, and then taking over said group as a dungeon master, with my own campaign. And as we have been forced to move online, I find myself returning to my thoughts from creating those presentations. It might be worth reflecting a bit more, especially if I can find more documentation of my research leading up to them. Immersion is one possible goal for role-playing, and I think this connects well with my thoughts on radio. The need to be able to control immersion is also relevant, especially when considering the importance of safety tools in role-playing games.