Projects

Background

This is an ongoing project that has been in the works since summer 2018. It started as part of my internship under the guidance of Prof. Jayesh Pillai and in collaboration with my colleague and filmmaker, Amal Dev . We were initially exploring the idea of dynamically altering the visuals of a VR film in areas that are beyond the field of view of the viewer. This approach led to the current framework, Cinévoqué, whose name is a portmanteau of the words Cinema and Evoke, which espouses the ability of the framework to evoke a narrative that corresponds to the viewer's gaze behavior over the course of the movie.

Introduction

Virtual Reality as a medium of storytelling is relatively less developed than storytelling in traditional films. The viewer is empowered with the ability to change the framing in VR, and they may not follow all the points of interest intended by the storyteller. As a result, the filmmaking and storytelling practices of traditional films do not directly translate. Researchers and filmmakers have studied how people watch VR films and have suggested guidelines to nudge the viewer to follow along with the story. However, the viewers are still in control, and they can consciously choose to rebel against such nudges. Accounting for this, and taking advantage of the affordances of VR, Cinévoqué alters the narrative shown to the viewers based on the events of the movie they have followed or missed. Furthermore, it also estimates their interest in particular events by measuring the time they spend gazing at it and shows them an appropriate storyline. Consequently, the experience doesn't have to be interrupted for the viewer to make conscious choices about the storyline branching, unlike existing interactive VR films.
This project is being built as a plugin for Unity 3D that could be used by filmmakers to create a responsive live-action immersive film. We have chosen to focus on live-action film over real-time 3d movies as passively responsive narratives have been explored in the context of games and interactive experiences previously. Additionally, the technical implementation is more novel in the case of live-action films as the content (videos) cannot be changed dynamically like real-time rendered scenes. Using a game engine such as unity to power live-action Cinemative VR brings forth extra features that weren't implementable before, for example, we could add a virtual body that orients to the viewer's physical body by using rotational data from 6DOF controllers, which allow for the viewer to be a more integrated character in the narrative than before.

To learn more about the design and implementation of the framework please refer to my publications. We had the opportunity to present our work at reputed international conferences such as VRST, VRCAI and INTERACT. We have also been invited speakers at national and international events such as SIGCHI Asian Symposium, UNITE India and IndiaHCI.

Raytracing

This is my first graphics project. Following the Raytracing in one weekend ebook series, I implemented a Raytracer from scratch in C++. The project served as an useful refresher for the maths used in graphics and contexualized the concepts. It also helped me learn some advanced and new concepts in C++. The first book focused on implementing vector math functions, rays, ray-sphere interactions, shading, aliasing, positionable camera and lens blur. The following images show scenes that implement all the above mentioned features.

The second book covered more advanced topics such as motion blur, BVH, procedural textures, image texture mapping, lights and volumes. Beyond the contents of the book I had implemented multi-threading and non-uniform volumes. The code for the project can be found here.

Shaders

After the raytracing projects I have implemented a couple of shaders in Shadertoy. The first was an interactive mandelbrot set, which served as a refresher for complex math and its uses in graphics. The second shader is an interactive mandelbulb, which helped me understand raymarching and SDFs. Additionally, I have implemented custom shaders in Unity as part of my work in IMXD lab. Which would be updated here after their completion.

Next Steps

Recently I've come across multiple graphics courses that were published publicly, and have been going through them while trying to complete the assignments. Currently I'm following Prof Keenan Crane's Computer Graphics course (CMU 15-462/662) and Prof Cem Yuksel's Interactive Computer Graphics (CS 5610/6610). The completed assignments of the Interactive CG course are being updated here.

ScholAR is a research project that is exploring how AR-based educational content can be democratized and deployed in schools. The project is being undertaken as part of Pratiti Sarkar's Ph.D. and is funded by the Tata Center for Technology and Design. I have been developing the AR applications used in the experiments while also assisting in conducting them.

Scholar for Classrooms

This part of the project focused on scaffolding classroom sessions with AR based content moderated by the teacher. This is proposed to provide a more enganging and interactive learning experience compared to solutions like smartboards. We have focused on building and testing AR content for maths, specifically for geometry over the last three years. Our expeirments explored learning efficacy, collaboration and interactions in rural schools where classes were held using our applications. Our apps were also built based on pedagogical models to help the students better grasp the concepts. I was reponsible for building the apps and further helped with content creation and experiments.

Scholar for Remote Learning

Due to the COVID-19 pandemic we starting working on a remote learning solution that could bridge the work we have done for physical classrooms. I created a prototype that focused on creating a virtual classroom where the teacher is able to control a shared AR artefact and teach concepts to students who are spatially present in the virtual space. Visual cues & spatial audio were added to give a better sense of other users’ relative position. Depending on the active artifact, the users could place a marker or draw on top, and these interactions are reflected for everyone in the session. We had conducted preliminary tests with students from our department to better understand the opportunities and challenges that arise as a consequence.

Apart from education this project also brought out challenges that extend beyond the educational use case of Scholar, for example, the best avatar representation for mobile AR. With just the positional information of the hand held device it may not be possible to create avatars with the similar accuracy as those in VR or HMD based devices. I have taken up the lead in this work, where we study avatars that vary in both visual and behavioural fidelity. The following image shows the avatar space in our study. The following video demonstrates both direct and procedural networked avatars in our current prototype

Background

This project was started during Prof Jayesh Pillai's VR course; after I joined IDC School of Design as an RA. While assisting the M.Des student groups with project development, I got the opportunity to join one of them and work on this project. My teammates Maulashree Shanbhag and Rishi Vanukuru and I wanted to explore the possibility of creating a visual intuition for music theory by immersing users into an experience where the virtual environment changes according to the music played by the user.

implementation

We used an HTC VIVE with leap motion, and an extra vive tracker attached to the keyboard for aligning the virtual and physical keyboard, the midi data from the same was sent to our application to provide feedback.


The experience consists of two levels.Initially, to onboard the user, an underwater environment is shown with the virtual keyboard in front of them; when they press a key, a school of fish is spawned, which is driven by the BOIDS algorithm. As the user continues to play the instrument, the school changes its location accordingly to indicate that they are affected by the input. Once the user is comfortable with the experience, we move them to the next level, where they surface near an archipelago. Each island is mapped to an octave, and the topology of the island changes when a key within its octave is pressed.

Future Work

The project is a work in progress. We would like to add more features and improve the quality of the experience. For the technical side, I am planning to implement this with recent features of unity such as, ECS, Job System and the Burst Compiler for possible optimizations. We also intended to do a more rigorous literature review for finding better mappings between the keyboard inputs and terrain modifications.

This is a compilation of three different projects that are multiplayer experiences. The first application is an AR game that I implemented as part of a student's (Shraddha Dhodi) project, which was about exploring new co-op game mechanics in a virtually co-located AR space. The second application is an android game that is an implementation of another student's project (Aishwary Khobragade), it sought to identify elements that foster cooperation in a game and how they differ from the ones that create conflict. Finally, the last application was built as part of a hackathon; it is an experience that allows musical collaboration through VR using digital twins of the instrument.

Door2Door

The motivation of this project was to leverage AR to create a multiplayer collaborative game for users who are not physically co-located. Door2Door was the result of the project; it's a puzzle-based AR game where two players are able to see the avatar of the other through a portal. Clues to solve the puzzles are shown to the players on the other side portal, but the clues on their side wouldn't be visible to them1. As a result, they have to communicate with each other through text or voice and solve the puzzles together.

Bait

This an android game that was built to create a cooperative gaming experience. Each player assumes control over a school of fish, and they are tasked with collecting small crystals spread around the map. These collectibles must be deposited at a single location on the map as they replenish the health of the central deity; in this case, it is a larger crystal. Failing to deposit the collectibles frequently would end the game. The map is also riddled with predators that attack the fishes if they are close. They could be evaded by getting out of the line of sight, or they could be repelled when two or more players bring together their schools of fishes. This mechanic serves as the cooperative aspect of the game and presents the players with the options to either help the deity or the player(s) under attack.

Midichlorians

This is a hack I worked on along with Rishi Vanukuru during Music Hack Day, Mumbai. We wanted to use the hackathon as a means to prototype new features for our previous project on VR and music. The final application worked as follows: on one end, a person with a midi controller played the notes and was able to see a digital twin of the same reacting on a screen; another person in a different physical location was able to see and interact with the same digital twin in a VR environment. The goal of this was to create a collaborative VR environment for jam sessions or for teaching music remotely. To espouse the teaching side of the hack, we added a feature for the person using the physical midi controller, which highlighted keys they could play next so that it sounds good with the keys they played prior. Our application won the best music hack in the end.

Gro is a concept that was formulated during Prof Anirudha Joshi's Monsoon HCI Course at IIT Bombay. It is an intensive two-week course offered to professionals as part of the institute's Continuing Education Program. I was one of the student volunteers, so my responsibilities included participant recruitments and dealing with some aspects of logistics. Otherwise, I had the same opportunities as the participants of the course. I was assigned to a group consisting of a Product Manager, Tech Lead, and two designers; over the two weeks, we were tasked with proposing a solution that would inculcate financial literacy to kids.

During the first week, we were taught about the UX process and models. We learned about identifying problems through contextual interviews and consolidating data from these interviews through affinity mapping. My team interviewed a variety of people who would possibly be involved with the financial aspects of a child, so our interviewees included parents, teachers, kids, and bank officials. Post interviews, our affinity mapping brought out some interesting insights; for example, we noticed that parents are interested in teaching financial literacy to their children, but they rarely allow their kids to handle money and make independent decisions about their spendings.

With the data aggregated from affinity, we moved on to prototyping and evaluation in the second week. We ideated around the concerns and constraints that were brought out and arrived at the concept of a digital school ID card. To prototype and better convey our idea, I created an AR application that displays our UI over an actual ID card. Heuristic evaluation was conducted to address issues with the design we had. At the end of the course, we presented our final concept, along with personas and scenarios. The course gave me a deeper understanding of the UX process, and working alongside experienced professionals was a very valuable exposure.

AR/VR is gaining traction in the commercial sector, and content creators in the medium are left with limited options in monetizing their experiences. I made AdsXR as a POC that could serve as a possible solution to this problem. The project was undertaken as part of Accenture's Hack Your Reality hackathon, where one of the themes was immersive marketing. This idea was one of the hundred or so shortlisted submissions from over two thousand.

AdsXR aims to serve as a platform for selling ads for virtual real estate created in immersive experiences. It is platform agnostic and can be integrated with both mobile AR/VR and within high-end VR or MR systems. The project address two issues with the current state of immersive marketing. The first is to open up a new revenue stream for developers and content creators of immersive content, which would allow them to enjoy the same luxury the smartphone market is leveraging. The second issue is that any company looking to use Ar/Vr for advertising has to invest in XR development of some sort and allocate extra resources to it. This platform could help solve this issue and exist as a one-stop solution to buy ads in Ar/Vr applications.

The locations where these ads are shown is determined by the content creators; they could mark the 3D models of posters, billboards, newspapers, etc. as an advertisement, an AdsXR would dynamically apply a texture of an Ad during runtime to these models. The asset also tracks the user gaze to measure the amount of time the user spends looking at the location of these Ads and uses this data to adjust to the cost of placing an advertisement at the spot. The asset is made for Unity, and the site which allows for buying the Ads was also made in unity and exported as a WebGL application. The data from unity was stored in a database, and the communication was done through PHP and SQL. The technical architecture of the asset is as follows:

Smart India Hackathon 2017, the first edition of the competition conducted by the Indian government, was the largest hackathon in the world when it happened. The issues faced by each ministry of the government that could be solved with tech were put forth as problem statements for this hackathon. Me and five other members of Next Tech Lab formed a team and chose the problem statement, "Smart solution for providing commentary of exhibits in Museum in India."

We initially considered the usual approach of using sensors to trigger commentary for each exhibit, but after analyzing the cost of implementation and maintenance, and issues with scalability, we decided against it. Instead, we chose to go with a QR code for each exhibit, which, when scanned, would display information about the same and start the commentary. Our app also maintained a floor plan of the museum with the position of each exhibit. They were highlighted in different colors to show if the visitor has seen them. This system also helped us estimate the visitor's location within the museum and opened up the possibility of providing better navigational instructions.

All six of us were part of the AR/VR research group within Next Tech, so we understood the value these mediums would bring to a museum experience. So our other use cases for the hack included:

• superimposing a reconstructed digital twin on a damaged artifacts.
• Recreating the working mechanisms of old machinery and events in AR and VR

I was responsible for implementing the code architecture that would combine the modules my teammates, and I have worked on, unsurprisingly I had to deal with many bugs and pull an all-nighter to get everything working. The above-mentioned use cases, along with relatively our low-cost solution, gave us an edge over the other teams. Ultimately, it led to our first hackathon win.

The final demo:

Our win even got us some attention from the press!

if you're interested in learning more about our journey, do go though this article written by my teammate: Tackling inconveniences and winning the world’s largest hackathon: A detailed account

This was my bachelor's thesis project, which I worked on along with my classmate Balaji Ganesh. It was undertaken in collaboration with SRM Medical Colege. The broader goal of this project was to create a platform for mixed reality that holds educational content that can be consumed in a multi-user environment. Furthermore, the platform had to support addition of new content post-deployment.

We were able to implement dynamic content addition and retrieval in the networked environment, with extra transform interactions added as well. We used Dreamworld's Dreamglass AR HMD for the hardware and developed the application in Unity. The networked experience was created through Unity's in-built networking solution Unet (it is currently deprecated). We used Amazon AWS to store and retrieve the content that would be shown on the platform. For this project, we had 3d models of the heart and lungs obtained from MRI scans, which were optimized further for use in AR. I also created models of cellular structures such as alveoli to support a use case of explaining a concept at different levels of scales.

This project was built based on a proof of concept mixed application I developed during my internship in the year before for the hololens, the demo of the same is shown below.

This is an ongoing project that has been in the works since summer 2018. It started as part of my internship under the guidance of Prof. Jayesh Pillai. We were initially exploring the idea of dynamically altering the visuals of a VR film in areas that are beyond the field of view of the viewer. This approach led to the current framework, Cinévoqué, whose name is a portmanteau of the words Cinema and Evoke, which espouses the ability of the framework to evoke a narrative that corresponds to the viewer's gaze behavior over the course of the movie.