Discussion about 3D tool development and mainstream development

In the course of a discussion on a SIGGRAPH alias about something else (Plone), someone asked the question below, and my response got kind of long. While I'm sure most people's eyes will simply glaze over (a perfectly justifiable response), it reminded me of many a Sunday morning conversation on a ridgetop above Marin, so I'm forwarding here.

------------------------

Gustavo's question:

This is Gustavo, I am working on the migration to Plone 3. Quick question, reading your post I found this line quite intriguing:

"develop web apps in Django all day at this point -- computer animation pipeline programming is converging with mainstream development frighteningly quickly"

I am quite intrigued about that, what sort of applications are you talking about? Do you have any pointers or links to pages that talk about it? I am really interested on it

Thanks,

Gustavo

---------------------
My answer:

Hi Gustavo,

Sure, here's what I was referring to. None of it is exactly public, so I can't point you to web pages or anything, but it's not exactly secret either.

The previous generation of CG pipeline tools was largely implemented inside of the DCC (Digital Content Creation) tools (Maya, Max, XSi, Modo, etc.). Those tools all have pretty robust scripting environments nowadays, so you can implement tools in some high-level scripting language with some UI toolkit available to you. And I mean, really pretty robust environments -- we use Django as a object-relational mapper for our code that runs in Python inside of Maya.

That's all good, but when you think about it, that's fundamentally implementing tools in the client/server approach that dominated in the 1980s and 1990s. Those tools (including what Polygon was using up until this year, and the tools I implemented at EA before that):

• connect to the data source via a LAN connection
  
• use a storage-level protocol (usually SQL supplemented by file system access via CIFS/NFS)
  
• require that the code be propagated out to the client machine

For sure there are huge advantages to the DCC integration, and it's still what I recommend for artist-facing tools. However, it has some problems that are becoming especially apparent as the computer animation production world becomes more outsourced and more collaboration-oriented:

• the storage-level protocols are inappropriate to extend outside the company (you don't expose your MySQL or Samba/NFS server to the public net these days)
  
• for a lot of the legitimate stakeholders in your project (notably project managers) the Maya interface is not a preferred means of use
  
• updates to the code require that the updated code be pushed to the clients
  
• those tools are not well-connected to the rest of the computing environment -- notably Excel. Getting data in or out required custom development in every case
  
• the tools are very dependent on the client OS/environment

So, increasingly, we're putting a lot (not all) of our tools into a web-facing mode. While the files themselves are still saved via the standard Maya interface, we're moving our status-tracking apps (that keep track of the status of every model, animation, and rendered shot) to be web-facing. This means:

• extending the app to be accessible from other companies is straightforward -- the security model for web-facing applications is very well-understood
  
• you get to use modern development tools. There's nothing running inside any DCC that compares to the productivity of developing with Django/Ruby on Rails/ASP.NET inside Kodomo/Eclipse/Visual Studio
  
• users have a lot more ad hoc means of data integration (when we haven't had time to code a custom query report for you, just copy-and-paste out of the HTML table into Excel)
  
• apart from that, it's easy to integrate with community software (wiki/forum/etc) to pick up the benefits thereof
  
• printing, etc. is free (I've never seen an in-Maya application that supported printing)
  
• we're out of the software installation business, notably including not caring whether you run Windows or Mac OS X or Linux (different studios have different answers), which version of those OSes you're running, and largely not caring even which version of Maya you run

In a CG production environment, web-facing apps still have some drawbacks, notably:

• you cannot send the actual 3D data over http -- it's just too heavy (a few hundred megabytes for a set is not unusual). So the metadata travels over http but the actual data can't
  
• UI development on the web is harder than locally (IMHO)
  
• in return for getting out of the software installation business, we have to deal with the browser wars. If it wasn't for toolkits like jQuery or YUI, it almost wouldn't be viable to do this

What made me comment on the convergence of application development is that most of the above comments (maybe absent the data volume) apply to almost any kind of custom tool development. I'm very firmly convinced that except for very data-heavy applications (video, 3D scenes) web-facing application development is a huge win.

Regarding the data-heavy stuff, we did something I think is pretty interesting this year. When you have multiple companies participating in CG production, you have to share the model assets, and as I said, those are fundamentally heavy, so you won't be uploading them over http. The traditional solution to this is production managers doing a lot of FTP or running around carrying hard drives between locations; the high-end solutions (which we couldn't afford) are the companies that sell you high-bandwidth shared workspaces that you can use online.

I implemented an interesting solution this year for version control of these massive data assets -- it's like a hybrid between CVS and git. There's a shared server holding the assets (accessible via SFTP) and a web-facing status-tracking application. It's like CVS in the sense that checkout is a global operation -- to be able to edit an asset, you need to check it out on the global web-facing database, either via the web interface or via a web services call from the in-DCC tool. However, it's like git in that the actual asset is assumed to take considerable time to propagate from the committing party to all parties. The artist uses an in-Maya tool to actually check in; that tool both copies the asset to the local file server, and updates the checked-out status on the global web-facing database. Then, it kicks off an asynchronous transfer of the asset data from the local file server to the shared SFTP server. Thus, while the metadata about checkin is instantly propagated via the central server, the actual asset transfer happens later. In the current implementation, downloading from the central server to the local file server is a periodic task kicked off once a day at each company. Of course, the checkout tool enforces the constraint that you can't check out an asset unless your local file server currently holds the version known to be the latest version, and no one else has that asset checked out on the central web-facing server.

Implementation-wise, by making some assumptions about the directory structure, we created a protocol that is atomic for both upload and download even when using nothing but standard FTP operations.

Again, putting this on the public internet is very straightforward: you're only dealing with http and SFTP security (we didn't even bother to use https although it wouldn't affect the architecture at all obviously). For the same reason, you can deploy this on any generic web server, it doesn't take a fussy installation or full virtual server -- just Apache and SFTP. For users, all of their file reads/writes are happening to the local file server, so they get a very high level of service. It uses a lot of disk storage -- but that's the cheapest element of the equation these days.

Sorry, that's probably way more answer than you wanted!

Take care Gustavo,

Leo