[thelist] Managing css, images, js versioning

[Ken Snyder]

Bill Moseley wrote:
> On Tue, Dec 04, 2007 at 09:24:12PM -0700, Ken Snyder wrote:
>   
>> - Compile and minify js/css on demand.  On each page load, the server
>> checks file version strings, timestamps or file md5's to see if that
>> combination has ever been compiled into a file.
>>     
>
> So the application source page might include:
>
>     include_js( 'foo' );
>
>
> Where load_js() might run out and find the current version needed, and
> minify/compress if it doesn't already exist and finally output:
>
>
> <script src="http://host.com/js/1223/foo.js" type="text/javascript"></script>
>
> Where 1223 is the version?
>   
What we did is more like this (PHP):
$tpl = new Templater();
$tpl->addJs('foo.js');
$tpl->addJs('/lib/mylib.js');
$tpl->addAllJsInDir('/global/');
...

<script src="<?php echo $tpl->getCachedJsPath(); ?>" type="text/javascript"></script>

where $tpl->getCachedJsPath() returns 
http://domain.com/cache/{md5}-{timestamp}.js

If you need to get different versions of foo.js for different apps, then 
the Templater::addJs() method can look in a config file or database to 
determine which foo.js to put into the compilation (e.g. /123/foo.js) 
for that particular app.  If all js and css are compiled on the fly, 
they don't even need to be publicly accessible by the web server.  They 
don't even need to be real files--you could export the files from svn on 
the fly. 

The part that you need to formulate is a quick algorithm for composing 
caching hashes that ensures that the total content gets a unique 
filename.  On our test server (CentOS Apache PHP5), we found that we 
could check file modification time and file md5 on 30 or so files in a 
few milliseconds so the performance hit after compilation was very small.

This approach is extremely developer friendly, because it is easier to 
manage lots of small css and js files within an intuitive directory 
structure.

> grabs a single minified file, then in development it's handy to have
> foo.js include other js by inserting <script> elements rather than
> having a it build on the fly every request -- mostly because the
> developer sometimes might be serving the files directly w/o a server
> that can build on the fly.
>   
I can see how it would be hard to manage a lot of compiled files if it 
is a requirement that developers be able to access compiled files 
without lazy compiling.  Are these developers designers by chance?  I'm 
not sure what your dev situation is here--it sounds pretty unique.

>
>   
>> You could also implement automation for updating of file/db configs
>> using subversion hooks.
>>
>> As far as shared files, we did a separate media repository.  With
>> apps, you likely have one repository for each app.  If you share
>> resources such as css, js, and images you may have success with one
>> media repository with multiple branches.  That way, each branch of
>> media can be merged or separated where needed.
>>     
>
> The issue with shared files between different applications is that
> it's very nice to have the item managed in one place (find out the
> designer stole that icon and it needs to be replace then just change
> it once and it's done for all sites.
>
> On the other hand, if something gets changes globally it may have
> unwanted side-effects on another site.  I see this happening with
> javascript.  Imagine hosting YUI and the same YUI code is referenced
> by a number of applications.  What if a new YUI version is released
> that you need for a new feature in one application?  Then you have to
> test all applications.
>
> I think it some ways it's a safer approach to not share the files
> across applications.
>   
It seems like subversion branches and merging are created to address 
these very situations.  We've had success using it on 5 or 6 branches at 
once.  It seems like it would save even more time if you needed many 
more branches.

>> Subversion really has the flexibility to complement your development
>> process, so start by examining your development and deployment
>> procedures.
>>     
>
> Anything like hard links in svn?  Where you have a thin copy of a tree
> of files and when they change on the original location they change in
> the copy.  But if the copy is modified then it's a private and
> different version.
>
>   
Subversion has no hard links AFAIK.  With subversion branches however, 
you can do exactly the same thing--maybe not as automatic as you are 
thinking.  You may already be familiar with SVN merging, but at least 
for the benefit of other readers, here are two examples of how I've 
managed multiple branches using Tortoise SVN:

1) Assume you add new js functionality to one or more /trunk/ files and 
commit to /trunk/ as r1001.  Your branches--say /app1/ and 
/app2/--remain untouched.  Then say you want those changes to apply to 
/app2/: you click on the root directory of /app2/ and choose merge 
/trunk/ range r1000 to r1001.  Assuming there are no conflicts, you can 
commit the change to /app2/.

2) Assume you add new js functionality to one or more /app2/ files and 
commit to /app2/ as r1002.  /trunk/ and the /app1/ branch remain 
untouched.  Then say you want those changes to apply to /app1/: you 
click on the root directory of /app1/ and choose merge /app2/ range 
r1001 to r1002.  Assuming there are no conflicts, you can commit the 
change to /app1/.

SVN branches contain truly separate copies, but with a few clicks you 
can copy changes to trunk or any branch.  Even if branches get quite 
diverged, it is not too cumbersome for developers to resolve conflicts.  
You can even diverge on purpose.  For example, you might fill a 
repository trunk with lots of reusable code.  Then you can create a 
branch each for several completely different applications and fill each 
branch with it's own non-common code.  When you need to change some of 
the common code, you only have to change it in /trunk/ and merge to each 
branch.

SVN allows you to take change sets across one or more files and resolve 
conflicts quickly.  Doing the same thing by copying and pasting file 
contents is quite cumbersome, especially when the change sets span 
several files or the applications are very different.