Best Practices for Speeding Up Your TBDev Tracker - Part 1 By pdq

[Hyperion (noobKID)]

hello biggy.

thanks for this, will read this for sure.
but im on vecation, and are not back before 2 weeks or so, is it possible to do the threads your making about faster tbdev tracker sticky?...

would be awsome, and not just for me. but for everyone, thanks in advance :)...

PS: if you got time, then can you take a look at my problem again?..
i do know that you did try to explain to me over at bvlist.com, but my login page does still not working, so i took the time to write for help here also... hoping that is ok.

thanks in advance if you can dude :).
See ya tomorrow or in 2 days, will go out and have some sun on my body ;)...

[Mindless]

Best Practices for Speeding Up Your TBDev Tracker - Part 1: Javascript Handling by pdq

First I'll outline some golden rules when it comes to making 'faster' web pages then I'll follow that up with how we can apply these rules on out TBDev.net trackers.
This thread will only be dealing with the handling and managing of Javascripts and is Part 1 out of a series of threads I will be posting about the 'best practices for a high performance TBDev front end'.

Part 2 will be Images
Part 3 will be CSS and CSS Sprites


These rules and techniques can be applied to all versions of tbdev source.



1. Minimize HTTP Requests

80% of the end-user response time is spent on the front-end. Most of this time is tied up in downloading all the components in the page: images, stylesheets, scripts, Flash, etc. Reducing the number of components in turn reduces the number of HTTP requests required to render the page. This is the key to faster pages.

Combined files are a way to reduce the number of HTTP requests by combining all scripts into a single script, and similarly combining all JS into a single script. Combining files is more challenging when the scripts vary from page to page, but making this part of your release process improves response times.

2. Minify Javascript

Minification is the practice of removing unnecessary characters from code to reduce its size thereby improving load times. When code is minified all comments are removed, as well as unneeded white space characters (space, newline, and tab). In the case of Javascript, this improves response time performance because the size of the downloaded file is reduced. Two popular tools for minifying Javascript code are JSMin and YUI Compressor. The YUI compressor can also minify CSS.

Obfuscation is an alternative optimization that can be applied to source code. It's more complex than minification and thus more likely to generate bugs as a result of the obfuscation step itself. In a survey of ten top U.S. web sites, minification achieved a 21% size reduction versus 25% for obfuscation. Although obfuscation has a higher size reduction, minifying Javascript is less risky.

In addition to minifying external scripts and styles, inlined <script> and <style> blocks can and should also be minified. Even if you gzip your scripts and styles, minifying them will still reduce the size by 5% or more. As the use and size of Javascript and CSS increases, so will the savings gained by minifying your code.


3. Remove Duplicate Scripts

It hurts performance to include the same Javascript file twice in one page. This isn't as unusual as you might think. A review of the ten top U.S. web sites shows that two of them contain a duplicated script. Two main factors increase the odds of a script being duplicated in a single web page: team size and number of scripts. When it does happen, duplicate scripts hurt performance by creating unnecessary HTTP requests and wasted Javascript execution.

Unnecessary HTTP requests happen in Internet Explorer, but not in Firefox. In Internet Explorer, if an external script is included twice and is not cacheable, it generates two HTTP requests during page loading. Even if the script is cacheable, extra HTTP requests occur when the user reloads the page.

In addition to generating wasteful HTTP requests, time is wasted evaluating the script multiple times. This redundant Javascript execution happens in both Firefox and Internet Explorer, regardless of whether the script is cacheable.


4. Put Scripts at the Bottom

The problem caused by scripts is that they block parallel downloads. The HTTP/1.1 specification suggests that browsers download no more than two components in parallel per hostname. If you serve your images from multiple hostnames, you can get more than two downloads to occur in parallel. While a script is downloading, however, the browser won't start any other downloads, even on different hostnames.

In some situations it's not easy to move scripts to the bottom. If, for example, the script uses document.write to insert part of the page's content, it can't be moved lower in the page. There might also be scoping issues. In many cases, there are ways to workaround these situations.


5. Make Javascript External

Many of these performance rules deal with how external components are managed. However, before these considerations arise you should ask a more basic question: Should Javascript be contained in external files, or inlined in the page itself?

Using external files in the real world generally produces faster pages because the Javascript files are cached by the browser. Javascript that are inlined in HTML documents get downloaded every time the HTML document is requested. This reduces the number of HTTP requests that are needed, but increases the size of the HTML document. On the other hand, if the Javascript are in external files cached by the browser, the size of the HTML document is reduced without increasing the number of HTTP requests.


6. No 404s

HTTP requests are expensive so making an HTTP request and getting a useless response (i.e. 404 Not Found) is totally unnecessary and will slow down the user experience without any benefit.

Some sites have helpful 404s "Did you mean X?", which is great for the user experience but also wastes server resources (like database, etc). Particularly bad is when the link to an external Javascript is wrong and the result is a 404. First, this download will block parallel downloads. Next the browser may try to parse the 404 response body as if it were Javascript code, trying to find something usable in it.


The following rules are general rules and should be applied as well:


A. Gzip Components

The time it takes to transfer an HTTP request and response across the network can be significantly reduced by decisions made by front-end engineers. It's true that the end-user's bandwidth speed, Internet service provider, proximity to peering exchange points, etc. are beyond the control of the development team. But there are other variables that affect response times. Compression reduces response times by reducing the size of the HTTP response.

Starting with HTTP/1.1, web clients indicate support for compression with the Accept-Encoding header in the HTTP request.

Accept-Encoding: gzip, deflate

If the web server sees this header in the request, it may compress the response using one of the methods listed by the client. The web server notifies the web client of this via the Content-Encoding header in the response.

Content-Encoding: gzip

Gzip is the most popular and effective compression method at this time. It was developed by the GNU project and standardized by RFC 1952. The only other compression format you're likely to see is deflate, but it's less effective and less popular.

Gzipping generally reduces the response size by about 70%. Approximately 90% of today's Internet traffic travels through browsers that claim to support gzip. If you use Apache, the module configuring gzip depends on your version: Apache 1.3 uses mod_gzip while Apache 2.x uses mod_deflate.

There are known issues with browsers and proxies that may cause a mismatch in what the browser expects and what it receives with regard to compressed content. Fortunately, these edge cases are dwindling as the use of older browsers drops off. The Apache modules help out by adding appropriate Vary response headers automatically.

Servers choose what to gzip based on file type, but are typically too limited in what they decide to compress. Most web sites gzip their HTML documents. It's also worthwhile to gzip your scripts and stylesheets, but many web sites miss this opportunity. In fact, it's worthwhile to compress any text response including XML and JSON. Image and PDF files should not be gzipped because they are already compressed. Trying to gzip them not only wastes CPU but can potentially increase file sizes.

Gzipping as many file types as possible is an easy way to reduce page weight and accelerate the user experience.


B. Add an Expires or a Cache-Control Header

There are two things in this rule:

* For static components: implement "Never expire" policy by setting far future Expires header
* For dynamic components: use an appropriate Cache-Control header to help the browser with conditional requests

Web page designs are getting richer and richer, which means more scripts, stylesheets, images, and Flash in the page. A first-time visitor to your page may have to make several HTTP requests, but by using the Expires header you make those components cacheable. This avoids unnecessary HTTP requests on subsequent page views. Expires headers are most often used with images, but they should be used on all components including scripts, stylesheets, and Flash components.

Browsers (and proxies) use a cache to reduce the number and size of HTTP requests, making web pages load faster. A web server uses the Expires header in the HTTP response to tell the client how long a component can be cached. This is a far future Expires header, telling the browser that this response won't be stale until April 15, 2010.

Expires: Thu, 15 Apr 2010 20:00:00 GMT

If your server is Apache, use the ExpiresDefault directive to set an expiration date relative to the current date. This example of the ExpiresDefault directive sets the Expires date 10 years out from the time of the request.

ExpiresDefault "access plus 10 years"

Keep in mind, if you use a far future Expires header you have to change the component's filename whenever the component changes.

Using a far future Expires header affects page views only after a user has already visited your site. It has no effect on the number of HTTP requests when a user visits your site for the first time and the browser's cache is empty. Therefore the impact of this performance improvement depends on how often users hit your pages with a primed cache. (A "primed cache" already contains all of the components in the page.) We measured this at Yahoo! and found the number of page views with a primed cache is 75-85%. By using a far future Expires header, you increase the number of components that are cached by the browser and re-used on subsequent page views without sending a single byte over the user's Internet connection.


C. Use Cookie-free Domains for Components

When the browser makes a request for a static image and sends cookies together with the request, the server doesn't have any use for those cookies. So they only create network traffic for no good reason. You should make sure static components are requested with cookie-free requests. Create a subdomain and host all your static components there.

If your domain is www.example.org, you can host your static components on static.example.org. However, if you've already set cookies on the top-level domain example.org as opposed to www.example.org, then all the requests to static.example.org will include those cookies. In this case, you can buy a whole new domain, host your static components there, and keep this domain cookie-free. Yahoo! uses yimg.com, YouTube uses ytimg.com, Amazon uses images-amazon.com and so on.

Another benefit of hosting static components on a cookie-free domain is that some proxies might refuse to cache the components that are requested with cookies. On a related note, if you wonder if you should use example.org or www.example.org for your home page, consider the cookie impact. Omitting www leaves you no choice but to write cookies to *.example.org, so for performance reasons it's best to use the www subdomain and write the cookies to that subdomain.


-----------------------------------------------------------------------------------------------

ok, now that we read all that and could see how our tracker may be breaking some of these rules,
let's do some changes on our code! :biggrin:

First off, I would suggest downloading the yslow firefox extension so you will be able to gauge performance improvements before and after.

It requires firebug firefox extension and it can be found here: http://developer.yahoo.com/yslow/

Now that we have installed firebug and yslow and restarted Firefox let's continue on. =]

-----------------------------------------------------------------------------------------------


First let's start with the easy ones, Rule A (Gzip Components) and Rule B (Add an Expires or a Cache-Control Header) and Rule C. (Use Cookie-free Domains for Components):


Rule C explains itself so let's do that and stroke it off of the list. :)

Rule A
(PHP 4 >= 4.0.4, PHP 5)

Quote
Note: While HTTP-EQUIV META tag appears to work properly with Netscape Navigator, other browsers may ignore them, and they are ignored by Web proxies, which are becoming more widespread. Use of the equivalent HTTP header, as supported by e.g. Apache server, is more reliable and is recommended wherever possible.


locate your stdhead() functions within bittorrent.php and:
find:

Code (php) Select Expand

function stdhead($title = "", $msgalert = true) {
    global $CURUSER, $TBDEV, $lang;



and add in $_NO_COMPRESS
like so for 09 source:

Code (php) Select Expand

function stdhead($title = "", $msgalert = true) {
    global $CURUSER, $TBDEV, $lang, $_NO_COMPRESS;


in 09/current source find:

Code (php) Select Expand

//header("Content-Type: text/html; charset=iso-8859-1");


change to:

Code (php) Select Expand

header('Content-Type: text/html; charset=utf-8');
header('Content-Language content="en-us"');

Above edit saw a lot of servers fail over on the content-language header so remove if you get a http 500


now find where you start your html output find:

Code (php) Select Expand

$htmlout = "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\"
\"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">



above that ^^ add:

Code (php) Select Expand

/** ZZZZZZZZZZZZZZZZZZZZZZZZZZip it! **/
if (!isset($_NO_COMPRESS))
    if (!ob_start('ob_gzhandler'))
        ob_start();


now find and remove these lines:

Code (php) Select Expand

<meta http-equiv='Content-Language' content='en-us' />
<meta http-equiv='Content-Type' content='text/html; charset=utf-8' />


if you have any pages you want to disable gzipping just add $_NO_COMPRESS = true; above require bittorrent.php in that specific page.

Now that Rule A is taken care of let's tackle Rule B.

Rule B
Required components http://httpd.apache....od_expires.html
(Apache HTTP Server Version 2.0)

* i'm sure this can be applied in other web servers such as lighty etc. but i only have applied this to my apache 2 server, feel free to post how to do this on other web servers.

Since this thread is in regards to Javascript i will only cover the config required for that:

1st off this will only work if your apache setup has mod_expires enabled so DO THAT FIRST, google will help with that :)

we will go the easy way here and create an .htaccess file in your scripts or js directory(s) and include this in it:

Code (php) Select Expand

<IfModule mod_expires.c>
  ExpiresActive On
  ExpiresByType application/x-javascript "access plus 1 week"
</IfModule>


i have only used 1 week in this example so by all means when you are comfortable with everything and wanna cache for longer, do it. Proper date/time syntax for doing this can be found at source

*NOTE*
Keep in mind, if you use a far future Expires header you have to change the component's filename whenever the component changes.
So if you plan on editing your Javascripts you will need to rename the filename each time to ensure the end user is downloading/using the updated code changes.

----------------------------------------------------------------------------------------------

THAT's IT! Congrats! :D we just hopefully met the standards to pass Rules A., B. and C. in regards to Javascript.

After a congratulatory beer or three we will move on to tackle Rules 1-6:

----------------------------------------------------------------------------------------------

open bittorrent.php again and find stdhead() function again and:

add in to stdhead() $stdhead = false like so:

Code (php) Select Expand

function stdhead($title = "", $msgalert = true, $stdhead = false) {


now find in 09 source:

$htmlout = "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\"

above that ^^ add:

Code (php) Select Expand

/** include js files needed only for the page being used by pdq **/
$js_incl = '
<!-- javascript goes here or in footer -->
';
if ($stdhead['js'] != false) {
    foreach ($stdhead['js'] as $JS)
        $js_incl .= '<script type='text/javascript' src='".$TBDEV['baseurl']."/scripts/".$JS.".js'></script>
';
}


then just below in the $htmlout after:

Code (html) Select Expand

<link rel='stylesheet' href='{$TBDEV['stylesheet']}' type='text/css' />


add:

Code Select Expand

".$js_incl."


now still in bittorrent.php find stdfoot() function:

change in 09 source:

Code (php) Select Expand

function stdfoot() {
  global $TBDEV;
 
    return "<p align='center'>
    <a href='http://www.tbdev.net'><img src='{$TBDEV['pic_base_url']}tbdev_btn_red.png' border='0' alt='Powered By TBDev &copy;2009' title='Powered By TBDev &copy;2009' /></a></p>
    </td></tr></table>\n
    </body></html>\n";
}

change to like:

Code (php) Select Expand

function stdfoot($stdfoot = false) {
  global $TBDEV;
 
  $htmlfoot = "<p align='center'>
    <a href='http://www.tbdev.net'><img src='{$TBDEV['pic_base_url']}tbdev_btn_red.png' border='0' alt='Powered By TBDev &copy;2009' title='Powered By TBDev &copy;2009' /></a></p>
    </td></tr></table>\n";
/** include js files needed only for the page being used by pdq **/
  $htmlfoot .= '
<!-- javascript goes here -->
';
if ($stdfoot['js'] != false) {
    foreach ($stdfoot['js'] as $JS)
        $htmlfoot .= '<script type="text/javascript" src="'.$TBDEV['baseurl'].'/scripts/'.$JS.'.js"></script>
';
}
  $htmlfoot .= "</body></html>\n"
  return $htmlfoot;
}

------------------------------------------------------------------------------------

Now it gets serious, let's minify and combine and put our scripts at the bottom of page which in turn will address Rules 1-6 of the above golden rules. :)

I would suggest AGAIN downloading the yslow firefox extension so you will be able to gauge performance improvements before and after.

It requires firebug firefox extension and it can be found here: http://developer.yahoo.com/yslow/


This is a MUST-DO for all your pages if your tracker has lot's of javascript files with different scripts required on different pages.

Even if your site has more than the standard basic install of popup.js I would still suggest doing the this.

in your js or scripts folder create some new js files, you can name them as such:
* global-v.1.js

the following are optional files and suggest real world page names you may need depending on the amount of javascript used on your site.
* browse-v.1.js
* details-v.1.js
* forum-v.1.js
* index-v.1.js
* my-v.1.js
* userdetails-v.1.js
and so on, on my tracker i use the above files plus one or 2 others such as faq-v.1.js and rules-v.1.js. :)

Use this tool or use yslows built in javascript minifier:
http://www.jslab.dk/tools.minify.php

The idea is that we take all global Javascripts, global as in ones that are used on every page of your site.

On a basic install of TBDev.net 09 we only have the one file:
scripts/popup.js.

If working on a modded source with many Javascripts this will take a little more thinking, say for example you are using jquery, and some other scripts like these examples:
scripts/jquery-1.3.2.min.js
scripts/some.script.js
scripts/some.different.script.js
scripts/some.other.script.js
scripts/yet.another.script.js
scripts/wow.i.got.lots.of.scripts.js

Now what we want to do is figure out which of these are 'global' scripts and which are only used on certain pages.

let's do the easy example first, let's do basic 09/current source:

since we only have one script 'scripts/popup.js' and it is not a global script (used on every page)
let's minify it and put it into the pages it is required on:
using yslow - tools - js minify OR http://www.jslab.dk/tools.minify.php minify the file:

Minified code is 33.74% smaller than entered source code.

Result:

Code (html) Select Expand

function PopUp(url,name,width,height,center,resize,scroll,posleft,postop){showx="";showy="";if(posleft!=0){X=posleft}if(postop!=0){Y=postop}if(!scroll){scroll=1}if(!resize){resize=1}if((parseInt(navigator.appVersion)>=4)&&(center)){X=(screen.width-width)/2;Y=(screen.height-height)/2;}if(X>0){showx=',left='+X;}if(Y>0){showy=',top='+Y;}if(scroll!=0){scroll=1}var Win=window.open(url,name,'width='+width+',height='+height+showx+showy+',resizable='+resize+',scrollbars='+scroll+',location=no,directories=no,status=no,menubar=no,toolbar=no');}

Now let's copy this over to the pages js files that require it:
/forums/forum_topicview.php is the only file/page that requires this Javascript (i thinks lol) so let's paste the minified code into scripts/forum-v.1.js and save it.

Now open forums/forum_topicview.php and remove:

Code (php) Select Expand

$HTMLOUT .= "<script type='text/javascript' src='./scripts/popup.js'></script>";

scroll to the bottom of this file and find:

Code (php) Select Expand

print stdhead("{$lang['forum_topic_view_view_topic']}") . $HTMLOUT . stdfoot();

change this to:

Code Select Expand

$stdfoot = array(/** include js **/
                         'js' => array('forum-v.1'
                         /*,'jquery-1.3.2.min'*/
                         /*,'some.other.script'*/
                         );
print stdhead("{$lang['forum_topic_view_view_topic']}") . $HTMLOUT . stdfoot($stdfoot);


we use the $stdfoot option to load it into stdfoot() since this script does not to be loaded into header, if this script does not function as it should or is required on page load you will have to add this into $stdhead and put it into stdhead() function.

Here is an example of this, say we have jquery script that we use and it is also needed in this page(forums/forum_topicview.php) but we must load this into header or some of it's functions will not work, so let's do that:

Code (php) Select Expand

$stdhead = array(/** include js **/
                         'js' => array(
                         'jquery-1.3.2.min'
                         );

          $stdfoot = array(/** include js **/
                         'js' => array('forum-v.1'
                         /*,'jquery-1.3.2.min'*/
                         /*,'some.other.script'*/
                         );

      print stdhead("{$lang['forum_topic_view_view_topic']}", true, $stdhead) . $HTMLOUT . stdfoot($stdfoot);


Now if you viewsource on this page you will notice that we load the file jquery-1.3.2.min.js in stdhead() and in stdfoot() we are loading the script forum-v.1.js.
Take your time and go through your pages deciding which javascript files are 'global' such as jquery in my example or which ones are only required on specific pages.
Then minify the javascript files and combine them into either the global js file we made (global-v.1.js) or into page specific js files such as:
* browse-v.1.js
* details-v.1.js
* forum-v.1.js
* index-v.1.js
* my-v.1.js
* userdetails-v.1.js
and so on...

Use this example to apply to your other pages such as index, browse, login, details and so on and so on. :)

If you have global javascript that requires inluding on each and every page (this is unlikely) then this is how we could deal with those using the file global-v.1.js.
from the edits made earlier to stdhead we would change:

09 source:

Code (php) Select Expand

$js_incl = '<!-- javascript goes here or in footer -->';

to like:

Code (php) Select Expand

$js_incl = '
<!-- javascript goes here or in footer -->
<script type="text/javascript" src="'.$TBDEV['baseurl'].'/scripts/global-v.1.js"></script>
';


The reason why we have added version numbers to out javascript files is so when we add new code into these files, since we are caching the javascripts in the users browser, everytime we edit a js file we must change the revision number as well as the version number in the file itself, example:
forum-v.1.js

would become:
forum-v.2.js

simple simple! Keep this in mind before applying the cache-control/mod_expires tips from above. :)

-----------------------------------------------------------------------------------------

Congrats! You are done! My hat's off to you.

Now Apply these techniques throughout your source for performance improvements.
Make sure and use yslow for before and after comparisons. :)

Feel free to post your before and after screens or results :)

Here are my current statistics:

index page:
Grade B
Overall performance score 89 Ruleset applied: YSlow(V2)

Primed Cache
HTTP Requests - 1
Total Weight - 47.0K
1 HTML/Text 47.0K

Empty Cache
HTTP Requests - 43
Total Weight - 796.7K
1 HTML/Text 47.0K
1 Javascript File 20.2K
1 Stylesheet File 2.9K
6 CSS Image 82.0K
34 Image 644.5K


browse page:
Grade A
Overall performance score 90 Ruleset applied: YSlow(V2)

Primed Cache
HTTP Requests - 1
Total Weight - 33.4K

1 HTML/Text 33.4K

Empty Cache
HTTP Requests - 42
Total Weight - 519.1K
1 HTML/Text 33.4K
3 Javascript File 21.7K
1 Stylesheet File 4.2K
6 CSS Image 82.0K
31 Image 377.7K

faq page:
Grade
A
Overall performance score 92 Ruleset applied: YSlow(V2)

Primed Cache
HTTP Requests - 1
Total Weight - 29.4K

1 HTML/Text 29.4K

Empty Cache
HTTP Requests - 31
Total Weight - 457.3K
1 HTML/Text 29.4K
1 Javascript File 19.7K
1 Stylesheet File 3.6K
6 CSS Image 82.0K
22 Image 322.4K

userdetails page
Grade A
Overall performance score 92 Ruleset applied: YSlow(V2)

Primed Cache
HTTP Requests - 1
Total Weight - 20.1K
1 HTML/Text 20.1K
Empty Cache
HTTP Requests - 34
Total Weight - 452.4K

1 HTML/Text 20.1K
1 Javascript File 20.6K
1 Stylesheet File 3.5K
6 CSS Image 82.0K
25 Image 325.9K

login page:
Grade A
Overall performance score 96 Ruleset applied: YSlow(V2)

Primed Cache
HTTP Requests - 1
Total Weight - 2.4K
1 HTML/Text 2.4K

Empty Cache
HTTP Requests - 6
Total Weight - 85.0K
1 HTML/Text 2.4K
1 Javascript File 7.3K
3 CSS Image 75.1K
1 Image 0.1K

If i have made mistakes in my post please correct me and of course feel free to add your own related suggestions and tips. :)
big thanks to the yahoo guys and gals for their great referance material found here:
http://developer.yahoo.com/yslow/

=]