Web Development

Web development is a much, much deeper subject than simply building websites. It involves project management, agile work methods, and knowledge of a multitude of technologies, from the behind the scenes software that make websites come to life, to the ‘font end’ technologies that make the finished product look and feel welcoming for the user.

My passion for software development has lead me in to the the field of large scale, enterprise development projects, specialising in high quality, high capacity, and high availability software.
While I have traditionally been designing or building ‘back end’ projects, I have recently taken on a full spectrum project, involving everything from system adminstration to interfacing with external services, and cutting up the design in Photoshop. Versatility is important since the luxury of starting a project from the very beginning is not always possible.

block facebook on network home router

use a sniffer likehttp://www.nirsoft.net/utils/dns_query_sniffer.html
This exactly shows in real-time what domains are being queried.

But to make your life easier, these are the domains being queried when using the Facebook Messenger for Windows:



to block chat



will update this post with the blocking ways

10 Immutable Laws of Security

Law #1: If a bad guy can persuade you to run his program on your computer, it’s not your computer anymore
Law #2: If a bad guy can alter the operating system on your computer, it’s not your computer anymore
Law #3: If a bad guy has unrestricted physical access to your computer, it’s not your computer anymore
Law #4: If you allow a bad guy to upload programs to your website, it’s not your website any more
Law #5: Weak passwords trump strong security
Law #6: A computer is only as secure as the administrator is trustworthy
Law #7: Encrypted data is only as secure as the decryption key
Law #8: An out of date virus scanner is only marginally better than no virus scanner at all
Law #9: Absolute anonymity isn’t practical, in real life or on the Web
Law #10: Technology is not a panacea

Law #1: If a bad guy can persuade you to run his program on your computer, it’s not your computer anymore

It’s an unfortunate fact of computer science: when a computer program runs, it will do what it’s programmed to do, even if it’s programmed to be harmful. When you choose to run a program, you are making a decision to turn over control of your computer to it. Once a program is running, it can do anything, up to the limits of what you yourself can do on the computer. It could monitor your keystrokes and send them to a website. It could open every document on the computer, and change the word “will” to “won’t” in all of them. It could send rude emails to all your friends. It could install a virus. It could create a “back door” that lets someone remotely control your computer. It could dial up an ISP in Katmandu. Or it could just reformat your hard drive.

That’s why it’s important to never run, or even download, a program from an untrusted source—and by “source,” I mean the person who wrote it, not the person who gave it to you. There’s a nice analogy between running a program and eating a sandwich. If a stranger walked up to you and handed you a sandwich, would you eat it? Probably not. How about if your best friend gave you a sandwich? Maybe you would, maybe you wouldn’t—it depends on whether she made it or found it lying in the street. Apply the same critical thought to a program that you would to a sandwich, and you’ll usually be safe.

Law #2: If a bad guy can alter the operating system on your computer, it’s not your computer anymore

In the end, an operating system is just a series of ones and zeroes that, when interpreted by the processor, cause the computer to do certain things. Change the ones and zeroes, and it will do something different. Where are the ones and zeroes stored? Why, on the computer, right along with everything else! They’re just files, and if other people who use the computer are permitted to change those files, it’s “game over”.

To understand why, consider that operating system files are among the most trusted ones on the computer, and they generally run with system-level privileges. That is, they can do absolutely anything. Among other things, they’re trusted to manage user accounts, handle password changes, and enforce the rules governing who can do what on the computer. If a bad guy can change them, the now-untrustworthy files will do his bidding, and there’s no limit to what he can do. He can steal passwords, make himself an administrator on the computer, or add entirely new functions to the operating system. To prevent this type of attack, make sure that the system files (and the registry, for that matter) are well protected. (The security checklists on the Microsoft Security website will help you do this).

Law #3: If a bad guy has unrestricted physical access to your computer, it’s not your computer anymore

Oh, the things a bad guy can do if he can lay his hands on your computer! Here’s a sampling, going from Stone Age to Space Age:

  • He could mount the ultimate low-tech denial of service attack, and smash your computer with a sledgehammer.
  • He could unplug the computer, haul it out of your building, and hold it for ransom.
  • He could boot the computer from a floppy disk, and reformat your hard drive. But wait, you say, I’ve configured the BIOS on my computer to prompt for a password when I turn the power on. No problem – if he can open the case and get his hands on the system hardware, he could just replace the BIOS chips. (Actually, there are even easier ways).
  • He could remove the hard drive from your computer, install it into his computer, and read it.
  • He could make a duplicate of your hard drive and take it back his lair. Once there, he’d have all the time in the world to conduct brute-force attacks, such as trying every possible logon password. Programs are available to automate this and, given enough time, it’s almost certain that he would succeed. Once that happens, Laws #1 and #2 above apply.
  • He could replace your keyboard with one that contains a radio transmitter. He could then monitor everything you type, including your password.

Always make sure that a computer is physically protected in a way that’s consistent with its value—and remember that the value of a computer includes not only the value of the hardware itself, but the value of the data on it, and the value of the access to your network that a bad guy could gain. At a minimum, business-critical computers like domain controllers, database servers, and print/file servers should always be in a locked room that only people charged with administration and maintenance can access. But you may want to consider protecting other computers as well, and potentially using additional protective measures.

If you travel with a laptop, it’s absolutely critical that you protect it. The same features that make laptops great to travel with – small size, light weight, and so forth—also make them easy to steal. There are a variety of locks and alarms available for laptops, and some models let you remove the hard drive and carry it with you. You also can use features like the Encrypting File System in Microsoft Windows® 2000 to mitigate the damage if someone succeeded in stealing the computer. But the only way you can know with 100% certainty that your data is safe and the hardware hasn’t been tampered with is to keep the laptop on your person at all times while traveling.

Law #4: If you allow a bad guy to upload programs to your website, it’s not your website any more

This is basically Law #1 in reverse. In that scenario, the bad guy tricks his victim into downloading a harmful program onto his computer and running it. In this one, the bad guy uploads a harmful program to a computer and runs it himself. Although this scenario is a danger anytime you allow strangers to connect to your computer, websites are involved in the overwhelming majority of these cases. Many people who operate websites are too hospitable for their own good, and allow visitors to upload programs to the site and run them. As we’ve seen above, unpleasant things can happen if a bad guy’s program can run on your computer.

If you run a website, you need to limit what visitors can do. You should only allow a program on your site if you wrote it yourself, or if you trust the developer who wrote it. But that may not be enough. If your website is one of several hosted on a shared server, you need to be extra careful. If a bad guy can compromise one of the other sites on the server, it’s possible he could extend his control to the server itself, in which he could control all of the sites on it—including yours. If you’re on a shared server, it’s important to find out what the server administrator’s policies are. (By the way, before opening your site to the public, make sure you’ve followed the security checklists for IIS 4.0 and IIS 5.0).

Law #5: Weak passwords trump strong security

The purpose of having a logon process is to establish who you are. Once the operating system knows who you are, it can grant or deny requests for system resources appropriately. If a bad guy learns your password, he can log on as you. In fact, as far as the operating system is concerned, he is you. Whatever you can do on the system, he can do as well, because he’s you. Maybe he wants to read sensitive information you’ve stored on your computer, like your e-mail. Maybe you have more privileges on the network than he does, and being you will let him do things he normally couldn’t. Or maybe he just wants to do something malicious and blame it on you. In any case, it’s worth protecting your credentials.

Always use a password—it’s amazing how many accounts have blank passwords. And choose a complex one. Don’t use your dog’s name, your anniversary date, or the name of the local football team. And don’t use the word “password”! Pick a password that has a mix of upper- and lower-case letters, number, punctuation marks, and so forth. Make it as long as possible. And change it often. Once you’ve picked a strong password, handle it appropriately. Don’t write it down. If you absolutely must write it down, at the very least keep it in a safe or a locked drawer—the first thing a bad guy who’s hunting for passwords will do is check for a yellow sticky note on the side of your screen, or in the top desk drawer. Don’t tell anyone what your password is. Remember what Ben Franklin said: two people can keep a secret, but only if one of them is dead.

Finally, consider using something stronger than passwords to identify yourself to the system. Windows 2000, for instance, supports the use of smart cards, which significantly strengthens the identity checking the system can perform. You may also want to consider biometric products like fingerprint and retina scanners.

Law #6: A computer is only as secure as the administrator is trustworthy

Every computer must have an administrator: someone who can install software, configure the operating system, add and manage user accounts, establish security policies, and handle all the other management tasks associated with keeping a computer up and running. By definition, these tasks require that he have control over the computer. This puts the administrator in a position of unequalled power. An untrustworthy administrator can negate every other security measure you’ve taken. He can change the permissions on the computer, modify the system security policies, install malicious software, add bogus users, or do any of a million other things. He can subvert virtually any protective measure in the operating system, because he controls it. Worst of all, he can cover his tracks. If you have an untrustworthy administrator, you have absolutely no security.

When hiring a system administrator, recognize the position of trust that administrators occupy, and only hire people who warrant that trust. Call his references, and ask them about his previous work record, especially with regard to any security incidents at previous employers. If appropriate for your organization, you may also consider taking a step that banks and other security-conscious companies do, and require that your administrators pass a complete background check at hiring time, and at periodic intervals afterward. Whatever criteria you select, apply them across the board. Don’t give anyone administrative privileges on your network unless they’ve been vetted – and this includes temporary employees and contractors, too.

Next, take steps to help keep honest people honest. Use sign-in/sign-out sheets to track who’s been in the server room. (You do have a server room with a locked door, right? If not, re-read Law #3). Implement a “two person” rule when installing or upgrading software. Diversify management tasks as much as possible, as a way of minimizing how much power any one administrator has. Also, don’t use the Administrator account—instead, give each administrator a separate account with administrative privileges, so you can tell who’s doing what. Finally, consider taking steps to make it more difficult for a rogue administrator to cover his tracks. For instance, store audit data on write-only media, or house System A’s audit data on System B, and make sure that the two systems have different administrators. The more accountable your administrators are, the less likely you are to have problems.

Law #7: Encrypted data is only as secure as the decryption key

Suppose you installed the biggest, strongest, most secure lock in the world on your front door, but you put the key under the front door mat. It wouldn’t really matter how strong the lock is, would it? The critical factor would be the poor way the key was protected, because if a burglar could find it, he’d have everything he needed to open the lock. Encrypted data works the same way—no matter how strong the crypto algorithm is, the data is only as safe as the key that can decrypt it.

Many operating systems and cryptographic software products give you an option to store cryptographic keys on the computer. The advantage is convenience – you don’t have to handle the key – but it comes at the cost of security. The keys are usually obfuscated (that is, hidden), and some of the obfuscation methods are quite good. But in the end, no matter how well-hidden the key is, if it’s on the computer it can be found. It has to be – after all, the software can find it, so a sufficiently-motivated bad guy could find it, too. Whenever possible, use offline storage for keys. If the key is a word or phrase, memorize it. If not, export it to a floppy disk, make a backup copy, and store the copies in separate, secure locations. (All of you administrators out there who are using Syskey in “local storage” mode—you’re going to reconfigure your server right this minute, right?)

Law #8: An out of date virus scanner is only marginally better than no virus scanner at all

Virus scanners work by comparing the data on your computer against a collection of virus “signatures”. Each signature is characteristic of a particular virus, and when the scanner finds data in a file, email, or elsewhere that matches the signature, it concludes that it’s found a virus. However, a virus scanner can only scan for the viruses it knows about. It’s vital that you keep your virus scanner’s signature file up to date, as new viruses are created every day.

The problem actually goes a bit deeper than this, though. Typically, a new virus will do the greatest amount of damage during the early stages of its life, precisely because few people will be able to detect it. Once word gets around that a new virus is on the loose and people update their virus signatures, the spread of the virus falls off drastically. The key is to get ahead of the curve, and have updated signature files on your computer before the virus hits.

Virtually every maker of anti-virus software provides a way to get free updated signature files from their website. In fact, many have “push” services, in which they’ll send notification every time a new signature file is released. Use these services. Also, keep the virus scanner itself—that is, the scanning software—updated as well. Virus writers periodically develop new techniques that require that the scanners change how they do their work.

Law #9: Absolute anonymity isn’t practical, in real life or on the Web

All human interaction involves exchanging data of some kind. If someone weaves enough of that data together, they can identify you. Think about all the information that a person can glean in just a short conversation with you. In one glance, they can gauge your height, weight, and approximate age. Your accent will probably tell them what country you’re from, and may even tell them what region of the country. If you talk about anything other than the weather, you’ll probably tell them something about your family, your interests, where you live, and what you do for a living. It doesn’t take long for someone to collect enough information to figure out who you are. If you crave absolute anonymity, your best bet is to live in a cave and shun all human contact.

The same thing is true of the Internet. If you visit a website, the owner can, if he’s sufficiently motivated, find out who you are. After all, the ones and zeroes that make up the Web session have to be able to find their way to the right place, and that place is your computer. There are a lot of measures you can take to disguise the bits, and the more of them you use, the more thoroughly the bits will be disguised. For instance, you could use network address translation to mask your actual IP address, subscribe to an anonymizing service that launders the bits by relaying them from one end of the ether to the other, use a different ISP account for different purposes, surf certain sites only from public kiosks, and so on. All of these make it more difficult to determine who you are, but none of them make it impossible. Do you know for certain who operates the anonymizing service? Maybe it’s the same person who owns the website you just visited! Or what about that innocuous website you visited yesterday, that offered to mail you a free $10 off coupon? Maybe the owner is willing to share information with other website owners. If so, the second website owner may be able to correlate the information from the two sites and determine who you are.

Does this mean that privacy on the Web is a lost cause? Not at all. What it means is that the best way to protect your privacy on the Internet is the same as the way you protect your privacy in normal life—through your behavior. Read the privacy statements on the websites you visit, and only do business with ones whose practices you agree with. If you’re worried about cookies, disable them. Most importantly, avoid indiscriminate Web surfing—recognize that just as most cities have a bad side of town that’s best avoided, the Internet does too. But if it’s complete and total anonymity you want, better start looking for that cave.

Law #10: Technology is not a panacea

Technology can do some amazing things. Recent years have seen the development of ever-cheaper and more powerful hardware, software that harnesses the hardware to open new vistas for computer users, as well as advancements in cryptography and other sciences. It’s tempting to believe that technology can deliver a risk-free world, if we just work hard enough. However, this is simply not realistic.

Perfect security requires a level of perfection that simply doesn’t exist, and in fact isn’t likely to ever exist. This is true for software as well as virtually all fields of human interest. Software development is an imperfect science, and all software has bugs. Some of them can be exploited to cause security breaches. That’s just a fact of life. But even if software could be made perfect, it wouldn’t solve the problem entirely. Most attacks involve, to one degree or another, some manipulation of human nature—this is usually referred to as social engineering. Raise the cost and difficulty of attacking security technology, and bad guys will respond by shifting their focus away from the technology and toward the human being at the console. It’s vital that you understand your role in maintaining solid security, or you could become the chink in your own systems’ armor.

The solution is to recognize two essential points. First, security consists of both technology and policy—that is, it’s the combination of the technology and how it’s used that ultimately determines how secure your systems are. Second, security is journey, not a destination—it isn’t a problem that can be “solved” once and for all; it’s a constant series of moves and countermoves between the good guys and the bad guys. The key is to ensure that you have good security awareness and exercise sound judgment. There are resources available to help you do this. The Microsoft Security website, for instance, has hundreds of white papers, best practices guides, checklists and tools, and we’re developing more all the time. Combine great technology with sound judgment, and you’ll have rock-solid security.

Patterns For Large-Scale JavaScript Application Architecture

Can you summarize this article in 140 characters?

In the event of you being short for time, here’s the tweet-sized summary of this article:

Decouple app. architecture w/module,facade & mediator patterns. Mods publish msgs, mediator acts as pub/sub mgr & facade handles security

What exactly is a ‘large’ JavaScript application?

Before we begin, let us attempt to define what we mean when we refer to a JavaScript application as being significantly ‘large’. This is a question I’ve found still challenges developers with many years of experience in the field and the answer to this can be quite subjective.

As an experiment, I asked a few intermediate developers to try providing their definition of it informally. One developer suggested ‘a JavaScript application with over 100,000 LOC’ whilst another suggested ‘apps with over 1MB of JavaScript code written in-house’. Whilst valiant (if not scary) suggestions, both of these are incorrect as the size of a codebase does not always correlate to application complexity – those 100,000 LOC could easily represent quite trivial code.

My own definition may or may not be universally accepted, but I believe that it’s closer to what a large application actually represents.

In my view, large-scale JavaScript apps are non-trivial applications requiring significant developer effort to maintain, where most heavy lifting of data manipulation and display falls to the browser.

The last part of this definition is possibly the most significant.

Let’s review your current architecture.

If working on a significantly large JavaScript application, remember to dedicate sufficient time to planning the underlying architecture that makes the most sense. It’s often more complex than you may initially imagine.

I can’t stress the importance of this enough – some developers I’ve seen approach larger applications have stepped back and said ‘Okay. Well, there are a set of ideas and patterns that worked well for me on my last medium-scale project. Surely they should mostly apply to something a little larger, right?’. Whilst this may be true to an extent, please don’t take it for granted – larger apps generally have greater concerns that need to be factored in. I’m going to discuss shortly why spending a little more time planning out the structure to your application is worth it in the long run.

Most JavaScript developers likely use a mixed combination of the following for their current architecture:

  • custom widgets
  • models
  • views
  • controllers
  • templates
  • libraries/toolkits
  • an application core.

You probably also break down your application’s functionality into blocks of modules or apply other patterns for this. This is great, but there are a number of potential problems you can run into if this represents all of your application’s structure.

1. How much of this architecture is instantly re-usable?

Can single modules exist on their own independently? Are they self-contained? Right now if I were to look at the codebase for a large application you or your team were working on and selected a random module, would it be possible for me to easily just drop it into a new page and start using it on its own?. You may question the rationale behind wanting to do this, however I encourage you to think about the future. What if your company were to begin building more and more non-trivial applications which shared some cross-over in functionality?. If someone said, ‘Our users love using the chat module in our mail client. Let’s drop that into our new collaborative editing suite’, would this be possible without significantly altering the code?.

2. How much do modules depend on other modules in the system?

Are they tightly coupled? Before I dig into why this is a concern, I should note that I understand it’s not always possible to have modules with absolutely no other dependencies in a system. At a granular level you may well have modules that extend the base functionality of others, but this question is more-so related to groups of modules with distinct functionality. It should be possible for all of these distinct sets of modules to work in your application without depending on too many other modules being present or loaded in order to function.

3. If specific parts of your application fail, can it still function?

If you’re building a GMail-like application and your webmail module (or modules) fail, this shouldn’t block the rest of the UI or prevent users from being able to use other parts of the page such as chat. At the same time, as per before, modules should ideally be able to exist on their own outside of your current application architecture. In my talks I mention dynamic dependency (or module) loading based on expressed user-intent as something related. For example, in GMail’s case they might have the chat module collapsed by default without the core module code loaded on page initialization. If a user expressed an intent to use the chat feature, only then would it be dynamically loaded. Ideally, you want this to be possible without it negatively affecting the rest of your application.

4. How easily can you test individual modules?

When working on systems of significant scale where there’s a potential for millions of users to use (or mis-use) the different parts it, it’s essential that modules which may end up being re-used across a number of different applications be sufficiently tested. Testing needs to be possible for when the module both inside and outside of the architecture for which it was initially built. In my view, this provides the most assurance that it shouldn’t break if dropped into another system.

Think Long Term

When devising the architecture for your large application, it’s important to think ahead. Not just a month or a year from now, but beyond that. What might change? It’s of course impossible to guess exactly how your application may grow, but there’s certainly room to consider what is likely. Here, there is at least one specific aspect of your application that comes to mind.

Developers often couple their DOM manipulation code quite tightly with the rest of their application – even when they’ve gone to the trouble of separating their core logic down into modules. Think about it..why is this not a good idea if we’re thinking long-term?

One member of my audience suggested that it was because a rigid architecture defined in the present may not be suitable for the future. Whilst certainly true, there’s another concern that may cost even more if not factored in.

You may well decide to switch from using Dojo, jQuery, Zepto or YUI to something entirely different for reasons of performance, security or design in the future. This can become a problem because libraries are not easily interchangeable and have high switching costs if tightly coupled to your app.

If you’re a Dojo developer (like some of the audience at my talk), you may not have something better to switch to in the present, but who is to say that in 2-3 years something better doesn’t come out that you’ll want to switch to?.

This is a relatively trivial decision in smaller codebases but for larger applications, having an architecture which is flexible enough to support not caring about the libraries being used in your modules can be of great benefit, both financially and from a time-saving perspective.

To summarize, if you reviewed your architecture right now, could a decision to switch libraries be made without rewriting your entire application?. If not, consider reading on because I think the architecture being outlined today may be of interest.

There are a number of influential JavaScript developers who have previously outlined some of the concerns I’ve touched upon so far. Three key quotes I would like to share from them are the following:

“The secret to building large apps is never build large apps. Break your applications into small pieces. Then, assemble those testable, bite-sized pieces into your big application” – Justin Meyer, author JavaScriptMVC

“The key is to acknowledge from the start that you have no idea how this will grow. When you accept that you don’t know everything, you begin to design the system defensively. You identify the key areas that may change, which often is very easy when you put a little bit of time into it. For instance, you should expect that any part of the app that communicates with another system will likely change, so you need to abstract that away.” – Nicholas Zakas, author ‘High-performance JavaScript websites’

and last but not least:

“The more tied components are to each other, the less reusable they will be, and the more difficult it becomes to make changes to one without accidentally affecting another” – Rebecca Murphey, author of jQuery Fundamentals.

These principles are essential to building an architecture that can stand the test of time and should always be kept in mind.


Let’s think about what we’re trying to achieve for a moment.

We want a loosely coupled architecture with functionality broken down into independent modules with ideally no inter-module dependencies. Modules speak to the rest of the application when something interesting happens and an intermediate layer interprets and reacts to these messages.

For example, if we had a JavaScript application responsible for an online bakery, one such ‘interesting’ message from a module might be ‘batch 42 of bread rolls is ready for dispatch’.

We use a different layer to interpret messages from modules so that a) modules don’t directly access the core and b) modules don’t need to directly call or interact with other modules. This helps prevent applications from falling over due to errors with specific modules and provides us a way to kick-start modules which have fallen over.

Another concern is security. The reality is that most of us don’t consider internal application security as that much of a concern. We tell ourselves that as we’re structuring the application, we’re intelligent enough to figure out what should be publicly or privately accessible.

However, wouldn’t it help if you had a way to determine what a module was permitted to do in the system? eg. if I know I’ve limited the permissions in my system to not allow a public chat widget to interface with an admin module or a module with DB-write permissions, I can limit the chances of someone exploiting vulnerabilities I have yet to find in the widget to pass some XSS in there. Modules shouldn’t be able to access everything. They probably can in most current architectures, but do they really need to be able to?

Having an intermediate layer handle permissions for which modules can access which parts of your framework gives you added security. This means a module is only able to do at most what we’ve permitted it do.

The Proposed Architecture

The solution to the architecture we seek to define is a combination of three well-known design patterns: the module, facade and mediator.

Rather than the traditional model of modules directly communicating with each other, in this decoupled architecture, they’ll instead only publish events of interest (ideally, without a knowledge of other modules in the system). The mediator pattern will be used to both subscribe to messages from these modules and handle what the appropriate response to notifications should be. The facade pattern will be used to enforce module permissions.

I will be going into more detail on each of these patterns below:

Module Theory

You probably already use some variation of modules in your existing architecture. If however, you don’t, this section will present a short primer on them.

Modules are an integral piece of any robust application’s architecture and are typically single-purpose parts of a larger system that are interchangeable.

Depending on how you implement modules, it’s possible to define dependencies for them which can be automatically loaded to bring together all of the other parts instantly. This is considered more scalable than having to track the various dependencies for them and manually load modules or inject script tags.

Any significantly non-trivial application should be built from modular components. Going back to GMail, you could consider modules independent units of functionality that can exist on their own, so the chat feature for example. It’s probably backed by a chat module, however, depending on how complex that unit of functionality is, it may well have more granular sub-modules that it depends on. For example, one could have a module simply to handle the use of emoticons which can be shared across both chat and mail composition parts of the system.

In the architecture being discussed, modules have a very limited knowledge of what’s going on in the rest of the system. Instead, we delegate this responsibility to a mediator via a facade.

This is by design because if a module only cares about letting the system know when something of interest happens without worrying if other modules are running, a system is capable of supporting adding, removing or replacing modules without the rest of the modules in the system falling over due to tight coupling.

Loose coupling is thus essential to this idea being possible. It facilitates easier maintainability of modules by removing code dependencies where possible. In our case, modules should not rely on other modules in order to function correctly. When loose coupling is implemented effectively, its straight-forward to see how changes to one part of a system may affect another.

In JavaScript, there are several options for implementing modules including the well-known module pattern and object literals. Experienced developers will already be familiar with these and if so, please skip ahead to the section on CommonJS modules.

The Module Pattern

The module pattern is a popular design that pattern that encapsulates ‘privacy’, state and organization using closures. It provides a way of wrapping a mix of public and private methods and variables, protecting pieces from leaking into the global scope and accidentally colliding with another developer’s interface. With this pattern, only a public API is returned, keeping everything else within the closure private.

This provides a clean solution for shielding logic doing the heavy lifting whilst only exposing an interface you wish other parts of your application to use. The pattern is quite similar to an immediately-invoked functional expression (IIFE) except that an object is returned rather than a function.

It should be noted that there isn’t really a true sense of ‘privacy’ inside JavaScript because unlike some traditional languages, it doesn’t have access modifiers. Variables can’t technically be declared as being public nor private and so we use function scope to simulate this concept. Within the module pattern, variables or methods declared are only available inside the module itself thanks to closure. Variables or methods defined within the returning object however are available to everyone.

Below you can see an example of a shopping basket implemented using the pattern. The module itself is completely self-contained in a global object called basketModule. The basket array in the module is kept private and so other parts of your application are unable to directly read it. It only exists with the module’s closure and so the only methods able to access it are those with access to its scope (ie. addItem(), getItem() etc).

  2. var basketModule =(function(){
  3. var basket =[];//private
  4. return{//exposed to public
  5. addItem:function(values){
  6. basket.push(values);
  7. },
  8. getItemCount:function(){
  9. return basket.length;
  10. },
  11. getTotal:function(){
  12. var q =this.getItemCount(),p=0;
  13. while(q–){
  14. p+= basket[q].price;
  15. }
  16. return p;
  17. }
  18. }
  19. }());

Inside the module, you’ll notice we return an object. This gets automatically assigned to basketModule so that you can interact with it as follows:

  2. //basketModule is an object with properties which can also be methods
  3. basketModule.addItem({item:‘bread’,price:0.5});
  4. basketModule.addItem({item:‘butter’,price:0.3});
  6. console.log(basketModule.getItemCount());
  7. console.log(basketModule.getTotal());
  9. //however, the following will not work:
  10. console.log(basketModule.basket);// (undefined as not inside the returned object)
  11. console.log(basket);//(only exists within the scope of the closure)

The methods above are effectively namespaced inside basketModule.

From a historical perspective, the module pattern was originally developed by a number of people including Richard Cornford in 2003. It was later popularized by Douglas Crockford in his lectures and re-introduced by Eric Miraglia on the YUI blog.

How about the module pattern in specific toolkits or frameworks?


Dojo attempts to provide ‘class’-like functionality through dojo.declare, which can be used for amongst other things, creating implementations of the module pattern. For example, if we wanted to declare basket as a module of the store namespace, this could be achieved as follows:

  1. //traditional way
  2. var store = window.store ||{};
  3. store.basket = store.basket ||{};
  5. //using dojo.setObject
  6. dojo.setObject(“store.basket.object”,(function(){
  7. var basket =[];
  8. function privateMethod(){
  9. console.log(basket);
  10. }
  11. return{
  12. publicMethod:function(){
  13. privateMethod();
  14. }
  15. };
  16. }()));

which can become quite powerful when used with dojo.provide and mixins.


The following example is heavily based on the original YUI module pattern implementation by Eric Miraglia, but is relatively self-explanatory.

  1. YAHOO.store.basket =function(){
  3. //”private” variables:
  4. var myPrivateVar =“I can be accessed only within YAHOO.store.basket .”;
  6. //”private” method:
  7. var myPrivateMethod =function(){
  8. YAHOO.log(“I can be accessed only from within YAHOO.store.basket”);
  9. }
  11. return{
  12. myPublicProperty:“I’m a public property.”,
  13. myPublicMethod:function(){
  14. YAHOO.log(“I’m a public method.”);
  16. //Within basket, I can access “private” vars and methods:
  17. YAHOO.log(myPrivateVar);
  18. YAHOO.log(myPrivateMethod());
  20. //The native scope of myPublicMethod is store so we can
  21. //access public members using “this”:
  22. YAHOO.log(this.myPublicProperty);
  23. }
  24. };
  26. }();


There are a number of ways in which jQuery code unspecific to plugins can be wrapped inside the module pattern. Ben Cherry previously suggested an implementation where a function wrapper is used around module definitions in the event of there being a number of commonalities between modules.

In the following example, a library function is defined which declares a new library and automatically binds up the init function to document.ready when new libraries (ie. modules) are created.

  2. function library(module){
  3. $(function(){
  4. if(module.init){
  5. module.init();
  6. }
  7. });
  8. returnmodule;
  9. }
  11. var myLibrary = library(function(){
  12. return{
  13. init:function(){
  14. /*implementation*/
  15. }
  16. };
  17. }());
Object Literal Notation

In object literal notation, an object is described as a set of comma-separated name/value pairs enclosured in curly braces ({}). Names inside the object may be either strings or identifiers that are followed by a colon. There should be no comma used after the final name/value pair in the object as this may result in errors.

Object literals don’t require instantiation using the new operator but shouldn’t be used at the start of a statement as the opening { may be interpreted as the beginning of a block. Below you can see an example of a module defined using object literal syntax. New members may be added to the object using assignment as follows myModule.property = 'someValue';

Whilst the module pattern is useful for many things, if you find yourself not requiring specific properties or methods to be private, the object literal is a more than suitable alternative.

  1. var myModule ={
  2.     myProperty :‘someValue’,
  3.     //object literals can contain properties and methods.
  4.     //here, another object is defined for configuration
  5.     //purposes:
  6.     myConfig:{
  7.         useCaching:true,
  8.         language:‘en’   
  9.     },
  10.     //a very basic method
  11.     myMethod:function(){
  12.         console.log(‘I can haz functionality?’);
  13.     },
  14.     //output a value based on current configuration
  15.     myMethod2:function(){
  16.         console.log(‘Caching is:’+(this.myConfig.useCaching)?‘enabled’:‘disabled’);
  17.     },
  18.     //override the current configuration
  19.     myMethod3:function(newConfig){
  20.         if(typeof newConfig ==‘object’){
  21.            this.myConfig = newConfig;
  22.            console.log(this.myConfig.language);
  23.         }
  24.     }
  25. };
  27. myModule.myMethod();//I can haz functionality
  28. myModule.myMethod2();//outputs enabled
  29. myModule.myMethod3({language:‘fr’,useCaching:false});//fr

CommonJS Modules

Over the last year or two, you may have heard about CommonJS – a volunteer working group which designs, prototypes and standardizes JavaScript APIs. To date they’ve ratified standards for modules and packages.The CommonJS AMD proposal specifies a simple API for declaring modules which can be used with both synchronous and asynchronous script tag loading in the browser. Their module pattern is relatively clean and I consider it a reliable stepping stone to the module system proposed for ES Harmony (the next release of the JavaScript language).

From a structure perspective, a CommonJS module is a reusable piece of JavaScript which exports specific objects made available to any dependent code. This module format is becoming quite ubiquitous as a standard module format for JS. There are plenty of great tutorials on implementing CommonJS modules, but at a high-level they basically contain two primary parts: an exports object that contains the objects a module wishes to make available to other modules and a require function that modules can use to import the exports of other modules.

  1. /*
  2. Example of achieving compatibility with AMD and standard CommonJS by putting boilerplate around the standard CommonJS module format:
  3. */
  5. (function(define){
  6. define(function(require,exports){
  7. // module contents
  8. var dep1 =require(“dep1”);
  9. exports.someExportedFunction =function(){…};
  10. //…
  11. });
  12. })(typeof define==“function”?define:function(factory){factory(require,exports)});

There are a number of great JavaScript libraries for handling module loading in the CommonJS module format, but my personal preference is RequireJS. A complete tutorial on RequireJS is outside the scope of this tutorial, but I can recommend reading James Burke’s ScriptJunkie post on it here. I know a number of people that also like Yabble.

Out of the box, RequireJS provides methods for easing how we create static modules with wrappers and it’s extremely easy to craft modules with support for asynchronous loading. It can easily load modules and their dependencies this way and execute the body of the module once available.

There are some developers that however claim CommonJS modules aren’t suitable enough for the browser. The reason cited is that they can’t be loaded via a script tag without some level of server-side assistance. We can imagine having a library for encoding images as ASCII art which might export a encodeToASCII function. A module from this could resemble:

  1. var encodeToASCII =require(“encoder”).encodeToASCII;
  2. exports.encodeSomeSource =function(){
  3. //process then call encodeToASCII
  4. }

This type of scenario wouldn’t work with a script tag because the scope isn’t wrapped, meaning our encodeToASCII method would be attached to the window, require wouldn’t be as such defined and exports would need to be created for each module separately. A client-side library together with server-side assistance or a library loading the script with an XHR request using eval() could however handle this easily.

Using RequireJS, the module from earlier could be rewritten as follows:

  1. define(function(require, exports,module){
  2. var encodeToASCII =require(“encoder”).encodeToASCII;
  3. exports.encodeSomeSource =function(){
  4. //process then call encodeToASCII
  5. }
  6. });

For developers who may not rely on just using static JavaScript for their projects, CommonJS modules are an excellent path to go down, but do spend some time reading up on it. I’ve really only covered the tip of the ice berg but both the CommonJS wiki and Sitepen have a number of resources if you wish to read further.

The Facade Pattern

Next, we’re going to look at the facade pattern, a design pattern which plays a critical role in the architecture being defined today.

When you put up a facade, you’re usually creating an outward appearance which conceals a different reality. The facade pattern provides a convenient higher-level interface to a larger body of code, hiding its true underlying complexity. Think of it as simplifying the API being presented to other developers.

Facades are a structural pattern which can often be seen in JavaScript libraries and frameworks where, although an implementation may support methods with a wide range of behaviors, only a ‘facade’ or limited abstract of these methods is presented to the client for use.

This allows us to interact with the facade rather than the subsystem behind the scenes.

The reason the facade is of interest is because of its ability to hide implementation-specific details about a body of functionality contained in individual modules. The implementation of a module can change without the clients really even knowing about it.

By maintaining a consistent facade (simplified API), the worry about whether a module extensively uses dojo, jQuery, YUI, zepto or something else becomes significantly less important. As long as the interaction layer doesn’t change, you retain the ability to switch out libraries (eg. jQuery for Dojo) at a later point without affecting the rest of the system.

Below is a very basic example of a facade in action. As you can see, our module contains a number of methods which have been privately defined. A facade is then used to supply a much simpler API to accessing these methods:

  1. varmodule=(function(){
  2.     var _private ={
  3.         i:5,
  4.         get:function(){
  5.             console.log(‘current value:’+this.i);
  6.         },
  7.         set:function( val ){
  8.             this.i = val;
  9.         },
  10.         run :function(){
  11.             console.log(‘running’);
  12.         },
  13. jump:function(){
  14. console.log(‘jumping’);
  15. }
  16.     };
  17.     return{
  18.         facade :function( args ){
  19.             _private.set(args.val);
  20.             _private.get();
  21.             if( args.run ){
  22.                 _private.run();
  23.             }
  24.         }
  25.     }
  26. }());
  29. module.facade({run:true, val:10});
  30. //outputs current value: 10, running

and that’s really it for the facade before we apply it to our architecture. Next, we’ll be diving into the exciting mediator pattern. The core difference between the facade pattern and the mediator is that the facade (a structural pattern) only exposes existing functionality whilst the mediator (a behavioral pattern) can add functionality.

The Mediator Pattern

The mediator pattern is best introduced with a simple analogy – think of your typical airport traffic control. The tower handles what planes can take off and land because all communications are done from the planes to the control tower, rather than from plane-to-plane. A centralized controller is key to the success of this system and that’s really what a mediator is.

Mediators are used when the communication between modules may be complex, but is still well defined. If it appears a system may have too many relationships between modules in your code, it may be time to have a central point of control, which is where the pattern fits in.

In real-world terms, a mediator encapsulates how disparate modules interact with each other by acting as an intermediary. The pattern also promotes loose coupling by preventing objects from referring to each other explicitly – in our system, this helps to solve our module inter-dependency issues.

What other advantages does it have to offer? Well, mediators allow for actions of each module to vary independently, so it’s extremely flexible. If you’ve previously used the Observer (Pub/Sub) pattern to implement an event broadcast system between the modules in your system, you’ll find mediators relatively easy to understand.

Let’s take a look at a high level view of how modules might interact with a mediator:

Consider modules as publishers and the mediator as both a publisher and subscriber.
Module 1 broadcasts an event notifying the mediator something needs to done.
The mediator captures this message and ‘starts’ the modules needed to complete this task
Module 2 performs the task that Module 1 requires and broadcasts a completion event back to the mediator. In the mean time, Module 3 has also been started by the mediator and is logging results of any notifications passed back from the mediator.

Notice how at no point do any of the modules directly communicate with one another. If Module 3 in the chain were to simply fail or stop functioning, the mediator could hypothetically ‘pause’ the tasks on the other modules, stop and restart Module 3 and then continue working with little to no impact on the system. This level of decoupling is one of the main strengths the pattern has to offer.

To review, the advantages of the mediator are that:

It decouples modules by introducing an intermediary as a central point of control.It allows modules to broadcast or listen for messages without being concerned with the rest of the system. Messages can be handled by any number of modules at once.

It is typically significantly more easy to add or remove features to systems which are loosely coupled like this.

And its disadvantages:

By adding a mediator between modules, they must always communicate indirectly. This can cause a very minor performance drop – because of the nature of loose coupling, its difficult to establish how a system might react by only looking at the broadcasts. At the end of the day, tight coupling causes all kinds of headaches and this is one solution.

This is a possible implementation of the mediator pattern based on previous work by @rpflorence

  1. var mediator =(function(){
  2.     var subscribe =function(channel, fn){
  3.         if(!mediator.channels[channel]) mediator.channels[channel]=[];
  4.         mediator.channels[channel].push({ context:this, callback: fn });
  5.         returnthis;
  6.     },
  8.     publish =function(channel){
  9.         if(!mediator.channels[channel])returnfalse;
  10.         var args =Array.prototype.slice.call(arguments,1);
  11.         for(var i =0, l = mediator.channels[channel].length; i < l; i++){
  12.             var subscription = mediator.channels[channel][i];
  13.             subscription.callback.apply(subscription.context, args);
  14.         }
  15.         returnthis;
  16.     };
  18.     return{
  19.         channels:{},
  20.         publish: publish,
  21.         subscribe: subscribe,
  22.         installTo:function(obj){
  23.             obj.subscribe = subscribe;
  24.             obj.publish = publish;
  25.         }
  26.     };
  28. }());

Here are two sample uses of the implementation from above. It’s effectively managed publish/subscribe:

  1. //Pub/sub on a centralized mediator
  3. mediator.name =“tim”;
  4. mediator.subscribe(‘nameChange’,function(arg){
  5.         console.log(this.name);
  6.         this.name = arg;
  7.         console.log(this.name);
  8. });
  10. mediator.publish(‘nameChange’,‘david’);//tim, david
  13. //Pub/sub via third party mediator
  15. var obj ={ name:‘sam’};
  16. mediator.installTo(obj);
  17. obj.subscribe(‘nameChange’,function(arg){
  18.         console.log(this.name);
  19.         this.name = arg;
  20.         console.log(this.name);
  21. });
  23. obj.publish(‘nameChange’,‘john’);//sam, john

Applying The Facade: Abstraction Of The Core

In the architecture suggested:

A facade serves as an abstraction of the application core which sits between the mediator and our modules – it should ideally be the only other part of the system modules are aware of.

The responsibilities of the abstraction include ensuring a consistent interface to these modules is available at all times. This closely resembles the role of the sandbox controller in the excellent architecture first suggested by Nicholas Zakas.

Components are going to communicate with the mediator through the facade so it needs to be dependable. When I say ‘communicate’, I should clarify that as the facade is an abstraction of the mediator which will be listening out for broadcasts from modules that will be relayed back to the mediator.

In addition to providing an interface to modules, the facade also acts as a security guard, determining which parts of the application a module may access. Components only call their own methods and shouldn’t be able to interface with anything they don’t have permission to. For example, a module may broadcast dataValidationCompletedWriteToDB. The idea of a security check here is to ensure that the module has permissions to request database-write access. What we ideally want to avoid are issues with modules accidentally trying to do something they shouldn’t be.

To review in short, the mediator remains a type of pub/sub manager but is only passed interesting messages once they’ve cleared permission checks by the facade.

Applying the Mediator: The Application Core

The mediator plays the role of the application core. We’ve briefly touched on some of its responsibilities but lets clarify what they are in full.

The core’s primary job is to manage the module lifecycle. When the core detects an interesting message it needs to decide how the application should react – this effectively means deciding whether a module or set of modules needs to be started or stopped.

Once a module has been started, it should ideally execute automatically. It’s not the core’s task to decide whether this should be when the DOM is ready and there’s enough scope in the architecture for modules to make such decisions on their own.

You may be wondering in what circumstance a module might need to be ‘stopped’ – if the application detects that a particular module has failed or is experiencing significant errors, a decision can be made to prevent methods in that module from executing further so that it may be restarted. The goal here is to assist in reducing disruption to the user experience.

In addition, the core should enable adding or removing modules without breaking anything. A typical example of where this may be the case is functionality which may not be available on initial page load, but is dynamically loaded based on expressed user-intent eg. going back to our GMail example, Google could keep the chat widget collapsed by default and only dynamically load in the chat module(s) when a user expresses an interest in using that part of the application. From a performance optimization perspective, this may make sense.

Error management will also be handled by the application core. In addition to modules broadcasting messages of interest they will also broadcast any errors experienced which the core can then react to accordingly (eg. stopping modules, restarting them etc).It’s important that as part of a decoupled architecture there to be enough scope for the introduction of new or better ways of handling or displaying errors to the end user without manually having to change each module. Using publish/subscribe through a mediator allows us to achieve this.

Tying It All Together

  • Modules contain specific pieces of functionality for your application. They publish notifications informing the application whenever something interesting happens – this is their primary concern. As I’ll cover in the FAQs, modules can depend on DOM utility methods, but ideally shouldn’t depend on any other modules in the system. They should not be concerned with:
    • what objects or modules are subscribing to the messages they publish
    • where these objects are based (whether this is on the client or server)
    • how many objects subscribe to notifications

  • The Facade abstracts the core to avoid modules touching it directly. It subscribes to interesting events (from modules) and says ‘Great! What happened? Give me the details!’. It also handles module security by checking to ensure the module broadcasting an event has the necessary permissions to pass such events that can be accepted.

  • The Mediator (Application Core) acts as a ‘Pub/Sub’ manager using the mediator pattern. It’s responsible for module management and starts/stops modules as needed. This is of particular use for dynamic dependency loading and ensuring modules which fail can be centrally restarted as needed.

The result of this architecture is that modules (in most cases) are theoretically no longer dependent on other modules. They can be easily tested and maintained on their own and because of the level of decoupling applied, modules can be picked up and dropped into a new page for use in another project without significant additional effort. They can also be dynamically added or removed without the application falling over.

Beyond Pub/Sub: Automatic Event Registration

As previously mentioned by Michael Mahemoff, when thinking about large-scale JavaScript, it can be of benefit to exploit some of the more dynamic features of the language. You can read more about some of the concerns highlighted on Michael’s G+ page, but I would like to focus on one specifically – automatic event registration (AER).

AER solves the problem of wiring up subscribers to publishers by introducing a pattern which auto-wires based on naming conventions. For example, if a module publishes an event called messageUpdate, anything with a messageUpdate method would be automatically called.

The setup for this pattern involves registering all components which might subscribe to events, registering all events that may be subscribed to and finally for each subscription method in your component-set, binding the event to it. It’s a very interesting approach which is related to the architecture presented in this post, but does come with some interesting challenges.

For example, when working dynamically, objects may be required to register themselves upon creation. Please feel free to check out Michael’s post on AER as he discusses how to handle such issues in more depth.

Frequently Asked Questions

Q: Is it possible to avoid implementing a sandbox or facade altogether?

A: Although the architecture outlined uses a facade to implement security features, it’s entirely possible to get by using a mediator and pub/sub to communicate events of interest throughout the application without it. This lighter version would offer a similar level of decoupling, but ensure you’re comfortable with modules directly touching the application core (mediator) if opting for this variation.

Q: You’ve mentioned modules not having any dependencies. Does this include dependencies such as third party libraries (eg. jQuery?)

A: I’m specifically referring to dependencies on other modules here. What some developers opting for an architecture such as this opt for is actually abstracting utilities common to DOM libraries -eg. one could have a DOM utility class for query selectors which when used returns the result of querying the DOM using jQuery (or, if you switched it out at a later point, Dojo). This way, although modules still query the DOM, they aren’t directly using hardcoded functions from any particular library or toolkit. There’s quite a lot of variation in how this might be achieved, but the takeaway is that ideally core modules shouldn’t depend on other modules if opting for this architecture.

You’ll find that when this is the case it can sometimes be more easy to get a complete module from one project working in another with little extra effort. I should make it clear that I fully agree that it can sometimes be significantly more sensible for modules to extend or use other modules for part of their functionality, however bear in mind that this can in some cases increase the effort required to make such modules ‘liftable’ for other projects.

Q: I’d like to start using this architecture today. Is there any boilerplate code around I can work from?

A: I plan on releasing a free boilerplate pack for this post when time permits, but at the moment, your best bet is probably the ‘Writing Modular JavaScript‘ premium tutorial by Andrew Burgees (for complete disclosure, this is a referral link as any credits received are re-invested into reviewing material before I recommend it to others). Andrew’s pack includes a screencast and code and covers most of the main concepts outlined in this post but opts for calling the facade a ‘sandbox’, as per Zakas. There’s some discussion regarding just how DOM library abstraction should be ideally implemented in such an architecture – similar to my answer for the second question, Andrew opts for some interesting patterns on generalizing query selectors so that at most, switching libraries is a change that can be made in a few short lines. I’m not saying this is the right or best way to go about this, but it’s an approach I personally also use.

Q: If the modules need to directly communicate with the core, is this possible?

A: As Zakas has previously hinted, there’s technically no reason why modules shouldn’t be able to access the core but this is more of a best practice than anything. If you want to strictly stick to this architecture you’ll need to follow the rules defined or opt for a looser architecture as per the answer to the first question.

copied http://addyosmani.com/largescalejavascript/

hard disk failure

No power. No Spin. No Sound. Burning smell. PCB is extremely hot. See burned mark on the printed circuit board. Power surge. Connect the hard drive case to wrong power supply. Hard drive makes “Clicking sound”.
Symptoms of PCB failure and the percentage
the following symptoms of a media failure are may be results of failed circuit board component (PCB). Percent that follows the description represents the chance of the failed PCB..
1. Hard drive does not spin up. Either no sound or a short, quiet tickling sound can be heard. (80 percent)
2. Hard drive does not spin up. Buzzing sound can be heard. (20 percent)
3. Hard drive spins up fine, but is not recognized in device manager or in BIOS. Hard drive recognized with default manufacturer value (like 0GB capacity, wrong SN or Model). (30percent)
4. There is a burning smell can be sensed coming from a hard drive or circuit board. (95percent)
5. You can see a burned component on the hard drive circuit board. (95percent)
6. Hard drive makes a clicking sound. (20percent)
7. Extremely slow reading. (10percent)
8. Hard drive, after some time disappears from Device Manager and from the system. May also be running very hot. (40percent)
How to inspect the PCB board
1. Inspect the circuit board for signs of visible physical damage. Watch for components that appear to be cracked or burned. Turn the board over and ensure that the conducting traces on the bottom of the board don’t appear scratched or cracked. Check all silver solder blobs to ensure that none have been damaged or come off of the board.
2. Turn off the power supply to the circuit board. Use a multimeter set to the resistance or ohms setting to test the connection between the positive and negative connections from the power supply; if there is continuity between these two points the power supply may be shorted out.
Symptoms of disk assembly failure:The computer can see the file, but cannot copy the file.The computer can see the hard drive, but the size is wrongThe hard drive does not spin but make “beeping sound”If the hard drive failure is because of Falling to the floor, most likely the PCB board is ok, but the disk assembly is damaged.Seagate 7200.11-12 firmware brick problem:
spin nicely, but No see in BIOS. Disk size shown as 0 MB.
hopefully this helps !

ساعلم ابنائي

ساعلم ابنائي

ان اللحيه من الاسلام وسنه من النبي العدنان
وانها للمسلم عنوان و امر من الله لخير الانام
وكيف القاه ان كانت فرضا وخالفناه

وان الحجاب فرض من الرحمن وعفة وايمان فكيف فسرناه
وان النقاب ان كان فرضا وفيناه
وان كان سنة اتبعناه
وكيف لا وهو عن امهات المؤمنين اقتفيناه
وبامر بالضرب علي الجيوب التزمناه
وكيف القاه ان كان فرضا وخالفناه

وانه من احسن قولا ممن دعا الي الله
وعمل صالحا وبين للمسلمين انتماه
هنيئا لمن ورث الانبياء واختاره الله واصطفاه
الا بني فادرك فضل الدعاه واهميتهم في الحياه
وان اهل السماء والارض يستغفرون للدعاه

وان الذين يؤذون المؤمنون فقد احتملو اثما وبهتان كيف تجرانا وحملناه
وان للدين اصول لا تتركك مع الفروع وانه جمله اخذناه
ليس كما نهوي انتقينا وما كرهنا تركناه
نستغفرك ربي ان نكون ممن كره جزءا مما اُمرناه

ولا تنسي بني ان الدين معامله وحياه

محمد شرف

why i choose one big table for the same sets of data

– Seek time with proper index, should be exactly the same, no matter how wide”table is.
Apart from performance issues,
this will simplify design and relations with other tables (foreign keys etc).
– code little

Assuming that table B is exact copy of table A, and that all necessary indexes are created, self-join of table A should be a bit faster than join of B with A simply because data from table A and its indexes can be reused from cache in order to perform self-join (this may also implicitly give more memory for self-join, and more rows will fit into working buffers).

If table B is not the same, then it is impossible to compare.

– one more thing wordpress , drupal and joomla uses the same way

db mysql youtube speed performance

Paul Tuckfield of YouTube is giving a keynote. He has been sysadmin for 25 years, Oracle DBA for 15 years and MySQL DBA for 8 months.

For them, MySQL is one (important) piece of scalability. Top reasons for YouTube Scalability includes drinking :).

Top reasons for YouTUbe database scalability include Python, Memcache and MySQL replication. WHat you can do with Python, you can’t do with other scripting languages. The fastest query on the database is that is never sent to the database.

They were doing click tracking on separate MyISAM whereas the read/write was being done on InnoDB. Everything was being written to one master and then being replicated to slaves. There were far fewer writes than reads. Every time replication slows down, you are screwed. Keeping replication going fast was hard.

They had a lot of replicas. One of the things that worked was separating pages for replicas. If you let entire pages built from a replica then there is cache affinity. The most important page is the watch page on YouTube. Most people coming to watch the video and if one thing they could make fast, it was that.

Standardizing on db boxes (they still crash daily)
4x2GHz Opteron core
12x10Krpm scsi
LSI hardware raid 10

Not fun to see corrupt in logs. InnoDB doesn’t recover very fast. When it crashed they had so much in cache that it would take hours to recover. Master for them was a SPOF. How many backups can you restore in 8-10 hours? When you went to switch replicas there are a lot of questions to answer. Did you test the backup? No body does. Are the binlog coordinates correct? They had to promote replicas to master.

Database Failures:
Replication was key to trying different hardware permutations. They got good at re-parenting;/promoting replicas really fast. They they built up ways to clone databases as fast as possible (shutdown and sync) which would make site run again.

Lessons: Go replication. It is an excellent way to test tuning changes or fixes. You can create pools for QOS and cache affinity. You can get good at creating/recreating replicas fast. Get good at re-parenting replicas. Keep intentional lag / stemcell replicas. It stops SQL thread to keep it hours behind. Four kids is when people start saying “You got four kids?”

Trying out 5.0 “mystery cache hits”
They introduced a 5.0 slave for 4.1 master. One of the 5.0 slave was going really fast and other replicas were lacking. they were surprised. If you dump and load in 5.0, you will get more performance than just upgrading because of the compact low format in 5.0. So don’t just upgrade but be sure to dump and load again for 5.0.

Top Heavy:
2×1.5GHz CPU
512M RAM
2x300G disk

Dream upgrade: (Put as much as memory as possible and put as many spindles as possible. Doesn’t matter how fast. The number is really important.
4x4GHz cpu
16G Ram
10x10kRPM disk

If your db fits in ram, that’s great otherwise here is what he says:
cache is king
writes should be cache by raid controller (buffered really) not the OS
only the db should cache reads (not raid and not even linux buffer cache).

Filesystem shouldn’t cache reads, only db should cache it. Hit in db cache means lower caches went unused. Miss in db cache can only miss in other caches since they’re smaller. Keep FS cache smaller than DB cache. RAID cache is even smaller. DB Cache is more useful. Read caching is pointless. Caching reads is worse than useless: they often serialize behind writes. Do not cache reads in the RAID controller.

Write engages 2 spindles (mirrors).

Don’t cache in linux
vm.swappiness = 1-5

Cache in InnoDB

He is concerned about mystery serialization.

Writes are parallel to master but serialized to replicas. They need true horizontal scaling. Starting to run out of bandwidth on master. They want true independent master each with their own set of replicas. How is it going to work? How do you move users from masters? You may screwed up and corrupt it.

EMD = Extreme Makeover: Database (Even more databases :)) codename for horizontal scaling

Slave exacts must serialize to preserve commit order.

oracle (lower case) caching algorithm to predict the future. Manually caching tables by doing selects. Fetch blocks into cache for changes. Replication get great cache hits by using mysqlbinlog + python (it got 3-4 times faster).

“The Diamond”: For golive, play shards binlogs back to original master for fallback.

Some questions I have been wanting to ask Paul:
How many total database servers?
How many Memcache servers?
Read-ahead script for replication?
How many concurrent threads?