Both of these firearms have pluses and minuses. Both platforms have seen extensive use in combat.
The AR15 has been in military service for several decades and is becoming more and more popular in law enforcement.
The shotgun devastated enemy troops in close quarters battle through both world wars, and worked particularly well in trench warfare. It continues to see use mostly in "specialized" circumstances with our troops. It is even more common in law enforcement, it's practically impossible to see a police cruiser anywhere in the US without a shotgun inside.
There are several factors to consider when deciding between the two weapons. One may even be, "which firearm if any, are near you?" I would more than likely grab the weapon closest to me in many cases, be it a handgun or a long gun, knowing that time is almost surely a greater factor to consider than caliber or platform.
That said, I prefer the AR15 and will make my case for it here.
One of the common misconceptions about the shotgun is that it is great in CQB because you can just aim in the general direction of the enemy and take them out with ease. The truth is, at CQB ranges, the patterns are still very very tight and you do in fact have to aim every shot.
Also, assuming you didn't have to aim because of the spread you would likely lose quite a bit of that power. A shotguns power doesn't comes from the effectiveness of one single pellet, but from the combined effect of 9 pellets hitting the target together. Almost like a .32 auto hitting you 9 times all at once.
However, it is a moot point since the patterns are tight and you do need to aim.
The AR15 shoots faster, has greater capacity (both of those help when engaging multiple targets), and fires more accurately. Also, my AR15 carbine is shorter than my HD shotgun making it better for tighter spaces.
Not as good stopping power shot for shot with the shotgun, but plenty more when compared to a pistol, the 5.56 round is no pushover.
Also, the 5.56 can defeat most commonly worn types of concealable and handgun rated body armor better than a shotgun or a pistol.
For many years police agencies stayed away from the AR15 for CQB/Entry, etc... thinking that the collateral damage of the .223/5.56 round was worse than that from a shotgun, pistol, or SMG. Now you have agencies like the FBI who are slowly doing away with the MP5's and going to the AR15 carbines.
FBI studies have indicated that over penetration at CQB ranges have in the past been exaggerated in 5.56/.223 and it penetrates less in human tissue than most common pistol rounds (including JHP's) on average.
Another thing for me to consider is the amount of trigger time and training I have with an AR15 compared to a shotgun. I am easily most proficient with my AR15. You must go with the weapon you have the most training and proficiency with, so consider that on a personal basis first and foremost.
Lastly, if the engagement takes you outside, you are equipped to handle targets at ranges far past typical CQB ranges.
I think a shotgun is a great HD weapon, but I wouldn't trade one for my AR15. To read more on the AR15 visit my AR15 blog.
That said, either will do. It's highly unlikely that either weapon would give you a significant advantage in home defense as it is almost always the way it is used that determines the outcome.