When I described my drawdown strategy in early retirement, I stated we will “sell from the taxable account first.” I also plan to collect quarterly dividends from my index funds, but if Vanguard were Burger King and I could have it my way, I’d hold the dividends and take bites out of my account only as needed.

When I discuss the downsides of dividends, note that I am only referring to receiving dividends in a plain old brokerage account, a.k.a a taxable account. Anything that happens in a tax-advantaged account like a traditional or Roth IRA, 401(k) or similar with regards to dividends, capital gains, buying or selling occurs without tax repercussions. In a taxable account, one must be cognizant of the consequences with regards to taxation.

I love dividends!

Well, not exactly, but I do tilt toward dividend-paying stocks in my portfolio. I simply do so in my Roth IRA. Value stocks tend to be more likely to pay a dividend as compared to growth stocks, and I own Vanguard’s small cap and mid cap indexes in my Roth IRA. I also own the REIT index fund in Roth, which pays a higher dividend (trailing twelve-month yield of 3.76%) than many dividend-focused ETFs and mutual funds.

I’m an index fund investor. Those index funds actually pay dividends. In fact, when you compare the S&P 500 ETF  to the ProShares Dividend Darlings ETF, you’ll see remarkably similar dividend yields of 1.94% and 1.96% respectively. You’ll also see very similar returns. The biggest difference is in the fees, with the Proshares ETF costing investors nearly nine times as much (0.04% versus 0.35%). That’s an argument for buying the individual stocks rather than the ETF, I suppose.

It is fun to see the dividend income grow along with your portfolio. Truly passive income without so much as lifting a finger.

Dividends are one component of total return from an investment (the other being capital appreciation), and I like a good total return. I certainly wouldn’t want to give up a portion of my returns. If companies are paying dividends, I’ll certainly take them.

Dividend investors share data that show dividend payers have higher total returns than the total stock market, while some studies show the exact opposite.

While I believe it depends on the timeframe you choose to analyze, and that survivorship and hindsight bias play a role, I also believe that any edge (a.k.a. alpha) that exists rapidly disappears in a reasonably efficient market, and I’m content with the return of the indexes. We may also be seeing a value premium, a term that describes the slightly better performance of value stocks as compared to growth stocks that has persisted over long stretches of time and diverse geography.

I hate dividends!

Dividends create tax drag on my portfolio. Over the course of the year, the index funds I own will kick out roughly 2% in qualified dividends. I not only pay the 15% capital gains tax on those, but also a state income tax of 9.85% and the NIIT (ACA) surtax of 3.8% for a total of 28.65%.

With a 2% dividend, that’s  a tax drag of $573 for every $100,000 invested. I can expect to pay over $7,000 in taxes based on those dividends this year.

If I made 7-figures in California, my qualified dividends would be taxed at about 36% or $720 annually for every $100,000 invested in a taxable account with an average 2% qualified dividend.

And don’t get me started on ordinary (non-qualified) dividends. Those are taxed at your marginal tax rate, or somewhere between 40% and 50% or more for many high-income households. Last year, all of Vanguard’s S&P 500 fund’s dividends were qualified, but about 7% of the Total Stock Market Fund and nearly 25% of my international funds’ dividends were taxed at that lofty ordinary income tax rate.

An ideal allocation in a taxable account would be broadly diversified and pay as little dividend as possible. Vanguard has tax managed funds which attempt to do this, but with higher fees than the corresponding index funds that may negate some or all of the benefit. Additionally, tax loss harvesting becomes difficult when you don’t have suitable partners that are similarly tax-managed.

I’ve written about my favorite stock, Berkshire Hathaway, and the fact that it has not only performed very well for decades, but also has the benefit of zero tax drag because it pays no dividends. There are rumblings that this wonderful feature could be going away at some point. I would view a dividend from my BRK.B as an unwelcome change.

Why selling shares is better

When you invest in a taxable account, you should plan to buy and hold. But the day will come when you want to spend some of that money. You may access that money to make a big purchase during your working years, in which case you’ll want to sell the shares with the smallest percentage gains, as I recently did.

The most common scenario for my audience, however, is living off the money in a taxable account in early retirement.

Would you rather own stocks and funds that automatically pay you enough in dividends to cover your spending needs or own funds that pay little to no dividend, forcing you to sell shares?

Let me explain why you I much prefer the latter, assuming the total return of the two options is equal.

When you sell shares, you get to decide which shares to sell. Be sure to set your “cost basis” to “Specific ID” or the equivalent. Having purchased shares over a number of years, you may have some shares that have quadrupled in value and others that have only appreciated 10% or not at all. If you’re selling shares to “create your own dividend,” you can also determine the tax impact of acquiring those dollars.

If you’re in danger of being bumped into a higher tax bracket, particularly the 25% federal income tax bracket in which capital gains taxes are levied, you can sell those shares that will give you very little capital gains. If you’re well below any thresholds of increased taxation, it may be advantageous to sell some shares that have seen tremendous gains in order to lower your potential future tax burden. This is similar to tax gain harvesting except you use the proceeds to live rather than to buy shares of a similar fund.

You can also decide when to sell shares. If you know you’ll need money in the winter, you get to decide not only how much of a tax impact your withdrawal will have, but whether or not it will occur in the current tax year (December) or the next one (January).

When you receive a dividend, you have no control over the size of the dividend or the timing of receiving it, and 100% of the money you receive will be considered income, whereas when you sell shares, only the difference in cost from your purchase price may be subject to taxes.

To better illustrate this concept, I’ll contrast selling shares that have seen various returns with receiving a dividend. You’ll see the same amount of money land in your bank account in all cases, but only the portion in red will be subject to taxes. When receiving a dividend, the whole enchilada will be subject to taxes.*

If you find yourself in the 25% federal income tax bracket and live in a state with an average sized state income tax of 5%, you can expect to pay 20% in taxes on dividends and long-term capital gains. In this scenario, how much tax will you pay to access $10,000?

• If it comes all in the form of dividends, $2,000
• If you sell shares that have quadrupled? $1,500
• If you sell shares that have doubled ? $1,000
• If you sell shares that have appreciated 10%? $181
• If you sell shares with no gain? $0
• If you’re stuck selling for a loss, you can earn a tax deduction when you collect the money from that sale.

* Of course, it is possible to keep income low enough to avoid taxes on dividends and long-term capital gains. Live in a state with no income tax and you may avoid taxes on dividends entirely. But numerous income streams can foil those plans, sending your taxable income north of the magic number of about $75,000 for couples (half that for singles) including Social Security Income, withdrawals from tax deferred accounts, unanticipated blog income, and yes, dividends themselves. The tax code could also change, and the tax-free dividends in the 10% and 15% federal income tax brackets could disappear.

Should one avoid dividends at all costs?

Dividends are not going to sink your portfolio, they’re simply suboptimal in a taxable investing account, and we want to do all we can to optimize. In my portfolio, I see a tax drag of about 0.6% of the portfolio annually with taxes on dividends of about 30% on a 2% dividend.

How could I improve this? I could continue to invest more in Berkshire Hathaway and other companies that have not been paying dividends. I could sell what I’ve got and start from scratch, but that would come with disastrous tax consequences, so there’s no way I’m actually entertaining such an idea.

What’s an investor to do?

For one, don’t focus on dividend-paying stocks or funds in a taxable account. If you have no interest in tax loss harvesting, consider tax-managed funds. If you’re really keen on avoiding dividends and willing to forego some diversification, load up on BRK.B and other no-dividend payers in taxable. If you choose to slice and dice, keep your higher dividend stocks and funds tucked away in a tax-advantaged account.

Recognize that avoiding dividends does come at a cost, even in your taxable account. In 2016 alone, I was able to take $46,000 in paper losses by tax loss harvesting (TLH). That’s good for a $1,300 to $1,400 per year tax break for the next 15 years in my current tax bracket. It’s easy to TLH with standard index funds, not so much with tax-managed funds, although it wouldn’t hurt to exchange into one knowing that you may eventually exchange out of it.

What’s worse than focusing on dividends?

Owning a dividend-focused portfolio in a taxable account is less than ideal, but there are many far more egregious sins in the investing world. For example, you could:

• Own actively managed funds in taxable that create far more taxable events due to high turnover.
• Invest haphazardly with no defined plan or goals. Buy high; sell low.
• Keep all of your savings in a savings account, slowly losing buying power to inflation.
• Fail to save enough money to have anything left over to invest.
• Buy loaded funds. You’re automatically down about 5% the first day.
• Trust your money to someone else without knowing or understanding how it’s invested.

“Physician on FIRE” is an anesthesiologist and can be reached at his self-titled site, Physician On FIRE.  

Image credit: Shutterstock.com