We can control most of the variables that will affect our financial futures. We can control our income by choosing to work more hours or in a less desirable location. We can control our spending by buying a smaller house or a less nice car.

However, one thing that we cannot fully control is our investment returns. And the problem is that our investment returns can significantly impact our net worth in retirement.

We all believe that the stock market has a positive expected return (otherwise we wouldn’t invest), but our actual returns can often deviate significantly from the expected return we input into a spreadsheet.

Let’s run a portfolio simulation

To highlight how big of a deal this is, let’s look at a person who saves 100,000 a year for 35 years, and see what their portfolio would look like with 2%, 4%, and 6% annual returns.

You’d have $6 million more to spend in retirement if the stock market gave you a 6% return over your career compared with a 2% return.Now let’s do it with actual historical market returnsOf course, if only the stock market rose the same percentage year-in and year-out. The real stock market is much more volatile, so let’s run the same analysis using actual historical market returns. Using the NYU database of historical S&P 500 returns, we run our portfolio simulation for every rolling 35-year time period from 1928-1962 to 1983-2017.

The final portfolio value ranges from as low as $27.8 million (1940-1974) to as high as $82.2 million (1965-1999). The jaggedness in a lot of the lines reflects the various time periods that contained the 1999 stock market peak.If the numbers seem a little bit high, that’s because I did not include inflation or taxes when running the numbers using historical market returns.Since there are a lot of lines on that graph, let’s look at the histogram of the final portfolio values to get a perspective on the range in possible portfolio values.Luck can overwhelm skill in investing

There is huge variability in your portfolio value in retirement, regardless of how many good investment decisions you make. Just like a bad bounce or an incorrect call by the referee may cause the better team to lose in a soccer or football game, bad luck (especially at the end of a career) can cause someone who has done everything right to end up with far less in retirement than they had planned.

Even with the best cards, you can lose with bad luck.

How do you manage stock market volatility?Assume conservative market returns in financial planning

The old adage goes, prepare for the worst and hope for the best. In our own financial plan, we assume a 2% (after-inflation, post-tax) return and build the plan designed to succeed in that market environment. If we have better returns like 6% or 8%, then we will have some extra spending money for charity or other luxury items.

Hold a diversified portfolio

The best way to reduce your risk is to hold a diversified portfolio. This is best done through the use of index funds such as the S&P 500 or Total Stock Market index.

Diversification is one the few free lunches in investing, and using low-cost index funds gets you the most diversification in the most efficient way possible. You can build a diversified portfolio with individual stocks, but this typically requires at least 30 stocks.

Be flexible in when you retire

If you’re able to work a few extra years in the event the market does poorly in the last few years of your career, then you can weather a period of poor market returns.

Be flexible in your retirement needs

If you’re able to live on less, then you can retire when you want even if you don’t end up with as much in your nest egg as you had planned. If you have great stock market returns, then you’ll be able to enjoy a few extra luxuries in retirement.

If luck goes your way, you might be able to enjoy vacations in the Maldives during retirement.

Optimize the parts of your investment returns that you can control

Your returns aren’t completely out of your control. You can improve your returns by minimizing the fees in your accounts. You can invest in a tax-efficient manner. You can control and minimize the amount of trading in your accounts. You can maximize your contributions to retirement accounts. The combination of these factors can boost your returns by up to 3% or 4% a year, which can be the difference between an adequate retirement and an awesome retirement.

Conclusion

Even if you do everything right, stock market volatility could cause you end up with significantly more or less than you expected in retirement. Plan accordingly, and enjoy the extra millions if you get lucky with above-average market returns.

What do you think? What do you expect your stock market returns to be? What would you do if you end up with way more than you need for retirement?

“Wall Street Physician,” a former Wall Street derivatives trader , is a physician who blogs at his self-titled site, the Wall Street Physician.

Image credit: Shutterstock.com