What are the risks of index investing?

When it comes to game changing developments in finances, the advent of low-cost index funds has to be up there.

Jack Bogle of Vanguard is largely credited with being the creator of index investing and he’s on the Mount Rushmore of personal finance, as far as I’m concerned.

Index funds offer a very compelling sales pitch:

• Instant investment in the stock market (whatever the fund represents)
• Exceptionally low fees (often less than 0.05%)
• Low maintenance

But are there risks to index funds? Are there any reasons why we shouldn’t be using index funds?

Today, I want to play a little Devil’s Advocate and point out some of the potential risks of index investing:

Index funds do not replace financial planning

When you read personal finance advice, you’ll often hear experts say “invest in low cost index funds.” Focus on how much you’re saving and then invest the surplus in index funds.

While technically correct, it glosses over a key aspect of your finances – your financial plan.

It’s important that you sit down and come up with a financial plan. You can do this with a fee-only financial planner or start one on your own. The basic gist is that you figure out your future financial goals and then build a savings and investment plan to help you achieve it.

From there, you can save and invest towards those longer term goals using investments such as index funds.

High concentration in technology

When experts talk about investing in index funds, they’re most often talking about the S&P 500 index. It’s a popular one because it offers a nice balance of risk and returns.

But it doesn’t give you the entire stock market (that would be a total market index fund). There are risks to be aware of in any index but this is what you need to know about the S&P 500.

The S&P 500 is a market cap weighted index, which means companies with larger market caps will make up a larger percentage of the index. When you have the likes of Microsoft, Apple, Google, and other tech giants worth billions… you end up with an index that is tech heavy.

How tech heavy? YCharts has a list of the SPY holdings and the first five companies are technology – Apple, Microsoft, Amazon, NVIDIA, and Alphabet (Google). SPY is the SPDR S&P 500 ETF Trust.

This graphical illustration of the S&P 500 Index from Visual Capitalist can give you a sense of how concentrated the index is in technology (it’s from July 2020):

That chart is almost three years old and we know a lot has changed within the market in that time but the general trends are similar.

A Financial Times article from April 25th, 2023 discussed how the S&P 500’s performance this year is largely due to technology companies involved in AI (large language models). These “six LLM Innovation companies MSFT, GOOGL, AMZN, META, NVDA, CRM explain 53% of S&P 500 performance.” Also, Microsoft and Apple account for 13.4% of the S&P 500 index!

The risk with the S&P 500 is that you’re getting a lot of explore to technology. One could argue that while this is a risk, due to the concentration, it’s also a strength because technology performs quite well.

You can’t beat the market

By definition, you can’t beat the market with an index fund.

But you can’t underperform either.

🤣

You lose some control on taxes

Index funds lack some tax advantages you have when you invest in individual companies. For example, you are unable to harvest tax losses from individual companies because you own shares of a fund, rather than of the underlying equities.

You can still harvest losses of the fund itself but if you are regularly contributing (and buying shares of the index fund), you’ll run afoul of wash sale rules if you aren’t careful.

Also, you are responsible for the net gains from the funds. Sometimes they’re required to buy and sell during the year, such as with an index change, and those are taxable events that get passed onto you. When you control your own investments, you can decide when to buy and sell.

With index funds, the manager decides (or they are forced to, such as if a lot of people take money out of the fund) and you may find yourself with a little bit of a tax surprise at the end of the year. This is a risk of all mutual funds though, not just index funds, but index funds are less likely to do this.

You may get lazy

The beauty of index investing is that you can set it and forget it.

Except you can’t completely forget it. A lot of your investing decisions are taken care of but you still need to stay on top of them.

You need to rebalance from time to time (once or twice a year) to make sure your allocations match your target. You also have to reassess your allocations every few years, to make sure they’re aligned with your goals.

But it can become very easy to fall into the trap of “Ok, I’m invested, now I can leave it alone forever.”

This risk is easy to mitigate with reminders to check in on your investments periodically. I perform a cursory review my investments each month (to track our net worth) and then a closer look each quarter to see if there are any adjustments we need to make.

Not all index funds are created equal

Vanguard and Fidelity has shown us that low cost index funds are available to everyone. There’s just one problem… not all index funds are low cost.

The Rydex S&P 500 Fund Class H (RYSPX) is an index fund that seeks to track the performance of the S&P 500 index.

OK great… except it charges an expense ratio of 1.56%!

Are you kidding me?

• Vanguard’s S&P 500 ETF (VOO) charges only 0.03%.
• Fidelity’s 500 Index Fund (FXAIX) charges 0.015%.
• Charles Schwab’s S&P 500 Index Fund (SWPPX) charges 0.02%.

Index funds are great but don’t assume that because it’s an index fund, it’s low cost.

Double check the prospectus. There are people invested with Rydex and paying 100x more than the folks at Fidelity. FOR NO REASON.

Index funds could get TOO big

This is not a risk for you personally but for the market as a whole – what happens if index funds are too big? This story in the Atlantic from 2021 took a look into the idea that index funds might wield too much power. Back then, index funds control 20-30% of the American stock market.

When companies are added to an index, as was the case recently with Tesla, it has a big impact on the stock price. Does that mean index funds are over-valued simply because they’re in an index? Does that mean fund managers are overpaying and returns will be muted in the future? It’s hard to say.

Are index funds worth it?

Yes. 100%.

Index funds are a fantastic investment option but they are not a panacea. When you determine your asset allocation, you have to think carefully about the role index funds will play in building your portfolio. While you can accomplish a lot with something simple, like the three-fund portfolio, it still requires some care and attention.

Also, as you age, your needs will change. You may be comfortable using a 120 minus your age allocation when you’re 25 or 30, but as you get older that simplicity may not be as attractive. You may accumulate more wealth and not be comfortable with it all being in the market. You may wish to add other asset classes to the mix or other hedges based on your needs and goals. An index may not be appropriate.

Index funds are a good tool in the financial toolkit but they’re not the only one.