Substack stopped giving this feature to new publications (Sitemaps on Substack)
I found about an important feature that Substack stopped giving to new Substack writers.
[Note: This post is too long for email, so some content may not appear. Read this article in a browser or Substack app, for a better reading experience].
Hello!
I had a different topic planned for today, but I discovered something important about Substack, and I wanted everyone to know about it, so I'm writing this article.
The Sitemap in my Substack website has disappeared??
A few days ago, I decided to tweak some things in my Substack to improve it. While I was at it, I also changed the layout of my website/publication using the customization options that Substack provides.
After I was done, I opened my website and I noticed that– a small section that was at the bottom of the page had disappeared?? It looked like this:
This little section had links to my newsletter’s Archive, About page, Recommendations, and the Sitemap. (Only the links to those webpages disappeared; the pages themselves still exist, and I was able to locate all of them, except for the sitemap).
Now, that section is just gone, and the bottom of my website looks like this:
I looked to see if it was moved to a different section or webpage, but it was nowhere to be found.
Understandably, when it didn't see it reappear after a while, I panicked. Sitemaps are kind of an important feature. If you don't know why that is– don’t worry, I’ll explain.
Spoiler alert: I later found that my sitemap exists, and it’s just invisible. But your website may not have one because of a recent change on Substack that they didn’t tell us about.
What is a Sitemap?
Let’s start from the top. You have to know some basic things about SEO to understand why sitemaps are important.
When you search for something on Google, you get results– links to websites, Webpages, and posts.
These results are pulled from a big database called the “Search Index”.
So for your website and articles to show up on Google search, they need to be ‘indexed’ by Google. (Which means they need to get in Google’s search index).
There are programs called “Web Crawlers” or “Bots” that travel through URLs (or links) to discover websites and posts, and they'll automatically index them for us. The web crawlers of Google are called “Googlebot”.
Sitemaps are useful because they help the web crawlers navigate through your website and discover most of your webpages.
Your website will show up on Google search, even if it doesn't have a sitemap. But not having one puts smaller publications at a disadvantage, so that's why we're looking into this today.
Important notes:
Crawlers won't index every webpage they crawl on. They process the content on the page first and see if it's worthy of indexing (this step is called “Processing”). If they find the content useful, they'll index the page. (Google hasn't told us what the requirements are– for a page or site to get indexed by crawlers).
Another thing to note is that your site or webpage won't immediately get crawled and indexed as soon as you publish it online. It can take anywhere from hours to months until it starts showing up on search.
Sitemaps help crawlers crawl and process faster and more efficiently.
Is the homepage of your website indexed on Google?
To check if your Substack publication is indexed on Google, you can simply search:
“site:yourwebsite.substack.com”
Or search this if you’re using a custom domain:
“site:yourwebsite.com”
My substack’s name is “Harshini writes” and I’m not currently using a custom domain, so
I searched:
"site:harshiniwrites.substack.com"
The main homepage of your website should show up, even if your website doesn't have a sitemap. Mine showed up as you can see in the above image. (If it didn't show up, try reading this comment and see if it helps: Link).
A sitemap’s purpose is to help Google find the webpages and posts that are within your website.
Your website consists of several Webpages and posts. And Google needs to crawl and index all the webpages and posts separately– not just the homepage.
Your website might show up on search, but your articles may not, if they haven't been crawled and indexed by Google.
Having a sitemap on your website will tell Google which pages of your website are important, making sure that Google indexes all of the important ones, including the posts. It will automatically link newer articles you post to your sitemap.
Not having a sitemap isn’t a big deal if your Substack is a medium-sized or bigger publication. Because (later we learn that) Substack automatically gives site-maps to certain publications that meet their quality standards, which they haven’t specified to the public yet.
This official Google article says:
“A sitemap helps search engines discover URLs on your site, but it doesn't guarantee that all the items in your sitemap will be crawled and indexed. However, in most cases, your site will benefit from having a sitemap.”
Mine is a smaller publication, and I value most of my articles. I want more of my articles to get found by people. So yes, if my publication will benefit from having a sitemap, I want one. And assume you want to learn more about this, too, so I’m writing this article.
Why did my Sitemap disappear? (Asking Substack Support)
I decided to contact the Substack Support Bot to see if it knows something. Here’s how it went:
(Of course, my newsletter is set to Public mode. How else will I write to my readers?)
As it suggested, I already waited for a few days and checked back to see if the Sitemap had reappeared. Still nothing.
Back then, sitemaps were automatically given to new Substack publications as soon as they were created. Not anymore, it seems.
Luckily, I made a Substack publication back when this was true, so I had a sitemap from the start.
Now, it seems that there’s a minimum quality requirement that new Substack writers have to meet to get the feature? (At least that’s what the bot says).
Michael Wolf also mentions this change in his article:
“I’ve been back and forth Substack support after I managed to authenticate with Google Tag Manager and discovered no sitemap. They have all been nice, but all say the same thing: promote it on social media or put links on your site, or whatever and get enough traffic and that may lead Google to index. And maybe if you get famous enough, Substack will generate a sitemap.xml and you can get your whole site indexed. But otherwise you’re out of luck.”
This is important to note for later. Here, the bot says that if a sitemap is not visible on our website, it doesn't exist at all.
I also looked at some other Substack publications. Interestingly, the websites that were customized didn't have a visible sitemap or that little section. And the publications that used the default Substack website layout– had that whole section.
After some research, I learned this: If you don't see a sitemap, it either exists in the backend of your website (so it’s invisible) or– it doesn’t exist at all!
That begs the question:
How do we (actually) know if our website has a sitemap?
Thanks to this Reddit post, I learned another way to check if our website has a sitemap. For the sake of simplicity, I call this the “Backend sitemap”.
First, go to your website’s homepage, or type this in your browser: https://Yourwebsite.substack.com (Or search this if you’re using a custom domain: https://Yourwebsite.com)
Example: my Substack publication’s name is “Harshini writes”, so I have to search “harshiniwrites.substack.com”.
Next, add this to the end of your website’s URL: “/sitemap.xml” to find your sitemap. I searched: “harshiniwrites.substack.com/sitemap.xml” to find mine.
If your website has a backend sitemap, you should see something like this:
Or, you should see something like this:
If your website doesn't have a sitemap, this is what you'll see:
How to locate your Sitemap that was generated by Substack?
I didn’t find anyone online mentioning this, but there’s a way to locate the sitemap that was generated for you by Substack.
Well, you still have to check if you have a backend sitemap by doing the previous check. But if you find that you do have one, just add “/sitemap” to the end of your publication’s URL.
And there it is! The neatly arranged sitemap that went missing in the first place:
After looking at several perfectly fine Substacks that didn’t have this sitemap visible on their website, now I’m thinking— The backend sitemap is probably what’s been helping us all along, so I don’t know what to do with this page?
Maybe I’ll just link it on my homepage, because that’s how it was before.
Did Substack really stop giving Sitemaps to new publications?
All clues point to the same thing: Substack has stopped giving sitemaps to new Substack publications, and new writers and bloggers have to meet certain requirements to get a sitemap.
I was curious, so I wanted to test it myself and see if this is true.
I made a new Substack account, finished setting up the basic settings, made a publication named “Demo newsletter”, scrolled to the bottom, and…
Yep! There’s no sitemap.
(Link to the recommendations also doesn’t exist, but that’s because I’m not recommending any Substacks in my new publication yet).
Okay, what if the sitemap exists, but it’s invisible? I added “/sitemap.xml” to the end of the website’s URL to check. And look what shows up:
This proves it. Newer Substack publications truly don’t get a sitemap. And their publication has to grow a bit, to be qualified to get one. (How much do they need to grow? I don’t know. That’s what the Substack Support Bot is refusing to tell us).
Luckily, I created my Substack last year, so my site still has a sitemap. But what about newer publications that were created this year, and they don’t have a sitemap yet? Will they still be able to get found by Google? The answer is yes, but now it’s harder to do than before. That’s what we’ll see next.
How do we get our website to be indexed without a sitemap?
I recently discovered a powerful tool that enables us to index our webpages manually without depending on sitemaps. The tool is called “Google Search Console” or GSC for short.
It’s a time-and-effort consuming process to manually index every article, especially if you have a lot of them.
There was an easier way to connect your Substack to the Google Search Console, but apparently, that option doesn’t exist for newer publications anymore.
Here’s how to check if you have that option: Go to your publication’s settings, and scroll down to a section called “Analytics”. You may see an option named “Google Site Verification.”
As you can see, my main newsletter (the older one), has that option. But the newer publication doesn’t.
If you have it, congratulations! You can connect GSC to your Substack publication, and you can manually index your articles by following the steps in this article by Michael Wolf , or you can follow this article by Colin Gardiner.
If you don’t have that Google Site Verification option, you can still connect to Google Search Console (GSC) by using another tool called “Google Tag Manager” (GTM).
I recommend reading this article by Kristi Koeter , to know how to use GTM to connect your Substack to GSC.
I'm linking some useful articles that might help you, because I haven’t used Google Tag Manager before.
So yeah! I hope this article has helped you.
Side note:
Maybe the Substack team is cutting corners to help small writers in a different way? Maybe Substack publications grow normally even without sitemaps? Or maybe Google set a rule not to give Sitemaps to small websites? (If so, this change makes sense). Who knows? There’s a lot to theorize about, haha.
Related articles I found useful:
Thank you for reading! I hope you liked reading this article, and I hope it was helpful. Come again next week on Friday for a ✨new post✨
- Harshini
thats actually incorrect- maybe for someone who cant validate their account .
Those that can you go directly through google- for those bot spam accounts yes it has gotten more difficult here, and you have access to play with your URL and how you are searched in google which they do not allow spambot suspected accounts access to you must verify the account.
know a little in basic HTML, and SEO options- bam you can actually appear right at the top of google’s engine return right on day 1 unless you took a name that is highly searched and burried.
checked my newsletter all old has sitemap, new one has not. I don't understand substack's these decision. are they not interested in free seo traffic? or do they not want the newsletter get crawled?