Why does TalkYard's robot.txt exclude user profile pages?

Is that intentional? Those seem like reasonable pages for Google to index.

Yes it is intentional — although maybe it's better to change this no-index thing, or make it optional.

I was thinking as follows: Somewhat often, people reuse the same usernames, across different places on the Internet. Then, one can websearch for someone's username, and find many places that person has joined — but maybe some people don't like this?

Maybe they'd want their user profile pages, to be not-indexed by the search engines, for privacy reasons.

So that's why it's not indexed. (Other things below /-/ is API endpoints and admin/moderation things, and shouldn't be indexed b.t.w.)

At the same time, if one posts a reply, then one's username appears next to the reply, anyway. And can be found via a websearch anyway.

Now I'm thinking:

• Maybe it's better that, by default, the user profile pages get search-engine-indexed. And admins could disable indexing, and individual members could disable indexing for themselves.

• If disabled, I thought that maybe there was a way to tell Googlebot to not index one's username, when it appears next to one's replies,
  however seems that's not possible:
  https://stackoverflow.com/questions/14314111/avoid-crawling-part-of-a-page-with-googleoff-and-googleon

  [...] googleon" and "googleoff" are only supported by the Google Search Appliance (when you host your own search results, [...]

  Seems Googlebot indexes all, or nothing.

• Maybe instead one could choose to display only one's first name & avatar, but not one's username, next to one's replies. Then, only one's first name got indexed (and first names is pretty pointless to websearch for). But maybe then it's better if people simply choose different usernames.

Actually no one has asked about their profile pages and search engines, so I suspect that it's ok to index profile pages, by default.

they're indexing pages that are disallowed by robots.txt (weird that they're ignoring the robots.txt, but anyway...).

Yes that was surprising. Looking in the "Affected pages" graph, seems they started doing this recently? Hmm.

Anyway, for now, I can add a <meta name="robots" content="noindex" /> tag — and, as mentioned in a reply to your Twitter tweet, then they also need to be removed from robots.txt:

For the noindex directive to be effective, the page must not be blocked by a robots.txt file
https://developers.google.com/search/docs/advanced/crawling/block-indexing

***

Apparently Googlebot can index robots.txt-disallowed pages, if they're linked to: (which sounds a bit weird to me)

https://www.searchenginejournal.com/google-pages-blocked-robots-txt-will-get-indexed-theyre-linked/255911/

if these pages are blocked by robots.txt, then it could theoretically happen that someone randomly links to one of these pages. And if they do that then it could happen that we index this URL without any content because its blocked by robots.txt. So we wouldn’t know that you don’t want to have these pages actually indexed.

Whereas if they’re not blocked by robots.txt you can put a noindex meta tag on those pages ... then we would know that these pages don’t need to be indexed and we can just skip them

says John Mueller, Senior Webmaster Trends Analyst at Google.

***

( There was a Noindex: ... instruction one could add in robots.txt, but it's been deprecated — the meta tag is recommended instead:

the use of noindex within robots.txt will no longer be supported by Google.
Gary Illyes [Google Webmaster Trends Analyst at Google] explained that after running analysis around the use of noindex in robots.txt files, Google found “the number of sites that were hurting themselves was very high.”

https://www.deepcrawl.com/blog/best-practice/robots-txt-noindex-the-best-kept-secret-in-seo/ )

Status update: In the upcoming version (in some days), there'll be a <meta name="robots" content="noindex" /> tag, so the Google Search Console warnings should go away.

Later, in the next-next? version, I'd like to add a per user and per group (incl the Everyone group) setting, so one can configure the search engine visibility, so user profile pages can get indexed.

And also, because I need this myself, settings for making group membership invisible, so one cannot see if someone is a member of a "hidden group", or that the group even exists.
And settings for making group members invisible to others in the forum.
(I'm wondering if it's not-so-easy to guess what the background for these invisible group and members is :- ))