#readthedocs

It seems like after I build a version of my docs the search returns worse results. Is there a way to trigger it to build a new full text index right away?

begriffs: regarding your previous question, that's exactly the behavior of tags?

if you tag HEAD, and advance HEAD...

(i.e. commit something on top of HEAD)

untitaker: thanks for getting back to me about that. I discovered that I under Admin -> Advanced Settings there is a field to configure the version that latest points at. We're using long-lived branches for each version and don't work on master directly.

So that part worked fine.

oh, you mean for each x.y-release, not each concrete x.y.z version?

But the poor search performance is strange. I remember it was pretty bad at one point but then improved over time like it was being index. Now it's bad again.

I have no idea about the search index (not affiliated with RTD)

begriffs: note that "default version" only defines what / will redirect to

Actually what we're doing now is exposing two versions v5.0 and v5.1 and hiding stable and latest. Then using cloudflare to do a 302 redirect from latest to v5.1 for SEO.

Actually discovered another interesting thing about search: it works well unless we access the page behind a custom subdomain. Same RTD server, just different domain in the url and search differs.

could you share a link?

Yeah just a sec, verifying the behavior again

begriffs: there's an issue with custom subdomain usage and search, the api endpoints aren't exposed on the subdomain and we disallow the queries from unknown domains

agj: is there a setting I need to change? Is it a CORS thing?

CORS on our end, yup

Is there a way to whitelist our particular RTD instance and its CNAME?

wait, but this seems to work (judging from the dev console): https://vdirsyncer.pimutils.org/en/stable/searc...

untitaker: i get the expected warning about the origin being denied

oh, nvm

untitaker: I think it falls back to a front-end search, and you get fewer results than the index. I also see the telltale raw markup in the results

since the search JS hits the custom domain before the redirect, you might be able to intercept the request and edit the CORS headers yourself.

unless you're pointing your CNAME directly to RTD

Cool, our domain is behind cloudflare...I wonder what kind of rule we'll need to implement to change CORS.

or no, it actually *does* just hit readthedocs.org directly.

weird, I thought I saw something else before

correct, search js will fall back to sphinx search on failure, explaining the poor results

ultimately, we need to serve the same search on the custom domain to get around this in a secure way

you could also only whitelist the primary hostname of each project.

If I call search?project=vdirsyncer, vdirsyncer.pimutils.org is the primary hostname, therefore Access-Control-Allow-Origin should be set to that.

or check if the Origin request header matches a known hostname for the given project.

untitaker: this is a very good point

that said I don't see the security risk in allowing all hosts.

at least for search

Looks like the API CORS problem is documented here https://github.com/rtfd/readthedocs.org/issues/...

untitaker: points of our api have side effects, and so we blanket limit access to the api to protect against csrf attacks

more security is better than less, and all that

ultimately, we'd rather the api endpoints are hosted on the project domains, but that will take a little of operational change on the community site, and a large amount on the commerical site

It doesn't look like an insurmountable problem though. Inside Admin -> Domains it knows which domains are legit for my docs page, so it could use that value for the Access-Control-Allow-Origin response header.

yup, i think that's a great idea that does solve a few problems around api usage from docs on both

we didn't use to have the domain mappings, rtd only tried to make an intelligent guess about the domain

so that was never added to that api

but now we do have explicit domains to map to projects, so that makes this all possible

agj: is the relevant API code open source?

(Also one workaround might be to allow all hosts to access certain safe routes but keep the strict cors on other more dangerous routes)

begriffs: i can outline the work in an issue, but the bulk of the work will be addings the cors headers on OPTIONS, via the cors middleware

ie https://github.com/ottoyiu/django-cors-headers/...