#readthedocs

hi

i used the following install guide to install rtd on ubuntu 16.04

the installation is working alright apart from the "build"

the build keep hanging on status "triggering' (web interface), in the logs the only entries are

[08/Feb/2017 07:02:09] INFO [readthedocs.core.views.hooks:295] (Incoming Generic Build) docs [latest]

[08/Feb/2017 07:02:09] INFO [readthedocs.core.views.hooks:55] (Version build) Building docs:latest

but no errors

what is going on? :/

guys, can someone help me with my build issue? :/

would be appreciated , seems that docu is lacking info

hoobaman: please link the actual build

untitaker:thx for your reply, really appreciated

i am testing with an existing project that is working on thr rtd online service

hoobaman: ok and which build fails? https://readthedocs.org/projects/bob-build-tool... indicates they all pass

untitaker:i am using that project on my local rtd install to verify the functionality

but when i import the project and press build, i only see "triggering" and no output

however the django admin console says success :/ but no docs were generated

hmm

how can i debug the build process? the rtd.log only gives me INFO that the task is started

and no errors :/ nothing

hoobaman: I don't think I can help you with that, sorry. I am only using the hosted service.

untitaker: ok , thx anway, your effort is/was appreciated

i'm trying to set up a private RTD instance... i kept getting stuff in the logs with virtualenv failures because virtualenv.py: error: no such option: --no-download

it was using my system python, not the python in the virtualenv i'm running manage.py runserver with. after a long investigation, i finally realized that the python environment builder has the executable hardcoded as 'python', so it seems that it's assuming i'm running it in an activated virtualenv

instead of just using sys.executable or some configurable thing :(

i made a bug https://github.com/rtfd/readthedocs.org/issues/...

cdunklau: wouldn't you just use the docker image?

the issue is the wrong version of virtualenv

you can run rtd off system level packages, but the versions need to match what we've specified in our reqs

agj: i _am_ running it in a virtualenv, it's just not activated

agj: so venv/bin/python manage.py runserver and everything's fine, except since the build step uses the string 'python' for the executable instead of sys.executable, it doesn't work

agj: oh, you put that on the github ticket. i'll answer there

sys.executable also doesn't work because we support multiple versions of python

a patch i just put up also expands this to support arbitrary versions of python, so the interpreter can be python{2.7,3.4,3.5,3.6}, not just the interpreter version of the running instance

agj: sure, sure, my suggested fix doesn't interfere with that

agj: did you merge it or is it a PR? can you give me a link?

agj: oh it's this one isn't it

roger, i might be missing something with your purposed fix then, perhaps a PR would be better

cdunklau: yep, that's the one

agj: okay, so with that PR the core issue is the same: the virtualenv gets created using the specified interpreter and whatever version of virtualenv it has installed

i'd assume, though this is also following our pattern of using docker for encapsulation

agj: instead of the specific version of virtualenv that RTD's requirements pinning specifies

that's up to the docker container for the virtualenv requirement

all other reqs are installed to specific versions

agj: so the fix for that is to use sys.executable (the python binary RTD is running on), -mvirtualenv, -p <python_to_use_in_the_created_env>

agj: oh docker. hmm

okay i know basically nothing about docker apart from "ooh containers pretty"

well, virtualenv is executed on the host system, all other python calls are inside the docker container

aha okay. so this thing https://github.com/rtfd/readthedocs.org/blob/ma...

agj: that's the thing that runs outside of docker, then you do some magical docker stuff to use the env it creates?

ah, `-p` is a virtualenv option, that's the piece i was overlooking

yes

cdunklau: boiling it down, yes

-p tells it what python to use in the env it _creates_

because it doesn't (really) matter which python makes the venv

i've seen some oddball corner cases (hence the (really)) but i'm pretty sure my python install was just fubared

(when i hit the oddball case)

this makes more sense now, though it would definitely only address a limited use case

feel free to put together a PR, we'd want to see that normal operation (activated venv) works, along with unactivated

what do you mean? i have strong doubts most people running private instances use docker builds

agj: sure, will do. The main thing preventing me from doing that was lack of knowledge about the context

and i think i have a better grasp of it now. I won't hack the hardcoded 'python' things in the rest of the module, just the actual virtualenv creation

yeah, that's the only one really

even if you are running outside a venv for the application, venvs will be created/used for builds

regardless of docker usage or not

disclaimer: "i think"

we don't test the local (non-docker) build pattern anymore

oh wait.

agj: this agrees with my observations

agj: it certainly tried to create a virtualenv locally

sorry, i'm not with it, this damn cold has me all messed up

the virtualenv call is in the docker container

so after i figured the issue out and ran it with an activated virtualenv, it broke in other ways that i didn't have time to look into. but i recall it was complaining about something with elasticsearch so it was probably just something i hadn't set up

and the purposed fix will likely break things, as this binary isn't guaranteed to be in the docker container

agj: which virtualenv call, the one that makes it

?

correct

or whatever call that _uses_ it

okay

pattern is this:

code is checked out on host system

we open a docker container

all python calls are executed in the container

agj: okay, and that docker container has what python libs installed?

local (non-docker) build environments, your fix works in theory

the RTD requirements?

a subset of, there's actually no rtd code in the docker container

the docker container acts as a thick wrapper to shell execution

i really do need to play with docker. i know what it does at a really high level but not in any useful detail for a discussion like this :(

so where the local build env executes using subprocess, the docker container executes the same commands, but through the docker api and eventually a shell

yeah, i'm not quite sure what the fix looks like if it addresses both the local and the docker build environments

agj: oh, shit, soooo /home/foo/rtd/rtdvenv/bin/python would certainly be the wrong thing

in the docker case, yes