#dat

karissa: did routes on the datfolder username/dataset change or something? getting a full html page back from that route now in the dat-next tests.

jhand: what are you looking for?

jhand: https://dat.land/api/v1/karissa/more-tweets-mor...

karissa: the test is hitting http://localhost:3000/api/v1/joe/awesome but I get an HTML page back

huh

jhand: one sec

let me reinstall that dependency

jhand: ive been working dilligently on the master branch yesterday and today

I didn't think this happened on travis, just seeing it locally.

karissa: ya thats why I had trouble seeing if it was me or some change I missed

danger of using master for testing in the cli

jhand: what database do you have set up?

jhand: a sqlite one?

karissa: https://github.com/joehand/dat-next/blob/master...

ya

jhand: hm i have a test for it and its passing locally

karissa: ya im confused. I just reinstalled and getting different error

let me sit with it a bit more

jhand: which?

"{ statusCode: 400, message: 'A dat already exists with that name on that account.' } ")

trying to clone from joe/dataset

yeah cause the db already exists

oh no wrong error

are you trying to create a dat

("http://localhost:3000/api/v1/joe/awesome {"statusCode":400,"message":"Dat with that name not found."}

yeah maybe the username is wrong or the dat never got created or something?

karissa: ya I think its doing it in the test before but I'll make sure.

karissa: agh. So if I run `npm test` its all fine. I was running the specific auth.js tests only and that was failing.

so it wasn't creating the dat or something?

karissa: ya I thought each test was independent but must have missed something

its definitely creating it, since thats the test right before it. but something obv missing

ok

ogd: I think this might relate to what you were talking about earlier, with the archiver server being passive https://github.com/joehand/hypercloud/issues/39

[dat] yoshuawuyts created greenkeeper-discovery-swarm-4.2.0 (+1 new commit): https://git.io/vMOue

dat/greenkeeper-discovery-swarm-4.2.0 f23b719 greenkeeperio-bot: chore(package): update discovery-swarm to version 4.2.0...

4.2.0 blaze it

lol max

npm version swag

datproject/dat - chore(package): update discovery-swarm to version 4.2.0 - errored - https://travis-ci.org/datproject/dat/builds/188...

[dat] greenkeeperio-bot opened pull request #594: discovery-swarm@4.2.0 breaks build ⚠️ (master...greenkeeper-discovery-swarm-4.2.0) https://github.com/datproject/dat/pull/594

dangit

haha

ogd: am I making sense?

pfrazee: i am just wondering if the 'many hypercores over one connection' is the issues

issue*

pfrazee: and if we can just have them each be a separate process

ogd: yeah, it's not exactly. The issue is *too* many hypercores

pfrazee: but say i have a million hypercores on a machine

yeah

pfrazee: and a client wants to push or clone one

pfrazee: that should scale fine right?

ogd: right, and it'll be fine if it goes like this

client connects you. client says, "can I have one?" you give

but badly if it goes like this:

client connects. you say, "hey I have a million cores, here they all are." client dead

afaik, in that second scenario, it wont try to replicate all million, but it will announce them

and that's still a rpblem

in what situation would the client be asking for > 2 feeds?

pfrazee: i think announcing a million things is fine

jhand: I'm not sure. I think it depends on how the client is written

pfrazee: we'll need to scale discovery-channel anyway when we hit that point

pfrazee: mafintosh was worried about doing lots of announced so had me add this timeout https://github.com/joehand/archiver-server/blob...

ogd: it's fine for now but that's a huge amount of traffic

pfrazee: yea im sure we can optimize announcing

jhand: that's a different flood but yeah

pfrazee: what i dont get is the protocol level many .replicate() over one socket thing

ogd: it's just multiplexing

I think that's a good thing

"just multiplexing" lol

pfrazee: so im confused why this line will flood anything https://github.com/mafintosh/hypercore-archiver...

so is the issue that if passive is off, it will spam the socket with all the multiplexed dat headers?

jhand: to clarify, there's two places where a flood could happen. The one you linked to is announcing on the discovery network. The other one, which max and I are discussing, is announcing feeds once a connection is established between peers

handshakes/headers

ah

ogd: correct

https://github.com/mafintosh/hypercore-archiver... is what causes it to call .replicate a bunch

yeah

pfrazee: so all im saying is, why announce over the socket

pfrazee: when we have a separate stage for announcing

(discovery channel)

ogd: oh I see. Yeah not sure. Perhaps an optimization

so thats our question for when mafintosh wakes up :D

ogd: in my code in beaker, I do what you suggest

hah yeah

i think by using the same ip/port for multiple dat keys its causing this issue

well actually i take that back

for reference: https://github.com/beakerbrowser/beaker/blob/ma...

so if the archiver by default announced all its keys over discovery-channel, with the same ip/port

by did not do this eager .replicate() thing

then i think everything would work right?

it would just wait for the other side to call .replicate() for the key

and since it has announced itself, the other side should find it and be able to initiate

*but did not

I think it should

code I linked works that way, and I think it would work, just perhaps be slower than optimistic announcements

does hypercore-archiver announce over discovery-channel now?

yes

well

if we had two bots would we get Canadian replication with that?

its consumers do

what do you mean consumers?

optimistic announcements arent necessarily bad, they just cant go overboard

consumers - users of the library

hypercloud being one

the discovery stuff is not implemented in hypercore-archiver

ok so in hypercore-archiver-bot it must not do announcing

cause that would explain its canadianness

I'll check

right

it can either announce over discovery-channel or announce via eager .replicate()

but if it does neither its a canadian

it uses discovery-channel

i hope someones reading this right now wondering what im talking about

it just doesnt ask for anything

ogd: too polite to ask?

lol

so its announcing (disc.join with a port)

I see I see

ok this should be totally solveable then

ah i remember now

the canadian scenario is

two hypercore-archiver-bots walk into a bar

but the passive is true so it doesn't do the eager replication

both are passive

neither call .replicate()

<time passes forever>

some bots just keep waiting for the right moment

poor bot

so i kinda think

to solve the canadian bot problem

when you call ! add with the bot it should have it call .replicate() in that instance

otherwise it should act as it currently acts

that would make sense

jhand: last piece of the puzzle on hypercloud, I think https://github.com/joehand/hypercloud/wiki/Trig...

this wont scale horizontally as I'm imagining it, or even vertically too well, but we'll just have to plan to rewrite it in the future