#dat

I'm unable to test

ok it looks like my issues had to do with using an old dat, created by an old version

at least with bigger files it never runs long enough

hmm

perhaps my linux system on the vps doesn't work well with dat

TheLink: how recently did you try this?

you remember the version?

dunno, last week or week before

I reported everything here

existing issues should cover my findings afaik

I'll just wait till all the new stuff gets put in

hmm whats broken in the new release?

core wise not much changed. jhand knows more about the cli stuff these days

ok, let me test again then

yes, I talked to him

don't want to badmouth anything, the hole punching and transferring per se works fine

I just never managed to transfer my 1gb test file successfully recently

hmm

but it used to work you said?

do you know which version if used to work in?

*it

no, sorry

I was busy for a while and didn't test anything

restarted doing so in the last 2 months or so

I'll do my test with current 11.5.0 again and see

cool. if you find an old version where it works better it's easier for us to identify the problem

ok

mafintosh: I definitely got some weirdness trying to get a dat to work that was, I believe, created in the current major (11)

hmm (!)

but I updated a couple minor revisions and things got weird. I have no issues on clean dats

yeah. Not sure what that's about

seeding side: https://www.dropbox.com/s/6kntvvt4egsbnmb/Scree...

progress bar and size are off

item count is correct

but it errors out after a minute or so

okay, we need to make that way more stable

receiving side: item count is off (says 3 items but 1 is correct), says 2 peers but there's only one

i'll see if i can make some time to look into it

yikes

TheLink: did you delete the .dat folder before trying that?

pfrazee: I always do that ;)

TheLink: ok

having to manage the .dat folders is also suboptimal but I think that's an obvious issue

that all is about live mode btw.

well sure, you only have to right now because there's some kind of bug that might be due to incompatible .dats

they shouldn't be imcompatible though ...

right

well, who knows whats up, but fishy behavior for me persisted until I deleted the .dat and started fresh

archive-bot: add 41573dda3143e48f0f9e819a9e4310aa5985f12691edde85e8f4ea866871d739

Adding 41573dda3143e48f0f9e819a9e4310aa5985f12691edde85e8f4ea866871d739

mafintosh: oh wait, I might still be on the test version jhand created for me

ah

I'll do a clean npm install

wait

the count being off is fixable by just being way less agreesive about adding files that are watched

like if mtime is the same just skip it

(yes there are edge cases where people change their own mtimes, but meh :))

just upgraded to newest dat and works fine for me at least

oh wait now i get the download count being off

yea acting up big time now...

I'm on 11.5.3 now

it didn't error out yet but all the other issues I mentioned are still valid

yep, seing the same now

peer count on receiving side was correct now though

but therefor the download progress (xx/xx) appears late and disappears and re-appears again repeatedly

^ on receiving side that is

on seeding side it seems to be fine

jhand: what was the switch to avoid the emfile error again?

yea the file watching code is being triggered a bunch

jhand: okay we need to make the cli just never re-add the same file for now

once thats stable we can look into making it better

generally speed seems to be slower than it used to be once

I can normally download with up to 9 MB/s from my server

with recent dat I hardly ever reach 1 MB/s

dat-cli I mean

testing that is difficult for me now since I'm at my parents place now and wifi sucks here

might be related to this re-add issue

everytime something is re-added it might lower your bandwidth quite a bit

ah, ok

... also could be something else, just saying :)

since I test with this 1 gb file, it might make things even worse

its a good test though

ok

did you notice if the "seeder" said more than 1 item added?

hmm, I don't think so but I can just try and see

btw, using --snapshot avoids most of the mentioned issues

but doesn't improve speed and receiver still says 2 peers

the rest seems fine

snapshot still does some weird things in latest dat because of the re-add

it broke a local dat of mine i just noticed. like pfrazee mentioned

ah, ok

visually it's better though :P

mafintosh: with this watch complexity, I wonder if we shouldnt be doing user-constructed commits ala git?

pfrazee: yea...

mafintosh: just tested again regarding count of added items on seeding side – it says "Adding 1 items (871.68 GB/1.07 GB)"

the count is correct

TheLink: yea that was the snapshot thing i was seeing

same bug

mafintosh: that's with live mode I mean

oh :/

dunno about the "(871.68 GB/1.07 GB)" part

it stays like that

now it says "Updating 3 items (871.68 GB/3.21 GB)"

so it seems to think that I added files

jhand: does dat-js persist any progress data? or is it always calcuated on the fly?

okay have a fix coming up

\o/

mafintosh: I think it persists some things

mafintosh: or hyperdrive-stats does I don't remember where I saw it but I remember seeing persistence somewhere

That's not very helpful, sorry ;)

I'm persistent

juliangruber: https://github.com/juliangruber/hyperdrive-impo...

o/ should be first step into making it more stable

dat-js pr, https://github.com/joehand/dat-js/pull/25

pfrazee, TheLink with o/ it seems to work quite reliable for me now

mafintosh: just hopping back in, was giving a demo

mafintosh: ok great

trade-off is it'll only add new files to the dat

that's less great

well its a tmp workaround

its better than it exploding right now

mafintosh: no stats are persistent right now.

yeah it's a start. I'm sure you've got other places you need to focus, can you toss the rest of the fix to jhand or someone else?

jhand: okay good

jhand: see the above convo. file watching bugs is breaking cli, a lot right now.

cool

jhand: my workaround is to just only add new files to the dat until the watching bugs are fixed, redesigned around etc

fixed the stats being off as well it seems

I'll test as soon as it's pushed to npm

TheLink: darn, I forgot to publish that fix I had you test before.

jhand: fix is not ideal, but stability first imo

hehe

jhand: as I said at http://irclog.iq.harvard.edu/dataverse/2016-10-... I'm looking forward to your talk next week! Nice overview today.

mafintosh: ya. I'm going to remove some stats stuff to make it more stable in next release.

pdurbin: thanks! nice to meet y'all.

jhand: stats seems to work with the overwrite fix though

just been tested it a bunch right now

mafintosh: were you getting file overwrites before? I was never able to reproduce that

jhand: a bunch yea

thought it was OS specific

hmm

jhand: but i've getting those for a while i think

files being re-added over and over again

switch slowed down upload/download a lot

i think this is the root cause of some TheLink perf regressions

(but i need to test that part more)

definitly would hurt perf

anyway, i think we should:

get a cli release with o/ tmp fixes

investigate + test the overwrite stuff and make sure that actually works as expected