#dat

(decarola_twitter) Hello everyone, is the a guide or something to cuild your own site over dat? I mean from getting the URL.... or whatever it takes. I'm trying to fully get it

kemitchell: Is the licensezero website out of sync with the cli?

millette: pfrazee Thanks for links, reading now

(tusharmath) Hi — Can we build transactional systems using dat?

(tusharmath) For instance we have a `Contest` which only `N` users can join and at any moment a lot of users are trying to join the contest. I want to make sure they are served on a first come first serve basis.

(RangerMauve) @decarola_twitter dat://beakerbrowser.com/docs/tutorials/ there's some docs there on how you can make sites

(RangerMauve) @tusharmath You'd probably be better of using a centralized service for that use case. But you could serve the site through dat and just use centralization for registering the contest users.

^ that portion of the site is temporarily janked fyi

(tusharmath) @RangerMauve do you think Block chain is a better choice for such transactional setup?

(RangerMauve) @tusharmath I'm personally not a fan of most blockchain things since it's very inefficient and there are alternatives that don't require global consensus or expensive proof-of-work functions. But if you're set on it, you could try looking into etherium since it's smart contracts are a great place to put code that's semi P2P and they're working on making the network accessible in more places.

(RangerMauve) What's your use case?

(RangerMauve) A contest with first-come-first-serve seems like the ideal thing to use a centralized service for.

(tusharmath) Yeah I work with a fantasy gaming company and we have a highly transactional system. We host thousands of contests and each contest can have upto a million users joining. More often than not we reach a concurrency of 1M and it become insanely difficult for us to scale.

(tusharmath) There are a lot of problems that were thinking of doing in a decentralised way but none of us I think have a solid understanding of how to get started.

(RangerMauve) Yeah, blockchain will probably just make things more inefficient and harder to scale. (and more expensive)

(RangerMauve) For massive scale, I've been reading up a lot on "event sourcing" https://martinfowler.com/eaaDev/EventSourcing.html

(tusharmath) One more problem is that our leaderboard shows every user their rank. This setup has been built on top of Spark and Cassandra right now. IMO doing this kind of sorting on the client would be really really slow.

(RangerMauve) Yeah, Spark with Cassandra paired with sending requests to your service is a good approach

(RangerMauve) With event sourcing you could potentially have the calculation needed for the leaderboards done in real time as events come in instead of running batch jobs

(tusharmath) I personally feel we should go in the direction of a completely decentralized internet but at this scale it seems like we are probably not there yet :(

(tusharmath) DHTs can solve our distribution problem

(tusharmath) Event Sourcing sounds like — Redux

(RangerMauve) Yeah, Redux has some similar concepts!

(RangerMauve) But instead of having a central state that's mutated with events, you just have events and intermediate states are also events

(tusharmath) But I will be performing the calculations on the client directly?

(tusharmath) that can take lot of time and if the user is on a mobile device it would completely drain out the battery

(RangerMauve) Probably not, but you can scale horizontally more easily since you could have independent processing that won't create single bottlenecks

(RangerMauve) Users will send events to your back end, and subscribe to result events from your backend

(RangerMauve) Then in your backend you can split up the events with filters and combinators to create your result events

(tusharmath) which is how it works right now

(RangerMauve) Then I guess you're probably on the right path. 😅

(tusharmath) :D

(RangerMauve) http://reactivex.io/ is great for structuring your code in a event-driven style, but I think https://kafka.apache.org/ is the thing being used a bunch of event sourcing in Java land

(tusharmath) yup

(RangerMauve) What's the biggest bottleneck in your system right now?

(RangerMauve) Processing incoming data? Fetching data?

(RangerMauve) If fetching data is expensive, you could make use of dat to distribute it among the p2p network and get rid of a bunch of load on your servers

(tusharmath) 1. Computing ranks of a millions of users every minute

(tusharmath) 2. Serve user rank and points to a million user at the same time.

(tusharmath) The problem is the spiky nature of the traffic.

(tusharmath) Our concurrency goes from 100K to 1M in about 2-3mins.

(tusharmath) This happens when a popular player hit a shot etc.

(tusharmath) depending on the game.

(tusharmath) We generally over provision our servers so that they can handle max load expected during the game.

(tusharmath) One the game is over we scale them down :sweat_smile:

(RangerMauve) Jeeze. :sweat_smile: I haven't had the pleasure with working with those loads, sadly. Have you looked into the "serverless" trend? It seems like people were having good results with auto-scaling their serverless functions. But you'd probably want a self-hosted alternative to amazon lambda.

(RangerMauve) If you distribute read-only data through dat or via torrents you could probably decrease some of the load on the peers actually loading the data

(tusharmath) Yup we have evaluated lambda but unfortunately it take 5-10 mins for them to come up. Which is too slow for us.

(tusharmath) For data distribution we try to make sure most of it is immutable and infinitely cacheable

(RangerMauve) Yeah, and that's perfect for serving over a p2p network

(tusharmath) That's why I think DHTs would work for distributing data

(tusharmath) reduce some load on our CDNs

(tusharmath) I am just thinking about latency

(RangerMauve) DHTs aren't the best if you're storing a lot of data. You should store data identifiers in the DHT, but then use something else like torrents/dat/ipfs for fetching

(tusharmath) DAT is a DHT right ?

(RangerMauve) No, dat is a protocol for distributing versioned data

(RangerMauve) You create a "dat", which creates a private/public key pair used for adding data to it, and then people can use the public key to download the data from the p2p network

(RangerMauve) It can use a DHT to discovery peers that hold the data for a "dat", but it doesn't store the actual data in the DHT

(tusharmath) Ooooooh

(tusharmath) I thought the data is also on the DHT

(RangerMauve) The basic Dat, the one that comes with the CLI acts kinda like a filesystem

(RangerMauve) Nah, DHTs aren't very good for storing lots of data.

(RangerMauve) They're usually used for peer discovery to get the actual data from the p2p network

(tusharmath) but torrents are DHTs right?

(RangerMauve) No, Torrents don't store data in the DHT. They take data, split it into pieces, and then publish to the DHT which pieces they have.

(RangerMauve) Then peers look up who has what pieces and connect to them to get the actual content

(tusharmath) Make sense

(RangerMauve) You can visualize your leaderboard as a nested folder structure where each folder has some hundreds of users represented with files

(RangerMauve) Then you can have your back-end use that to store the rank, and have clients download just the parts that are relevant for them (the top group)

(RangerMauve) Instead of calculating the full order each time, whenever a person's points change, calculate the changes needed to the folders. (Maybe group these with multiple people at once or whatever)

(RangerMauve) Then you only have to touch those users whose rank changed

(RangerMauve) Users are likely to be close-ish to each other in rank, so you probably won't have to make huge changes to your structure.

(tusharmath) How can I stress test ?

(RangerMauve) You could simulate lots of peers replicating the data and simulate lots of rank changes

(tusharmath) okay

(RangerMauve) One caveat is that Dat only really works in Node.js at the moment, but there's a rust implementation in progress that should be easier to integrate with different environments

(tusharmath) Yeah :)

(RangerMauve) You can also look into IPFS since it can be used for representing filesystems, too. The main difference is that it's way less efficient for discovering peers for data.

(tusharmath) ok

(RangerMauve) Do let me know if you end up making something cool with it! It sounds like a really exciting project.

(RangerMauve) Also https://github.com/datrs

#dat-gitter

who needs irc ;-)

(RangerMauve) 😅

just you wait until we have a dat-cabalist going

"I'm just trying to find the bridge... Has anybody seen the bridge?"

hey does anyone know how to find out if their university's network is blocking connexions through dat:// ?

bgcarlisle, dat doctor

amazing

lemme just search that a sec

it's an internal dat command

ahh

FAIL - unable to load utp-native, utp connections will not work

uh oh

TCP ONLY - success!

that's good though

what os ?

elementary os

weird that utp-native wouldn't load though

maybe i'm missing a dependency or something