#influxdb

xmj, I managed to fix the issue by dropping that database, I am not sure why it was having issues

though I had to manualy delete the files in /var/opt/influxdb/meta and wal

then boot up the influxdb service again

lol

i had that one before..

Sora: how large was your WAL ?

xmj I forgot to check, the database in question probably had around 5 mil entries (in a single measurement) other measurements in it probably had a few hundred thousand each (dont think that should have been a problem?)

dunno

Sora: at work we tried feeding influxdb lots and lots of metrics but eventually we always ran into an artificial bottleneck

xmj that doesnt sound particuarly good, may I ask what your workaround was?

well, the bottleneck wasn't as artificial -- it's that influxdb ended up eating all the machine's ram

so yes, chrischris and I have found a way to bring influx to its limits :->

well I will be adding roughly 100k enries into this measurement daily and then several other databases (originally planned to use the same influx instance) would be getting a total of a few mil entries daily

heh

we were feeding this one db somewhere between 25k and 50k metrics..

per second

for how long a period?

i'm sorry?

how long a period did you feed the db the data

you mean how long until the instance waved its hands and said "ok, bye" ?

Sora: if you're seeing an issue in the web UI but not the CLI, I'm wondering if your query is just trying to pull back too many records? If so, that'll bring your browser to its knees

paauldix the shell should be able to handle a query of 4m-5m entries should it not?

xmj, yeah

Sora: I don't remember

I do remember patching out a thing that made it bottleneck earlier

pauldix: how's the new backend engine coming along?

xmj: I'm very much hoping to get it into a nightly build by Friday. That will be for testing purposes only, but I want to get it into people's hands as quickly as possible

we'll see, I was hacking away all weekend

cool

i like the idea of taking it for a spin, hopefully it has better memory allocation than the boltdb one

btw, seems the right place to ask, how many entries would you recommend sending in a single write post?

Sora: are you bringing back the raw records or doing an aggregate?

aggregating

Sora: really depends on your ingestion rate. I run tests anywhere from 1 to 100,000 in a single request

with the latter obviously taking quite a bit more time

perf seems best with around 5-10k per, but right now the index won't be able to keep up with the ingestion rate that the WAL lets you hit

I currently have a script running which takes data from a mssql database, aggregates it and then pushes to the influxdb

you guys need to make the wal be faster

the main bottleneck being sending curl requests

seriously faster

xmj: the new engine is MUCH faster

well not curl now, web requests sorry

originally ran it on curl for prototyping for simplicity

i was getting a little curious why i'd see wal growing, data thrown away, and yet -- CPU% in the low single digits, disk IO likewise, low single digits

xmj: really depends on which version of the WAL you were using. Also, were you batching writes? The WAL in 0.9.4 can handle hundreds of thousands of points per second

i dunno

that is until the index gets backed up on flushing the WAL, then it starts rejecting writes

but we had 10'000 different servers feeding influx things

the WAL fsyncs on every write

maybe

we'll see with the new version

we can add an option to turn off fsyncing which would speed things up at the cost of potential data losss

xmj: how are the servers sending data?

via the opentsdb plugin

xmj: do you have batching enabled in the config?

in which?

yeah

-size 10k -pending 200 -timeout 60s

when sending multiple points to influx, should it be a new line or a space character that is sent between each point?

nvm, says in the first point

pauldix: lowering tag cardinality lowered memory usage considerbly. up to 10gb of data, no crash, plenty of continuous queries and only 11gb of mem usage on bz storage

also no times out since running since last night :)

cool

ParisHolley: nice

pauldix, I did a stupid and while clearing up my influxdb deleted the "test" db, is there any way to restore it?

Sora: not if you didn't back up the files

zfs rollback :->

Ill be honest, no clue what that is xmj

pauldix: so basically I need to reinstall the demux server to regain the functionality?

Sora: ZFS is a filesystem and 'zfs rollback' is the way to roll back to a snapshot previously taken

that would require that this machine had both ZFS and a previous snapshot :)

it may have ZFS, no clue about that, but it def doesnt have a previous snapshot

if the machine ran zfs you'd know about it :p

probably ^^

welp, to the reinstall

if I can figure out how to uninstall

pauldix: spoke too soon, the timeouts are back :( though at least I know it isn’t a memory issue

i’ll stop bugging ya though, i know you are cranking on the new storage :)

xmj, Pauldix: How do I delete influxdb from the server O.o I try erasing the rpm but after installing the rpm again it is just as it was before the erase

nevermind, I figured it out :)

yay

Sora: what OS are you on?

:D and it even kept the old data

centOS

ah, yum remove..

"rpm -ivh --replacepkgs influxdb-0.9.3-1.x86_64.rpm"

was what I ended up using

that'd work as well, so many options :->

x3 yup

should probably upgrade to .9.4.1 but meh, tomorrow

haha