#influxdb

hoover_damm: eats a lot of CPU actually :)

hoover_damm: 80k per second on m4.large?

or per minute?

per 10s

Civil, is it your graphite templates?

so it's only 8k per second?

hoover_damm: m?

no it's 80k per 10seconds; every 10s we get 80s

that's 480k per minute

and that gaves you 8k per second avg

sure

but it's not really true

anyway my target is a little bit more than that

i don't send data per second

thundering hurd scenario you know?

everyone barks at once

hoover_damm: yeah, I understand, right

still good approximation

and if your going to send data every second or sub-second you might need a point of aggregation so you can do bulk insertion

nah it's not because there's not data every second

you likely don't understand but it's okay

when someone says i get 80k every 10s they mean that; they don't mean i get 8k every second... because they don't

when you have 25 or hundreds of servers pumping every 10s on the clock it's a lot more to keep up with

as if your clocks are right; everyone's doing it at the same time

hoover_damm: I understand what you are saying. In my case it's just bunch of stuff thats written per second, bunch for 10s, bunch for 30s and rest is for 60s and it's really much simplier for me to operate either as per minute or per second value, cause I can't reliably tell what's the real distribution

right that's why maybe using telegraf to aggregate and have it insert bulk

would be good

i collect every 10s; but my switch is every 60s

hoover_damm: but in real world it's not like that - your clock is not perfectly in sync, your network have a delay and blah blah blah

my world is semi-perfect

they all have ntp on each other; my metrics are pumped through rabbitmq via sensu if it's graphite

otherwise it's telegraf

having your data go through rabbitmq makes it fun when hosts go down

:)

hoover_damm: my world is not that perfect and I've already have graphite installation that handle some load. It's just yet another turn to look around and to see maybe I can replace whisper with something else

you can absolutely replace whisper with this

well, I'm not quite sure with that :)

I used to have a 12 carbon mix with 4 relays and haproxy in front

hoover_damm: we've got several magnitudes more load than you have

the amount of data you can injest is pretty ridiculus with whisper

nah i doubt it; my lowest was 800k at one point highest was 2.1million

on a single host

hoover_damm: we have not really standart setup

btw you seem to be really acting high and mighty; you'd get a better response if you didn't come off like you were better

on a single host we've got 4.8M per minute tops

hoover_damm: well, maybe I was just too tired to formulate the question right, sorry for that

I've just setted up a influxdb 0.13 and tried to push some data into that, using it's graphite receiver. And see that it's cpu consumption is actually 2x higher than for whisper

so I'm curious if it's influxdb's graphtie listner

or influxdb itself or I missed something in the config

hoover_damm: and well, my target idealy is something around 10M per host

10M per minute per host

with some reading

the question about ineffectiveness was from 0.9 when they've made the graphite stuff to handle the tags and implemented parsing with regexps which was slow and influxdb developers I think explicitly said that graphite's listner is not very effective

and the native interface should be much better in temrs of performance

you may want to look into the enterprise product then

and talk to sales

because there is a clustering product that was removed in 0.12

that let you have more than one host for reading and writing

past a single container you need some kind of clustering

afaik there's something in open source but i haven't looked at it

and if it's something you depend on; enterprise gives you support for emergencies

better than irc

this is a /very/ slow channel

I'm not in hurry. For now I'm not sure that influxdb is the thing I really want to use

hoover_damm: "a clustering product that was removed in 0.12"?

ucb, well 0.13

it existed in 0.12

right, but what is it? is it the ability to cluster instances of influxdb, or?

look at the blog posts if you are interested