#logstash

" on the command line. The /opt/logstash/bin/logstash file is itself the original Logstash bourne shell script. Does anyone already have an init script?

is there any way to pass environment variables into a logstash config? It would be really helpful for parameterizing them.

davecore82: did you install from the repo, or the tar.gz?

right now it seems like I'm hardcoding everything, from file paths to connection details, which is pretty painful

rekibnikufesin, from the .tar.gz

davecore82: try from the repo: http://logstash.net/docs/1.4.2/repositories — I haven't used the yum repo, but the apt repo includes init scripts in the package

Title: logstash - open source log management (at logstash.net)

I would imagine the yum repo is similar

plus you get the other benefits of using repos for management

rekibnikufesin, ok thanks I'll look at the repos

hmmm anbody know if there's a way to make elasticsearch balance shards on the most recent index?

"make"?

JDShu: in what way?

Elasticsearch should automatically balance shard count between nodes

unless you've deployed settings to force-route data to specific shards, they should be fairly evenly sized amongst themselves too

untergeek: how are things?

Just wrapped up two days of training in Minneapolis

Did private Elasticsearch training at a company there

untergeek: nice; However, I could think of better places to be than Minneapolis

it was actually really pleasant this time of year

and the Elasticsearch Meetup we did last night was awesome

one of the best I've ever been to

Well, the weather in general has been kinda screwed up kind of around the country, so I heard the mid-west is experiencing fall like temps lately

cooler than the 90°F weather I've been experiencing in Texas :)

Do you happen to know off hand what the highest version of server that an 0.90.7 native client can talk to is? We have some developers that decided they weren't going to upgrade their cluster, and then when I went to put the code into staging (on a 1.3.1 server) it's all ... uhm, nope. And I'm refusing to install a server that is < 1.2 and they are complaining that the APIs in 1.3.x .jar are different enough to break their code....

ronnocol: tell them to implement bulk http inputs or use logstash =p

(Cuz I’m evil like that) ;)

ronnocol: 0.90.x has to talk to 0.90.x, and even then, there was a version that couldn't talk to other 0.90.x versions

Coolacid[cloud]: yes, well, this isn't for logstash, this is for actual search like things, and I would have typed /join #elasticsearch, but most of the people there are here so I opted for the less typing route :)

meh, if I was going to complain, I would have =p I was more making a dev joke :)

You mean people use elasticsearch for SEARCH?

Wow.. who knew ;)

(I’m in a mood)

torrancew: That's what I thought... so now the question becomes in which 1.x version did the APIs become different enough to require massive re-work vs. a little re-work

VacationMode, Defcon, VacationMode and possible happy dance ;)

Coolacid[cloud]: hitting BH as well, or just DC?

I’m staff @ BH

oh, that's right. pretty sure we talked about that.

yep :)

sorry. There's not enough caffeine in the world to compensate for my brain today.

Aagin - VacationMode ;)

:)

Left $work stuff at home, and killed $work email on ipad.. but, I can’t stay away from ES stuff. ;)

:)

already en route to LV/there?

weee

Suyog's working on some grok perf improvements. Currently we've got a 47% perf improvement

(it's nearly twice as fast)

percentages are stupid. silly me :P

and we're working on even more changes that should help improve perf by more than that as well

they are, that

so I installed the logstash RPM from the logstash repo, but when I try to start logstash it says it started but then it seems like it dies instantaneously and in /var/log/logstash/logstash.err I have the message chroot: invalid option -- -

davecore82: RHEL 5?

yes

or CentOS 5

there's an open issue for that

RHEL 5

missed an l in that OS name

https://logstash.jira.com/browse/LOGSTASH-2225 is that it?

Title: [LOGSTASH-2225] Logstash 1.4.x init script "deficiencies" - logstash.jira.com (at logstash.jira.com)

ya, I believe so

Silly aunt’s network not letting me send email on port 25.. House VPN to the resuce! :)

Oh gawd, not RL%

RL5

Please let it die.. please.

not gonna happen :S

PR specificly for CentOS/RHEL5 — The problem is RHEL is old and busted. https://github.com/elasticsearch/logstash/pull/...

<http://tinyurl.com/ljp2rjp>; (at github.com)

RHEL5 is old and busted.

RHEL5 is what the cool kids use

So, after vacation mode I need to look at my PR to fix GOOD Linux versions, and mix that in with RHEL detection in the script.

RHEL5 needs to DIAF

:)

whack: Not logged into Github, can you assign that PR to me so I can look at it and try and come up with a good “mix”

anyone tried RHEL7 yet?

shoerain: ping re: #1572

Same file, diffent bug: https://github.com/elasticsearch/logstash/pull/... — I need to merge those two “fixes” and document the problems with running RHEL5 apps

<http://tinyurl.com/qa5w3g6>; (at github.com)

Specificly, chroot and sudo can’t be run in tandum - as such, in RHEL5 you can have sudo, but not chroot.

ok, I guess I'll go get some lunch and figure out how I'm going to tell the developers that they have unplanned work to do because they were to lazy to type "yum update elasticsearch" on their dev cluster

yum is hard

untergeek: it balances total shards, but not shards per index

I'm searching for a list of logstash patterns (e.g. NUMBER, WORD, URIPATHPARAM) but I'm faililng. clues please?

dvl: http://grokdebug.herokuapp.com/

(sorry was away from the computer when you replied)

Title: Grok Debugger (at grokdebug.herokuapp.com)

i think there is some github pages too with the list, but that tool is very useful if you've not used it.

JDShu: What do you mean by balance shards per index? You want a certain number of shards from index A on each computer?

untergeek: yeah

ah

you have to set a template option for that

jpyth: wonderful. thank you.

"index.routing.allocation.total_shards_per_node" : #

untergeek: I have a ton of logstash data hitting my cluster, today's index happened to sharded in such a way that they're all on the same few machines and now elasticsearch has fallen over

this setting will distribute them, but only for new indices created.

sweet, that sounds like exactly what I need, thanks!

you could also use rack awareness

to guarantee that shards and replicas are in different "racks" (real or virtual)

the PR #1555 works for me

RHEL5 can live on!

not too sure how to set that in ec2

Oh.. untergeek that’s what I was going to ask you — (and I won’t even think about it right now) how would I modify LS template to default a tag require (in Elasticsearch) — IE: the shard allocation tag.

Also - does the LS template handle subfields (per the convo in GSwithELK)

davecore82:

grr.r.

davecore82: yes, it would work - but it reduces security for those of us with non RHEL systems :) So, I need to figure out a clean way to detect RHEL5 and build the correct command.. (or better, would be to have an EL5 rpm — hint hint whack )

:)

Thanks guys