#logstash

No help with a "execution expired" "multiJson::ParseError" ?

Hello :)

I have ELK running with docker-compose.

t4nk475, with no errors ? That is exactly what I am trying to do

BlackCrypto: yeah, this works great: https://github.com/deviantony/docker-elk

Title: GitHub - deviantony/docker-elk: The ELK stack powered by Docker and Docker-compose. (at github.com)

I'm having trouble with nested json, though.

cat docker-compose.yml elasticsearch: image: elasticsearch:latest command: elasticsearch -Des.network.host=0.0.0.0 ports: - "9200:9200" - "9300:9300" volumes: - /mnt/elk:/usr/share/elasticsearch/data logstash: image: logstash:latest command: logstash -f /etc/logstash/conf.d/logstash.conf volumes: - ./logstash/config:/etc/logstash/conf.d ports: - "5000:5000" links: - elasticsearch kibana:

kibana/config/kibana.yml:/opt/kibana/config/kibana.yml ports: - "5601:5601" links: - elasticsearch

logstash.conf input { tcp { port => 5000 } } ## Add your filters here filter { json { source => "elements" } } output { elasticsearch { hosts => "elasticsearch:9200" } }

curl -XPUT 'http://elk.aws.p0.com:9200/autotest/manualnested/3?pretty' -d ' { "name": "Jack Doe", "elements": [{ "name": "Can Jack Haz donuts?", "keyword": "Scenario", "tags": [{ "line": 4, "name": "@HAD-555" }] }], "fields": { "@timestamp": [ 1457682128003 ] } } ' { "_index" : "autotest", "_type" : "manualnested", "_id" : "3", "_version" : 2, "_shards" : { "total" :

<http://tinyurl.com/htvrjhy>; (at elk.aws.p0.com:9200)

t4nk475: use pastebin for configs and stuff

ed" : 0 }, "created" : false }

t4nk475: and maybe ask a question first...

http://pastebin.com/ZVN8SkTf http://pastebin.com/zX63c6gG http://pastebin.com/VpxRxPy9 The nested 'elements' JSON object gets indexed as a string.

Title: cat docker-compose.yml elasticsearch: image: elasticsearch:latest command - Pastebin.com (at pastebin.com)

I want the nested objects in 'elements' to also be indexed and show up as such in Kibana.

Can this be done?

Sorry for the config spam.

t4nk475: your third pastebin shows that you're inserting the document straight into ES. LS can't help you in this case...

rastro: thank you. when I try to do this is just hangs forever http://pastebin.com/4Y44fTGx what is the right way to send to LS? Can I use curl?

Title: curl -XPUT 'http://elk.aws.p0.com:5000/autotest/manualnested/4?pretty' -d ' { - Pastebin.com (at pastebin.com)

t4nk475: seems like curl would open the port and put stuff, which might work. For testing, I'd use a stdin{} input and a stdout{} output until I got the output to be what I wanted.

double-p: yes

when I try and do a use docker-compose up I get errors about pkg_resources import

and some python 2.6 messages

ajmartinez: thanks for the link. I could verify there that my pattern works. What I really wanted to know though was where to put it in the grok config

curl -Ss http://localhost:9200/_nodes/stats/jvm?pretty | grep heap_used_percent

"heap_used_percent" : 33,

My Lord! What an output! curl -XGET 'http://localhost:9200/_nodes/stats'

fatdragon: you get all the stats :)

I am gonna have to feed that back through ELK to understand , parse it again :)

fatdragon: eh, more useful to pipe that through to either a separate, dedicated elk cluster, or an external system altogether

b/c otherwise, it won't be there when you need it most (when ELK is having issues)

lol

where is a good ubuntu link to generate java thread dumps?

fatdragon: jstack

how can I see those stats from ES in a gui? install bigdesk?

fatdragon: marvel

torrancew: ping

akp: pong

was looking for you yesterday

wanted to ask what OS/distro you're working through this on

what it means that if I'm consuming from an idex of ES (that has always incoming data) Logstash says : "Logstash shutdown completed"? (it stops importing)

torrancew: the logstash "server" is on centos 6.5

akp: where are you generating the certs?

the LSF has been tried on CentOS 6.5 and 7

on the centos 6.5 machine

k, rules out that hunch

(was wondering if you had great openssl version disparity, seems you do not)

yeah, i wondered that too, it was actually one of the first things i checked.

i'm still stumbling through the plugin + ruby

*nod* didn't have time to repro last week

probably can this week though

to figure out how to get it to give me some info about the certs it has loaded

no worries.

you probably want to look at https://github.com/elastic/ruby-lumberjack for that

Title: GitHub - elastic/ruby-lumberjack: Ruby lumberjack protocol, contains both client and server (at github.com)

that's teh underlying lib where most ssl stuff is done

though the plugin would work too, you just may have to do a little more work there (don't recall which class loads the cert/key into ssl objects)

yeah, thats been part of my issue =/

figuring out what lumberjack.rb to actually look at.

<http://tinyurl.com/o8cfkpm>; (at github.com)

word. thans

np. logstash/bin/plugin list may show you the active version of that gem, not sure if it filters non-plugin gems or not

cool