#logstash

I missed logstash-web listing in /etc/init, which is why it would still start on startup. I managed to narrow down the problem to needing and older version of a plugin using the debug run like you suggested.

However, an unendable process is never good.

Hi again, I'm having a weird problem. I set up my lsf on windows server and made a test file in the folder lsf is point at. I created a file and put 1 line into it but it didn't send. BUT when I add a 2nd line then it sends the first. When I add a 3rd line it sends the 2nd etc...

t4nk714, this is normal, because in case of log file, each line adds with new line character

t4nk714, so app that generate logs normally add new line after each log record

Ah I see, thank's again for the response

urw

olivier__, are you still here?

Logs shows that "pipeline is blocked" and I want to try another settings. Sadly, logstash doesn't stop with SIGTERM, do you know any reasons?

olivier__, http://pastebin.com/PTygprXT here is minimal to work geoip filter

Title: filter { if [server_address] { ### (remove this) check if your field with IP - Pastebin.com (at pastebin.com)

Zesk, I have same issue when something wrong with config or plugins

Zesk, just killall logstash proc

and try to test config with debug output: /opt/logstash/bin/logstash -f CONFIG_PATH --debug

tiv, it work for sometime but when Kibana seems to use heavily elasticsearch, logstash get stuck and lumberjack cannot send anyting...

tiv, ok for kill -9 logstash ^^

Zesk, do you use some additional plugins?

with logstash

there is some issues with some of them in some situation, so you'll need to check similar issue with google

I use /opt/logstash/bin/logstash -t -f myconf.conf to test but haven't use "--debug" will try

hope this will help

filter part has date, geoip, mutate, ruby

ruby can do the problem (I have one with timestamp field manipulation)

but it depends

dig your debug logs

Hm, will do that. But I don't understand why logstash are failing (due to blocked pipeline) when I use kibana. Logically, it should elasticsearch the problem ^^

Weird, "Plugin not defined in namespace, checking for plugin file" for filter{}...

Zesk, there is some info in google :) https://www.google.com/search?q=Plugin+not+defi...

<http://tinyurl.com/o9nqjhj>; (at www.google.com)

question on logs forwarding: if I want to send my nginx logs to logstash, the simplest way is to forward things using https://github.com/elastic/logstash-forwarder ?

Title: elastic/logstash-forwarder · GitHub (at github.com)

xowap, it depends

xowap, you'll need to choose it by yourself - syslog, syslog-ng, lsf, ....

lsf gives you ability to parse logs on client side and send securely ready data in JSON (generally)

tiv, thanks... :D I already had this page open. Note sure if this point is the origine of the issue since everything "seems" to work fine if nothing is stress-out. Maybe it's something about output protocol elasticsearch { host => localhost} should i add protocol => http or transport?

I'm running at small scale so far (and I output my nginx logs in kv format)

but lsf sounds pretty easy in the sense that it's doing SSL authentication natively

Zesk, in my case http://pastebin.com/Jgn1tjWm

Title: elasticsearch { host => "localhost" embedded => false port => " - Pastebin.com (at pastebin.com)

and if network (or remote server) fails, logs are sent later, right?

xowap, thats what we want to think so ))

just try it, it's must be suits all your needs

if not, you'll know about it first :)

tiv: yep thanks, I just wanted to make sure I wasn't missing something huge

going to give it a try

xowap, urw, it's all about your needs

also to confirm, there is no need for HA on the ELK server unless I want it, right?

aka, 5 minutes of logs are not going to vanish if I reboot my server

Thanks tiv, my elasticsearch is not on cluster (single node), the original config should be okay. I will dig deeper with debug mode. Hope to come with a fix soon ^^

xowap, logstash-forwarder won't send anyting if the server don't respond, if it's what you ask :p

Zesk, urw, I was write embedded => false, when there is conflict in elasticsearch versions in logstash and ES

with this option LS don't act as ES node, so this solve my problem with that

xowap, I'm running only one node, it's normal for me :)

have small setup too (>100 hosts that sends logs)

How can I give multiple patterns to grok and have it picks the one that matches for a given message?

but in linux I use syslog-ng, on Windows nxlog for log shippment, just because it's simplier for me to deploy and maintain

GLM, you can just %{PATTERN1}|%{PATTERN2} in grok filter

first, that match will yours

the same you can do in your patterns: GLOBAL_PATTERN %{PATTERN1}|%{PATTERN2}, then use %{GLOBAL_PATTERN} in grok filter

just look at patterns in LS install dir, or here: http://grokdebug.herokuapp.com/patterns

Title: Grok Debugger (at grokdebug.herokuapp.com)

and you can figure out how it works

it's all about regexp

tiv:How would I convert what I have here to work with that?

<http://tinyurl.com/nlhovyk>; (at gist.github.com)

tiv, Zesk: yup thanks

GLM, you can try just like that: http://pastebin.com/EgCYSB3H

Title: filter { grok { match => { "message" => "%{DATA:a - Pastebin.com (at pastebin.com)

GLM: pattern1|pattern2 can be very expensive (running a really long pattern across your whole message).

GLM: in your case, i'd pull the common stuff off first (a smaller pattern), putting the results back into 'message', then running the other groks against the shorter message.

How would I do that?

GLM: well, lookie, i have a blog post :) http://svops.com/blog/processing-common-event-i...

<http://tinyurl.com/nrw8oqa>; (at svops.com)

rastro:You're too good to me