#logstash

that is, from different hosts, if you write in one file without proper locking

look up multiline logstash

log-courier does multiline

cek, you should separate files for each host

that will create a mess

if it's mess, even by reading the logs you cannot make sense of it if you have multiline

yeah your logs have to at least be understandable

can anything accept HTTP PUT and feed logstash?

we had some guys wanting us to process stuff that had lines truncated by other lines

it was messy

cek: why will individual files be messy?

million of hosts, for ex

millions?

yeah, why not

how many do you actually have?

millions of hosts writing multiline to a single file is gonna be a headache too

i don't want it to write to that 1 file, i want data to be fed to logstash

>Codec support is available which allows multiline processing at the sender side

log-courier sounds promissing

<http://tinyurl.com/pt84vpk>; (at github.com)

it's not gonna sort out interleaved multiline for you though

have you seen modsec audit log ?

nope

but I can tell you the multiline bit in LC will just match a pattern to decide if the line continues

so if there's anything else in between it's not going to work very well

depending on the number of hosts you have (i.e., < 1 million) if you write to separate files and give LC a wildcard path it will pick them all up

https://gist.github.com/celesteking/aa6b8467fa6... 1 modsec audit log entry

ovbiously it will have to be much less than 1 million

<http://tinyurl.com/llcyxz5>; (at gist.github.com)

so there's a line feed between entries?

or they all start with "--"

yep. and "entries" themselves aren't predefined, they can contain multiple events

logcourier can be used to ship each server's log to logstash

i just wanted to use mlogc to http PUT to logstash server via some app possibly

I think the kv filter can help there if you get them all on one line

i won't get them in one line, that's impossible

"[`;\\|&\\r\\n].*?(\\.exe)?(\\s+[-/])?.+[&<>\\|]*?" you won't get this in one sane line

that should all be structured/wrapped in json or something

Now I'm thinking, is there any app that would accept HTTP PUT request with data and ship that to logstash?

millions of PUTs (thus, no memleaks please)

it's gonna be a pain to parse that thing

you might want to write some intermediate parser that converts this thing to json

and outputs a json file

there's one already

well, not json, its a logstash thing

which one?

<http://tinyurl.com/na285bn>; (at github.com)

cek: can't you just send multiline based on lines that don't start with "--" and parse it with that then?

i.e., if a line doesn't start with "--" then it's part of the previous line

...based on your paste

that paste is only 1 event

oic

I suppose there's a regex that can match those based on the first capture

but I suck at writing regexes

maybe someone else here can confirm that

that you suck at regexes? ;)

BaM`: confirm that you suck at regexps? :_

why not both?

lol

do people use ELK stack w Heroku and similar stacks? or is it more common if you're using a more custom infrastructure? i'm new to this stuff and can't seem to get a straight answer

<cek> https://github.com/bitsofinfo/logstash-modsecurity -- thanks

<http://tinyurl.com/na285bn>; (at github.com)

make sure to add an 'if [type] == "mod_security" in the whole filter if you are parsing more than just mod_sec

hey, can someone help me with a big picture view of elkstack?

tell me if I'm doing this right?

daidoji: try us!

logstash-forwarder or writing via say https://github.com/bigfootproject/python-logstash writes to my logstash instance, which should have open port?

Title: bigfootproject/python-logstash · GitHub (at github.com)

logstash then stores that to /var/logs?

then elasticsearch picks that up somehow...

and then I can view it with Kibanna?

logstash should be listening with the lumberjack input

I guess what I'm having trouble with is configuring it correctly because I apparently need indexes?

daidoji: nope. logstash doesn't write files (usually), it just filters the messages it receives and forwards them to ES.

rastro: oh okay

daidoji: LS will create any indexes that don't exist.

rastro: roger. How do I hook up elasticsearch to kibanna then?

daidoji: other way around - you hook kibana (the UI) to ES (the data store).

daidoji: http://svops.com/blog/elk-system-overview/

daidoji: ES accepts requests via an interface, and kibana talks to that interface to pull data to draw.

ahhh thanks for the link. Don't know why that doesn't pop up in my top list of google results

rastro: but okay

rastro: so if my kibanna is complaining that no indexes are set, I should be looking in kibanna config?

daidoji: no time for SEO :)

rastro: ahhh

daidoji: you do need to tell kibana which indexes (or, usually, index patterns) you want to use. But it should walk you through that if it finds indexes.

daidoji: i would guess that your data isn't flowing LSF->LS->ES

rastro: ahhh okay

daidoji: some debugging tips: http://svops.com/blog/debugging-your-elk-cluster/

rastro: ahhh, thanks a lot

daidoji: yw

well I'll keep plugging away, I appreciate all your help

daidoji: go slowly and ask lots of questions!

rastro: is it your blog?

RobertDupont: svops? yeah.

RobertDupont: comments welcome!

I came across it several times before, very helpful

RobertDupont: ah, so it *does* show up in google :)

RobertDupont: glad it helped.

;)

I got to go

bye