#logstash

socket-: well, to put it another way, generating json from yaml with ruby is this simple:

Ok, so now the data is getting to my LS server, http://apaste.info/2Tk but it's not pre-indexed properly, so now i need to figure out how to use the CSV filter to pre-assign field values

Title: Paste #2Tk - Apache Paste Bucket (at apaste.info)

ruby -rjson -ryaml -e "File.open('/path/to/config.json', 'w'){ |f| f.puts JSON.pretty_generate(YAML.load(File.read('/path/to/config.yml'))) }"

can LSF assign fields, or do i have to do that on the LS server?

torrancew: Is there by chance like a diagram of sorts somewhere that shows the flow of logs from a client into logstash? Kind of trying to get my head around the big picture of LS

Alayde_: there's not really a super canonical one, because you can build almost any architecture you want; however, a lot of the common architectures are well documented

and diagrammed, etc

Alright, fair enough. I'll poke around a bit

The way I describe it to my dev team is that Logstash is more or less a way to create something similar to a complex shell one-liner with pipes and such

it gives you different inputs and outputs (<foo >bar), as well as many different filters (| baz | qux)

that's not a perfect description, but it's an analogy some folks are comfortable with

socket-: it can assign fields

but only of type string

not arrays

Alayde_: example:

<http://tinyurl.com/lqjfw9x>; (at www.google.com)

0&bih=1003&ved=0CDIQyjc&ei=63BRVISRKNLsoATFnYLIBQ

ugh, google

will fix.

<http://tinyurl.com/mczg4yp>; (at michael.bouvy.net)

that's one common arch

in that case, they're using logstash as the shipper

hey is there a way to disable logstash-forwarder from spamming to syslog?

copying that hyperlink is gonna be a fail for me, can you pm it my way?

sure

torrancew: so how would I modify my LSF to do something like this.. http://apaste.info/sHe

Title: Paste #sHe - Apache Paste Bucket (at apaste.info)

torrancew: success! thanks

socket-: ah, we misunderstood one another

you can add fields all you want, but you don't get to inspect the event like that

think of it more as FilePath-wide fields, rather than per-event fields. Yes, they come in on each event, but they're scoped more loosely.

so, i would have to assign the values on the LS server instead of LSF?

if they're coming from event data, yes

LSF isn't a parsing engine, just a shipping engine

it allows you to add static data to help route that stuff when it lands somewhere else

So if i wanted to offload the burden of tagging hundrends of servers all on my central server, i would have to install LS on each client for it to pre-field, right?

correct

k

torrancew: so something like this on my centeal log server? http://apaste.info/Fl9

Title: Paste #Fl9 - Apache Paste Bucket (at apaste.info)

socket-: yeah, generally so

could you help me assign one field, and i can probably figure out the others?

sure

i can try, at leaest

I make no promises regarding my success

lol, k

something like this?? http://apaste.info/8Pn

Title: Paste #8Pn - Apache Paste Bucket (at apaste.info)

I believe so, yes (don't use csv, but will try to repro here shortly)

cool

socket-: that second item I believe needs containing brackets around the array

"columns"

so, here is my full attempt, and i get some errors as shown.. http://apaste.info/Xmu

Title: Paste #Xmu - Apache Paste Bucket (at apaste.info)

socket-: see above comment regarding 'columns'

should be ['an', 'array', 'of', 'things']

Oh, ok i added the [] around it, and now everything is getting stuffed into field ocsp_date

so it looks like it doesnt like the seperator

yeah, that's what I'm messing with lcoally, too

"ocsp_date" => "2014-10-29\t23:25:02\t192.168.0.22\t1\tCN%3DCryptosec-CA2\t75daed0c275dcb51\t6\t1\t997f63bac384aba235cbdce17e3bce7a8cc71add\t200\t1\t9\tPOST\t127.0.0.1\t-\t-\t-\t-\t-\t-"

I think this has to do with a bug in the logstash config syntax, IIRC

Ohh, i pushed "tab" on the keyboard for separator => " " and it looks like its working

yeah, was about to suggest inserting a hard tab

^I

or whatever your editor supports

Title: Paste #avw - Apache Paste Bucket (at apaste.info)

thanks, for helping me

np

torrancew: Possibly a stupid question, but is there a file somewhere within the kibana hierarchy that contains a version number?

Trying to figure out the current version this server is using

hmm

the one in logstash-web?

or one you've installed?

The former, it's currently located at /opt/logstash/vendor/kibana I believe

ya

not my /tmp/kibana new hotness

so, here's the ugly, ugly truth

hit me

IIRC the logstash process did a git checkout

see if you have an /opt/logstash/vendor/kibana/.git

negative

ok, then apologies, but I don't know of a way to isolate it other than finding the build date of your logstash

cafuego: xtfer: drush alias just needs an update

torrancew: Alright, no worries

Hey guys, I'm getting a crash on startup with the error "invalid byte sequence in UTF-8' . At this point just trying to figure out how to dig deeper to get it running

logstash 1.4.2 output to elasticsearch 1.1.1

even with no conf files in /etc/logstash/conf.d/ i'm getting the same UTF-8 error and crash

JWade: are you certain you don't have any weird binary files in that directory?

JWade: like, if you're editing with emacs or something, it'll have a backup file in that directory.

ls -la /etc/logstash/conf.d/

Thank you...

I shall now remove me face from my desk