#logstash

untergeek: i see your name on a lot of rspec-related issues... might you have some insight on why I get a grokparsefailure in rspec (but not in my full LS config or in a test LS harness)?

not without any context

hi all, i'm seeing a (potential)? issue where the logstash.stdout log file on my logstash server is growing quite large

i read from a redis server as the input into the logstash server

is this normal?

shaggy_ is stdout a configured output?

valeech: yes

shaggy_ the log file will continue to grow until you rotate it or disable the output

okay

so it figures out where the logdir is?

# grep stdout *

20-output.conf: stdout { codec => rubydebug }

that's all i have

ahh okay, ogdir is in the startup script

allrighty, looks like i have some logrotate to setup

untergeek: http://pastebin.com/tDDtkK1Q (thanks in advance)

shaggy_ indeed

shaggy_: or don't run stdout output once you've gotten things working.

rastro: for one, please do not use insist in rspec. It will be removed in future releases. It is Rspec <= 2.0, and we're aiming for RSpec 3.x

shaggy_ rastro is spot on. stdout is generally used for debugging an installation or config change. Once up and running, you shouldn’t need stdout.

untergeek: any chance of updating the doc then? https://github.com/elastic/logstash/wiki/Tips:T...

whoa, do NOT rely on that old wiki

we're working on some new docs for the main site

oh lame. the included logrotate config gets dropped in the wrong mode for logrotate to use it

untergeek: well, all i have to go on are the tests that shipped with the product (which all use insist), other stuff elastic publishes, and the random wiki.

included in the logstash rpm that is

we're getting away from insist by degrees, just saying

untergeek: would the use of insist cause the GPF?

rastro: i will likely remove the stdout once i'm done

no

but I wouldn't rule it out

lol

I recommend using %{SPACE} instead of \s+

even if they're syntactically identical behind the scenes

and I worry about (%{URIPROTO}://)

untergeek: the grok works in my production config and in a small LS test harness. it GPFs in rspec.

slashes and [] tend to do much better when escaped

it could be that rspec is more strict

how come you're using @message syntax?

rastro: ^^^^

that

I think that is probably the source of your immediate problem

rastro: You're matching: match => [ "message",

but then you're using @message later

why?

it can't find a message field because you're passing in your sample line as "@message"

but grok is looking for a "message" field (no @)

untergeek: as to why, it's because every example on the internet uses it. will go test...

rastro: they're just outdated (pre-1.2)

they're stale, as torrancew pointed out

changed to sample("message" => message...) and it produces the same error. is there some other syntax there?

in the debug, shows this nesting: @lut={"message"=>[{"message"=>"1448797802.050....

update the paste?

indeed

k

updated: http://pastebin.com/tDDtkK1Q

i imagine line 23 isn't right...

rastro: say I wanted to reproduce this - how would I go about it?

torrancew: put the spec script in /opt/logstash/spec and run it: bin/logstash rspec spec/test.rb

rastro: no custom modules needed or anything? k

rastro: @lut={"message"=>[{"message"=>"1448797802.050 511 1.2.3.4 TCP_MISS/201 824 PUT http://2.3.4.5:1234/example.html - HIER_DIRECT/69.78.224.182 text/html", "type"=>"squid-access", "@version"=>"1", "@timestamp"=>"2015-12-01T19:23:23.517Z", "tags"=>["_grokparsefailure_squidaccess"]}, "message"],

torrancew: the real version will read my real config files in @config :)

I'm puzzled by this

and this:

"timestamp"=>[{"message"=>"1448797802.050 511 1.2.3.4 TCP_MISS/201 824 PUT http://2.3.4.5:1234/example.html - HIER_DIRECT/69.78.224.182 text/html", "type"=>"squid-access", "@version"=>"1", "@timestamp"=>"2015-12-01T19:23:23.517Z", "tags"=>["_grokparsefailure_squidaccess"]}, "timestamp"]

untergeek: indeed.

and this:

"tags"=>[{"message"=>"1448797802.050 511 1.2.3.4 TCP_MISS/201 824 PUT http://2.3.4.5:1234/example.html - HIER_DIRECT/69.78.224.182 text/html", "type"=>"squid-access", "@version"=>"1", "@timestamp"=>"2015-12-01T19:23:23.517Z", "tags"=>["_grokparsefailure_squidaccess"]}, "tags"]}

it does not seem to be inputting properly

untergeek: is sample() expecting message to be json?

other blogs had it as plain text...

to test grok in rspec, I'd be more inclined to use something like this: https://github.com/logstash-plugins/logstash-fi...

these are what we use to test the grok plugin

as you can see, sample is just passing the message as a string, and that's all

you can use expect instead of insist

untergeek: i can test that, but i will need to still set [type] in the input.

rastro: try "message_text", {hash_of_extra_fields}, possibly

I can't find the method definition for sample

rastro: that's irrelevant for grok testing

rastro: I would completely bypass that as it's not being set in the grok filter anyway

if you need to test it, test it in a different block

untergeek: i will be testing my entire config (which includes conditionals), not just a single grok.

use a different test block

one test at a time

i want it to read my production config file and test the final output based on a given input.

it's tempting to try two birds with one stone, but that's clearly not working here, where working examples of just the grok rules do exist

rastro: fwiw, looking at the definition of sample now, it may be a matter of wrapping {} around the input

that aside though, I'd really lean more towards doing integration tests by running your filter config through a real LS instance with more simplified input/output plugins

ie stdin -> {real_filters} -> file { codec => json }

rastro: 109 of test_utils.rb is assert_equal(expected_lines, actual_lines)

then you can build a sample input file and its known equivalent

torrancew: i was hoping to use the rspec expect stuff and not have to write my own.

well, you're definitely going to find writing a single rspec test for a whole config pipeline very frustrating

may be easier to wire up a harness of running ls, recording output to a file, and using ruby/python/whatever to parse the json ls generates and compare it to known good json

torrancew: my config is split into 30 different files. i was expecting to have 30 rspec tests to match them, each reading the production config file.

(ordering may change, so you'd want to actually parse it into objects and compare those)

rastro: you got that right? didnt see it discussed but its the error in http://pastebin.com/tDDtkK1Q

rastro: sure, but that's really not the point I'm trying to make; rspec is for unit testing, and lends itself to such (where units tend to be small and focused)

I'm assuming that your 30 files are basically 30 different filter pipelines constrained by type/tag/whatever, no?

torrancew: rspec takes a sample input and allows you to examine your output. seems like precisely what i need.

torrancew: yes, that's the layout.

let me be clear

you *can* do it

but it's probably going to feel clunky and drive you half mad in the process

torrancew: considering there are no working examples, and two of the smartest people that i know can't figure it out....

BIGkbs1: no, i didn't follow that.

rastro: so your pastebin.. im catching up...

Are there any developers for the logstash-plugins package around?

rastro: think BIGkbs1 is pointing out lines that threw traces

looks like youre running this "1448797802.050 511 1.2.3.4 TCP_MISS/201 824 PUT http://2.3.4.5:1234/example.html - HIER_DIRECT/69.78.224.182 text/html"

thru the logstash config

BIGkbs1: that's the sample input, yes.

and it throws error

BIGkbs1: indeed.

so thats a good test

BIGkbs1: i thought so :)

what more did you need?

was your idea to verify the data before sending it in?

BIGkbs1: the pattern works for that input, but it throws a GPF and other fields get all jinky (as discussed earlier).