#logstash

Rumbles, disk is OK - I've upgraded to 1.50.0rc3

it looks like a Lumberjack/logstash-forwarder process is leaking memory and dying

but I can't confirm because it's not logging at all

Title: FLP_logstash - Pastebin.com (at pastebin.com)

this is what i've made till now but still i'm getting massed up data in logstash

i have 3 domains XXX1 XXX2 nad XXX3 and all of them should write to one output

I know i could do it by just merging the files in one but i tried this way because it was easier for me to use it with puppet

talos: i have one output

ok so if i would put an if at the start of filter section and then in outut it should work right ?

Rumbles: maybe you could help me on another problem, i got two log types which should be written into one daily index. this works so far and there are several indexes with test-YYYY-MM-DD. But in kibana i would like to search all those indexes, so i created a template namend test-* and assigned it to both indexes, but that doesnt work. Here is my config and template, do you got a hint for me? http://pastebin.com/JqguYZfw

Title: test - Pastebin.com (at pastebin.com)

hmmmm can't say I've tinkered much with indexes in kibana

what does it come back with, anything, errors?

in kibana

FLP that condtional in the config you patebined would mean only the files from XXX1 would go to your output

i.e. if [path] =~ "XXX1" would only match events which can from a path which matches the regex XXX1 in some way

so if they are XXX1 XXX2 XXX3 the regex =~ "XXX" should work for all?

Rumbles that was just one file and that was just for domain XXX1 now i have three files each for XXXn

but i think i know now the way

will try and get back

THX

!!

Rumbles: there is no error, i guess i will have to use aliases?

Rumbles works perfectly

not sure FlashDel, might be worth hanging around and waiting for the USA to come online, or ask in #kibana ?

i don't understand why kibana4 doesn't show tags of my file, instead logstash in logstash.stdout i saw that all parsing is correct... bah

Hi

hello

I was wondering if someone could tell me if its possible to use grok to add a field with an value matched from message but with its own regexp specifically for that field.

yep

create your own pattern

or

Rumbles: yeah figured that ;) but the question more is, how to apply that pattern again :)

let me show you a small example to clarify what i would like to accomplish

http://logstash.net/docs/1.4.2/filters/grok custom patterns on this page, anchor doesn't work :/

Title: logstash - open source log management (at logstash.net)

if you want to use the same pattern a number of times it would be best to define it in a pattern file

normally in /opt/logstash/patterns (but depends on your OS)

but yeah pastebin what you have

Rumbles: https://gist.github.com/smeevil/22f071087f9a9e8...

<http://tinyurl.com/psm6fgu>; (at gist.github.com)

hope that makes a bit more sense :)

so, I'm not sure this is what you wanted: add_field => [ "contract_id", "%{contract_id}"] << this would add a field called contract_id which holds the content of the captured field %{contract_id}, but as far as I can see you haven't captured %{contract_id} in your grok patterncaptured

exactly my problem :) how can i give a secondary search patter to extract that ?

if you have a pattern called CONTRACT_ID, and have it in your grok match, and capture using %{CONTRACT_ID:contract_id} it will generate the field contract_id when it captures the data

can you give some example log lines? you say that the contract id information can be in any part of the message?

assuming you have CONTRACT_ID defined in a pattern file, you could use (?:%{CONTRACT_ID:contract_id}|) anywhere in your pattern that that information could be found, that will match the pattern or nothing at all (using (?:%{PATTERN}|) the bit after the | is nothing, so will match one thing or the other)

let me rephrase that

(using (?:%{PATTERN}|) the bit after the | is nothing, so will match %{PATTERN} or nothing)

Rumbles http://pastebin.com/ZMuFj3Tz

Title: input { file { path => "C:\\main.log.2015- - Pastebin.com (at pastebin.com)

tigryss did you look at what I sent you back last time? you can match onpath, but you may as well just define the type on your input and use a conditional for the type

then forget about using mutate to add your type

ok

thats why i love the kv filter ;)

also, if you use 2 spaces for each indentation, stick to that, don't use random amounts of spaces to indent, it makes it really difficult to read

each { or } should be on a new line

Hi, I've imported a few million files into logstash and then realised I didn't have any grok filters against it. I want to reimport the files again so that the filters are associated against the data properly. I don't have a $HOME/.sincedb path, is there a way to reset the filter?

I don't have OCD but that config makes my skin crawl :/

Rumbles: to clear up what i want to achieve : We have quite a big stack of services and servers. these will all processes stuff in one way or the other but what they all have in common is that most of the time they will have a reverence to a contract_id somewhere in they logline. now I'd like to search for a given contract id and trace the whole path of that

given contract id throughout all the moving parts. does that make sense ?

mdempsey what input did you use?

smeevil do you have an example of a few of your loglines =

I see what you mean smeevil

that should be possible

Rumbles i can't change the logs :))))

@Rumbles: I used file

but yeah ^^ examples and I can advise further

so tahts why i'd like to apply a second match to see if i can extract...arbitrary data... and add that to fields if it has been found

with start_position => beginning

meaning it might or might not be in there :D but if it is, extract it and add it as a field

mdempsey if you run "locate .sincedb" you should find your sncedb file (assuming you're on linux)

mine live in /var/lib

yeah smeevil look at my earlier posts, use (?:%{PATTERN}|)

it will match it if is there, and still match if it isn't

if there is data your field will be populated

if there is nothing there the field won't be populated

Rumbles thanks, yeah on linux will have a look now

if it can appear in 3 or 4 different places in the log entry, you just need to place that pattern in all the places it might occur

Rumbles: thanks for that suggestion, that was my initial approch, but might make the patters even more unreadable :) also it might accor in information that is already being match as a whole.

I'll try and investigate further :) i think it should be possible to run multiple matches / operations against a single line

yeah you can

within your grok you can have one match, then a second, then a thrid etc

third :/

oh ! that might be it then :)

do you happen to have an example of that ?

if i understand correct its like : if this match, do this if not go on, then if this matches, do this... etc:)

1 sec

Rumbles Looks like its not there, getting no files found.

updatedb mdempsey ?

I'll leave you be after that :P thanks for you help man, appreciated !

Rumbles yeah tried that as well, still nothing

:(

not sure then tbh...

you could define a sincedb path in your input

then restart ls

it should then start a new one?

yeah that was my next thought

will give that a shot and see what happens

gl :)

:)

when it comes to elk, is there a way to modify an existing mapping of a key? I've modified the template but I have some misunderstanding on how to modify the existing mapping on the indices https://gist.github.com/dariusjs/f263dad5d9a209...