#logstash

Hello! I'm using last versions of logstash and filebeat. My goal is usual - harvest logs from windows machine. But I have an unusual case: proprietary software write logs to file every 5 seconds with ~2,5mb of nulls (to avoid disk fragmentation). Because of that, a lot of lines are missing. Can you, please, advise me, what to do?

linjan: the software writes 2.5MB of nulls to the log file or does it overwrite the existing log lines with nulls?

Depending on your log shipper, you might be able to exclude empty lines. What's your shipper setup on your Windows nodes? Filebeat?

Xylakant: first one. writes a lot of nulls, and next time replaces nulls with actual log lines

bjorn_: yes, filebeat

linjan: well, that sucks. and I don't think there's a way to fix that

other than fixing the software.

filebeat remembers where it stopped in the file, so after it read all the nulls and forwarded them, it will never re-read that portion of the file.

Is the log file location fixed?

you could try and patch filebeat to not continue reading the file if it encounters null bytes.

Xylakant: this software writes a lot of data in log. About 1Gb per day per one file. They trying to prevent fragmentation. So we cant fix that proprietary software ( I'm thinking about write simple parser in python and send to logstash

bjorn_: fixed, yep

Perhaps there's a way to catch the file content in transit

1GB per day is less than 12KB per second

that's not a lot.

that's actually tiny.

Xylakant: for ntfs fragmentation is like crazy )

Xylakant: for linux is tiny )))

Xylakant: thank you for advice, maybe there is another solution, except filebeat patching?

I would've tried to intercept the file location, but I have no idea whether that's doable in Windows.

Filtering out null bytes and only forwarding to filebeat when there's real content.

linjan: build your own log shipper. maybe it's also possible to convince the application to log to something like syslog or similar.

I'd also be willing to place bets that this is a case of premature optimization.

bjorn_: i found only that: https://www.eldos.com/cbflt/virtual-files-and-f... . thanks for idea

Xylakant: i didn't catch last message

the software tries to optimize for a problem that does not truly exist.

but in practice, that doesn't matter since you can't change the software.

I have no idea whether this would work, so I'm just throwing it out there: Read the full log file in regular intervals, and implement something that only picks up events that haven't been seen before. Perhaps the fingerprint filter could be used.

To make this work you might need to install Logstash on the Windows node, not only Filebeat.

Xylakant: this is very old financial software ) they still supporting it since 2000th. But you're right, some of their decisions a little bit stupid)

Filebeat can accept data on STDIN, so you can essentially pipe the log file to Filebeat once a minute. Then you'd have to do some deduplication on the receiving side.

i'd probably rather go and change filebeat :)

it currently reads until EOF, so it shouldn't be too hard to add "read until EOF or NULL"

(last famous words of a programmer)

:D

or maybe have that something that reads the file and writes to filebeat stall on null bytes

advantage: vanilla filebeat.

disadvantage: yet another moving part.

I don't think this case can be solved without having to accept some disadvantage :-)

life's a trade.

¯\_(ツ)_/¯

Xylakant: bjorn_: a lot of thanks, guys

how can i add a timestamp field for when the event is processed by logstash? not by logentry timestamp

to see latency between shippers/indexers

rofl____: Unless a message has a @timestamp field when it enters Logstash it'll create that field and initialize it with the current time. Depending on your configuration you might be able to just save that timestamp (possibly in another field)