#logstash

so once filebeat sends the source logs into elastic search the sources can be destroyed yet elasticsearch still have a database of ALL * those logs, that is understood.,

hal: elastic themselves at one point even acknowledged that they shouldn't be the system of record (see "Robustness"): https://www.elastic.co/blog/found-elasticsearch...

next question how do you prevent elasticsearch from growing too big, can you logrotate elasticsearch?

hal: assuming filebeat processed them as you intended and ES indexed them as you intended, yes, ES would have a copy of them.

do you have much experience running logrotate from a single linux machine to a NFS NAS appliance? Not sure if that is even doable

hal: but one small mapping discrepancy, and ES won't index your documents any more :(

hal: the typical trick for managing ES indexes is to have daily indexes and delete "old" ones with `curator`.

hal: if the share is mounted as writable, i imagine that logrotate would be able to manage the original files for you.

untergeek! :)

still alive

:)

i was tempted to write c-u-r-a-t-o-r to not summon you :)

hahahaha

okay lets back up if I delete the source NFS logs that filebeat is reading from then I will lose my ES indexes once the NFS is purged?

so you advised that I logrotate?

hal: let's try to focus on one thing at a time.

yes I backed up on that last question for a second : )

thx for youre help btw

hal: you have a bunch of files in a directory. doesn't matter if it's SSD, spinning disk, NFS, or a thumb drive.

ok got it

hal: you point filebeat at that directory, and tell it to send those logs directly to elasticsearch...

got it

hal: elasticsearch will try to index the logs that are being sent. in that indexing, ES will store a copy of the data in its own place.

check.

hal: if the logs are successfully loaded into ES, and you have no further need of the original logs, you can compress, rotate, and/or delete them as you wish.

at this point if the source is delete the indexes are still in tact correct?

*deleted

hal: "a bunch of files in a directory" are different than "[ES storing] a copy of the data in its own place".

hal: you have, perhaps, made some large assumption...

so then yes once ES has indexed my data, the source and those indexes have no relationship based on what your are saying.

hal: that's true, yes.

yes I did assume there : )

so think of the panama canal once you are in a new lock the previous lock can be drained with not threat to the water level where the boat is currently.

hal: but if ES isn't running, or doesn't like your data, or ..., then it might not index the info sent from filebeat.

hal: or you might want to change the mapping on a field, which you can't do without re-indexing.

in other words you do not like depending on ES in the event of an issue where ES is not longer reachable so are youre logs.

hal: or your company may have a retention policy for old information.

so I was thinking about implementing the following let me know what you think.

1. Logrotate is setup on a single apache web server in our cluster of 4 e.g. web1 will have an entry in /etc/logrotate.d/new_rotate

2. Have filebeat read /var/log/web/logs*

into ES as we discussed.

to be indexed

3. I will have AWS backup and archive the NFS logs while rotate is keeping 30 days logs, so AWS will have more than 30 days 60 90 120 etc.

this is a seperate task

** so we would have a running local 30 day rotation, AWS for long term reading the /var/log/*, and ES ingesting the NFS logs from filebeat.

sound good?

hal: sure.

hal: i would add step 2.5: have logrotate compress the files.

yes that was assumed ; )

but agreed