Project

General

Profile

Actions

User story #2974

closed

Archive reports does not free space on disk

Added by Vincent MEMBRÉ over 11 years ago. Updated about 9 years ago.

Status:
Released
Priority:
N/A
Category:
Documentation
Target version:
UX impact:
Suggestion strength:
User visibility:
Effort required:
Name check:
Fix check:
Regression:

Description

When archiving reports, reports size on disk does not change.

lines from "ruddersysevents" are correctly inserted in "archivedsysevents" and are correctly deleted, but the datatable size stays the same.

processing a request "vacuum full ruddersysevents" free it effectively.

problem, a vacuum full you have to be veeeeeeeery careful when processing a vacuum full, as it locks the database.

A solution would be to process "vacuum" regurlarly and to add the possibility to proceed a vacuum full rarely (one time per month), by default no vacuum full

We should add a temporary table to store datas, when a vacuum full is at work.


Related issues 1 (0 open1 closed)

Related to Rudder - User story #2972: Allow to delete report from databaseReleasedVincent MEMBRÉ2012-10-30Actions
Actions #1

Updated by Nicolas CHARLES over 11 years ago

Indeed, VACUUM FULL is frowned upon if it's not done outside on maintenance windows ( http://serverfault.com/questions/264221/postgresql-8-4-auto-vacuum-is-enabled-to-get-the-best-performance last answer) for it locks the database.
Erasing gigs of data inside a table can be a very long process (in terms of dozens of minutes), effectively blocking the whole application (and log collection)

The archived reports can be, I feel, safely vacuum fulled, it would prevent other archiving during this time, and people could leave with it

Setting a temporary table would probably not solve the issue, and would have too much impact to allow the release of Rudder 2.4 in November (it would impact log collection and reporting)

Actions #2

Updated by Vincent MEMBRÉ over 11 years ago

Yes for temporary table, I was just keeping a track about that idea to explore in a following version (2.5).

I'm ok with the idea of having a vacuum full only on the archived reports, it should not be a problem

Actions #3

Updated by François ARMAND over 11 years ago

  • Status changed from New to 2

The chosen solution is:

- before each cleaning, check if a full vaccuum is currently running, and if so, fail the cleaning, adding a log message (and an error message for the user if the clean request was interactive) explaining why the cleaning can not occures at that time.
- after each cleaning (archive or delete), do:
- a (simple) vaccuum on table Reports
- a full caccuum on table Archive Reports

These actions have to be performed by the Scala / SQL code performing the cleaning.

Actions #4

Updated by Nicolas CHARLES over 11 years ago

François ARMAND wrote:

The chosen solution is:

- before each cleaning, check if a full vaccuum is currently running, and if so, fail the cleaning, adding a log message (and an error message for the user if the clean request was interactive) explaining why the cleaning can not occures at that time.

I've looked in the Postgres documentation and forum, there doesn't seems to be an easy way to know if a vacuum is in progress. Here there might be something that could be an inspiration on how to do it: http://pgsql.inb4.se/2009/february/follow-up-vacuum-full-progress.html

Actions #5

Updated by Vincent MEMBRÉ over 11 years ago

  • Status changed from 2 to Pending technical review

We decided to perform a vacuum full on archived reports and a vacuum on "active" reports

so disk space will be freed by rudder on archived reports, but not block the active reports table, I think this is the best compromise we can do for now

I made a pull request about that, https://github.com/Normation/rudder/pull/4

I change this ticket status into "Technical review"

Actions #6

Updated by Nicolas PERRON over 11 years ago

  • Target version changed from 2.4.0~rc1 to 2.5.0~beta1

The new feature about automation of DB management tasks will not be included in Rudder 2.4. Then, postponed to Rudder 2.5

Actions #7

Updated by Vincent MEMBRÉ over 11 years ago

Since targent branch has changed, I had to make another pull request, technical review can be done here https://github.com/Normation/rudder/pull/8

Actions #8

Updated by Nicolas CHARLES over 11 years ago

  • Assignee set to Nicolas CHARLES
Actions #9

Updated by Nicolas CHARLES over 11 years ago

  • Tracker changed from Bug to Enhancement
  • Status changed from Pending technical review to 12

This is valid, thank you Vincent

Actions #10

Updated by Nicolas CHARLES over 11 years ago

I have no idea why this ticket is in rudder-doc, and I can't change its project

Actions #11

Updated by Vincent MEMBRÉ over 11 years ago

  • Status changed from 12 to Pending release
  • % Done changed from 0 to 100
Actions #12

Updated by Matthieu CERDA over 11 years ago

  • Status changed from Pending release to Released
Actions #13

Updated by Benoît PECCATTE about 9 years ago

  • Project changed from 30 to Rudder
  • Category set to Documentation
Actions #14

Updated by Benoît PECCATTE about 9 years ago

  • Tracker changed from Enhancement to User story
Actions

Also available in: Atom PDF